[MDEV-27524] Incorrect binlogs after Galera SST using rsync and mariabackup - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.2.41, 10.3.32, 10.4.22, 10.5.13, 10.6.5, 10.7.1
Fix Version/s: 10.9.0, 10.3.35, 10.4.25, 10.5.16, 10.6.8, 10.7.4, 10.8.3
Component/s: Galera SST, mariabackup
Labels:
None

Description

In many scenarios, nodes crash after SST if the donor node uses multiple binlogs files and then SST is performed using rsync or mariabackup. Especially often the failure occurs after accessing binlogs from the joiner side. These failures are due to the fact that currently neither SST with rsync nor SST with mariabackup transfers all binlogs (not just the latest one), and the binlogs handling code in SST scripts is very simplistic and cannot correctly work in a number of configurations, for example, in configurations including absolute paths to binlogs in an index file, etc. An example of such a failure due to incorrect transmission of binlogs after SST:

021-12-28  1:48:20 45 [Warning] Access denied for user 'testuser'@'127.0.0.1' (using password: YES)

2021-12-28  2:11:01 0 [Note] InnoDB: Buffer pool(s) load completed at 211228  2:11:01

2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) desyncs itself from group

2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) resyncs itself to group.

2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) synced with group.

2021-12-28  9:11:13 10 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 0, event_type: 0

2021-12-28  9:11:13 10 [ERROR] WSREP: applier could not read binlog event, seqno: 196201, len: 18446744072749699206

2021-12-28  9:11:13 0 [Note] WSREP: Member 1(ddmsmariadb03) initiates vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,8c41510586499f3b:

2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:

   0000000000000000:   1/3

   8c41510586499f3b:   1/3

Waiting for more votes.

2021-12-28  9:11:13 5 [Note] WSREP: Got vote request for seqno 68ed27ac-671d-11ec-bed4-924ee3878f93:196201

2021-12-28  9:11:13 0 [Note] WSREP: Member 0(ddmsmariadb01) responds to vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,0000000000000000: Success

2021-12-28  9:11:13 0 [Warning] WSREP: Received bogus VOTE message: 196201.0, from node 52e83762-673c-11ec-b091-279e19995029, expected > 196203. Ignoring.

2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:

   0000000000000000:   1/3

   8c41510586499f3b:   1/3

Waiting for more votes.

2021-12-28  9:11:13 0 [Note] WSREP: Member 2(ddmsmariadb02) initiates vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,8c41510586499f3b:

2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:

   0000000000000000:   1/3

   8c41510586499f3b:   2/3

Winner: 8c41510586499f3b

2021-12-28  9:11:13 10 [ERROR] WSREP: Failed to apply write set: gtid: 68ed27ac-671d-11ec-bed4-924ee3878f93:196201 server_id: 52e83762-673c-11ec-b091-279e19995029 client_id: 5863 trx_id: 6901439 flags: 3 (start_transaction | commit)

2021-12-28  9:11:13 10 [Note] WSREP: Closing send monitor...

2021-12-28  9:11:13 10 [Note] WSREP: Closed send monitor.

2021-12-28  9:11:13 10 [Note] WSREP: gcomm: terminating thread

2021-12-28  9:11:13 10 [Note] WSREP: gcomm: joining thread

2021-12-28  9:11:13 10 [Note] WSREP: gcomm: closing backend

2021-12-28  9:11:15 10 [Note] WSREP: view(view_id(NON_PRIM,52e83762-b091,3) memb {

	6e8d8876-803a,0

} joined {

} left {

} partitioned {

	52e83762-b091,0

	5ef9bf5e-bd37,0

})

Attachments

Issue Links

blocks

MDEV-27719 Syntax error in wsrep_sst_common: [: readable: integer expression expected

Closed

MDEV-27777 Some Galera tests fail on FreeBSD with "Could not find target log"

Closed

causes

MDEV-27777 Some Galera tests fail on FreeBSD with "Could not find target log"

Closed

MDEV-28758 Mariabackup copies binary logs to backup directory

Closed

MDEV-28781 mariabackup rsync fails to copy /var/lib/mysql/var/lib/mysql/ib_buffer_pool

Closed

MDEV-29109 mariabackup --rsync option broken since 10.3

Closed

includes

MDEV-24365 datadir cannot be changed even when specified via CLI

Stalled

relates to

MDEV-17921 Galera SST successfully finishing with error "Error in `mariabackup': double free or corruption (out)"

Closed

MDEV-19874 Unable to join galera node. SST ERROR with mariabackup

Closed

MDEV-26201 MariaDB 10.5.11 cannot do SST when binary log is enabled on donor

Closed

MDEV-28583 Galera: binlogs disappear after rsync IST

Closed

MDEV-28968 mariabackup SST doesn't properly handle --innodb-buffer-pool-filename

Closed

MDEV-28015 Mariabackup | GTID value is missing, Galera Cluster , adding async slave to it

Closed

split from

MDEV-24097 galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed

Closed

split to

MDEV-27534 mariabackup: missing line in the compress_qpress.result

Closed

MDEV-27602 Automatic creation of working directories in mtr

Stalled

MDEV-27692 Waiting for a port before trying to connect in buildbot scripts

Stalled

MDEV-27740 Joiner node failed to join the cluster when saving redo_logs in a directory different from the data directory

Closed

(1 causes, 1 includes, 6 relates to, 1 split from, 4 split to)

Activity

Ascending order - Click to sort in descending order

Julius Goryavsky created issue - 2022-01-17 03:05

Julius Goryavsky made changes - 2022-01-17 03:05

Field	Original Value	New Value
Link		This issue relates to ~~MDEV-24097~~ [ ~~MDEV-24097~~ ]

Julius Goryavsky made changes - 2022-01-17 03:10

Link

This issue split from ~~MDEV-24097~~ [ ~~MDEV-24097~~ ]

Julius Goryavsky made changes - 2022-01-17 03:10

Link

This issue relates to ~~MDEV-24097~~ [ ~~MDEV-24097~~ ]

Julius Goryavsky made changes - 2022-01-17 03:10

Status

Open [ 1 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2022-01-17 03:15

Assignee	Julius Goryavsky [ sysprg ]	Jan Lindström [ jplindst ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2022-01-18 09:00

Assignee	Jan Lindström [ jplindst ]	Julius Goryavsky [ sysprg ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-18 11:32

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2022-01-18 11:55

Link

This issue split to ~~MDEV-27534~~ [ ~~MDEV-27534~~ ]

Julius Goryavsky made changes - 2022-01-20 15:00

Assignee	Julius Goryavsky [ sysprg ]	Brandon Nesterenko [ JIRAUSER48702 ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Brandon Nesterenko made changes - 2022-01-20 15:48

Assignee	Brandon Nesterenko [ JIRAUSER48702 ]	Julius Goryavsky [ sysprg ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-25 01:00

Assignee	Julius Goryavsky [ sysprg ]	Brandon Nesterenko [ JIRAUSER48702 ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Julius Goryavsky made changes - 2022-01-25 02:37

Status

In Review [ 10002 ]

In Testing [ 10301 ]

Brandon Nesterenko made changes - 2022-01-25 03:47

Assignee

Brandon Nesterenko [ JIRAUSER48702 ]

Julius Goryavsky [ sysprg ]

Julius Goryavsky made changes - 2022-01-25 13:19

Status

In Testing [ 10301 ]

Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-25 13:24

Assignee	Julius Goryavsky [ sysprg ]	Jan Lindström [ jplindst ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2022-01-26 07:33

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2022-01-26 07:35

Assignee	Jan Lindström [ jplindst ]	Sergei Golubchik [ serg ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Julius Goryavsky made changes - 2022-01-27 19:31

Assignee

Sergei Golubchik [ serg ]

Julius Goryavsky [ sysprg ]

Julius Goryavsky made changes - 2022-01-27 19:31

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-27 19:34

Status

Stalled [ 10000 ]

In Testing [ 10301 ]

Julius Goryavsky made changes - 2022-01-27 19:34

Status

In Testing [ 10301 ]

Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-27 19:36

Assignee	Julius Goryavsky [ sysprg ]	Brandon Nesterenko [ JIRAUSER48702 ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Brandon Nesterenko made changes - 2022-01-27 20:01

Assignee	Brandon Nesterenko [ JIRAUSER48702 ]	Julius Goryavsky [ sysprg ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-27 23:56

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2022-01-27 23:56

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-01-27 23:57

Link

This issue split to MDEV-27602 [ MDEV-27602 ]

Julius Goryavsky made changes - 2022-01-27 23:58

Link

This issue is blocked by MDEV-27602 [ MDEV-27602 ]

Julius Goryavsky made changes - 2022-01-28 00:09

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Julius Goryavsky made changes - 2022-01-31 13:02

Link

This issue split to MDEV-27692 [ MDEV-27692 ]

Julius Goryavsky made changes - 2022-02-02 18:56

Link

This issue blocks ~~MDEV-27719~~ [ ~~MDEV-27719~~ ]

Julius Goryavsky made changes - 2022-02-09 03:57

Status

Stalled [ 10000 ]

In Testing [ 10301 ]

Julius Goryavsky made changes - 2022-02-09 03:57

Assignee

Julius Goryavsky [ sysprg ]

Ramesh Sivaraman [ JIRAUSER48189 ]

Julius Goryavsky made changes - 2022-02-09 04:46

Link

This issue split to ~~MDEV-27740~~ [ ~~MDEV-27740~~ ]

Julius Goryavsky made changes - 2022-02-14 03:57

Link

This issue causes ~~MDEV-27777~~ [ ~~MDEV-27777~~ ]

Julius Goryavsky made changes - 2022-02-14 04:13

Link

This issue blocks ~~MDEV-27777~~ [ ~~MDEV-27777~~ ]

Julius Goryavsky made changes - 2022-02-16 00:03

Assignee	Ramesh Sivaraman [ JIRAUSER48189 ]	Julius Goryavsky [ sysprg ]
Status	In Testing [ 10301 ]	Stalled [ 10000 ]

Julius Goryavsky made changes - 2022-02-17 18:12

Link

This issue relates to ~~MDEV-19874~~ [ ~~MDEV-19874~~ ]

Julius Goryavsky made changes - 2022-02-18 13:21

Link

This issue relates to ~~MDEV-17921~~ [ ~~MDEV-17921~~ ]

Julius Goryavsky made changes - 2022-02-18 17:14

Link

This issue relates to MDEV-24365 [ MDEV-24365 ]

Julius Goryavsky made changes - 2022-02-18 17:19

Link

This issue includes MDEV-24365 [ MDEV-24365 ]

Julius Goryavsky made changes - 2022-02-18 17:19

Link

This issue relates to MDEV-24365 [ MDEV-24365 ]

Julius Goryavsky made changes - 2022-02-22 12:11

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2022-02-22 12:12

Fix Version/s		10.9.0 [ 27113 ]
Fix Version/s		10.2.44 [ 27514 ]
Fix Version/s		10.3.35 [ 27512 ]
Fix Version/s		10.4.25 [ 27510 ]
Fix Version/s		10.5.16 [ 27508 ]
Fix Version/s		10.6.8 [ 27506 ]
Fix Version/s		10.7.4 [ 27504 ]
Fix Version/s		10.8.3 [ 27502 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.3 [ 22126 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Fix Version/s	10.7 [ 24805 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Julius Goryavsky made changes - 2022-02-22 12:35

Link

This issue is blocked by MDEV-27602 [ MDEV-27602 ]

Julien Fritsch made changes - 2022-02-24 06:59

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Julius Goryavsky made changes - 2022-03-22 04:38

Link

This issue relates to MENT-1448 [ MENT-1448 ]

Julius Goryavsky made changes - 2022-04-11 23:25

Link

This issue relates to ~~MDEV-26201~~ [ ~~MDEV-26201~~ ]

Julius Goryavsky made changes - 2022-05-16 20:09

Link

This issue relates to ~~MDEV-28583~~ [ ~~MDEV-28583~~ ]

Marko Mäkelä made changes - 2022-06-21 06:40

Link

This issue causes ~~MDEV-28758~~ [ ~~MDEV-28758~~ ]

Sergei Golubchik made changes - 2022-06-22 08:22

Fix Version/s

10.2.44 [ 27514 ]

Julius Goryavsky made changes - 2022-06-27 23:27

Link

This issue relates to ~~MDEV-28968~~ [ ~~MDEV-28968~~ ]

Sergei Golubchik made changes - 2022-06-28 10:24

Link

This issue causes ~~MDEV-28781~~ [ ~~MDEV-28781~~ ]

Elena Stepanova made changes - 2022-07-20 12:31

Link

This issue causes ~~MDEV-29109~~ [ ~~MDEV-29109~~ ]

Richard Stracke made changes - 2023-05-25 05:10

Link

This issue relates to ~~MDEV-28015~~ [ ~~MDEV-28015~~ ]

Jira Automation (IT) made changes - 2024-07-04 01:55

Zendesk Related Tickets

185775

People

Assignee:: Julius Goryavsky

Reporter:: Julius Goryavsky

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2022-01-17 03:05

Updated:: 2024-07-07 20:41

Resolved:: 2022-02-22 12:12

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration