Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27524

Incorrect binlogs after Galera SST using rsync and mariabackup

Details

    Description

      In many scenarios, nodes crash after SST if the donor node uses multiple binlogs files and then SST is performed using rsync or mariabackup. Especially often the failure occurs after accessing binlogs from the joiner side. These failures are due to the fact that currently neither SST with rsync nor SST with mariabackup transfers all binlogs (not just the latest one), and the binlogs handling code in SST scripts is very simplistic and cannot correctly work in a number of configurations, for example, in configurations including absolute paths to binlogs in an index file, etc. An example of such a failure due to incorrect transmission of binlogs after SST:

      021-12-28  1:48:20 45 [Warning] Access denied for user 'testuser'@'127.0.0.1' (using password: YES)
      		 
      2021-12-28  2:11:01 0 [Note] InnoDB: Buffer pool(s) load completed at 211228  2:11:01
      		 
      2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) desyncs itself from group
      		 
      2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) resyncs itself to group.
      		 
      2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) synced with group.
      		 
      2021-12-28  9:11:13 10 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 0, event_type: 0
      		 
      2021-12-28  9:11:13 10 [ERROR] WSREP: applier could not read binlog event, seqno: 196201, len: 18446744072749699206
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Member 1(ddmsmariadb03) initiates vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,8c41510586499f3b: 
      2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:
      		 
         0000000000000000:   1/3
      		 
         8c41510586499f3b:   1/3
      		 
      Waiting for more votes.
      		 
      2021-12-28  9:11:13 5 [Note] WSREP: Got vote request for seqno 68ed27ac-671d-11ec-bed4-924ee3878f93:196201
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Member 0(ddmsmariadb01) responds to vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,0000000000000000: Success
      		 
      2021-12-28  9:11:13 0 [Warning] WSREP: Received bogus VOTE message: 196201.0, from node 52e83762-673c-11ec-b091-279e19995029, expected > 196203. Ignoring.
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:
      		 
         0000000000000000:   1/3
      		 
         8c41510586499f3b:   1/3
      		 
      Waiting for more votes.
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Member 2(ddmsmariadb02) initiates vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,8c41510586499f3b: 
      2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:
      		 
         0000000000000000:   1/3
      		 
         8c41510586499f3b:   2/3
      		 
      Winner: 8c41510586499f3b
      		 
      2021-12-28  9:11:13 10 [ERROR] WSREP: Failed to apply write set: gtid: 68ed27ac-671d-11ec-bed4-924ee3878f93:196201 server_id: 52e83762-673c-11ec-b091-279e19995029 client_id: 5863 trx_id: 6901439 flags: 3 (start_transaction | commit)
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: Closing send monitor...
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: Closed send monitor.
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: gcomm: terminating thread
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: gcomm: joining thread
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: gcomm: closing backend
      		 
      2021-12-28  9:11:15 10 [Note] WSREP: view(view_id(NON_PRIM,52e83762-b091,3) memb {
      		 
      	6e8d8876-803a,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
      	52e83762-b091,0
      		 
      	5ef9bf5e-bd37,0
      		 
      })

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27719 Syntax error in wsrep_sst_common: [: readable: integer expression expected

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27777 Some Galera tests fail on FreeBSD with "Could not find target log"

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27777 Some Galera tests fail on FreeBSD with "Could not find target log"

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28758 Mariabackup copies binary logs to backup directory

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28781 mariabackup rsync fails to copy /var/lib/mysql/var/lib/mysql/ib_buffer_pool

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29109 mariabackup --rsync option broken since 10.3

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          includes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24365 datadir cannot be changed even when specified via CLI

          • Major - Major loss of function.
          • Stalled
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17921 Galera SST successfully finishing with error "Error in `mariabackup': double free or corruption (out)"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19874 Unable to join galera node. SST ERROR with mariabackup

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26201 MariaDB 10.5.11 cannot do SST when binary log is enabled on donor

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28583 Galera: binlogs disappear after rsync IST

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28968 mariabackup SST doesn't properly handle --innodb-buffer-pool-filename

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28015 Mariabackup | GTID value is missing, Galera Cluster , adding async slave to it

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          split from

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24097 galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          split to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27534 mariabackup: missing line in the compress_qpress.result

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. MDEV-27602 Automatic creation of working directories in mtr

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Stalled

          Task - A task that needs to be done. MDEV-27692 Waiting for a port before trying to connect in buildbot scripts

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27740 Joiner node failed to join the cluster when saving redo_logs in a directory different from the data directory

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Transition Time In Source Status Execution Times
            Julius Goryavsky made transition -
            Open In Progress
            		5m 14s 1
            Julius Goryavsky made transition -
            In Progress In Review
            		2d 3h 32m 2
            Julius Goryavsky made transition -
            In Review In Testing
            		1h 37m 1
            Julius Goryavsky made transition -
            Stalled In Review
            		4d 9h 20m 4
            Brandon Nesterenko made transition -
            In Review Stalled
            		3d 13h 4m 5
            Julius Goryavsky made transition -
            In Progress Stalled
            		10s 1
            Julius Goryavsky made transition -
            Stalled In Testing
            		12d 4h 3m 2
            Julius Goryavsky made transition -
            In Testing Stalled
            		7d 6h 48m 3
            Julius Goryavsky made transition -
            Stalled In Progress
            		6d 18h 36m 3
            Julius Goryavsky made transition -
            In Progress Closed
            		1m 1

            People

              sysprg Julius Goryavsky
              sysprg Julius Goryavsky
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.