Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27524

Incorrect binlogs after Galera SST using rsync and mariabackup

Details

    Description

      In many scenarios, nodes crash after SST if the donor node uses multiple binlogs files and then SST is performed using rsync or mariabackup. Especially often the failure occurs after accessing binlogs from the joiner side. These failures are due to the fact that currently neither SST with rsync nor SST with mariabackup transfers all binlogs (not just the latest one), and the binlogs handling code in SST scripts is very simplistic and cannot correctly work in a number of configurations, for example, in configurations including absolute paths to binlogs in an index file, etc. An example of such a failure due to incorrect transmission of binlogs after SST:

      021-12-28  1:48:20 45 [Warning] Access denied for user 'testuser'@'127.0.0.1' (using password: YES)
      		 
      2021-12-28  2:11:01 0 [Note] InnoDB: Buffer pool(s) load completed at 211228  2:11:01
      		 
      2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) desyncs itself from group
      		 
      2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) resyncs itself to group.
      		 
      2021-12-28  3:31:32 0 [Note] WSREP: Member 1.0 (ddmsmariadb03) synced with group.
      		 
      2021-12-28  9:11:13 10 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 0, event_type: 0
      		 
      2021-12-28  9:11:13 10 [ERROR] WSREP: applier could not read binlog event, seqno: 196201, len: 18446744072749699206
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Member 1(ddmsmariadb03) initiates vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,8c41510586499f3b: 
      2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:
      		 
         0000000000000000:   1/3
      		 
         8c41510586499f3b:   1/3
      		 
      Waiting for more votes.
      		 
      2021-12-28  9:11:13 5 [Note] WSREP: Got vote request for seqno 68ed27ac-671d-11ec-bed4-924ee3878f93:196201
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Member 0(ddmsmariadb01) responds to vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,0000000000000000: Success
      		 
      2021-12-28  9:11:13 0 [Warning] WSREP: Received bogus VOTE message: 196201.0, from node 52e83762-673c-11ec-b091-279e19995029, expected > 196203. Ignoring.
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:
      		 
         0000000000000000:   1/3
      		 
         8c41510586499f3b:   1/3
      		 
      Waiting for more votes.
      		 
      2021-12-28  9:11:13 0 [Note] WSREP: Member 2(ddmsmariadb02) initiates vote on 68ed27ac-671d-11ec-bed4-924ee3878f93:196201,8c41510586499f3b: 
      2021-12-28  9:11:13 0 [Note] WSREP: Votes over 68ed27ac-671d-11ec-bed4-924ee3878f93:196201:
      		 
         0000000000000000:   1/3
      		 
         8c41510586499f3b:   2/3
      		 
      Winner: 8c41510586499f3b
      		 
      2021-12-28  9:11:13 10 [ERROR] WSREP: Failed to apply write set: gtid: 68ed27ac-671d-11ec-bed4-924ee3878f93:196201 server_id: 52e83762-673c-11ec-b091-279e19995029 client_id: 5863 trx_id: 6901439 flags: 3 (start_transaction | commit)
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: Closing send monitor...
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: Closed send monitor.
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: gcomm: terminating thread
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: gcomm: joining thread
      		 
      2021-12-28  9:11:13 10 [Note] WSREP: gcomm: closing backend
      		 
      2021-12-28  9:11:15 10 [Note] WSREP: view(view_id(NON_PRIM,52e83762-b091,3) memb {
      		 
      	6e8d8876-803a,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
      	52e83762-b091,0
      		 
      	5ef9bf5e-bd37,0
      		 
      })

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27719 Syntax error in wsrep_sst_common: [: readable: integer expression expected

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27777 Some Galera tests fail on FreeBSD with "Could not find target log"

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27777 Some Galera tests fail on FreeBSD with "Could not find target log"

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28758 Mariabackup copies binary logs to backup directory

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28781 mariabackup rsync fails to copy /var/lib/mysql/var/lib/mysql/ib_buffer_pool

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29109 mariabackup --rsync option broken since 10.3

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          includes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24365 datadir cannot be changed even when specified via CLI

          • Major - Major loss of function.
          • Stalled
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17921 Galera SST successfully finishing with error "Error in `mariabackup': double free or corruption (out)"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19874 Unable to join galera node. SST ERROR with mariabackup

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26201 MariaDB 10.5.11 cannot do SST when binary log is enabled on donor

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28583 Galera: binlogs disappear after rsync IST

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28968 mariabackup SST doesn't properly handle --innodb-buffer-pool-filename

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28015 Mariabackup | GTID value is missing, Galera Cluster , adding async slave to it

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          split from

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24097 galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          split to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27534 mariabackup: missing line in the compress_qpress.result

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. MDEV-27602 Automatic creation of working directories in mtr

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Stalled

          Task - A task that needs to be done. MDEV-27692 Waiting for a port before trying to connect in buildbot scripts

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27740 Joiner node failed to join the cluster when saving redo_logs in a directory different from the data directory

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            jplindst Jan Lindström (Inactive) added a comment -

            Code changes are ok, you could have avoided white space changes on mariabackup. However, rsync method seems to still have problems based on buildbot.

            jplindst Jan Lindström (Inactive) added a comment - Code changes are ok, you could have avoided white space changes on mariabackup. However, rsync method seems to still have problems based on buildbot.
            bnestere Brandon Nesterenko added a comment -

            The refinements look good. I left a couple minor suggestions on the new commit, feel free to push after their consideration.

            bnestere Brandon Nesterenko added a comment - The refinements look good. I left a couple minor suggestions on the new commit, feel free to push after their consideration.
            jplindst Jan Lindström (Inactive) added a comment -

            mariabackup, test and script changes ok to push

            jplindst Jan Lindström (Inactive) added a comment - mariabackup, test and script changes ok to push
            jplindst Jan Lindström (Inactive) added a comment -

            serg Can you review https://github.com/MariaDB/server/commit/6777c260c22e16fb9a76a60f124c2d9337d04585 . My Perl knowledge is too thin to make any kind of review.

            jplindst Jan Lindström (Inactive) added a comment - serg Can you review https://github.com/MariaDB/server/commit/6777c260c22e16fb9a76a60f124c2d9337d04585 . My Perl knowledge is too thin to make any kind of review.
            bnestere Brandon Nesterenko added a comment -

            I left one last question, otherwise it looks good. Thanks!

            bnestere Brandon Nesterenko added a comment - I left one last question, otherwise it looks good. Thanks!
            sysprg Julius Goryavsky added a comment -

            Fixed, https://github.com/MariaDB/server/commit/17e0f5224c8339ec08707a6ad0397bbf8c19bbd3

            sysprg Julius Goryavsky added a comment - Fixed, https://github.com/MariaDB/server/commit/17e0f5224c8339ec08707a6ad0397bbf8c19bbd3

            People

              sysprg Julius Goryavsky
              sysprg Julius Goryavsky
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.