Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27437

Galera snapshot transfer fails to upgrade between some major versions

Details

    Description

      mariabackup SST is failing sporadically in prepare stage. The problem only occurs when node1 and node2 have an active read / write workload.

      ramesh@galapq:~/qa/galera-qa$ cat /home/ramesh/qa/node3/mariabackup.prepare.log
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup based on MariaDB server 10.8.0-MariaDB Linux (x86_64)
      		 
      [00] 2022-01-06 15:00:03 cd to /home/ramesh/qa/node3/.sst/
      		 
      [00] 2022-01-06 15:00:03 open files limit requested 0, set to 1048576
      		 
      [00] 2022-01-06 15:00:03 This target seems to be not prepared yet.
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '33' csname 'utf8' trying to replace existing csname 'utf8mb3'
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '83' csname 'utf8' trying to replace existing csname 'utf8mb3'
      		 
      [00] 2022-01-06 15:00:03 mariabackup: using the following InnoDB configuration for recovery:
      		 
      [00] 2022-01-06 15:00:03 innodb_data_home_dir = .
      		 
      [00] 2022-01-06 15:00:03 innodb_data_file_path = ibdata1:12M:autoextend
      		 
      [00] 2022-01-06 15:00:03 innodb_log_group_home_dir = .
      		 
      [00] 2022-01-06 15:00:03 InnoDB: Using Linux native AIO
      		 
      [00] 2022-01-06 15:00:03 Starting InnoDB instance for recovery.
      		 
      [00] 2022-01-06 15:00:03 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Using transactional memory
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Number of transaction pools: 1
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Using Linux native AIO
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000MiB, chunk size = 100.000MiB
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Completed initialization of buffer pool
      		 
      2022-01-06 15:00:03 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.7.2-MariaDB.
      		 
      2022-01-06 15:00:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      		 
      [00] 2022-01-06 15:00:03 mariadb-backup: srv_start() returned 11 (Generic error).

      Testcase

      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node1 > /home/ramesh/qa/log/startup1.log 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node2 > /home/ramesh/qa/log/startup2.log 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node3 > /home/ramesh/qa/log/startup3.log 2>&1
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node1.cnf  --wsrep-new-cluster > /home/ramesh/qa/node1/node1.err 2>&1 &
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -Bse"delete from mysql.user where user='';" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node2.cnf  > /home/ramesh/qa/node2/node2.err 2>&1 &
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf  > /home/ramesh/qa/node3/node3.err 2>&1 &
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test; create database test;" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      		 
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node2/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      		 
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node3/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      		 
      sleep 10
      		 
       
      		 
      Terminate sysbench initiated on node3 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqladmin --user=root --socket=/home/ramesh/qa/node3/mysql.sock shutdown > /dev/null 2>&1
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_one; create database test_one;" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_two; create database test_two;" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_three; create database test_three;" > /dev/null 2>&1
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_three  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf --wsrep-provider=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/lib/libgalera_smm.so --datadir=/home/ramesh/qa/node3 --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt >> /home/ramesh/qa/node3/node3.err 2>&1 &

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25975 Turning on innodb_disallow_writes will cause mysqladmin shutdown to hang

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28256 Galera SST fails with 'InnoDB: Upgrade after a crash is not supported'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31506 Updating cluster to a new major version Reports Upgrade after a crash is not supported with rsync sst

          • Major - Major loss of function.
          • Confirmed

          Task - A task that needs to be done. MDEV-31536 Galera mariadb-backup to work as SST between major versions

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12353 Efficient InnoDB redo log record format

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-14425 Change the InnoDB redo log format to reduce write amplification

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15912 InnoDB: Failing assertion: purge_sys.tail.commit <= purge_sys.rseg->last_commit upon upgrade from 10.0 or 10.1 to 10.3

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-29162 Major upgrade instructions missing for Galera

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-36202 rsync and mariabackup SST impossible if joiner has newer major version than donor

          • Major - Major loss of function.
          • Open

          Activity

            Descending order - Click to sort in ascending order
            marko Marko Mäkelä added a comment -

            Thank you, otto. This ticket is indeed about upgrades between major versions. We do avoid changes to file formats after a release series has been declared generally available (GA). There are some exceptions, such as MDEV-11623, MDEV-13564, MDEV-25004, perhaps also MDEV-25440 if you consider it to be a file format change. When implementing such changes, special attention must be (and has been) paid on upgrades, as well as downgrades within the same release series (say, 10.3.35 to 10.3.34).

            I see that the default setting is wsrep_sst_method=rsync. According to the tests conducted by ramesh, major version upgrades using that method will work reliably if the donor (running the older major version) includes the fixes of MDEV-25975 and MDEV-28247.

            I updated the following pages to document the limitations around major version upgrades:
            https://mariadb.com/kb/en/introduction-to-state-snapshot-transfers-ssts/
            https://mariadb.com/kb/en/mariabackup-sst-method/

            marko Marko Mäkelä added a comment - Thank you, otto . This ticket is indeed about upgrades between major versions. We do avoid changes to file formats after a release series has been declared generally available (GA). There are some exceptions, such as MDEV-11623 , MDEV-13564 , MDEV-25004 , perhaps also MDEV-25440 if you consider it to be a file format change. When implementing such changes, special attention must be (and has been) paid on upgrades, as well as downgrades within the same release series (say, 10.3.35 to 10.3.34). I see that the default setting is wsrep_sst_method=rsync . According to the tests conducted by ramesh , major version upgrades using that method will work reliably if the donor (running the older major version) includes the fixes of MDEV-25975 and MDEV-28247 . I updated the following pages to document the limitations around major version upgrades: https://mariadb.com/kb/en/introduction-to-state-snapshot-transfers-ssts/ https://mariadb.com/kb/en/mariabackup-sst-method/
            otto Otto Kekäläinen added a comment -

            I read this issue on request by marko but I am not sure if I understood it fully. Event with potentially lacking understanding I would strongly advice the following:

            1. Only change redo log format on major releases, never in minor releases. If this happens, make sure the release notes of the major version says that the new version redo log format is no longer backwards compatible and sysadmins and automation software needs to do X, Y, and Z when upgrading to new major version.
            2. This is hard to fix in Linux distro packaging and anyways doing such tricks in the package manager is against the purpose of packaging scripts. To begin with, there is no guarantee that the server is even running when the upgrade happens. The distro packages primary function is to update plain files. Packagign scripts should not have code about data format detection and conversion. Secondly, any fix in packaging scripts in deb must also be done in rpm and in all other packaging scripts whatever they are. Would be hard to maintain and likely cause a lot of failures anyway.
            3. The only place outside the server and client/utility binaries that handles the start/stop/restart process is the systemd and init files. They could potentially have some logic about doing graceful restart in certain situations, and those files are shared across all Linux distros and container systems etc so the maintenance burden could be manageable.
            4. Best place to fix this is in the server and client/helper binaries by the same people do introduce a potential backwards incompatible change. This is identical to what Marko wrote in the comment from April 14th about solutions 1&2.

            But indeed, priority number 1 is to document such changes in release notes of major releases. Those are read by sysadmins and that is the place when sysadmins expect they might have to occasionally do something extra to make the upgrade work right.

            otto Otto Kekäläinen added a comment - I read this issue on request by marko but I am not sure if I understood it fully. Event with potentially lacking understanding I would strongly advice the following: Only change redo log format on major releases, never in minor releases. If this happens, make sure the release notes of the major version says that the new version redo log format is no longer backwards compatible and sysadmins and automation software needs to do X, Y, and Z when upgrading to new major version. This is hard to fix in Linux distro packaging and anyways doing such tricks in the package manager is against the purpose of packaging scripts. To begin with, there is no guarantee that the server is even running when the upgrade happens. The distro packages primary function is to update plain files. Packagign scripts should not have code about data format detection and conversion. Secondly, any fix in packaging scripts in deb must also be done in rpm and in all other packaging scripts whatever they are. Would be hard to maintain and likely cause a lot of failures anyway. The only place outside the server and client/utility binaries that handles the start/stop/restart process is the systemd and init files. They could potentially have some logic about doing graceful restart in certain situations, and those files are shared across all Linux distros and container systems etc so the maintenance burden could be manageable. Best place to fix this is in the server and client/helper binaries by the same people do introduce a potential backwards incompatible change. This is identical to what Marko wrote in the comment from April 14th about solutions 1&2. But indeed, priority number 1 is to document such changes in release notes of major releases. Those are read by sysadmins and that is the place when sysadmins expect they might have to occasionally do something extra to make the upgrade work right.
            ramesh Ramesh Sivaraman added a comment -

            marko Agree with using rsync SST during major upgrades. The 10.7> 10.8 upgrade works well on rsync SST.

            ramesh Ramesh Sivaraman added a comment - marko Agree with using rsync SST during major upgrades. The 10.7> 10.8 upgrade works well on rsync SST.
            marko Marko Mäkelä added a comment -

            ramesh, thank you. Automating the solution in something like a package installation script would seem to be somewhat challenging, so maybe this work-around should only be documented with a detailed example. An easier way could be that users must simply enable rsync based snapshot transfer before major-version upgrades.

            Theoretically, an upgrade with mariabackup could work if the user made a copy (or hard link) of the old backup executable and then pointed an environment variable or a configuration parameter to it, for the duration of upgrading the server. An interface for specifying the full path to the executable would be needed in wsrep_sst_mariabackup, as demonstrated by the patch that I posted. After the upgrade, the old executable could be removed and the configuration restored to normal (so that a snapshot transfer will continue to work from a current-version donor).

            I quickly checked the Debian packaging scripts. Baking such upgrade logic directly into those scripts would be very challenging to implement and test. Something like the following would happen:

            1. mariadb-server-10.7.prerm prepares for the removal of the 10.7 executables (tries to shut down the server)
            2. the 10.7 executables are removed
            3. the 10.8 executables are installed
            4. mariadb-server-10.8.preinst tries to shut down any old server process again
            5. mariadb-server-10.8.postinst starts the 10.8 server
            marko Marko Mäkelä added a comment - ramesh , thank you. Automating the solution in something like a package installation script would seem to be somewhat challenging, so maybe this work-around should only be documented with a detailed example. An easier way could be that users must simply enable rsync based snapshot transfer before major-version upgrades . Theoretically, an upgrade with mariabackup could work if the user made a copy (or hard link) of the old backup executable and then pointed an environment variable or a configuration parameter to it, for the duration of upgrading the server. An interface for specifying the full path to the executable would be needed in wsrep_sst_mariabackup , as demonstrated by the patch that I posted. After the upgrade, the old executable could be removed and the configuration restored to normal (so that a snapshot transfer will continue to work from a current-version donor). I quickly checked the Debian packaging scripts. Baking such upgrade logic directly into those scripts would be very challenging to implement and test. Something like the following would happen: mariadb-server-10.7.prerm prepares for the removal of the 10.7 executables (tries to shut down the server) the 10.7 executables are removed the 10.8 executables are installed mariadb-server-10.8.preinst tries to shut down any old server process again mariadb-server-10.8.postinst starts the 10.8 server
            ramesh Ramesh Sivaraman added a comment - - edited

            marko with older version mariabackup(hard-coded older binary location in 10.8 sst script) SST worked fine in 10.7 > 10.8 upgrade

            ramesh@galapq:/dev/shm/qa$ cat node3/mysql_upgrade_info 
            10.8.3-MariaDB
            		 
            ramesh@galapq:/dev/shm/qa$ cat mariabackup.prepare.log.2022.04.12-14.33.21.938721816
            		 
            /dev/shm/qa/mariadb-10.7.4-linux-x86_64//bin/mariabackup based on MariaDB server 10.7.4-MariaDB Linux (x86_64)
            		 
            [00] 2022-04-12 14:28:45 cd to /dev/shm/qa/node3/.sst/
            		 
            [00] 2022-04-12 14:28:45 open files limit requested 0, set to 1048576
            		 
            [00] 2022-04-12 14:28:45 Renaming ./mysql/wsrep_streaming_log.new to ./mysql/wsrep_streaming_log.ibd
            		 
            [..]
            		 
            2022-04-12 14:28:45 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed
            		 
            2022-04-12 14:28:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=41933,41933
            		 
            2022-04-12 14:28:45 0 [Note] InnoDB: Starting final batch to recover  25 pages from redo log.
            		 
            [00] 2022-04-12 14:28:45 Last binlog file , position 0
            		 
            [00] 2022-04-12 14:28:45 completed OK!
            		 
            ramesh@galapq:/dev/shm/qa$

            ramesh Ramesh Sivaraman added a comment - - edited marko with older version mariabackup(hard-coded older binary location in 10.8 sst script) SST worked fine in 10.7 > 10.8 upgrade ramesh@galapq:/dev/shm/qa$ cat node3/mysql_upgrade_info 10.8.3-MariaDB ramesh@galapq:/dev/shm/qa$ cat mariabackup.prepare.log.2022.04.12-14.33.21.938721816 /dev/shm/qa/mariadb-10.7.4-linux-x86_64//bin/mariabackup based on MariaDB server 10.7.4-MariaDB Linux (x86_64) [00] 2022-04-12 14:28:45 cd to /dev/shm/qa/node3/.sst/ [00] 2022-04-12 14:28:45 open files limit requested 0, set to 1048576 [00] 2022-04-12 14:28:45 Renaming ./mysql/wsrep_streaming_log.new to ./mysql/wsrep_streaming_log.ibd [..] 2022-04-12 14:28:45 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed 2022-04-12 14:28:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=41933,41933 2022-04-12 14:28:45 0 [Note] InnoDB: Starting final batch to recover 25 pages from redo log. [00] 2022-04-12 14:28:45 Last binlog file , position 0 [00] 2022-04-12 14:28:45 completed OK! ramesh@galapq:/dev/shm/qa$
            View 13 older comments

            People

              marko Marko Mäkelä
              ramesh Ramesh Sivaraman
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.