Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.8(EOL)
-
None
Description
mariabackup SST is failing sporadically in prepare stage. The problem only occurs when node1 and node2 have an active read / write workload.
ramesh@galapq:~/qa/galera-qa$ cat /home/ramesh/qa/node3/mariabackup.prepare.log
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup based on MariaDB server 10.8.0-MariaDB Linux (x86_64)
|
[00] 2022-01-06 15:00:03 cd to /home/ramesh/qa/node3/.sst/
|
[00] 2022-01-06 15:00:03 open files limit requested 0, set to 1048576
|
[00] 2022-01-06 15:00:03 This target seems to be not prepared yet.
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '33' csname 'utf8' trying to replace existing csname 'utf8mb3'
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '83' csname 'utf8' trying to replace existing csname 'utf8mb3'
|
[00] 2022-01-06 15:00:03 mariabackup: using the following InnoDB configuration for recovery:
|
[00] 2022-01-06 15:00:03 innodb_data_home_dir = .
|
[00] 2022-01-06 15:00:03 innodb_data_file_path = ibdata1:12M:autoextend
|
[00] 2022-01-06 15:00:03 innodb_log_group_home_dir = .
|
[00] 2022-01-06 15:00:03 InnoDB: Using Linux native AIO
|
[00] 2022-01-06 15:00:03 Starting InnoDB instance for recovery.
|
[00] 2022-01-06 15:00:03 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
|
2022-01-06 15:00:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2022-01-06 15:00:03 0 [Note] InnoDB: Using transactional memory
|
2022-01-06 15:00:03 0 [Note] InnoDB: Number of transaction pools: 1
|
2022-01-06 15:00:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2022-01-06 15:00:03 0 [Note] InnoDB: Using Linux native AIO
|
2022-01-06 15:00:03 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000MiB, chunk size = 100.000MiB
|
2022-01-06 15:00:03 0 [Note] InnoDB: Completed initialization of buffer pool
|
2022-01-06 15:00:03 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.7.2-MariaDB.
|
2022-01-06 15:00:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
|
[00] 2022-01-06 15:00:03 mariadb-backup: srv_start() returned 11 (Generic error).
|
Testcase
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node1 > /home/ramesh/qa/log/startup1.log 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node2 > /home/ramesh/qa/log/startup2.log 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node3 > /home/ramesh/qa/log/startup3.log 2>&1
|
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node1.cnf --wsrep-new-cluster > /home/ramesh/qa/node1/node1.err 2>&1 &
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -Bse"delete from mysql.user where user='';" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node2.cnf > /home/ramesh/qa/node2/node2.err 2>&1 &
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf > /home/ramesh/qa/node3/node3.err 2>&1 &
|
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test; create database test;" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
|
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log &
|
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node2/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log &
|
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node3/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log &
|
|
|
sleep 10
|
|
|
Terminate sysbench initiated on node3
|
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqladmin --user=root --socket=/home/ramesh/qa/node3/mysql.sock shutdown > /dev/null 2>&1
|
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_one; create database test_one;" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_two; create database test_two;" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_three; create database test_three;" > /dev/null 2>&1
|
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_three --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf --wsrep-provider=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/lib/libgalera_smm.so --datadir=/home/ramesh/qa/node3 --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt >> /home/ramesh/qa/node3/node3.err 2>&1 &
|
Attachments
Issue Links
- is blocked by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Confirmed
-
-
MDEV-31536 Galera mariadb-backup to work as SST between major versions
-
- Open
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Open
-
Activity
ramesh, thank you. Automating the solution in something like a package installation script would seem to be somewhat challenging, so maybe this work-around should only be documented with a detailed example. An easier way could be that users must simply enable rsync based snapshot transfer before major-version upgrades.
Theoretically, an upgrade with mariabackup could work if the user made a copy (or hard link) of the old backup executable and then pointed an environment variable or a configuration parameter to it, for the duration of upgrading the server. An interface for specifying the full path to the executable would be needed in wsrep_sst_mariabackup, as demonstrated by the patch that I posted. After the upgrade, the old executable could be removed and the configuration restored to normal (so that a snapshot transfer will continue to work from a current-version donor).
I quickly checked the Debian packaging scripts. Baking such upgrade logic directly into those scripts would be very challenging to implement and test. Something like the following would happen:
- mariadb-server-10.7.prerm prepares for the removal of the 10.7 executables (tries to shut down the server)
- the 10.7 executables are removed
- the 10.8 executables are installed
- mariadb-server-10.8.preinst tries to shut down any old server process again
- mariadb-server-10.8.postinst starts the 10.8 server
marko Agree with using rsync SST during major upgrades. The 10.7> 10.8 upgrade works well on rsync SST.
I read this issue on request by marko but I am not sure if I understood it fully. Event with potentially lacking understanding I would strongly advice the following:
- Only change redo log format on major releases, never in minor releases. If this happens, make sure the release notes of the major version says that the new version redo log format is no longer backwards compatible and sysadmins and automation software needs to do X, Y, and Z when upgrading to new major version.
- This is hard to fix in Linux distro packaging and anyways doing such tricks in the package manager is against the purpose of packaging scripts. To begin with, there is no guarantee that the server is even running when the upgrade happens. The distro packages primary function is to update plain files. Packagign scripts should not have code about data format detection and conversion. Secondly, any fix in packaging scripts in deb must also be done in rpm and in all other packaging scripts whatever they are. Would be hard to maintain and likely cause a lot of failures anyway.
- The only place outside the server and client/utility binaries that handles the start/stop/restart process is the systemd and init files. They could potentially have some logic about doing graceful restart in certain situations, and those files are shared across all Linux distros and container systems etc so the maintenance burden could be manageable.
- Best place to fix this is in the server and client/helper binaries by the same people do introduce a potential backwards incompatible change. This is identical to what Marko wrote in the comment from April 14th about solutions 1&2.
But indeed, priority number 1 is to document such changes in release notes of major releases. Those are read by sysadmins and that is the place when sysadmins expect they might have to occasionally do something extra to make the upgrade work right.
Thank you, otto. This ticket is indeed about upgrades between major versions. We do avoid changes to file formats after a release series has been declared generally available (GA). There are some exceptions, such as MDEV-11623, MDEV-13564, MDEV-25004, perhaps also MDEV-25440 if you consider it to be a file format change. When implementing such changes, special attention must be (and has been) paid on upgrades, as well as downgrades within the same release series (say, 10.3.35 to 10.3.34).
I see that the default setting is wsrep_sst_method=rsync. According to the tests conducted by ramesh, major version upgrades using that method will work reliably if the donor (running the older major version) includes the fixes of MDEV-25975 and MDEV-28247.
I updated the following pages to document the limitations around major version upgrades:
https://mariadb.com/kb/en/introduction-to-state-snapshot-transfers-ssts/
https://mariadb.com/kb/en/mariabackup-sst-method/
marko with older version mariabackup(hard-coded older binary location in 10.8 sst script) SST worked fine in 10.7 > 10.8 upgrade
ramesh@galapq:/dev/shm/qa$ cat node3/mysql_upgrade_info10.8.3-MariaDBramesh@galapq:/dev/shm/qa$ cat mariabackup.prepare.log.2022.04.12-14.33.21.938721816/dev/shm/qa/mariadb-10.7.4-linux-x86_64//bin/mariabackup based on MariaDB server 10.7.4-MariaDB Linux (x86_64)[00] 2022-04-12 14:28:45 cd to /dev/shm/qa/node3/.sst/[00] 2022-04-12 14:28:45 open files limit requested 0, set to 1048576[00] 2022-04-12 14:28:45 Renaming ./mysql/wsrep_streaming_log.new to ./mysql/wsrep_streaming_log.ibd[..]2022-04-12 14:28:45 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed2022-04-12 14:28:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=41933,419332022-04-12 14:28:45 0 [Note] InnoDB: Starting final batch to recover 25 pages from redo log.[00] 2022-04-12 14:28:45 Last binlog file , position 0[00] 2022-04-12 14:28:45 completed OK!ramesh@galapq:/dev/shm/qa$