Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.8(EOL)
-
None
Description
mariabackup SST is failing sporadically in prepare stage. The problem only occurs when node1 and node2 have an active read / write workload.
ramesh@galapq:~/qa/galera-qa$ cat /home/ramesh/qa/node3/mariabackup.prepare.log
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup based on MariaDB server 10.8.0-MariaDB Linux (x86_64)
|
[00] 2022-01-06 15:00:03 cd to /home/ramesh/qa/node3/.sst/
|
[00] 2022-01-06 15:00:03 open files limit requested 0, set to 1048576
|
[00] 2022-01-06 15:00:03 This target seems to be not prepared yet.
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '33' csname 'utf8' trying to replace existing csname 'utf8mb3'
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '83' csname 'utf8' trying to replace existing csname 'utf8mb3'
|
[00] 2022-01-06 15:00:03 mariabackup: using the following InnoDB configuration for recovery:
|
[00] 2022-01-06 15:00:03 innodb_data_home_dir = .
|
[00] 2022-01-06 15:00:03 innodb_data_file_path = ibdata1:12M:autoextend
|
[00] 2022-01-06 15:00:03 innodb_log_group_home_dir = .
|
[00] 2022-01-06 15:00:03 InnoDB: Using Linux native AIO
|
[00] 2022-01-06 15:00:03 Starting InnoDB instance for recovery.
|
[00] 2022-01-06 15:00:03 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
|
2022-01-06 15:00:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2022-01-06 15:00:03 0 [Note] InnoDB: Using transactional memory
|
2022-01-06 15:00:03 0 [Note] InnoDB: Number of transaction pools: 1
|
2022-01-06 15:00:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2022-01-06 15:00:03 0 [Note] InnoDB: Using Linux native AIO
|
2022-01-06 15:00:03 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000MiB, chunk size = 100.000MiB
|
2022-01-06 15:00:03 0 [Note] InnoDB: Completed initialization of buffer pool
|
2022-01-06 15:00:03 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.7.2-MariaDB.
|
2022-01-06 15:00:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
|
[00] 2022-01-06 15:00:03 mariadb-backup: srv_start() returned 11 (Generic error).
|
Testcase
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node1 > /home/ramesh/qa/log/startup1.log 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node2 > /home/ramesh/qa/log/startup2.log 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node3 > /home/ramesh/qa/log/startup3.log 2>&1
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node1.cnf --wsrep-new-cluster > /home/ramesh/qa/node1/node1.err 2>&1 &
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -Bse"delete from mysql.user where user='';" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node2.cnf > /home/ramesh/qa/node2/node2.err 2>&1 &
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf > /home/ramesh/qa/node3/node3.err 2>&1 &
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test; create database test;" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log &
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node2/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log &
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node3/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log &
|
|
sleep 10
|
|
Terminate sysbench initiated on node3
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqladmin --user=root --socket=/home/ramesh/qa/node3/mysql.sock shutdown > /dev/null 2>&1
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_one; create database test_one;" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_two; create database test_two;" > /dev/null 2>&1
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_three; create database test_three;" > /dev/null 2>&1
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_three --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
|
|
/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf --wsrep-provider=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/lib/libgalera_smm.so --datadir=/home/ramesh/qa/node3 --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt >> /home/ramesh/qa/node3/node3.err 2>&1 &
|
Attachments
Issue Links
- is blocked by
-
MDEV-25975 Turning on innodb_disallow_writes will cause mysqladmin shutdown to hang
- Closed
- relates to
-
MDEV-28256 Galera SST fails with 'InnoDB: Upgrade after a crash is not supported'
- Closed
-
MDEV-31506 Updating cluster to a new major version Reports Upgrade after a crash is not supported with rsync sst
- Confirmed
-
MDEV-31536 Galera mariadb-backup to work as SST between major versions
- Open
-
MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
- Closed
-
MDEV-12353 Efficient InnoDB redo log record format
- Closed
-
MDEV-14425 Change the InnoDB redo log format to reduce write amplification
- Closed
-
MDEV-15912 InnoDB: Failing assertion: purge_sys.tail.commit <= purge_sys.rseg->last_commit upon upgrade from 10.0 or 10.1 to 10.3
- Closed
-
MDEV-29162 Major upgrade instructions missing for Galera
- Closed