Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.6, 10.11
Description
SST failure occurs when gtid_strict_mode is enabled and OLTP load is active on the primary node.
Error info
2024-09-06 15:16:03 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 218950)
|
2024-09-06 15:16:03 0 [Note] WSREP: Processing event queue:... 0.0% (0/90729 events) complete.
|
2024-09-06 15:16:04 2 [ERROR] mariadbd: Error writing file 'binlog' (errno: 1950 "Unknown error 1950")
|
2024-09-06 15:16:04 2 [ERROR] WSREP: Failed to apply write set: gtid: 20146161-6c49-11ef-ab34-3e5b83e9db28:188601 server_id: 20134002-6c49-11ef-bb60-2346bc116142 client_id: 34 trx_id: 3772519 flags: 3 (start_transaction | commit)
|
2024-09-06 15:16:04 8 [ERROR] WSREP: Failed to apply write set: gtid: 20146161-6c49-11ef-ab34-3e5b83e9db28:188602 server_id: 20134002-6c49-11ef-bb60-2346bc116142 client_id: 36 trx_id: 3772477 flags: 3 (start_transaction | commit)
|
2024-09-06 15:16:04 2 [Note] WSREP: Closing send monitor...
|
|
Test case
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadb-admin --user=root --socket=/home/ramesh/framework/node2/mysql.sock shutdown
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadb-admin --user=root --socket=/home/ramesh/framework/node1/mysql.sock shutdown
|
|
|
rm -rf /home/ramesh/framework/node*
|
|
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64 --datadir=/home/ramesh/framework/node1 > /home/ramesh/framework/log/startup1.log 2>&1
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64 --datadir=/home/ramesh/framework/node2 > /home/ramesh/framework/log/startup2.log 2>&1
|
|
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadbd --defaults-file=/home/ramesh/framework/conf/node1.cnf --wsrep-provider=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/lib/libgalera_smm.so --basedir=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64 --wsrep-new-cluster > /home/ramesh/framework/node1/node1.err 2>&1 &
|
|
|
sleep 10
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadb --user=root --socket=/home/ramesh/framework/node1/mysql.sock -Bse"SET SESSION sql_log_bin=0;delete from mysql.user where user='';" > /dev/null 2>&1
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadbd --defaults-file=/home/ramesh/framework/conf/node2.cnf --wsrep-provider=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/lib/libgalera_smm.so --basedir=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64 > /home/ramesh/framework/node2/node2.err 2>&1 &
|
|
|
sleep 20
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadb --user=root --socket=/home/ramesh/framework/node2/mysql.sock -Bse"SET SESSION sql_log_bin=0;delete from mysql.user where user='';" > /dev/null 2>&1
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadb --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
|
|
|
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/framework/node1/mysql.sock prepare >/home/ramesh/framework/log/sysbench_prepare.log
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadb-admin --user=root --socket=/home/ramesh/framework/node2/mysql.sock shutdown
|
|
|
|
|
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql --mysql-socket=/home/ramesh/framework/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/framework/log/sysbench_read_write_10.log &
|
|
|
|
|
rm -rf node2/grastate.dat
|
|
|
sleep 10
|
|
|
/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/bin/mariadbd --defaults-file=/home/ramesh/framework/conf/node2.cnf --wsrep-provider=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64/lib/libgalera_smm.so --basedir=/home/ramesh/framework/mariadb-10.11.10-linux-x86_64 > /home/ramesh/framework/node2/node2.err 2>&1 &
|
|
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-
Activity
How to reproduce :
/mtr galera.galera_sst_mariabackup --mysqld=--gtid_strict_mode=ON --mysqld=--wsrep-gtid_mode=ON --mysqld=--log-bin --mysqld=--log-slave_updates --rr=--bind-to-cpu=0 --mysqld=--wsrep-debug=1
|
- Similar assertion using galera.galera_sst_rsync test case with same additional parameters.
2024-12-21 9:41:55 2 [Note] WSREP: wsrep_before_prepare: thread=2 real=1 killed=0 trx_seqno=27 xid:ec8e7b6b-bf6e-11ef-9604-93ab853bd41a:27 gtid=0:1:0 ret=Success (0)2024-12-21 9:41:55 2 [Note] WSREP: wsrep_after_prepare: 12024-12-21 9:41:55 2 [Note] WSREP: wsrep_before_commit: thread=2 real=1 killed=0 trx_seqno=27 xid:ec8e7b6b-bf6e-11ef-9604-93ab853bd41a:27 gtid=0:1:19 ret=Success (0)2024-12-21 9:41:55 2 [Note] WSREP: wsrep_ordered_commit: 1 272024-12-21 9:41:55 2 [Note] WSREP: wsrep_ordered_commit: thread=2 real=1 killed=0 trx_seqno=27 xid:ec8e7b6b-bf6e-11ef-9604-93ab853bd41a:27 gtid=0:1:19 ret=Success (0)2024-12-21 9:41:55 2 [Note] WSREP: rpl_binlog_state::update: domain_id=0 server_id=1 seq_no=192024-12-21 9:41:55 2 [Note] WSREP: rpl_binlog_state::update_nolock failed: domain_id=0 server_id=1 seq_no=19 !! last_domain_id=0 last_server_id=1 last_seq_no=192024-12-21 9:41:55 2 [ERROR] mariadbd: Error writing file 'mysqld-bin' (errno: 1950 "Unknown error 1950")2024-12-21 9:41:55 2 [Note] WSREP: log_and_order has failed 2 02024-12-21 9:41:55 2 [Note] WSREP: wsrep_before_rollback: thread=2 real=1 killed=0 trx_seqno=27 xid:ec8e7b6b-bf6e-11ef-9604-93ab853bd41a:27 ret=Success (0)mariadbd: /home/jan/work/mariadb/10.11/wsrep-lib/src/transaction.cpp:665: int wsrep::transaction::before_rollback(): Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed.- galera.galera_sst_mysqldump also fails
galera.galera_sst_mysqldump 'debug' [ fail ]Test ended at 2024-12-21 10:06:07CURRENT_TEST: galera.galera_sst_mysqldumpmysqltest: In included file "./suite/galera/include/galera_sst_set_mysqldump.inc":included from /home/jan/work/mariadb/10.11/mysql-test/suite/galera/t/galera_sst_mysqldump.test at line 4:At line 12: query 'CREATE USER 'sst'' failed: ER_GTID_STRICT_OUT_OF_ORDER (1950): An attempt was made to binlog GTID 0-1-1 which would create an out-of-order sequence number with existing GTID 0-1-2, and gtid strict mode is enabled
janlindstrom Thanks, the fix has been merged with the head revision https://github.com/MariaDB/server/commit/b167730499b359f247015447e985319814342bf1
Workaround : Problem does not reproduce if rsync SST method is used.