[MDEV-28252] IST is failing after 10.3 > 10.4 upgrade Created: 2022-04-07  Updated: 2022-04-07

Status: Open
Project: MariaDB Server
Component/s: Galera SST
Affects Version/s: 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Ramesh Sivaraman Assignee: Teemu Ollakka
Resolution: Unresolved Votes: 0
Labels: None


 Description   

Error info

2022-04-07 12:20:37 0 [Note] /home/ramesh/framework/GAL_MD070422-mariadb-10.4.25-linux-x86_64-opt/bin/mysqld: ready for connections.
Version: '10.4.25-MariaDB-log'  socket: '/home/ramesh/framework/node3/mysql.sock'  port: 5700  MariaDB Server
2022-04-07 12:20:37 14 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 4 (Interrupted system call)
	 at /test/mtest/10.4_galera_opt/galera/src/replicator_smm.hpp:pop_front():336. Null event.
2022-04-07 12:20:37 1 [Note] WSREP: Lowest cert index boundary for CC from sst: 16414
2022-04-07 12:20:37 1 [Note] WSREP: Min available from gcache for CC from sst: -1 

Test case

/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/scripts/mysql_install_db --no-defaults --force --basedir=/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt --datadir=/home/ramesh/framework/node1 > /home/ramesh/framework/log/startup1.log 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/scripts/mysql_install_db --no-defaults --force --basedir=/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt --datadir=/home/ramesh/framework/node2 > /home/ramesh/framework/log/startup2.log 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/scripts/mysql_install_db --no-defaults --force --basedir=/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt --datadir=/home/ramesh/framework/node3 > /home/ramesh/framework/log/startup3.log 2>&1
 
 
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/framework/conf/node1.cnf  --wsrep-new-cluster > /home/ramesh/framework/node1/node1.err 2>&1 &
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -Bse"SET SESSION sql_log_bin=0;delete from mysql.user where user='';" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/framework/conf/node2.cnf  > /home/ramesh/framework/node2/node2.err 2>&1 &
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/framework/conf/node3.cnf  > /home/ramesh/framework/node3/node3.err 2>&1 &
 
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"drop database if exists test; create database test;" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
 
sysbench data load 
 
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=10000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node1/mysql.sock prepare >/home/ramesh/framework/log/sysbench_prepare.log
 
Initiate multi master sysbench oltp load in background 
 
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=10000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/framework/log/sysbench_read_write_10.log & 
 
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=10000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node2/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/framework/log/sysbench_read_write_10.log & 
 
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=10000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node3/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/framework/log/sysbench_read_write_10.log & 
 
Terminate OLTP sysbench run initiated on node3
 
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysqladmin --user=root --socket=/home/ramesh/framework/node3/mysql.sock shutdown > /dev/null 2>&1
 
Initiate DATA load on node1
 
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"drop database if exists test_one; create database test_one;" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"drop database if exists test_two; create database test_two;" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"drop database if exists test_three; create database test_three;" > /dev/null 2>&1
/home/ramesh/framework/GAL_MD070422-mariadb-10.3.35-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/framework/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
 
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node1/mysql.sock prepare >/home/ramesh/framework/log/sysbench_prepare.log
 
 
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node1/mysql.sock prepare >/home/ramesh/framework/log/sysbench_prepare.log
 
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_three  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/framework/node1/mysql.sock prepare >/home/ramesh/framework/log/sysbench_prepare.log
 
Start node3
 
/home/ramesh/framework/GAL_MD070422-mariadb-10.4.25-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/framework/conf/node3.cnf --wsrep-provider=/home/ramesh/framework/GAL_MD070422-mariadb-10.4.25-linux-x86_64-opt/lib/libgalera_smm.so --datadir=/home/ramesh/framework/node3 --basedir=/home/ramesh/framework/GAL_MD070422-mariadb-10.4.25-linux-x86_64-opt >> /home/ramesh/framework/node3/node3.err 2>&1 &
 
run mysql upgrade on node3
 
/home/ramesh/framework/GAL_MD070422-mariadb-10.4.25-linux-x86_64-opt/bin/mysql_upgrade -uroot --socket=/home/ramesh/framework/node3/mysql.sock --skip-write-binlog > /home/ramesh/framework/log/node3_upgrade.log 2>&1
0
 
Check IST log in node3 error log


Generated at Thu Feb 08 09:59:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.