Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27437

Galera snapshot transfer fails to upgrade between some major versions

    XMLWordPrintable

Details

    Description

      mariabackup SST is failing sporadically in prepare stage. The problem only occurs when node1 and node2 have an active read / write workload.

      ramesh@galapq:~/qa/galera-qa$ cat /home/ramesh/qa/node3/mariabackup.prepare.log
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup based on MariaDB server 10.8.0-MariaDB Linux (x86_64)
      [00] 2022-01-06 15:00:03 cd to /home/ramesh/qa/node3/.sst/
      [00] 2022-01-06 15:00:03 open files limit requested 0, set to 1048576
      [00] 2022-01-06 15:00:03 This target seems to be not prepared yet.
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '33' csname 'utf8' trying to replace existing csname 'utf8mb3'
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '83' csname 'utf8' trying to replace existing csname 'utf8mb3'
      [00] 2022-01-06 15:00:03 mariabackup: using the following InnoDB configuration for recovery:
      [00] 2022-01-06 15:00:03 innodb_data_home_dir = .
      [00] 2022-01-06 15:00:03 innodb_data_file_path = ibdata1:12M:autoextend
      [00] 2022-01-06 15:00:03 innodb_log_group_home_dir = .
      [00] 2022-01-06 15:00:03 InnoDB: Using Linux native AIO
      [00] 2022-01-06 15:00:03 Starting InnoDB instance for recovery.
      [00] 2022-01-06 15:00:03 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      2022-01-06 15:00:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2022-01-06 15:00:03 0 [Note] InnoDB: Using transactional memory
      2022-01-06 15:00:03 0 [Note] InnoDB: Number of transaction pools: 1
      2022-01-06 15:00:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2022-01-06 15:00:03 0 [Note] InnoDB: Using Linux native AIO
      2022-01-06 15:00:03 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000MiB, chunk size = 100.000MiB
      2022-01-06 15:00:03 0 [Note] InnoDB: Completed initialization of buffer pool
      2022-01-06 15:00:03 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.7.2-MariaDB.
      2022-01-06 15:00:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      [00] 2022-01-06 15:00:03 mariadb-backup: srv_start() returned 11 (Generic error).
      

      Testcase

      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node1 > /home/ramesh/qa/log/startup1.log 2>&1
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node2 > /home/ramesh/qa/log/startup2.log 2>&1
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node3 > /home/ramesh/qa/log/startup3.log 2>&1
       
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node1.cnf  --wsrep-new-cluster > /home/ramesh/qa/node1/node1.err 2>&1 &
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -Bse"delete from mysql.user where user='';" > /dev/null 2>&1
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node2.cnf  > /home/ramesh/qa/node2/node2.err 2>&1 &
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf  > /home/ramesh/qa/node3/node3.err 2>&1 &
       
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test; create database test;" > /dev/null 2>&1
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
       
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
       
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node2/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node3/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      sleep 10
       
      Terminate sysbench initiated on node3 
       
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqladmin --user=root --socket=/home/ramesh/qa/node3/mysql.sock shutdown > /dev/null 2>&1
       
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_one; create database test_one;" > /dev/null 2>&1
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_two; create database test_two;" > /dev/null 2>&1
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_three; create database test_three;" > /dev/null 2>&1
       
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
       
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
       
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_three  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
       
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf --wsrep-provider=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/lib/libgalera_smm.so --datadir=/home/ramesh/qa/node3 --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt >> /home/ramesh/qa/node3/node3.err 2>&1 &
      

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              ramesh Ramesh Sivaraman
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.