Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27437

Galera snapshot transfer fails to upgrade between some major versions

Details

    Description

      mariabackup SST is failing sporadically in prepare stage. The problem only occurs when node1 and node2 have an active read / write workload.

      ramesh@galapq:~/qa/galera-qa$ cat /home/ramesh/qa/node3/mariabackup.prepare.log
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup based on MariaDB server 10.8.0-MariaDB Linux (x86_64)
      		 
      [00] 2022-01-06 15:00:03 cd to /home/ramesh/qa/node3/.sst/
      		 
      [00] 2022-01-06 15:00:03 open files limit requested 0, set to 1048576
      		 
      [00] 2022-01-06 15:00:03 This target seems to be not prepared yet.
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '33' csname 'utf8' trying to replace existing csname 'utf8mb3'
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt//bin/mariabackup: Warning: Charset id '83' csname 'utf8' trying to replace existing csname 'utf8mb3'
      		 
      [00] 2022-01-06 15:00:03 mariabackup: using the following InnoDB configuration for recovery:
      		 
      [00] 2022-01-06 15:00:03 innodb_data_home_dir = .
      		 
      [00] 2022-01-06 15:00:03 innodb_data_file_path = ibdata1:12M:autoextend
      		 
      [00] 2022-01-06 15:00:03 innodb_log_group_home_dir = .
      		 
      [00] 2022-01-06 15:00:03 InnoDB: Using Linux native AIO
      		 
      [00] 2022-01-06 15:00:03 Starting InnoDB instance for recovery.
      		 
      [00] 2022-01-06 15:00:03 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Using transactional memory
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Number of transaction pools: 1
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Using Linux native AIO
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000MiB, chunk size = 100.000MiB
      		 
      2022-01-06 15:00:03 0 [Note] InnoDB: Completed initialization of buffer pool
      		 
      2022-01-06 15:00:03 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.7.2-MariaDB.
      		 
      2022-01-06 15:00:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      		 
      [00] 2022-01-06 15:00:03 mariadb-backup: srv_start() returned 11 (Generic error).

      Testcase

      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node1 > /home/ramesh/qa/log/startup1.log 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node2 > /home/ramesh/qa/log/startup2.log 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/scripts/mariadb-install-db --no-defaults --force  --auth-root-authentication-method=normal  --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt --datadir=/home/ramesh/qa/node3 > /home/ramesh/qa/log/startup3.log 2>&1
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node1.cnf  --wsrep-new-cluster > /home/ramesh/qa/node1/node1.err 2>&1 &
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -Bse"delete from mysql.user where user='';" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node2.cnf  > /home/ramesh/qa/node2/node2.err 2>&1 &
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf  > /home/ramesh/qa/node3/node3.err 2>&1 &
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test; create database test;" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"create user if not exists sysbench@'localhost' identified  by 'sysbench';grant all on *.* to sysbench@'localhost';" > /dev/null 2>&1
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      		 
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node2/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      		 
      sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node3/mysql.sock --time=1000 --db-ps-mode=disable run >> /home/ramesh/qa/log/sysbench_read_write_10.log & 
       
      		 
      sleep 10
      		 
       
      		 
      Terminate sysbench initiated on node3 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysqladmin --user=root --socket=/home/ramesh/qa/node3/mysql.sock shutdown > /dev/null 2>&1
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_one; create database test_one;" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_two; create database test_two;" > /dev/null 2>&1
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.7.2-linux-x86_64-opt/bin/mysql --user=root --socket=/home/ramesh/qa/node1/mysql.sock -e"drop database if exists test_three; create database test_three;" > /dev/null 2>&1
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      sysbench /usr/share/sysbench/oltp_insert.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_three  --mysql-user=sysbench --mysql-password=sysbench --db-driver=mysql  --mysql-socket=/home/ramesh/qa/node1/mysql.sock prepare >/home/ramesh/qa/log/sysbench_prepare.log
      		 
       
      		 
      /home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/bin/mysqld --defaults-file=/home/ramesh/qa/conf/node3.cnf --wsrep-provider=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt/lib/libgalera_smm.so --datadir=/home/ramesh/qa/node3 --basedir=/home/ramesh/qa/GAL_MD060122-mariadb-10.8.0-linux-x86_64-opt >> /home/ramesh/qa/node3/node3.err 2>&1 &

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25975 Turning on innodb_disallow_writes will cause mysqladmin shutdown to hang

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28256 Galera SST fails with 'InnoDB: Upgrade after a crash is not supported'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31506 Updating cluster to a new major version Reports Upgrade after a crash is not supported with rsync sst

          • Major - Major loss of function.
          • Confirmed

          Task - A task that needs to be done. MDEV-31536 Galera mariadb-backup to work as SST between major versions

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12353 Efficient InnoDB redo log record format

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-14425 Change the InnoDB redo log format to reduce write amplification

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15912 InnoDB: Failing assertion: purge_sys.tail.commit <= purge_sys.rseg->last_commit upon upgrade from 10.0 or 10.1 to 10.3

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-29162 Major upgrade instructions missing for Galera

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-36202 rsync and mariabackup SST impossible if joiner has newer major version than donor

          • Major - Major loss of function.
          • Open

          Activity

            Transition Time In Source Status Execution Times
            Marko Mäkelä made transition -
            Open Needs Feedback
            		90d 1h 42m 1
            Marko Mäkelä made transition -
            Open In Progress
            		15h 57m 1
            Marko Mäkelä made transition -
            In Progress Needs Feedback
            		8m 3s 1
            Ramesh Sivaraman made transition -
            Needs Feedback Open
            		3d 2h 43m 2
            Marko Mäkelä made transition -
            Open Closed
            		1d 22h 7m 1

            People

              marko Marko Mäkelä
              ramesh Ramesh Sivaraman
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.