Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28880

Upgrade from MariaDB 10.3.32 to MariaDB 10.6.8 has errors/issues/documentation

    XMLWordPrintable

Details

    Description

      Hello,
      We upgraded 5 slaves and 1 master from MariaDB 10.3.32 to MariaDB 10.6.8
      Here are error, issue, documentation suggestions we encountered.

      1.
      4 of 5 of the slaves had the following error:

      [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.3.32.
      

      preventing startup after upgrade. so mariadb_upgrade could not be run and server could not be started

      Therefore we downgraded back to 10.3.32 and ran MySQL commands

      SET GLOBAL innodb_fast_shutdown = 0;
      SET GLOBAL innodb_buffer_pool_dump_at_shutdown=OFF;
      

      then shutdown then upgraded to 10.6.8 again with the same problem.

      We had to delete the ib_logfile0 file, upgrade to 10.6.8 and restart to upgrade.

      Here are related logs:

      2022-06-15 15:17:56 0 [Note] InnoDB: FTS optimize thread exiting.
      2022-06-15 15:17:56 1 [Note] InnoDB: to purge 2 transactions
      2022-06-15 15:17:56 0 [Note] InnoDB: Buffer pool(s) load aborted due to user instigated abort at 220615 15:17:56
      2022-06-15 15:17:56 0 [Note] InnoDB: Starting shutdown...
      2022-06-15 15:18:07 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      2022-06-15 15:18:07 0 [Note] InnoDB: Shutdown completed; log sequence number 30535472345669; transaction id 71002773175
      2022-06-15 15:18:07 0 [Note] /usr/sbin/mysqld: Shutdown complete
       
      2022-06-15 15:24:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2022-06-15 15:24:17 0 [Note] InnoDB: Using transactional memory
      2022-06-15 15:24:17 0 [Note] InnoDB: Number of pools: 1
      2022-06-15 15:24:17 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2022-06-15 15:24:17 0 [Note] InnoDB: Using Linux native AIO
      2022-06-15 15:24:17 0 [Note] InnoDB: Initializing buffer pool, total size = 332859965440, chunk size = 134217728
      2022-06-15 15:24:18 0 [Note] InnoDB: Completed initialization of buffer pool
      2022-06-15 15:24:20 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.3.32.
      2022-06-15 15:24:20 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      2022-06-15 15:24:20 0 [Note] InnoDB: Starting shutdown...
      2022-06-15 15:24:20 0 [ERROR] Plugin 'InnoDB' init function returned error.
      2022-06-15 15:24:20 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      2022-06-15 15:24:20 0 [Note] Plugin 'FEEDBACK' is disabled.
      2022-06-15 15:24:20 0 [ERROR] Failed to initialize plugins.
      2022-06-15 15:24:20 0 [ERROR] Aborting
      

      Here are related my.cnf settings:

      innodb_log_files_in_group      = 1
      innodb_log_file_size           = 16G
      innodb_flush_log_at_trx_commit = 1
      

      What we did not try: Stop slave replication manually before shutdown and then upgrade.
      Other notes: We had major problems in the past before changing innodb_change_buffering to none. Basically restarting slaves broke some index in the database nearly all the time. We ran scripts to check all tables rebuild all the broken table indexes using OPTIMIZE TABLE. On the other hand one of the servers that did not get this error was one rebuild recently from mariadbbackup. However the backup server running the mariadbbackup had the same error upgrading. Just adding in case some recovery we did broke the database before upgrade, but we don't think that is the issue.
      Other notes: Mariadb database is about 1.5 TB, with peak about 50,000 selects per second, 50 deletes per second, 500 inserts per second, 500 updates per second on master, with averages over half the peak

      2.
      On four out of five of the slaves after upgrade, we got

      2022-06-16 13:57:57 8 [ERROR] mariadbd: Can't find record in 'TABLENAMEREDACTED'
      

      one server had it twenty times in the last month, one server had it four times, one server had it twice, one server had it once.
      CHECK TABLE TABLENAMEREDACTED shows no error.
      percona checksum showed no data difference.
      There was no other slave replication problems.
      TABLENAMEREDACTED is the same table on all servers, and it is actually relatively not a big table or busy table.

      We performed OPTIMIZE TABLE TABLENAMEREDACTED on the master and slaves to recreate table just in case.
      However, we are still gettings errors on slaves with the same table, even after performing OPTIMIZE TABLE TABLENAMEREDACTED.
      Also, percona checksum still shows no data difference between servers.
      CHECK TABLE TABLENAMEREDACTED still shows no error.
      There are still no other slave replication problems.

      Other notes: all servers are runing with the following setting which enable parallel replication.

      slave_parallel_threads = 8
      

      3.
      The documentation in MariaDB suggests running mariadb_upgrade after updating MariaDB.
      I would suggest putting in the documentation to restart mariadb after running mariadb_upgrade as that seems to potentially remove some problems that may occur because of startup without the fixes mariadb_upgrade does.

      Attachments

        Issue Links

          Activity

            People

              thiru Thirunarayanan Balathandayuthapani
              FK F K
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.