Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33540

mariabackup --prepare: [ERROR] InnoDB: Crash recovery is broken due to insufficient innodb_log_file_size

Details

    Description

      Hello,
      in data center 1, there are 4 nodes:
      3 node galera cluster -> async slave

      From the async slave I'm trying to create a new slave in DC2 that will then form a new galera cluster with 2 more nodes.

      cd /var/lib/mysql
      ssh slave-server 'mariabackup --backup --stream=xbstream --parallel=4' | mbstream -x --parallel=4
      

      ...
      [00] 2024-02-26 03:25:59 MySQL binlog position: filename 'mysqld-bin.003284', position '666302125', GTID of the last change '0-8-18213972007'
      [00] 2024-02-26 03:25:59 Streaming backup-my.cnf
      [00] 2024-02-26 03:25:59         ...done
      [00] 2024-02-26 03:25:59 Streaming xtrabackup_info
      [00] 2024-02-26 03:25:59         ...done
      [00] 2024-02-26 03:25:59 Redo log (from LSN 31507622579405 to 31508457792375) was copied.
      [00] 2024-02-26 03:25:59 completed OK!
      

      # mariabackup --prepare --use-memory=40G --target-dir=/var/lib/mysql
      

      mariabackup based on MariaDB server 10.6.14-MariaDB Linux (x86_64)
      [00] 2024-02-26 08:34:36 cd to /var/lib/mysql/
      [00] 2024-02-26 08:34:36 open files limit requested 0, set to 1024
      [00] 2024-02-26 08:34:36 This target seems to be not prepared yet.
      [00] 2024-02-26 08:34:36 mariabackup: using the following InnoDB configuration for recovery:
      [00] 2024-02-26 08:34:36 innodb_data_home_dir = .
      [00] 2024-02-26 08:34:36 innodb_data_file_path = ibdata1:12M:autoextend
      [00] 2024-02-26 08:34:36 innodb_log_group_home_dir = .
      [00] 2024-02-26 08:34:36 InnoDB: Using liburing
      [00] 2024-02-26 08:34:36 Starting InnoDB instance for recovery.
      [00] 2024-02-26 08:34:36 mariabackup: Using 42949672960 bytes for buffer pool (set by --use-memory parameter)
      2024-02-26  8:34:36 0 [Note] InnoDB: Compressed tables use zlib 1.3
      2024-02-26  8:34:36 0 [Note] InnoDB: Using transactional memory
      2024-02-26  8:34:36 0 [Note] InnoDB: Number of pools: 1
      2024-02-26  8:34:36 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2024-02-26  8:34:36 0 [Note] InnoDB: Using liburing
      2024-02-26  8:34:36 0 [Note] InnoDB: Initializing buffer pool, total size = 42949672960, chunk size = 42949672960
      2024-02-26  8:34:36 0 [Note] InnoDB: Completed initialization of buffer pool
      2024-02-26  8:34:36 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=31507622579405,31507622976800
      2024-02-26  8:34:48 0 [ERROR] InnoDB: Crash recovery is broken due to insufficient innodb_log_file_size; last checkpoint LSN=31507622579405, current LSN=31508457793604. Shutdown is in progress.
      2024-02-26  8:34:48 0 [Note] InnoDB: Starting final batch to recover 237985 pages from redo log.
      2024-02-26  8:34:51 0 [Note] InnoDB: To recover: 174243 pages from log
      2024-02-26  8:35:00 0 [Note] InnoDB: Last binlog file './mysqld-bin.003284', position 666209873
      [00] 2024-02-26 08:35:01 Last binlog file ./mysqld-bin.003284, position 666209873
      [00] 2024-02-26 08:35:01 mariabackup: Recovered WSREP position: 2430d56f-1879-11e8-9a58-defae87a2441:6397994274
       
      [00] 2024-02-26 08:35:02 completed OK!
      

      Then the node is started and it works ok, but this line troubles me:

      2024-02-26  8:34:48 0 [ERROR] InnoDB: Crash recovery is broken due to insufficient innodb_log_file_size; last checkpoint LSN=31507622579405, current LSN=31508457793604. Shutdown is in progress.
      

      Is this expected? All I could find to this was: https://jira.mariadb.org/browse/MDEV-29982

      I've tried MariaDB 10.6.14 and 10.6.17, same error, multiple times.

      Thank you!

      Attachments

        Issue Links

          Activity

            hydrapolic Tomas Mozes added a comment -

            This project/database has been running for some time, it started on MySQL 5.1 or 5.5 (not sure). We then migrated to MariaDB.

            Unfortunately the migration is now complete and the temporary disk was already removed

            hydrapolic Tomas Mozes added a comment - This project/database has been running for some time, it started on MySQL 5.1 or 5.5 (not sure). We then migrated to MariaDB. Unfortunately the migration is now complete and the temporary disk was already removed

            sciascid, can you please create and upload a copy of a mariabackup --backup of a minimal 10.2 data directory where Galera has been enabled? I would use that to debug and simplify the upgrade code path.

            marko Marko Mäkelä added a comment - sciascid , can you please create and upload a copy of a mariabackup --backup of a minimal 10.2 data directory where Galera has been enabled? I would use that to debug and simplify the upgrade code path.

            marko I attached file backup.tar.gz. I created this from MariaDB 10.2, with Galera enabled. I created table and inserted one record before taking the backup.

            sciascid Daniele Sciascia added a comment - marko I attached file backup.tar.gz. I created this from MariaDB 10.2, with Galera enabled. I created table and inserted one record before taking the backup.

            marko Attempting to mariabackup --prepare on the backup I attached with mariabackup 10.6, leads to error "Upgrade after a crash is not supported" suggesting I use 10.4 or earlier. I then tried with 10.4 which produced the following assertion:

            mariabackup: /home/sciascid/Workspace/src/mariadb-server-test/storage/innobase/mtr/mtr0mtr.cc:429: void mtr_t::commit(): Assertion `!m_modifications || !recv_no_log_write' failed.
            

            This is on the path that clears the wsrep xid from trx sys page.
            I can prepare a pull request with the additional checks you suggested.

            sciascid Daniele Sciascia added a comment - marko Attempting to mariabackup --prepare on the backup I attached with mariabackup 10.6, leads to error "Upgrade after a crash is not supported" suggesting I use 10.4 or earlier. I then tried with 10.4 which produced the following assertion: mariabackup: /home/sciascid/Workspace/src/mariadb-server-test/storage/innobase/mtr/mtr0mtr.cc:429: void mtr_t::commit(): Assertion `!m_modifications || !recv_no_log_write' failed. This is on the path that clears the wsrep xid from trx sys page. I can prepare a pull request with the additional checks you suggested.

            sciascid, yes, that is what I had expected to see in a debug build. I am looking forward to the pull request.

            marko Marko Mäkelä added a comment - sciascid , yes, that is what I had expected to see in a debug build. I am looking forward to the pull request.

            People

              sciascid Daniele Sciascia
              hydrapolic Tomas Mozes
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.