Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40835

InnoDB: Malformed log record

    XMLWordPrintable

Details

    • Can result in data loss

    Description

      Similar to MDEV-40833, I am unable to find when this type of failure first started to occur. It is reproducible also with the BACKUP SERVER implementation of MDEV-14992. Here is a recent example of a failure:

      main 87bcaaaa983d4b1f2163fc36e50d7b48acfd0f81

      binlog_in_engine.mariabackup_slave_provision_nolock w3 [ fail ]
              Test ended at 2026-08-05 16:05:50
       
      CURRENT_TEST: binlog_in_engine.mariabackup_slave_provision_nolock
      [00] 2026-08-05 16:05:42 Connecting to MariaDB server host: localhost, user: root, password: not set, port: 19180, socket: /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/tmp/3/mysqld.1.sock
      [00] 2026-08-05 16:05:42 Using server version 13.1.0-MariaDB-log
      /home/buildbot/amd64-opensuse-1600/build/extra/mariabackup/mariadb-backup based on MariaDB server 13.1.0-MariaDB Linux (x86_64)
      [00] 2026-08-05 16:05:42 uses posix_fadvise().
      [00] 2026-08-05 16:05:42 cd to /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/mysqld.1/data/
      [00] 2026-08-05 16:05:42 open files limit requested 1024, set to 1024
      [00] 2026-08-05 16:05:42 mariabackup: using the following InnoDB configuration:
      [00] 2026-08-05 16:05:42 innodb_data_home_dir = 
      [00] 2026-08-05 16:05:42 innodb_data_file_path = ibdata1:12M:autoextend
      [00] 2026-08-05 16:05:42 innodb_log_group_home_dir = ./
      2026-08-05 16:05:42 0 [Note] InnoDB: Number of transaction pools: 1
      /home/buildbot/amd64-opensuse-1600/build/extra/mariabackup/mariadb-backup: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
      create_uring failed: falling back to libaio
      2026-08-05 16:05:42 0 [Note] InnoDB: Using Linux native AIO
      2026-08-05 16:05:42 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
      [00] 2026-08-05 16:05:42 mariabackup: Generating a list of tablespaces
      [00] 2026-08-05 16:05:42 DDL tracking : create 8 "./test/t1.ibd"
      [00] 2026-08-05 16:05:42 DDL tracking : create 9 "./test/t2.ibd"
      [00] 2026-08-05 16:05:42 Connecting to MariaDB server host: localhost, user: root, password: not set, port: 19180, socket: /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/tmp/3/mysqld.1.sock
      [00] 2026-08-05 16:05:42 Connecting to MariaDB server host: localhost, user: root, password: not set, port: 19180, socket: /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/tmp/3/mysqld.1.sock
      [00] 2026-08-05 16:05:42 Starting BACKUP STAGE START
      [00] 2026-08-05 16:05:42 Copying /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/mysqld.1/data//aria_log_control to /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/tmp/backup/aria_log_control
      [00] 2026-08-05 16:05:42         ...done
      [00] 2026-08-05 16:05:42 Loading aria_log_control.
      [00] 2026-08-05 16:05:42 aria_log_control: last_log_number: 1
      [00] 2026-08-05 16:05:42 Start scanning aria tables.
      [00] 2026-08-05 16:05:42 Start scanning aria log files.
      [00] 2026-08-05 16:05:42 Found 1 aria log files, minimum log number 1, maximum log number 1
      [00] 2026-08-05 16:05:42 Stop scanning aria tables.
      [00] 2026-08-05 16:05:42 Copying .//undo001 to /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/tmp/backup/undo001
      [00] 2026-08-05 16:05:44 Writing mariadb_backup_info
      [00] 2026-08-05 16:05:44         ...done
      [00] 2026-08-05 16:05:44 Redo log (from LSN 6601253 to 23166766) was copied.
      2026-08-05 16:05:44 0 [Note] InnoDB: Starting shutdown...
      [00] 2026-08-05 16:05:45 completed OK!
      /home/buildbot/amd64-opensuse-1600/build/extra/mariabackup/mariadb-backup based on MariaDB server 13.1.0-MariaDB Linux (x86_64)
      [00] 2026-08-05 16:05:45 cd to /home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/tmp/backup/
      [00] 2026-08-05 16:05:45 open files limit requested 1024, set to 1024
      [00] 2026-08-05 16:05:45 Renaming ./test/t2.new to ./test/t2.ibd
       
      [00] 2026-08-05 16:05:45 Renaming ./test/t1.new to ./test/t1.ibd
       
      [00] 2026-08-05 16:05:45 This target seems to be not prepared yet.
      [00] 2026-08-05 16:05:45 mariabackup: using the following InnoDB configuration for recovery:
      [00] 2026-08-05 16:05:45 innodb_data_home_dir = .
      [00] 2026-08-05 16:05:45 innodb_data_file_path = ibdata1:12M:autoextend
      [00] 2026-08-05 16:05:45 innodb_log_group_home_dir = .
      [00] 2026-08-05 16:05:45 Starting InnoDB instance for recovery.
      [00] 2026-08-05 16:05:45 mariabackup: Using 100663296 bytes for buffer pool (set by --use-memory parameter)
      2026-08-05 16:05:45 0 [Note] InnoDB: Compressed tables use zlib 1.3.1.zlib-ng
      2026-08-05 16:05:45 0 [Note] InnoDB: Number of transaction pools: 1
      2026-08-05 16:05:45 0 [Note] InnoDB: Using AVX512 instructions
      /home/buildbot/amd64-opensuse-1600/build/extra/mariabackup/mariadb-backup: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
      create_uring failed: falling back to libaio
      2026-08-05 16:05:45 0 [Note] InnoDB: Using Linux native AIO
      2026-08-05 16:05:45 0 [Note] InnoDB: innodb_buffer_pool_size_max=96m, innodb_buffer_pool_size=96m
      2026-08-05 16:05:45 0 [Note] InnoDB: Completed initialization of buffer pool
      2026-08-05 16:05:45 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
      2026-08-05 16:05:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=47152
      2026-08-05 16:05:45 0 [ERROR] InnoDB: Missing recovery record in file_no=54, page_no=1, LSN 22467950
      2026-08-05 16:05:45 0 [ERROR] InnoDB: Malformed log record at LSN 22468302; set innodb_force_recovery=1 to ignore.
      2026-08-05 16:05:45 0 [Note] InnoDB: End of log at LSN=22468302
      2026-08-05 16:05:45 0 [ERROR] InnoDB: Log scan aborted at LSN 22468302
      [00] 2026-08-05 16:05:45 mariadb-backup: srv_start() returned 11 (Generic error).
      /home/buildbot/amd64-opensuse-1600/build/extra/mariabackup/mariadb-backup: Warning: 0 files and 1 streams is left open
       
      Error in my_thread_global_end(): 2 threads didn't exit
      mysqltest: In included file "/home/buildbot/amd64-opensuse-1600/build/mysql-test/suite/binlog_in_engine/include/mariabackup_slave_provision.inc": 
      included from /home/buildbot/amd64-opensuse-1600/build/mysql-test/suite/binlog_in_engine/mariabackup_slave_provision_nolock.test at line 5:
      At line 46: exec of '/home/buildbot/amd64-opensuse-1600/build/extra/mariabackup/mariadb-backup --defaults-file=/home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/my.cnf --prepare --target-dir=/home/buildbot/amd64-opensuse-1600/build/mysql-test/var/3/tmp/backup ' failed, error: 256, status: 1, errno: 38
      

      This is also failing rather often in the test mariabackup.undo_truncate.

      To analyze this, we'd ideally need a run that includes rr record of the mariadbd process that was being backed up, as well as copies of the backup before any mariadb-backup --prepare was executed.

      Attachments

        Issue Links

          Activity

            People

              saahil Saahil Alam
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 0.5h
                  0.5h

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.