Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34233

InnoDB crashes due to corrupted ibdata1 (Assertion failure in innodb.undo_page)

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.6.17
    • N/A
    • Platform RedHat
    • CPanel 11.118.0.11, CloudLinux 8, Virtual Machine (KVM), 178G RAM

    Description

      During the day with no events happening at the same time, innodb1 turns out corrupted.

      2024-05-23 12:12:01 8668229 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
      2024-05-23 12:12:01 8668229 [ERROR] InnoDB: File './ibdata1' is corrupted
      2024-05-23 12:12:01 0x7f0944bad700  InnoDB: Assertion failure in file /builddir/build/BUILD/cl-MariaDB106-10.6.17/mariadb-10.6.17/storage/innobase/trx/trx0purge.cc line 268
      InnoDB: Failing assertion: undo_page
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mariadbd startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      240523 12:12:01 [ERROR] mysqld got signal 6 ;
      Sorry, we probably made a mistake, and this is a bug.
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
      Server version: 10.6.17-MariaDB-cll-lve source revision: 15c75ad083a55e198ae78324f22970694b72f22b
      key_buffer_size=402653184
      read_buffer_size=2097152
      max_used_connections=307
      max_threads=802
      thread_count=33
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3699403 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      Thread pointer: 0x7f078463b9a8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f0944bacb98 thread_stack 0x49000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ec22da47ae]
      /usr/sbin/mariadbd(handle_fatal_signal+0x475)[0x55ec2287ed95]
      /lib64/libpthread.so.0(+0x12cf0)[0x7f0c8d514cf0]
      2024-05-23 12:12:05 0x7f0944ed1700  InnoDB: Assertion failure in file /builddir/build/BUILD/cl-MariaDB106-10.6.17/mariadb-10.6.17/storage/innobase/buf/buf0lru.cc line 281
      InnoDB: Failing assertion: !block->page.in_file()
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mariadbd startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      mariadb.service: Main process exited, code=killed, status=6/ABRT
      mariadb.service: Failed with result 'signal'.
      

      The size of the ibdata1 is about 6931087360 bytes for reference.

      This resulted in broken mini-transaction which was later recovered from via innodb_force_recovery=1 (setting the value, starting, stopping, removing the value, starting). The database is running without issues after that.

      There is no core dump logged.

      What is concerning is that the previous week the database was rsynced and migrated from CloudLinux 7. However no problems where noticed while migrating the database.

      I would like to offer more information about the incident, but non is available. If you feel that information about the following mini-transaction problem is interesting I can post that too. I post this issue hoping that someone can shed some light on what our next step would be. Known bug? Need to upgrade?

      My.cnf

      [mysqld]
      character_set_server = utf8mb4
      collation_server = utf8mb4_unicode_ci
      ft_min_word_len = 3
      innodb_buffer_pool_size = 12G
      innodb_file_per_table = 1
      key_buffer_size = 384M
      max_allowed_packet = 268435456
      max_connections = 800
      max_user_connections = 100
      max_heap_table_size = 256M
      myisam_sort_buffer_size = 64M
      myisam_use_mmap = 1
      open_files_limit = 80000
      performance-schema = 1
      read_buffer_size = 2M
      read_rnd_buffer_size = 8M
      sort_buffer_size = 2M
      table_open_cache = 4000
      thread_cache_size = 8
      tmp_table_size = 256M
      join_buffer_space_limit = 24M
      join_cache_level = 8
      join_buffer_size = 24M
      query_cache_size = 128M
      query_cache_limit = 8M
      query_cache_type = 0
      tmp_disk_table_size = 805306368
      ignore_db_dirs = lost+found
      explicit_defaults_for_timestamp = On
      optimizer_switch = ""
      sql_mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE
      log_warnings=4
      tmpdir = "/var/cache/mysql"
      ssl_cert = /etc/mysql/ssl/fullchain.pem
      ssl_key = /etc/mysql/ssl/privkey.pem
      

      /var/cache/mysql is an ext4 fs with 10G space.

      Attachments

        Issue Links

          Activity

            martin-tag24 Martin added a comment -

            @marko the error occurred again on our cluster on a different server this time. Can we somehow help?

            Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 1 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
            Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 1 [ERROR] InnoDB: File './ibdata1' is corrupted
            Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 0x7f461847b6c0  InnoDB: Assertion failure in file ./storage/innobase/trx/trx0purge.cc line 262
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Failing assertion: undo_page
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: We intentionally generate a memory trap.
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: If you get repeated assertion failures or crashes, even
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: immediately after the mariadbd startup, there may be
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: about forcing recovery.
            Sep 17 14:19:11 tag02 mariadbd[1615]: 240917 14:19:11 [ERROR] mysqld got signal 6 ;
            Sep 17 14:19:11 tag02 mariadbd[1615]: Sorry, we probably made a mistake, and this is a bug.
            Sep 17 14:19:11 tag02 mariadbd[1615]: Your assistance in bug reporting will enable us to fix this for the next release.
            Sep 17 14:19:11 tag02 mariadbd[1615]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Sep 17 14:19:11 tag02 mariadbd[1615]: We will try our best to scrape up some info that will hopefully help
            Sep 17 14:19:11 tag02 mariadbd[1615]: diagnose the problem, but since we have already crashed,
            Sep 17 14:19:11 tag02 mariadbd[1615]: something is definitely wrong and this may fail.
            Sep 17 14:19:11 tag02 mariadbd[1615]: Server version: 10.11.6-MariaDB-0+deb12u1 source revision:
            Sep 17 14:19:11 tag02 mariadbd[1615]: key_buffer_size=134217728
            Sep 17 14:19:11 tag02 mariadbd[1615]: read_buffer_size=131072
            Sep 17 14:19:11 tag02 mariadbd[1615]: max_used_connections=22
            Sep 17 14:19:11 tag02 mariadbd[1615]: max_threads=153
            Sep 17 14:19:11 tag02 mariadbd[1615]: thread_count=24
            Sep 17 14:19:11 tag02 mariadbd[1615]: It is possible that mysqld could use up to
            Sep 17 14:19:11 tag02 mariadbd[1615]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468057 K  bytes of memory
            Sep 17 14:19:11 tag02 mariadbd[1615]: Hope that's ok; if not, decrease some variables in the equation.
            Sep 17 14:19:11 tag02 mariadbd[1615]: Thread pointer: 0x7f45f8000c68
            Sep 17 14:19:11 tag02 mariadbd[1615]: Attempting backtrace. You can use the following information to find out
            Sep 17 14:19:11 tag02 mariadbd[1615]: where mysqld died. If you see no messages after this, something went
            Sep 17 14:19:11 tag02 mariadbd[1615]: terribly wrong...
            Sep 17 14:19:11 tag02 mariadbd[1615]: stack_bottom = 0x7f461847aca8 thread_stack 0x49000
            Sep 17 14:19:11 tag02 mariadbd[1615]: Printing to addr2line failed
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5626b9a5126e]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(handle_fatal_signal+0x2b3)[0x5626b95c1593]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f461a05b050]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae3c)[0x7f461a0a9e3c]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f461a05afb2]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f461a045472]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0x698064)[0x5626b9225064]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0x694423)[0x5626b9221423]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xdb8516)[0x5626b9945516]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0x6fb0a0)[0x5626b92880a0]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xdb694e)[0x5626b994394e]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xdb69f2)[0x5626b99439f2]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xcc9620)[0x5626b9856620]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xcc97fc)[0x5626b98567fc]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xa3860f)[0x5626b95c560f]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(_Z15ha_commit_transP3THDb+0x3da)[0x5626b95c639a]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(_Z12trans_commitP3THD+0x41)[0x5626b94b1ec1]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(_ZN27Wsrep_high_priority_service6commitERKN5wsrep9ws_handleERKNS0_7ws_metaE+0x138)[0x5626b981d838]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xf56f83)[0x5626b9ae3f83]
            Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd(+0xf674bf)[0x5626b9af44bf]
            Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 0x7f45c83f46c0  InnoDB: Assertion failure in file ./storage/innobase/buf/buf0lru.cc line 285
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Failing assertion: !block->page.in_file()
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: We intentionally generate a memory trap.
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: If you get repeated assertion failures or crashes, even
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: immediately after the mariadbd startup, there may be
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: about forcing recovery.
            

            martin-tag24 Martin added a comment - @marko the error occurred again on our cluster on a different server this time. Can we somehow help? Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 1 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file : . /ibdata1 Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 1 [ERROR] InnoDB: File './ibdata1' is corrupted Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 0x7f461847b6c0 InnoDB: Assertion failure in file . /storage/innobase/trx/trx0purge .cc line 262 Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Failing assertion: undo_page Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: We intentionally generate a memory trap . Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Submit a detailed bug report to https: //jira .mariadb.org/ Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: If you get repeated assertion failures or crashes, even Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: immediately after the mariadbd startup, there may be Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: corruption in the InnoDB tablespace. Please refer to Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: https: //mariadb .com /kb/en/library/innodb-recovery-modes/ Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: about forcing recovery. Sep 17 14:19:11 tag02 mariadbd[1615]: 240917 14:19:11 [ERROR] mysqld got signal 6 ; Sep 17 14:19:11 tag02 mariadbd[1615]: Sorry, we probably made a mistake, and this is a bug. Sep 17 14:19:11 tag02 mariadbd[1615]: Your assistance in bug reporting will enable us to fix this for the next release. Sep 17 14:19:11 tag02 mariadbd[1615]: To report this bug, see https: //mariadb .com /kb/en/reporting-bugs Sep 17 14:19:11 tag02 mariadbd[1615]: We will try our best to scrape up some info that will hopefully help Sep 17 14:19:11 tag02 mariadbd[1615]: diagnose the problem, but since we have already crashed, Sep 17 14:19:11 tag02 mariadbd[1615]: something is definitely wrong and this may fail. Sep 17 14:19:11 tag02 mariadbd[1615]: Server version: 10.11.6-MariaDB-0+deb12u1 source revision: Sep 17 14:19:11 tag02 mariadbd[1615]: key_buffer_size=134217728 Sep 17 14:19:11 tag02 mariadbd[1615]: read_buffer_size=131072 Sep 17 14:19:11 tag02 mariadbd[1615]: max_used_connections=22 Sep 17 14:19:11 tag02 mariadbd[1615]: max_threads=153 Sep 17 14:19:11 tag02 mariadbd[1615]: thread_count=24 Sep 17 14:19:11 tag02 mariadbd[1615]: It is possible that mysqld could use up to Sep 17 14:19:11 tag02 mariadbd[1615]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468057 K bytes of memory Sep 17 14:19:11 tag02 mariadbd[1615]: Hope that's ok; if not, decrease some variables in the equation. Sep 17 14:19:11 tag02 mariadbd[1615]: Thread pointer: 0x7f45f8000c68 Sep 17 14:19:11 tag02 mariadbd[1615]: Attempting backtrace. You can use the following information to find out Sep 17 14:19:11 tag02 mariadbd[1615]: where mysqld died. If you see no messages after this, something went Sep 17 14:19:11 tag02 mariadbd[1615]: terribly wrong... Sep 17 14:19:11 tag02 mariadbd[1615]: stack_bottom = 0x7f461847aca8 thread_stack 0x49000 Sep 17 14:19:11 tag02 mariadbd[1615]: Printing to addr2line failed Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (my_print_stacktrace+0x2e)[0x5626b9a5126e] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (handle_fatal_signal+0x2b3)[0x5626b95c1593] Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc .so.6(+0x3c050)[0x7f461a05b050] Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc .so.6(+0x8ae3c)[0x7f461a0a9e3c] Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc .so.6(gsignal+0x12)[0x7f461a05afb2] Sep 17 14:19:11 tag02 mariadbd[1615]: /lib/x86_64-linux-gnu/libc .so.6(abort+0xd3)[0x7f461a045472] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0x698064)[0x5626b9225064] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0x694423)[0x5626b9221423] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xdb8516)[0x5626b9945516] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0x6fb0a0)[0x5626b92880a0] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xdb694e)[0x5626b994394e] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xdb69f2)[0x5626b99439f2] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xcc9620)[0x5626b9856620] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xcc97fc)[0x5626b98567fc] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xa3860f)[0x5626b95c560f] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (_Z15ha_commit_transP3THDb+0x3da)[0x5626b95c639a] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (_Z12trans_commitP3THD+0x41)[0x5626b94b1ec1] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (_ZN27Wsrep_high_priority_service6commitERKN5wsrep9ws_handleERKNS0_7ws_metaE+0x138)[0x5626b981d838] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xf56f83)[0x5626b9ae3f83] Sep 17 14:19:11 tag02 mariadbd[1615]: /usr/sbin/mariadbd (+0xf674bf)[0x5626b9af44bf] Sep 17 14:19:11 tag02 mariadbd[1615]: 2024-09-17 14:19:11 0x7f45c83f46c0 InnoDB: Assertion failure in file . /storage/innobase/buf/buf0lru .cc line 285 Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Failing assertion: !block->page.in_file() Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: We intentionally generate a memory trap . Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: Submit a detailed bug report to https: //jira .mariadb.org/ Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: If you get repeated assertion failures or crashes, even Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: immediately after the mariadbd startup, there may be Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: corruption in the InnoDB tablespace. Please refer to Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: https: //mariadb .com /kb/en/library/innodb-recovery-modes/ Sep 17 14:19:11 tag02 mariadbd[1615]: InnoDB: about forcing recovery.
            martin-tag24 Martin added a comment -

            @marko and the error happened again on our cluster. this time it corrupted the InnoDB files and the server couldn't fix itself
            same error as above, but here the failed start after the crash if that helps.
            Can we implement a workaround maybe?

            Oct 04 17:34:58 tag04 systemd[1]: Starting mariadb.service - MariaDB 10.11.6 database server...
            Oct 04 17:34:58 tag04 sh[1241]: WSREP: Failed to start mysqld for wsrep recovery: '2024-10-04 17:34:58 0 [Note] Starting MariaDB 10.11.6-MariaDB-0+deb12u1 source revision  as process 1327
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Number of transaction pools: 1
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Using liburing
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Completed initialization of buffer pool
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=814945293428
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] InnoDB: Corrupted page identifier at 814956922748; set innodb_force_recovery=1 to ignore the record.
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: End of log at LSN=814956922748
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Starting shutdown...
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] Plugin 'FEEDBACK' is disabled.
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Warning] 'innodb-locks-unsafe-for-binlog' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] Unknown/unsupported storage engine: InnoDB
            Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] Aborting'
            Oct 04 17:34:58 tag04 systemd[1]: mariadb.service: Control process exited, code=exited, status=1/FAILURE
            Oct 04 17:34:58 tag04 systemd[1]: mariadb.service: Failed with result 'exit-code'.
            Oct 04 17:34:58 tag04 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.6 database server.
            

            martin-tag24 Martin added a comment - @marko and the error happened again on our cluster. this time it corrupted the InnoDB files and the server couldn't fix itself same error as above, but here the failed start after the crash if that helps. Can we implement a workaround maybe? Oct 04 17:34:58 tag04 systemd[1]: Starting mariadb.service - MariaDB 10.11.6 database server... Oct 04 17:34:58 tag04 sh[1241]: WSREP: Failed to start mysqld for wsrep recovery: '2024-10-04 17:34:58 0 [Note] Starting MariaDB 10.11.6-MariaDB-0+deb12u1 source revision as process 1327 Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.13 Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Number of transaction pools: 1 Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Using liburing Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Completed initialization of buffer pool Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes) Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=814945293428 Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] InnoDB: Corrupted page identifier at 814956922748; set innodb_force_recovery=1 to ignore the record. Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: End of log at LSN=814956922748 Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] InnoDB: Starting shutdown ... Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Note] Plugin 'FEEDBACK' is disabled. Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [Warning] 'innodb-locks-unsafe-for-binlog' was removed. It does nothing now and exists only for compatibility with old my.cnf files. Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] Unknown /unsupported storage engine: InnoDB Oct 04 17:34:58 tag04 sh[1241]: 2024-10-04 17:34:58 0 [ERROR] Aborting' Oct 04 17:34:58 tag04 systemd[1]: mariadb.service: Control process exited, code=exited, status=1 /FAILURE Oct 04 17:34:58 tag04 systemd[1]: mariadb.service: Failed with result 'exit-code' . Oct 04 17:34:58 tag04 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.6 database server.
            mandrich M Andrich added a comment -

            I have the same problem. Is there any news yet?

            mandrich M Andrich added a comment - I have the same problem. Is there any news yet?

            Is this reproducible in a version of MariaDB Server that contains a fix of MDEV-34453?

            marko Marko Mäkelä added a comment - Is this reproducible in a version of MariaDB Server that contains a fix of MDEV-34453 ?
            martin-tag24 Martin added a comment -

            @marko we did update our servers today to the (maybe) fixed version and we will monitor if the error happens again.

            # mysql --version
            mysql  Ver 15.1 Distrib 10.11.10-MariaDB, for debian-linux-gnu (x86_64) using  EditLine wrapper
            

            martin-tag24 Martin added a comment - @marko we did update our servers today to the (maybe) fixed version and we will monitor if the error happens again. # mysql --version mysql Ver 15.1 Distrib 10.11.10-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper

            People

              marko Marko Mäkelä
              isodude Josef Johansson
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.