Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29302

Failing assertion: trx->lock.table_locks.empty()

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.6.5
    • N/A
    • production, docker, linux
      25GB of memory free
      CPU about 15%
      About 260 databases on the server

    Description

      Few seconds after creating new database and during table migration into it, mariadb server failed with following error lines.

      Our configuration:

      [mysqld]
      max_allowed_packet=64M
      back_log=1000
      log_warnings=3
      max_connections=1512
      max_connect_errors=4294967295
      key_buffer_size=1G
      tmp_table_size=512M
      thread_cache_size=200
      innodb_rollback_on_timeout=1
      innodb_buffer_pool_size=8G
      innodb_log_file_size=4G
      innodb_flush_log_at_trx_commit=1
      innodb_file_per_table=1
      

      First occurrence log:

      2022-08-15  7:30:26 273422829 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-15  7:30:26 273422829 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-15  7:30:26 273422829 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-15  7:30:27 273422829 [Note] InnoDB: Cannot close file ./db_195/user_sessions.ibd because of pending fsync
      2022-08-15 07:30:27 0x7efcc0474700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.5/storage/innobase/trx/trx0trx.cc line 918
      InnoDB: Failing assertion: trx->lock.table_locks.empty()
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mariadbd startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      220815  7:30:27 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.6.5-MariaDB-1:10.6.5+maria~focal
      key_buffer_size=1073741824
      read_buffer_size=131072
      max_used_connections=339
      max_threads=1026
      thread_count=200
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3307842 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7efa94000c58
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7efcc0473d98 thread_stack 0x49000
      mysqld(my_print_stacktrace+0x32)[0x5630e4505442]
      Printing to addr2line failed
      mysqld(handle_fatal_signal+0x485)[0x5630e3fb86a5]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7effa64163c0]
      

      UPDATE 2022-08-24:
      Today this error occurred again, under same conditions. We created new DB named "db_276" and started to migrate tables.

      2022-08-24  7:07:27 16520720 [Note] InnoDB: Cannot close file ./db_103/events.ibd because of pending fsync
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:27 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:28 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:29 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:29 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:30 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:31 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Cannot close file ./db_103/events.ibd because of pending fsync
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:32 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:33 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:34 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:35 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:35 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:35 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:35 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:36 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:36 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:36 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:36 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:37 16520720 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
      2022-08-24  7:07:37 16520720 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files
      2022-08-24  7:07:37 16520720 [Note] InnoDB: Online DDL : Applying log to index
      2022-08-24  7:07:37 16520720 [Note] InnoDB: Cannot close file ./db_103/events.ibd because of pending fsync
      2022-08-24 07:07:37 0x7fb5dc3de700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.5/storage/innobase/trx/trx0trx.cc line 918
      InnoDB: Failing assertion: trx->lock.table_locks.empty()
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mariadbd startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      220824  7:07:37 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.6.5-MariaDB-1:10.6.5+maria~focal
      key_buffer_size=1073741824
      read_buffer_size=131072
      max_used_connections=641
      max_threads=1514
      thread_count=201
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4382437 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7fb3ac000c58
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fb5dc3ddd98 thread_stack 0x49000
      mysqld(my_print_stacktrace+0x32)[0x559b6f5e7442]
      mysqld(handle_fatal_signal+0x485)[0x559b6f09a6a5]
      

      Attachments

        Activity

          People

            vlad.lesin Vladislav Lesin
            theese Patrik Vaclavek
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.