[MDEV-22397] repeatable crashes: InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.12/storage/innobase/btr/btr0cur.cc line 311 Created: 2020-04-28  Updated: 2021-05-05  Resolved: 2020-11-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.4
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Eugene Z. Zheganin Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

Centos Linux 7.7
MariaDB 10.4.12
Linux 3.10.0-1062.9.1.el7.x86_64


Issue Links:
Relates
relates to MDEV-18223 Crash during insert (?) Closed
relates to MDEV-18223 Crash during insert (?) Closed
relates to MDEV-24014 Assertion failure in btr0pcur.cc duri... Open

 Description   

Got repeatable crashes with the following traces in logs.
So far got no cores, will supply bts and cores asap.

Apr 28 14:20:39 sso-db02 mysqld: 2020-04-28 14:20:39 0 [Note] WSREP:  cleaning up 75554af7 (tcp://188.186.154.182:4567)
Apr 28 15:26:20 sso-db02 mysqld: 2020-04-28 15:26:20 0x7efa894c8700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.12/storage/innobase/btr/btr0cur.cc line 311
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: Failing assertion: btr_page_get_next(latch_leaves.blocks[0]->frame) == page_get_page_no(page)
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: We intentionally generate a memory trap.
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: If you get repeated assertion failures or crashes, even
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: immediately after the mysqld startup, there may be
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
Apr 28 15:26:20 sso-db02 mysqld: InnoDB: about forcing recovery.
Apr 28 15:26:20 sso-db02 mysqld: 200428 15:26:20 [ERROR] mysqld got signal 6 ;
Apr 28 15:26:20 sso-db02 mysqld: This could be because you hit a bug. It is also possible that this binary
Apr 28 15:26:20 sso-db02 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Apr 28 15:26:20 sso-db02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Apr 28 15:26:20 sso-db02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Apr 28 15:26:20 sso-db02 mysqld: We will try our best to scrape up some info that will hopefully help
Apr 28 15:26:20 sso-db02 mysqld: diagnose the problem, but since we have already crashed,
Apr 28 15:26:20 sso-db02 mysqld: something is definitely wrong and this may fail.
Apr 28 15:26:20 sso-db02 mysqld: Server version: 10.4.12-MariaDB
Apr 28 15:26:20 sso-db02 mysqld: key_buffer_size=16777216
Apr 28 15:26:20 sso-db02 mysqld: read_buffer_size=8388608
Apr 28 15:26:20 sso-db02 mysqld: max_used_connections=194
Apr 28 15:26:20 sso-db02 mysqld: max_threads=1026
Apr 28 15:26:20 sso-db02 mysqld: thread_count=190
Apr 28 15:26:20 sso-db02 mysqld: It is possible that mysqld could use up to
Apr 28 15:26:20 sso-db02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8684712 K  bytes of memory
Apr 28 15:26:20 sso-db02 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Apr 28 15:26:20 sso-db02 mysqld: Thread pointer: 0x7ef9b8124ff8
Apr 28 15:26:20 sso-db02 mysqld: Attempting backtrace. You can use the following information to find out
Apr 28 15:26:20 sso-db02 mysqld: where mysqld died. If you see no messages after this, something went
Apr 28 15:26:20 sso-db02 mysqld: terribly wrong...
Apr 28 15:26:20 sso-db02 mysqld: stack_bottom = 0x7efa894c7cf0 thread_stack 0x49000
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563680ae2a1e]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563680576d8f]
Apr 28 15:26:20 sso-db02 mysqld: /lib64/libpthread.so.0(+0xf630)[0x7efcfdd32630]
Apr 28 15:26:20 sso-db02 mysqld: /lib64/libc.so.6(gsignal+0x37)[0x7efcfc002377]
Apr 28 15:26:20 sso-db02 mysqld: /lib64/libc.so.6(abort+0x148)[0x7efcfc003a68]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0x5b18be)[0x56368026a8be]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xbcb9dc)[0x5636808849dc]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xbd6d20)[0x56368088fd20]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xb48a3b)[0x563680801a3b]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xb4b2f3)[0x5636808042f3]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xcbfe46)[0x563680978e46]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xcc0f81)[0x563680979f81]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xb55950)[0x56368080e950]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xb00f5d)[0x5636807b9f5d]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xb96f4b)[0x56368084ff4b]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xb971ae)[0x5636808501ae]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(+0xa5d308)[0x563680716308]
Apr 28 15:26:20 sso-db02 mysqld: /usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0x143)[0x56368057a373]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x57)[0x56368045adf7]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x338f)[0x56368036320f]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x24b)[0x5636803694eb]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(+0x6b0d99)[0x563680369d99]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2b06)[0x56368036d296]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56368036da4c]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56368044c32a]
Apr 28 15:26:21 sso-db02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56368044c40d]
Apr 28 15:26:21 sso-db02 mysqld: /lib64/libpthread.so.0(+0x7ea5)[0x7efcfdd2aea5]
Apr 28 15:26:21 sso-db02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7efcfc0ca8cd]
Apr 28 15:26:21 sso-db02 mysqld: Trying to get some variables.
Apr 28 15:26:21 sso-db02 mysqld: Some pointers may be invalid and cause the dump to abort.
Apr 28 15:26:21 sso-db02 mysqld: Query (0x7ef9b822c0d0): ROLLBACK
Apr 28 15:26:21 sso-db02 mysqld: Connection ID (thread ID): 295
Apr 28 15:26:21 sso-db02 mysqld: Status: NOT_KILLED
Apr 28 15:26:21 sso-db02 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
Apr 28 15:26:21 sso-db02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Apr 28 15:26:21 sso-db02 mysqld: information that should help you find out what is causing the crash.
Apr 28 15:26:21 sso-db02 mysqld: Writing a core file...
Apr 28 15:26:21 sso-db02 mysqld: Working directory at /var/lib/mysql
Apr 28 15:26:21 sso-db02 mysqld: Resource Limits:
Apr 28 15:26:21 sso-db02 mysqld: Limit                     Soft Limit           Hard Limit           Units
Apr 28 15:26:21 sso-db02 mysqld: Max cpu time              unlimited            unlimited            seconds
Apr 28 15:26:21 sso-db02 mysqld: Max file size             unlimited            unlimited            bytes
Apr 28 15:26:21 sso-db02 mysqld: Max data size             unlimited            unlimited            bytes
Apr 28 15:26:21 sso-db02 mysqld: Max stack size            8388608              unlimited            bytes
Apr 28 15:26:21 sso-db02 mysqld: Max core file size        0                    unlimited            bytes
Apr 28 15:26:21 sso-db02 mysqld: Max resident set          unlimited            unlimited            bytes
Apr 28 15:26:21 sso-db02 mysqld: Max processes             63374                63374                processes
Apr 28 15:26:21 sso-db02 mysqld: Max open files            16364                16364                files
Apr 28 15:26:21 sso-db02 mysqld: Max locked memory         65536                65536                bytes
Apr 28 15:26:21 sso-db02 mysqld: Max address space         unlimited            unlimited            bytes
Apr 28 15:26:21 sso-db02 mysqld: Max file locks            unlimited            unlimited            locks
Apr 28 15:26:21 sso-db02 mysqld: Max pending signals       63374                63374                signals
Apr 28 15:26:21 sso-db02 mysqld: Max msgqueue size         819200               819200               bytes
Apr 28 15:26:21 sso-db02 mysqld: Max nice priority         0                    0
Apr 28 15:26:21 sso-db02 mysqld: Max realtime priority     0                    0
Apr 28 15:26:21 sso-db02 mysqld: Max realtime timeout      unlimited            unlimited            us
Apr 28 15:26:21 sso-db02 mysqld: Core pattern: core



 Comments   
Comment by Marko Mäkelä [ 2020-10-23 ]

A similar bug MDEV-24014 was filed today. In order to be able to proceed, we would need more information, and something repeatable. Did the server ever crash before this? Were any DDL statements executed on the table in the past?

Generated at Thu Feb 08 09:14:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.