Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25344

Two hosts crashed at the same time with: Failing assertion: !cursor->index->is_committed()

Details

    Description

      Two hosts running 10.4.15 crashed exactly at the same time after replicating a certain query via replication, these are the logs:

      Apr 06 01:25:08 db2147 mysqld[12450]: 2021-04-06 01:25:08 0x7f7ac15d4700  InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: Failing assertion: !cursor->index->is_committed()
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: We intentionally generate a memory trap.
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: If you get repeated assertion failures or crashes, even
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: immediately after the mysqld startup, there may be
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: about forcing recovery.
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: 210406  1:25:08 [ERROR] mysqld got signal 6 ;
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: This could be because you hit a bug. It is also possible that this binary
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: or one of the libraries it was linked against is corrupt, improperly built,
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: We will try our best to scrape up some info that will hopefully help
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: diagnose the problem, but since we have already crashed,
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: something is definitely wrong and this may fail.
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: Server version: 10.4.15-MariaDB-log
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: key_buffer_size=134217728
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: read_buffer_size=131072
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: max_used_connections=22
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: max_threads=2010
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: thread_count=20
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: It is possible that mysqld could use up to
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K  bytes of memory
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: Hope that's ok; if not, decrease some variables in the equation.
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: Thread pointer: 0x7f7968000ce8
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: Attempting backtrace. You can use the following information to find out
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: where mysqld died. If you see no messages after this, something went
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: terribly wrong...
      		 
      Apr 06 01:25:08 db2147 mysqld[12450]: stack_bottom = 0x7f7ac15d3698 thread_stack 0x30000
      		 
      Apr 06 01:25:14 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55d0c89976fe]
      		 
      Apr 06 01:25:14 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55d0c849057d]
      		 
      Apr 06 01:25:26 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fdb16ddc730]
      		 
      Apr 06 01:25:36 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fdb164b87bb]
      		 
      Apr 06 01:25:36 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fdb164a3535]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x55d0c8188568]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x55d0c817683e]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x55d0c86dafce]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb377f8)[0x55d0c87127f8]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb37e2f)[0x55d0c8712e2f]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb1297c)[0x55d0c86ed97c]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xa5bfcf)[0x55d0c8636fcf]
      		 
      Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x55d0c849c76b]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x55d0c8335cb7]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2695)[0x55d0c8288145]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55d0c828df89]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x750)[0x55d0c8588000]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x55d0c81e2a52]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x55d0c81eba22]
      		 
      Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x55d0c894819b]
      		 
      Apr 06 01:25:56 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fdb16dd1fa3]
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdb1657a4cf]
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Trying to get some variables.
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Some pointers may be invalid and cause the dump to abort.
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Query (0x7f754a00925a): UPDATE /* WikiPage::updateCategoryCounts  */  `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = 'Images_from_Wiki_Loves_Monuments_missing_SDC_participant_in'
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Connection ID (thread ID): 965915
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Status: NOT_KILLED
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: information that should help you find out what is causing the crash.
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Writing a core file...
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Working directory at /srv/sqldata
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Resource Limits:
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Limit                     Soft Limit           Hard Limit           Units
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max cpu time              unlimited            unlimited            seconds
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max file size             unlimited            unlimited            bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max data size             unlimited            unlimited            bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max stack size            8388608              unlimited            bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max core file size        0                    0                    bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max resident set          unlimited            unlimited            bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max processes             2058337              2058337              processes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max open files            200001               200001               files
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max locked memory         65536                65536                bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max address space         unlimited            unlimited            bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max file locks            unlimited            unlimited            locks
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max pending signals       2058337              2058337              signals
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max msgqueue size         819200               819200               bytes
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max nice priority         0                    0
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max realtime priority     0                    0
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Max realtime timeout      unlimited            unlimited            us
      		 
      Apr 06 01:26:08 db2147 mysqld[12450]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
      		 
      Apr 06 01:26:33 db2147 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT

      This is the second host:

      Apr 06 01:25:08 db2106 mysqld[27097]: 2021-04-06 01:25:08 0x7f975e686700  InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: Failing assertion: !cursor->index->is_committed()
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: We intentionally generate a memory trap.
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: If you get repeated assertion failures or crashes, even
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: immediately after the mysqld startup, there may be
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: about forcing recovery.
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: 210406  1:25:08 [ERROR] mysqld got signal 6 ;
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: This could be because you hit a bug. It is also possible that this binary
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: or one of the libraries it was linked against is corrupt, improperly built,
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: We will try our best to scrape up some info that will hopefully help
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: diagnose the problem, but since we have already crashed,
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: something is definitely wrong and this may fail.
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: Server version: 10.4.15-MariaDB-log
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: key_buffer_size=134217728
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: read_buffer_size=131072
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: max_used_connections=22
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: max_threads=2010
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: thread_count=20
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: It is possible that mysqld could use up to
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K  bytes of memory
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: Hope that's ok; if not, decrease some variables in the equation.
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: Thread pointer: 0x7f948b158af8
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: Attempting backtrace. You can use the following information to find out
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: where mysqld died. If you see no messages after this, something went
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: terribly wrong...
      		 
      Apr 06 01:25:08 db2106 mysqld[27097]: stack_bottom = 0x7f975e685698 thread_stack 0x30000
      		 
      Apr 06 01:25:16 db2106 mysqld[27097]: 2021-04-06  1:25:16 968179 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
      		 
      Apr 06 01:25:16 db2106 mysqld[27097]: 2021-04-06  1:25:16 968179 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2090-bin.006718' at position 516814095; GTID position '0-180359175-3375835514,171978876-171978876-2496072471,171966557-171966557-1948492266,180363436-180363436-1169124439,171978775-
      		 
      Apr 06 01:25:16 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55e74f97a6fe]
      		 
      Apr 06 01:25:16 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55e74f47357d]
      		 
      Apr 06 01:25:32 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7ff7b4f54730]
      		 
      Apr 06 01:25:48 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7ff7b46307bb]
      		 
      Apr 06 01:25:48 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7ff7b461b535]
      		 
      Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x55e74f16b568]
      		 
      Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x55e74f15983e]
      		 
      Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x55e74f6bdfce]
      		 
      Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb377f8)[0x55e74f6f57f8]
      		 
      Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb37e2f)[0x55e74f6f5e2f]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb1297c)[0x55e74f6d097c]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xa5bfcf)[0x55e74f619fcf]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x55e74f47f76b]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x55e74f318cb7]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2695)[0x55e74f26b145]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55e74f270f89]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x750)[0x55e74f56b000]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x55e74f1c5a52]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x55e74f1cea22]
      		 
      Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x55e74f92b19b]
      		 
      Apr 06 01:26:20 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7ff7b4f49fa3]
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7ff7b46f24cf]
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Trying to get some variables.
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Some pointers may be invalid and cause the dump to abort.
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Query (0x7f9261fae06a): UPDATE /* WikiPage::updateCategoryCounts  */  `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = 'Images_from_Wiki_Loves_Monuments_missing_SDC_participant_in'
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Connection ID (thread ID): 968180
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Status: NOT_KILLED
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: information that should help you find out what is causing the crash.
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Writing a core file...
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Working directory at /srv/sqldata
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Resource Limits:
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Limit                     Soft Limit           Hard Limit           Units
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max cpu time              unlimited            unlimited            seconds
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max file size             unlimited            unlimited            bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max data size             unlimited            unlimited            bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max stack size            8388608              unlimited            bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max core file size        0                    0                    bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max resident set          unlimited            unlimited            bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max processes             2058365              2058365              processes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max open files            200001               200001               files
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max locked memory         65536                65536                bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max address space         unlimited            unlimited            bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max file locks            unlimited            unlimited            locks
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max pending signals       2058365              2058365              signals
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max msgqueue size         819200               819200               bytes
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max nice priority         0                    0
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max realtime priority     0                    0
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Max realtime timeout      unlimited            unlimited            us
      		 
      Apr 06 01:26:36 db2106 mysqld[27097]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
      		 
      Apr 06 01:27:04 db2106 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
      		 
      -- Subject: Unit process exited

      The table schema is:

             Table: category
      		 
      Create Table: CREATE TABLE `category` (
      		 
        `cat_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
      		 
        `cat_title` varbinary(255) NOT NULL DEFAULT '',
      		 
        `cat_pages` int(11) NOT NULL DEFAULT 0,
      		 
        `cat_subcats` int(11) NOT NULL DEFAULT 0,
      		 
        `cat_files` int(11) NOT NULL DEFAULT 0,
      		 
        PRIMARY KEY (`cat_id`),
      		 
        UNIQUE KEY `cat_title` (`cat_title`),
      		 
        KEY `cat_pages` (`cat_pages`)
      		 
      ) ENGINE=InnoDB AUTO_INCREMENT=351025254 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8

      The table was checked after the crash and it both reported ok

      root@db2147.codfw.wmnet[commonswiki]> check table category;
      		 
      +----------------------+-------+----------+----------+
      		 
      | Table                | Op    | Msg_type | Msg_text |
      		 
      +----------------------+-------+----------+----------+
      		 
      | commonswiki.category | check | status   | OK       |
      		 
      +----------------------+-------+----------+----------+
      		 
      1 row in set (33.649 sec)
      		 
       
      		 
       
      		 
      root@db2106.codfw.wmnet[commonswiki]> start slave;
      		 
      Query OK, 0 rows affected, 1 warning (0.032 sec)
      		 
       
      		 
      root@db2106.codfw.wmnet[commonswiki]> check table category;
      		 
      +----------------------+-------+----------+----------+
      		 
      | Table                | Op    | Msg_type | Msg_text |
      		 
      +----------------------+-------+----------+----------+
      		 
      | commonswiki.category | check | status   | OK       |
      		 
      +----------------------+-------+----------+----------+
      		 
      1 row in set (35.585 sec)

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22373 Unable to find a record to delete-mark ends up crashing mysqld process after upgrading from 10.1.43 to 10.4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24449 Corruption of system tablespace or last recovered page

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            marostegui Manuel Arostegui created issue -
            marostegui Manuel Arostegui made changes -
            Field Original Value New Value
            Description Two hosts running 10.4.15 crashed exactly at the same time after replicating a certain query via replication, these are the logs:

            {code}
            Apr 06 01:25:08 db2147 mysqld[12450]: 2021-04-06 01:25:08 0x7f7ac15d4700 InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: Failing assertion: !cursor->index->is_committed()
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: We intentionally generate a memory trap.
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: If you get repeated assertion failures or crashes, even
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: immediately after the mysqld startup, there may be
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: about forcing recovery.
            Apr 06 01:25:08 db2147 mysqld[12450]: 210406 1:25:08 [ERROR] mysqld got signal 6 ;
            Apr 06 01:25:08 db2147 mysqld[12450]: This could be because you hit a bug. It is also possible that this binary
            Apr 06 01:25:08 db2147 mysqld[12450]: or one of the libraries it was linked against is corrupt, improperly built,
            Apr 06 01:25:08 db2147 mysqld[12450]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Apr 06 01:25:08 db2147 mysqld[12450]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Apr 06 01:25:08 db2147 mysqld[12450]: We will try our best to scrape up some info that will hopefully help
            Apr 06 01:25:08 db2147 mysqld[12450]: diagnose the problem, but since we have already crashed,
            Apr 06 01:25:08 db2147 mysqld[12450]: something is definitely wrong and this may fail.
            Apr 06 01:25:08 db2147 mysqld[12450]: Server version: 10.4.15-MariaDB-log
            Apr 06 01:25:08 db2147 mysqld[12450]: key_buffer_size=134217728
            Apr 06 01:25:08 db2147 mysqld[12450]: read_buffer_size=131072
            Apr 06 01:25:08 db2147 mysqld[12450]: max_used_connections=22
            Apr 06 01:25:08 db2147 mysqld[12450]: max_threads=2010
            Apr 06 01:25:08 db2147 mysqld[12450]: thread_count=20
            Apr 06 01:25:08 db2147 mysqld[12450]: It is possible that mysqld could use up to
            Apr 06 01:25:08 db2147 mysqld[12450]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
            Apr 06 01:25:08 db2147 mysqld[12450]: Hope that's ok; if not, decrease some variables in the equation.
            Apr 06 01:25:08 db2147 mysqld[12450]: Thread pointer: 0x7f7968000ce8
            Apr 06 01:25:08 db2147 mysqld[12450]: Attempting backtrace. You can use the following information to find out
            Apr 06 01:25:08 db2147 mysqld[12450]: where mysqld died. If you see no messages after this, something went
            Apr 06 01:25:08 db2147 mysqld[12450]: terribly wrong...
            Apr 06 01:25:08 db2147 mysqld[12450]: stack_bottom = 0x7f7ac15d3698 thread_stack 0x30000
            Apr 06 01:25:14 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55d0c89976fe]
            Apr 06 01:25:14 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55d0c849057d]
            Apr 06 01:25:26 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fdb16ddc730]
            Apr 06 01:25:36 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fdb164b87bb]
            Apr 06 01:25:36 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fdb164a3535]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x55d0c8188568]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x55d0c817683e]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x55d0c86dafce]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb377f8)[0x55d0c87127f8]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb37e2f)[0x55d0c8712e2f]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb1297c)[0x55d0c86ed97c]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xa5bfcf)[0x55d0c8636fcf]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x55d0c849c76b]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x55d0c8335cb7]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2695)[0x55d0c8288145]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55d0c828df89]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x750)[0x55d0c8588000]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x55d0c81e2a52]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x55d0c81eba22]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x55d0c894819b]
            Apr 06 01:25:56 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fdb16dd1fa3]
            Apr 06 01:26:08 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdb1657a4cf]
            Apr 06 01:26:08 db2147 mysqld[12450]: Trying to get some variables.
            Apr 06 01:26:08 db2147 mysqld[12450]: Some pointers may be invalid and cause the dump to abort.
            Apr 06 01:26:08 db2147 mysqld[12450]: Query (0x7f754a00925a): UPDATE /* WikiPage::updateCategoryCounts */ `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = 'Images_from_Wiki_Loves_Monuments_missing_SDC_participant_in'
            Apr 06 01:26:08 db2147 mysqld[12450]: Connection ID (thread ID): 965915
            Apr 06 01:26:08 db2147 mysqld[12450]: Status: NOT_KILLED
            Apr 06 01:26:08 db2147 mysqld[12450]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on
            Apr 06 01:26:08 db2147 mysqld[12450]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            Apr 06 01:26:08 db2147 mysqld[12450]: information that should help you find out what is causing the crash.
            Apr 06 01:26:08 db2147 mysqld[12450]: Writing a core file...
            Apr 06 01:26:08 db2147 mysqld[12450]: Working directory at /srv/sqldata
            Apr 06 01:26:08 db2147 mysqld[12450]: Resource Limits:
            Apr 06 01:26:08 db2147 mysqld[12450]: Limit Soft Limit Hard Limit Units
            Apr 06 01:26:08 db2147 mysqld[12450]: Max cpu time unlimited unlimited seconds
            Apr 06 01:26:08 db2147 mysqld[12450]: Max file size unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max data size unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max stack size 8388608 unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max core file size 0 0 bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max resident set unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max processes 2058337 2058337 processes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max open files 200001 200001 files
            Apr 06 01:26:08 db2147 mysqld[12450]: Max locked memory 65536 65536 bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max address space unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max file locks unlimited unlimited locks
            Apr 06 01:26:08 db2147 mysqld[12450]: Max pending signals 2058337 2058337 signals
            Apr 06 01:26:08 db2147 mysqld[12450]: Max msgqueue size 819200 819200 bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max nice priority 0 0
            Apr 06 01:26:08 db2147 mysqld[12450]: Max realtime priority 0 0
            Apr 06 01:26:08 db2147 mysqld[12450]: Max realtime timeout unlimited unlimited us
            Apr 06 01:26:08 db2147 mysqld[12450]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
            Apr 06 01:26:33 db2147 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
            {code}

            This is the second host:
            {code}
            Apr 06 01:25:08 db2106 mysqld[27097]: 2021-04-06 01:25:08 0x7f975e686700 InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: Failing assertion: !cursor->index->is_committed()
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: We intentionally generate a memory trap.
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: If you get repeated assertion failures or crashes, even
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: immediately after the mysqld startup, there may be
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: about forcing recovery.
            Apr 06 01:25:08 db2106 mysqld[27097]: 210406 1:25:08 [ERROR] mysqld got signal 6 ;
            Apr 06 01:25:08 db2106 mysqld[27097]: This could be because you hit a bug. It is also possible that this binary
            Apr 06 01:25:08 db2106 mysqld[27097]: or one of the libraries it was linked against is corrupt, improperly built,
            Apr 06 01:25:08 db2106 mysqld[27097]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Apr 06 01:25:08 db2106 mysqld[27097]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Apr 06 01:25:08 db2106 mysqld[27097]: We will try our best to scrape up some info that will hopefully help
            Apr 06 01:25:08 db2106 mysqld[27097]: diagnose the problem, but since we have already crashed,
            Apr 06 01:25:08 db2106 mysqld[27097]: something is definitely wrong and this may fail.
            Apr 06 01:25:08 db2106 mysqld[27097]: Server version: 10.4.15-MariaDB-log
            Apr 06 01:25:08 db2106 mysqld[27097]: key_buffer_size=134217728
            Apr 06 01:25:08 db2106 mysqld[27097]: read_buffer_size=131072
            Apr 06 01:25:08 db2106 mysqld[27097]: max_used_connections=22
            Apr 06 01:25:08 db2106 mysqld[27097]: max_threads=2010
            Apr 06 01:25:08 db2106 mysqld[27097]: thread_count=20
            Apr 06 01:25:08 db2106 mysqld[27097]: It is possible that mysqld could use up to
            Apr 06 01:25:08 db2106 mysqld[27097]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
            Apr 06 01:25:08 db2106 mysqld[27097]: Hope that's ok; if not, decrease some variables in the equation.
            Apr 06 01:25:08 db2106 mysqld[27097]: Thread pointer: 0x7f948b158af8
            Apr 06 01:25:08 db2106 mysqld[27097]: Attempting backtrace. You can use the following information to find out
            Apr 06 01:25:08 db2106 mysqld[27097]: where mysqld died. If you see no messages after this, something went
            Apr 06 01:25:08 db2106 mysqld[27097]: terribly wrong...
            Apr 06 01:25:08 db2106 mysqld[27097]: stack_bottom = 0x7f975e685698 thread_stack 0x30000
            Apr 06 01:25:16 db2106 mysqld[27097]: 2021-04-06 1:25:16 968179 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
            Apr 06 01:25:16 db2106 mysqld[27097]: 2021-04-06 1:25:16 968179 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2090-bin.006718' at position 516814095; GTID position '0-180359175-3375835514,171978876-171978876-2496072471,171966557-171966557-1948492266,180363436-180363436-1169124439,171978775-
            Apr 06 01:25:16 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55e74f97a6fe]
            Apr 06 01:25:16 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55e74f47357d]
            Apr 06 01:25:32 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7ff7b4f54730]
            Apr 06 01:25:48 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7ff7b46307bb]
            Apr 06 01:25:48 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7ff7b461b535]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x55e74f16b568]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x55e74f15983e]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x55e74f6bdfce]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb377f8)[0x55e74f6f57f8]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb37e2f)[0x55e74f6f5e2f]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb1297c)[0x55e74f6d097c]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xa5bfcf)[0x55e74f619fcf]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x55e74f47f76b]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x55e74f318cb7]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2695)[0x55e74f26b145]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55e74f270f89]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x750)[0x55e74f56b000]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x55e74f1c5a52]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x55e74f1cea22]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x55e74f92b19b]
            Apr 06 01:26:20 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7ff7b4f49fa3]
            Apr 06 01:26:36 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7ff7b46f24cf]
            Apr 06 01:26:36 db2106 mysqld[27097]: Trying to get some variables.
            Apr 06 01:26:36 db2106 mysqld[27097]: Some pointers may be invalid and cause the dump to abort.
            Apr 06 01:26:36 db2106 mysqld[27097]: Query (0x7f9261fae06a): UPDATE /* WikiPage::updateCategoryCounts */ `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = 'Images_from_Wiki_Loves_Monuments_missing_SDC_participant_in'
            Apr 06 01:26:36 db2106 mysqld[27097]: Connection ID (thread ID): 968180
            Apr 06 01:26:36 db2106 mysqld[27097]: Status: NOT_KILLED
            Apr 06 01:26:36 db2106 mysqld[27097]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on
            Apr 06 01:26:36 db2106 mysqld[27097]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            Apr 06 01:26:36 db2106 mysqld[27097]: information that should help you find out what is causing the crash.
            Apr 06 01:26:36 db2106 mysqld[27097]: Writing a core file...
            Apr 06 01:26:36 db2106 mysqld[27097]: Working directory at /srv/sqldata
            Apr 06 01:26:36 db2106 mysqld[27097]: Resource Limits:
            Apr 06 01:26:36 db2106 mysqld[27097]: Limit Soft Limit Hard Limit Units
            Apr 06 01:26:36 db2106 mysqld[27097]: Max cpu time unlimited unlimited seconds
            Apr 06 01:26:36 db2106 mysqld[27097]: Max file size unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max data size unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max stack size 8388608 unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max core file size 0 0 bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max resident set unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max processes 2058365 2058365 processes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max open files 200001 200001 files
            Apr 06 01:26:36 db2106 mysqld[27097]: Max locked memory 65536 65536 bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max address space unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max file locks unlimited unlimited locks
            Apr 06 01:26:36 db2106 mysqld[27097]: Max pending signals 2058365 2058365 signals
            Apr 06 01:26:36 db2106 mysqld[27097]: Max msgqueue size 819200 819200 bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max nice priority 0 0
            Apr 06 01:26:36 db2106 mysqld[27097]: Max realtime priority 0 0
            Apr 06 01:26:36 db2106 mysqld[27097]: Max realtime timeout unlimited unlimited us
            Apr 06 01:26:36 db2106 mysqld[27097]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
            Apr 06 01:27:04 db2106 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
            -- Subject: Unit process exited
            {code}

            The table schema is:
            {code}


            		Two hosts running 10.4.15 crashed exactly at the same time after replicating a certain query via replication, these are the logs:

            {code}
            Apr 06 01:25:08 db2147 mysqld[12450]: 2021-04-06 01:25:08 0x7f7ac15d4700 InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: Failing assertion: !cursor->index->is_committed()
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: We intentionally generate a memory trap.
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: If you get repeated assertion failures or crashes, even
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: immediately after the mysqld startup, there may be
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            Apr 06 01:25:08 db2147 mysqld[12450]: InnoDB: about forcing recovery.
            Apr 06 01:25:08 db2147 mysqld[12450]: 210406 1:25:08 [ERROR] mysqld got signal 6 ;
            Apr 06 01:25:08 db2147 mysqld[12450]: This could be because you hit a bug. It is also possible that this binary
            Apr 06 01:25:08 db2147 mysqld[12450]: or one of the libraries it was linked against is corrupt, improperly built,
            Apr 06 01:25:08 db2147 mysqld[12450]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Apr 06 01:25:08 db2147 mysqld[12450]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Apr 06 01:25:08 db2147 mysqld[12450]: We will try our best to scrape up some info that will hopefully help
            Apr 06 01:25:08 db2147 mysqld[12450]: diagnose the problem, but since we have already crashed,
            Apr 06 01:25:08 db2147 mysqld[12450]: something is definitely wrong and this may fail.
            Apr 06 01:25:08 db2147 mysqld[12450]: Server version: 10.4.15-MariaDB-log
            Apr 06 01:25:08 db2147 mysqld[12450]: key_buffer_size=134217728
            Apr 06 01:25:08 db2147 mysqld[12450]: read_buffer_size=131072
            Apr 06 01:25:08 db2147 mysqld[12450]: max_used_connections=22
            Apr 06 01:25:08 db2147 mysqld[12450]: max_threads=2010
            Apr 06 01:25:08 db2147 mysqld[12450]: thread_count=20
            Apr 06 01:25:08 db2147 mysqld[12450]: It is possible that mysqld could use up to
            Apr 06 01:25:08 db2147 mysqld[12450]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
            Apr 06 01:25:08 db2147 mysqld[12450]: Hope that's ok; if not, decrease some variables in the equation.
            Apr 06 01:25:08 db2147 mysqld[12450]: Thread pointer: 0x7f7968000ce8
            Apr 06 01:25:08 db2147 mysqld[12450]: Attempting backtrace. You can use the following information to find out
            Apr 06 01:25:08 db2147 mysqld[12450]: where mysqld died. If you see no messages after this, something went
            Apr 06 01:25:08 db2147 mysqld[12450]: terribly wrong...
            Apr 06 01:25:08 db2147 mysqld[12450]: stack_bottom = 0x7f7ac15d3698 thread_stack 0x30000
            Apr 06 01:25:14 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55d0c89976fe]
            Apr 06 01:25:14 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55d0c849057d]
            Apr 06 01:25:26 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fdb16ddc730]
            Apr 06 01:25:36 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fdb164b87bb]
            Apr 06 01:25:36 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fdb164a3535]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x55d0c8188568]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x55d0c817683e]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x55d0c86dafce]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb377f8)[0x55d0c87127f8]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb37e2f)[0x55d0c8712e2f]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xb1297c)[0x55d0c86ed97c]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xa5bfcf)[0x55d0c8636fcf]
            Apr 06 01:25:46 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x55d0c849c76b]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x55d0c8335cb7]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2695)[0x55d0c8288145]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55d0c828df89]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x750)[0x55d0c8588000]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x55d0c81e2a52]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x55d0c81eba22]
            Apr 06 01:25:47 db2147 mysqld[12450]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x55d0c894819b]
            Apr 06 01:25:56 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fdb16dd1fa3]
            Apr 06 01:26:08 db2147 mysqld[12450]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdb1657a4cf]
            Apr 06 01:26:08 db2147 mysqld[12450]: Trying to get some variables.
            Apr 06 01:26:08 db2147 mysqld[12450]: Some pointers may be invalid and cause the dump to abort.
            Apr 06 01:26:08 db2147 mysqld[12450]: Query (0x7f754a00925a): UPDATE /* WikiPage::updateCategoryCounts */ `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = 'Images_from_Wiki_Loves_Monuments_missing_SDC_participant_in'
            Apr 06 01:26:08 db2147 mysqld[12450]: Connection ID (thread ID): 965915
            Apr 06 01:26:08 db2147 mysqld[12450]: Status: NOT_KILLED
            Apr 06 01:26:08 db2147 mysqld[12450]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on
            Apr 06 01:26:08 db2147 mysqld[12450]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            Apr 06 01:26:08 db2147 mysqld[12450]: information that should help you find out what is causing the crash.
            Apr 06 01:26:08 db2147 mysqld[12450]: Writing a core file...
            Apr 06 01:26:08 db2147 mysqld[12450]: Working directory at /srv/sqldata
            Apr 06 01:26:08 db2147 mysqld[12450]: Resource Limits:
            Apr 06 01:26:08 db2147 mysqld[12450]: Limit Soft Limit Hard Limit Units
            Apr 06 01:26:08 db2147 mysqld[12450]: Max cpu time unlimited unlimited seconds
            Apr 06 01:26:08 db2147 mysqld[12450]: Max file size unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max data size unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max stack size 8388608 unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max core file size 0 0 bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max resident set unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max processes 2058337 2058337 processes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max open files 200001 200001 files
            Apr 06 01:26:08 db2147 mysqld[12450]: Max locked memory 65536 65536 bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max address space unlimited unlimited bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max file locks unlimited unlimited locks
            Apr 06 01:26:08 db2147 mysqld[12450]: Max pending signals 2058337 2058337 signals
            Apr 06 01:26:08 db2147 mysqld[12450]: Max msgqueue size 819200 819200 bytes
            Apr 06 01:26:08 db2147 mysqld[12450]: Max nice priority 0 0
            Apr 06 01:26:08 db2147 mysqld[12450]: Max realtime priority 0 0
            Apr 06 01:26:08 db2147 mysqld[12450]: Max realtime timeout unlimited unlimited us
            Apr 06 01:26:08 db2147 mysqld[12450]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
            Apr 06 01:26:33 db2147 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
            {code}

            This is the second host:
            {code}
            Apr 06 01:25:08 db2106 mysqld[27097]: 2021-04-06 01:25:08 0x7f975e686700 InnoDB: Assertion failure in file /root/mariadb-10.4.15/storage/innobase/row/row0ins.cc line 218
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: Failing assertion: !cursor->index->is_committed()
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: We intentionally generate a memory trap.
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: If you get repeated assertion failures or crashes, even
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: immediately after the mysqld startup, there may be
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
            Apr 06 01:25:08 db2106 mysqld[27097]: InnoDB: about forcing recovery.
            Apr 06 01:25:08 db2106 mysqld[27097]: 210406 1:25:08 [ERROR] mysqld got signal 6 ;
            Apr 06 01:25:08 db2106 mysqld[27097]: This could be because you hit a bug. It is also possible that this binary
            Apr 06 01:25:08 db2106 mysqld[27097]: or one of the libraries it was linked against is corrupt, improperly built,
            Apr 06 01:25:08 db2106 mysqld[27097]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Apr 06 01:25:08 db2106 mysqld[27097]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Apr 06 01:25:08 db2106 mysqld[27097]: We will try our best to scrape up some info that will hopefully help
            Apr 06 01:25:08 db2106 mysqld[27097]: diagnose the problem, but since we have already crashed,
            Apr 06 01:25:08 db2106 mysqld[27097]: something is definitely wrong and this may fail.
            Apr 06 01:25:08 db2106 mysqld[27097]: Server version: 10.4.15-MariaDB-log
            Apr 06 01:25:08 db2106 mysqld[27097]: key_buffer_size=134217728
            Apr 06 01:25:08 db2106 mysqld[27097]: read_buffer_size=131072
            Apr 06 01:25:08 db2106 mysqld[27097]: max_used_connections=22
            Apr 06 01:25:08 db2106 mysqld[27097]: max_threads=2010
            Apr 06 01:25:08 db2106 mysqld[27097]: thread_count=20
            Apr 06 01:25:08 db2106 mysqld[27097]: It is possible that mysqld could use up to
            Apr 06 01:25:08 db2106 mysqld[27097]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4752266 K bytes of memory
            Apr 06 01:25:08 db2106 mysqld[27097]: Hope that's ok; if not, decrease some variables in the equation.
            Apr 06 01:25:08 db2106 mysqld[27097]: Thread pointer: 0x7f948b158af8
            Apr 06 01:25:08 db2106 mysqld[27097]: Attempting backtrace. You can use the following information to find out
            Apr 06 01:25:08 db2106 mysqld[27097]: where mysqld died. If you see no messages after this, something went
            Apr 06 01:25:08 db2106 mysqld[27097]: terribly wrong...
            Apr 06 01:25:08 db2106 mysqld[27097]: stack_bottom = 0x7f975e685698 thread_stack 0x30000
            Apr 06 01:25:16 db2106 mysqld[27097]: 2021-04-06 1:25:16 968179 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
            Apr 06 01:25:16 db2106 mysqld[27097]: 2021-04-06 1:25:16 968179 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'db2090-bin.006718' at position 516814095; GTID position '0-180359175-3375835514,171978876-171978876-2496072471,171966557-171966557-1948492266,180363436-180363436-1169124439,171978775-
            Apr 06 01:25:16 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x55e74f97a6fe]
            Apr 06 01:25:16 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x54d)[0x55e74f47357d]
            Apr 06 01:25:32 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7ff7b4f54730]
            Apr 06 01:25:48 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7ff7b46307bb]
            Apr 06 01:25:48 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7ff7b461b535]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x5ad568)[0x55e74f16b568]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x59b83e)[0x55e74f15983e]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xafffce)[0x55e74f6bdfce]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb377f8)[0x55e74f6f57f8]
            Apr 06 01:26:04 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb37e2f)[0x55e74f6f5e2f]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xb1297c)[0x55e74f6d097c]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xa5bfcf)[0x55e74f619fcf]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0xbb)[0x55e74f47f76b]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1b27)[0x55e74f318cb7]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2695)[0x55e74f26b145]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x55e74f270f89]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x750)[0x55e74f56b000]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0x607a52)[0x55e74f1c5a52]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1322)[0x55e74f1cea22]
            Apr 06 01:26:05 db2106 mysqld[27097]: /opt/wmf-mariadb104/bin/mysqld(+0xd6d19b)[0x55e74f92b19b]
            Apr 06 01:26:20 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7ff7b4f49fa3]
            Apr 06 01:26:36 db2106 mysqld[27097]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7ff7b46f24cf]
            Apr 06 01:26:36 db2106 mysqld[27097]: Trying to get some variables.
            Apr 06 01:26:36 db2106 mysqld[27097]: Some pointers may be invalid and cause the dump to abort.
            Apr 06 01:26:36 db2106 mysqld[27097]: Query (0x7f9261fae06a): UPDATE /* WikiPage::updateCategoryCounts */ `category` SET cat_pages = cat_pages - 1,cat_files = cat_files - 1 WHERE cat_title = 'Images_from_Wiki_Loves_Monuments_missing_SDC_participant_in'
            Apr 06 01:26:36 db2106 mysqld[27097]: Connection ID (thread ID): 968180
            Apr 06 01:26:36 db2106 mysqld[27097]: Status: NOT_KILLED
            Apr 06 01:26:36 db2106 mysqld[27097]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on
            Apr 06 01:26:36 db2106 mysqld[27097]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            Apr 06 01:26:36 db2106 mysqld[27097]: information that should help you find out what is causing the crash.
            Apr 06 01:26:36 db2106 mysqld[27097]: Writing a core file...
            Apr 06 01:26:36 db2106 mysqld[27097]: Working directory at /srv/sqldata
            Apr 06 01:26:36 db2106 mysqld[27097]: Resource Limits:
            Apr 06 01:26:36 db2106 mysqld[27097]: Limit Soft Limit Hard Limit Units
            Apr 06 01:26:36 db2106 mysqld[27097]: Max cpu time unlimited unlimited seconds
            Apr 06 01:26:36 db2106 mysqld[27097]: Max file size unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max data size unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max stack size 8388608 unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max core file size 0 0 bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max resident set unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max processes 2058365 2058365 processes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max open files 200001 200001 files
            Apr 06 01:26:36 db2106 mysqld[27097]: Max locked memory 65536 65536 bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max address space unlimited unlimited bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max file locks unlimited unlimited locks
            Apr 06 01:26:36 db2106 mysqld[27097]: Max pending signals 2058365 2058365 signals
            Apr 06 01:26:36 db2106 mysqld[27097]: Max msgqueue size 819200 819200 bytes
            Apr 06 01:26:36 db2106 mysqld[27097]: Max nice priority 0 0
            Apr 06 01:26:36 db2106 mysqld[27097]: Max realtime priority 0 0
            Apr 06 01:26:36 db2106 mysqld[27097]: Max realtime timeout unlimited unlimited us
            Apr 06 01:26:36 db2106 mysqld[27097]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
            Apr 06 01:27:04 db2106 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
            -- Subject: Unit process exited
            {code}

            The table schema is:
            {code}
                   Table: category
            Create Table: CREATE TABLE `category` (
              `cat_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
              `cat_title` varbinary(255) NOT NULL DEFAULT '',
              `cat_pages` int(11) NOT NULL DEFAULT 0,
              `cat_subcats` int(11) NOT NULL DEFAULT 0,
              `cat_files` int(11) NOT NULL DEFAULT 0,
              PRIMARY KEY (`cat_id`),
              UNIQUE KEY `cat_title` (`cat_title`),
              KEY `cat_pages` (`cat_pages`)
            ) ENGINE=InnoDB AUTO_INCREMENT=351025254 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8
            {code}

            The table was checked after the crash and it both reported ok
            {code}
            root@db2147.codfw.wmnet[commonswiki]> check table category;
            +----------------------+-------+----------+----------+
            | Table | Op | Msg_type | Msg_text |
            +----------------------+-------+----------+----------+
            | commonswiki.category | check | status | OK |
            +----------------------+-------+----------+----------+
            1 row in set (33.649 sec)


            root@db2106.codfw.wmnet[commonswiki]> start slave;
            Query OK, 0 rows affected, 1 warning (0.032 sec)

            root@db2106.codfw.wmnet[commonswiki]> check table category;
            +----------------------+-------+----------+----------+
            | Table | Op | Msg_type | Msg_text |
            +----------------------+-------+----------+----------+
            | commonswiki.category | check | status | OK |
            +----------------------+-------+----------+----------+
            1 row in set (35.585 sec)
            {code}

            marostegui Manuel Arostegui made changes -
            Component/s Replication [ 10100 ]
            Component/s Storage Engine - InnoDB [ 10129 ]
            marko Marko Mäkelä added a comment -

            Were both servers originally restored from the same physical backup? Could that backup have been created on data files that might have been corrupted due to MDEV-24449?

            I am rather sure that the bug that was fixed in MDEV-24449 can cause any corruption of:

            • any secondary indexes of any InnoDB tables
            • anything in the InnoDB system tablespace (as a consequence of doubly allocating some data pages, or failing to remove dangling links to freed pages)

            The only way to prevent such corruption is to never use anything else than innodb_change_buffering=none, or to never use a server that is missing the MDEV-24449 fix.

            I do not think that the corruption of the system tablespace can be fixed by anything else than by reinitializing the entire server from a logical backup. You might try ALTER TABLE…IMPORT TABLESPACE of all InnoDB tables; it could be a little faster. But you should probably drop all secondary indexes and create them anew.

            Note that CHECK TABLE will not necessarily find this kind of corruption.

            marko Marko Mäkelä added a comment - Were both servers originally restored from the same physical backup? Could that backup have been created on data files that might have been corrupted due to MDEV-24449 ? I am rather sure that the bug that was fixed in MDEV-24449 can cause any corruption of: any secondary indexes of any InnoDB tables anything in the InnoDB system tablespace (as a consequence of doubly allocating some data pages, or failing to remove dangling links to freed pages) The only way to prevent such corruption is to never use anything else than innodb_change_buffering=none , or to never use a server that is missing the MDEV-24449 fix. I do not think that the corruption of the system tablespace can be fixed by anything else than by reinitializing the entire server from a logical backup. You might try ALTER TABLE…IMPORT TABLESPACE of all InnoDB tables; it could be a little faster. But you should probably drop all secondary indexes and create them anew. Note that CHECK TABLE will not necessarily find this kind of corruption.
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ]
            Labels need_feedback
            marostegui Manuel Arostegui added a comment -

            Thanks Marko.
            Unfortunately we don't know whether they were cloned from the same host or not. It could be, as some other 10.4.15 hosts on the same replication chain didn't crash.
            I am in process of slowly deploying innodb_change_buffering=none across the fleet by the way.

            Thank you!

            marostegui Manuel Arostegui added a comment - Thanks Marko. Unfortunately we don't know whether they were cloned from the same host or not. It could be, as some other 10.4.15 hosts on the same replication chain didn't crash. I am in process of slowly deploying innodb_change_buffering=none across the fleet by the way. Thank you!
            serg Sergei Golubchik added a comment -

            marostegui, I'm tentatively closing it as incomplete, as there's not enough information to go by and it got no more info for a month. Please, don't hesitate to comment and add more data whenever you like — a bug can be reopened any time, if needed.

            serg Sergei Golubchik added a comment - marostegui , I'm tentatively closing it as incomplete, as there's not enough information to go by and it got no more info for a month. Please, don't hesitate to comment and add more data whenever you like — a bug can be reopened any time, if needed.
            serg Sergei Golubchik made changes -
            Fix Version/s N/A [ 14700 ]
            Resolution Incomplete [ 4 ]
            Status Open [ 1 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 120766 ] MariaDB v4 [ 159114 ]

            People

              marko Marko Mäkelä
              marostegui Manuel Arostegui
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.