Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.2.25, 10.3.13, 10.3.14, 10.3.15, 10.3.16
-
Debian Stretch guests inside VMWare host
Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
Description
Galera cluster was perfectly fine running 10.3.12. Upgraded to 10.3.14, and nodes are failing almost instantly. Logs shows assertion failure on all nodes listed below:
2019-04-25 10:09:19 2 [Note] WSREP: Receiving IST: 2 writesets, seqnos 260448-260450
|
2019-04-25 10:09:19 0 [Note] WSREP: Receiving IST... 0.0% (0/2 events) complete.
|
2019-04-25 10:09:20 0x7f16f83ac700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.14/storage/innobase/row/row0ins.cc line 270
|
InnoDB: Failing assertion: !cursor->index->is_committed()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
190425 10:09:20 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.3.14-MariaDB-1:10.3.14+maria~stretch-log
|
key_buffer_size=134217728
|
read_buffer_size=2097152
|
max_used_connections=0
|
max_threads=102
|
thread_count=9
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760012 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f16d40009a8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f16f83abd28 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55913783a92e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x41f)[0x5591372eba5f]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f1700c9c0e0]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f16ff33bfff]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f16ff33d42a]
|
/usr/sbin/mysqld(+0x49900a)[0x55913703500a]
|
/usr/sbin/mysqld(+0x988e2b)[0x559137524e2b]
|
/usr/sbin/mysqld(+0x9891b6)[0x5591375251b6]
|
/usr/sbin/mysqld(+0x9c3953)[0x55913755f953]
|
/usr/sbin/mysqld(+0x9c932d)[0x55913756532d]
|
/usr/sbin/mysqld(+0x99b374)[0x559137537374]
|
/usr/sbin/mysqld(+0x8d77a7)[0x5591374737a7]
|
/usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0x122)[0x5591372f64e2]
|
/usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x2b1)[0x5591373dce01]
|
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x24f)[0x5591373cfdff]
|
/usr/sbin/mysqld(wsrep_apply_cb+0x585)[0x559137269635]
|
/usr/lib/galera/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xfe)[0x7f16f8dd6f6e]
|
/usr/lib/galera/libgalera_smm.so(+0x22fd56)[0x7f16f8e28d56]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x156)[0x7f16f8e2bc16]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM8recv_ISTEPv+0x2b8)[0x7f16f8e3f1a8]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM22request_state_transferEPvRK10wsrep_uuidlPKvl+0x9f1)[0x7f16f8e44bb1]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM19process_conf_changeEPvRK15wsrep_view_infoiNS_10Replicator5StateEl+0x9bd)[0x7f16f8e2fc7d]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x52b)[0x7f16f8e039bb]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x76)[0x7f16f8e04cc6]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x8b)[0x7f16f8e2ee0b]
|
/usr/lib/galera/libgalera_smm.so(galera_recv+0x2b)[0x7f16f8e4ab6b]
|
/usr/sbin/mysqld(+0x6ce2c4)[0x55913726a2c4]
|
/usr/sbin/mysqld(start_wsrep_THD+0x373)[0x55913725c4c3]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f1700c924a4]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f16ff3f1d0f]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f16d00143ab): UPDATE `h12_part` SET `part_json`=JSON_SET(IFNULL(NULLIF(TRIM(`part_json`), ''), '{}'), '$.core_charge', 0, '$.core_text', NULL) WHERE `part_id`=10531
|
Connection ID (thread ID): 2
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Attachments
Issue Links
- relates to
-
MDEV-17466 Virtual column value not available during purge
-
- Closed
-
-
MDEV-19775 Bug in WSREP/Galera with virtual columns (keeps crashing on startup)
-
- Closed
-
-
MDEV-22759 Failing assertion: !cursor->index->is_committed() upon update on table with HASH index
-
- Closed
-
-
MDEV-14643 InnoDB: Failing assertion: !cursor->index->is_committed()
-
- Closed
-
-
MDEV-20396 Server crashes after DELETE with SET NULL Foreign key and a virtual column in index
-
- Closed
-
-
MDEV-22061 InnoDB: Assertion of missing row in sec index row_start upon REPLACE on a system-versioned table
-
- Closed
-
-
MDEV-22076 InnoDB: Failing assertion: !cursor->index->is_committed() or Assertion `0' in row_purge_remove_sec_if_poss_leaf after DML/DDL with XA
-
- Closed
-
-
MDEV-27744 LPAD in vcol created in ORACLE mode makes table corrupted in non-ORACLE
-
- Closed
-
Yet another variation, now on 10.4+ it doesn't require explicit virtual columns, unique keys for long blobs do the trick.
--source include/have_innodb.inc
# Cleanup
10.4 non-debug 9a621200
2019-12-10 21:50:20 9 [ERROR] InnoDB: Record in index `a` of table `test`.`t1` was not found on update: TUPLE (info_bits=0, 2 fields): {[8] |(0x000000110000097C),[4] (0x80000000)} at: COMPACT RECORD(info_bits=0, 2 fields): {[8] P(0x0000001100000950),[4] (0x80000000)}
2019-12-10 21:50:20 0x7f8030066700 InnoDB: Assertion failure in file /data/src/10.4/storage/innobase/row/row0ins.cc line 263
InnoDB: Failing assertion: !cursor->index->is_committed()
#6 0x000056029bd0ca7f in ut_dbg_assertion_failed (expr=expr@entry=0x56029c6a75c8 "!cursor->index->is_committed()", file=file@entry=0x56029c6a70e0 "/data/src/10.4/storage/innobase/row/row0ins.cc", line=line@entry=263) at /data/src/10.4/storage/innobase/ut/ut0dbg.cc:60
#7 0x000056029c206a10 in row_ins_sec_index_entry_by_modify (mtr=0x7f8030061960, thr=0x7f7fe40c4e68, entry=0x7f7fe4046d40, heap=0x7f7fe404d4f0, offsets_heap=<optimized out>, offsets=0x7f8030060a10, cursor=0x7f8030060a90, mode=2, flags=0) at /data/src/10.4/storage/innobase/row/row0ins.cc:263
#8 row_ins_sec_index_entry_low (flags=flags@entry=0, mode=mode@entry=2, index=index@entry=0x7f7fe40ca210, offsets_heap=<optimized out>, offsets_heap@entry=0x7f7fe404d070, heap=heap@entry=0x7f7fe404d4f0, entry=entry@entry=0x7f7fe4046d40, trx_id=<optimized out>, thr=0x7f7fe40c4e68) at /data/src/10.4/storage/innobase/row/row0ins.cc:3099
#9 0x000056029c206e7b in row_ins_sec_index_entry (index=index@entry=0x7f7fe40ca210, entry=entry@entry=0x7f7fe4046d40, thr=thr@entry=0x7f7fe40c4e68, check_foreign=check_foreign@entry=true) at /data/src/10.4/storage/innobase/row/row0ins.cc:3314
#10 0x000056029c2073f7 in row_ins_index_entry (thr=0x7f7fe40c4e68, entry=<optimized out>, index=<optimized out>) at /data/src/10.4/storage/innobase/row/row0ins.cc:3359
#11 row_ins_index_entry_step (thr=0x7f7fe40c4e68, node=<optimized out>) at /data/src/10.4/storage/innobase/row/row0ins.cc:3526
#12 row_ins (thr=<optimized out>, node=<optimized out>) at /data/src/10.4/storage/innobase/row/row0ins.cc:3662
#13 row_ins_step (thr=thr@entry=0x7f7fe40c4e68) at /data/src/10.4/storage/innobase/row/row0ins.cc:3805
#14 0x000056029c21621b in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7f7fe40809a8 "", prebuilt=0x7f7fe40c43c0, ins_mode=<optimized out>) at /data/src/10.4/storage/innobase/row/row0mysql.cc:1463
#15 0x000056029c168bbd in ha_innobase::write_row (this=0x7f7fe40cab80, record=0x7f7fe40809a8 "") at /data/src/10.4/storage/innobase/handler/ha_innodb.cc:8046
#16 0x000056029bfe794e in handler::ha_write_row (this=0x7f7fe40cab80, buf=0x7f7fe40809a8 "") at /data/src/10.4/sql/handler.cc:6674
#17 0x000056029bdb3fe4 in write_record (thd=thd@entry=0x7f7fe40009a8, table=table@entry=0x7f7fe407fba8, info=info@entry=0x7f8030062880) at /data/src/10.4/sql/sql_insert.cc:1732
#18 0x000056029bdbb179 in mysql_insert (thd=thd@entry=0x7f7fe40009a8, table_list=<optimized out>, fields=..., values_list=..., update_fields=..., update_values=..., duplic=<optimized out>, ignore=<optimized out>) at /data/src/10.4/sql/sql_insert.cc:1078
#19 0x000056029bde7f6c in mysql_execute_command (thd=thd@entry=0x7f7fe40009a8) at /data/src/10.4/sql/sql_parse.cc:4529
#20 0x000056029bdecb09 in mysql_parse (thd=thd@entry=0x7f7fe40009a8, rawbuf=<optimized out>, length=46, parser_state=parser_state@entry=0x7f80300651a0, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /data/src/10.4/sql/sql_parse.cc:7901
#21 0x000056029bdeef02 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f7fe40009a8, packet=packet@entry=0x7f7fe40079d9 "REPLACE INTO t1 VALUES (0,'',NULL),(0,'',NULL)", packet_length=packet_length@entry=46, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /data/src/10.4/sql/sql_parse.cc:1842
#22 0x000056029bdf0629 in do_command (thd=0x7f7fe40009a8) at /data/src/10.4/sql/sql_parse.cc:1360
#23 0x000056029bebf834 in do_handle_one_connection (connect=connect@entry=0x56029de7c1d8) at /data/src/10.4/sql/sql_connect.cc:1412
#24 0x000056029bebf8e4 in handle_one_connection (arg=arg@entry=0x56029de7c1d8) at /data/src/10.4/sql/sql_connect.cc:1316
#25 0x000056029c46be84 in pfs_spawn_thread (arg=0x56029de1c178) at /data/src/10.4/storage/perfschema/pfs.cc:1862
#26 0x00007f8037a0b4a4 in start_thread (arg=0x7f8030066700) at pthread_create.c:456
#27 0x00007f8035f52d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Debug builds fail with another infamous one
10.4 debug 9a621200
2019-12-10 21:51:42 9 [ERROR] InnoDB: Record in index `a` of table `test`.`t1` was not found on update: TUPLE (info_bits=0, 2 fields): {[8] |(0x000000110000097C),[4] (0x80000000)} at: COMPACT RECORD(info_bits=0, 2 fields): {[8] P(0x0000001100000950),[4] (0x80000000)}
mysqld: /data/src/10.4/storage/innobase/row/row0upd.cc:2425: dberr_t row_upd_sec_index_entry(upd_node_t*, que_thr_t*): Assertion `btr_validate_index(index, 0)' failed.
191210 21:51:42 [ERROR] mysqld got signal 6 ;
#7 0x00007fc3c4b1ef12 in __GI___assert_fail (assertion=0x555af35ddcf5 "btr_validate_index(index, 0)", file=0x555af35dd558 "/data/src/10.4/storage/innobase/row/row0upd.cc", line=2425, function=0x555af35e04c0 <row_upd_sec_index_entry(upd_node_t*, que_thr_t*)::__PRETTY_FUNCTION__> "dberr_t row_upd_sec_index_entry(upd_node_t*, que_thr_t*)") at assert.c:101
#8 0x0000555af2e018a5 in row_upd_sec_index_entry (node=0x7fc37418bfb0, thr=0x7fc37406f248) at /data/src/10.4/storage/innobase/row/row0upd.cc:2425
#9 0x0000555af2e0224d in row_upd_sec_step (node=0x7fc37418bfb0, thr=0x7fc37406f248) at /data/src/10.4/storage/innobase/row/row0upd.cc:2549
#10 0x0000555af2e0524b in row_upd (node=0x7fc37418bfb0, thr=0x7fc37406f248) at /data/src/10.4/storage/innobase/row/row0upd.cc:3327
#11 0x0000555af2e05691 in row_upd_step (thr=0x7fc37406f248) at /data/src/10.4/storage/innobase/row/row0upd.cc:3442
#12 0x0000555af2da102a in row_update_for_mysql (prebuilt=0x7fc37418b408) at /data/src/10.4/storage/innobase/row/row0mysql.cc:1889
#13 0x0000555af2c00b1c in ha_innobase::delete_row (this=0x7fc37400b3c8, record=0x7fc3741a0ed0 "") at /data/src/10.4/storage/innobase/handler/ha_innodb.cc:8972
#14 0x0000555af29c2526 in handler::ha_delete_row (this=0x7fc37400b3c8, buf=0x7fc3741a0ed0 "") at /data/src/10.4/sql/handler.cc:6782
#15 0x0000555af26092f7 in write_record (thd=0x7fc374000af0, table=0x7fc37400a560, info=0x7fc3b9ecc490) at /data/src/10.4/sql/sql_insert.cc:2007
#16 0x0000555af26064c7 in mysql_insert (thd=0x7fc374000af0, table_list=0x7fc374013260, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_REPLACE, ignore=false) at /data/src/10.4/sql/sql_insert.cc:1078
#17 0x0000555af2654615 in mysql_execute_command (thd=0x7fc374000af0) at /data/src/10.4/sql/sql_parse.cc:4529
#18 0x0000555af2660dc7 in mysql_parse (thd=0x7fc374000af0, rawbuf=0x7fc374013158 "REPLACE INTO t1 VALUES (0,'',NULL),(0,'',NULL)", length=46, parser_state=0x7fc3b9ecd160, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:7901
#19 0x0000555af264bf6c in dispatch_command (command=COM_QUERY, thd=0x7fc374000af0, packet=0x7fc3741361b1 "REPLACE INTO t1 VALUES (0,'',NULL),(0,'',NULL)", packet_length=46, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1842
#20 0x0000555af264a5f9 in do_command (thd=0x7fc374000af0) at /data/src/10.4/sql/sql_parse.cc:1360
#21 0x0000555af27d30e7 in do_handle_one_connection (connect=0x555af52e7180) at /data/src/10.4/sql/sql_connect.cc:1412
#22 0x0000555af27d2e36 in handle_one_connection (arg=0x555af52e7180) at /data/src/10.4/sql/sql_connect.cc:1316
#23 0x0000555af31d50ed in pfs_spawn_thread (arg=0x555af5264870) at /data/src/10.4/storage/perfschema/pfs.cc:1862
#24 0x00007fc3c66944a4 in start_thread (arg=0x7fc3b9ece700) at pthread_create.c:456
#25 0x00007fc3c4bdbd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97