Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.2.25, 10.3.13, 10.3.14, 10.3.15, 10.3.16
-
Debian Stretch guests inside VMWare host
Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
Description
Galera cluster was perfectly fine running 10.3.12. Upgraded to 10.3.14, and nodes are failing almost instantly. Logs shows assertion failure on all nodes listed below:
2019-04-25 10:09:19 2 [Note] WSREP: Receiving IST: 2 writesets, seqnos 260448-260450
|
2019-04-25 10:09:19 0 [Note] WSREP: Receiving IST... 0.0% (0/2 events) complete.
|
2019-04-25 10:09:20 0x7f16f83ac700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.14/storage/innobase/row/row0ins.cc line 270
|
InnoDB: Failing assertion: !cursor->index->is_committed()
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
190425 10:09:20 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.3.14-MariaDB-1:10.3.14+maria~stretch-log
|
key_buffer_size=134217728
|
read_buffer_size=2097152
|
max_used_connections=0
|
max_threads=102
|
thread_count=9
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760012 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f16d40009a8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f16f83abd28 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55913783a92e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x41f)[0x5591372eba5f]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f1700c9c0e0]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f16ff33bfff]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f16ff33d42a]
|
/usr/sbin/mysqld(+0x49900a)[0x55913703500a]
|
/usr/sbin/mysqld(+0x988e2b)[0x559137524e2b]
|
/usr/sbin/mysqld(+0x9891b6)[0x5591375251b6]
|
/usr/sbin/mysqld(+0x9c3953)[0x55913755f953]
|
/usr/sbin/mysqld(+0x9c932d)[0x55913756532d]
|
/usr/sbin/mysqld(+0x99b374)[0x559137537374]
|
/usr/sbin/mysqld(+0x8d77a7)[0x5591374737a7]
|
/usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhS1_+0x122)[0x5591372f64e2]
|
/usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x2b1)[0x5591373dce01]
|
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x24f)[0x5591373cfdff]
|
/usr/sbin/mysqld(wsrep_apply_cb+0x585)[0x559137269635]
|
/usr/lib/galera/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xfe)[0x7f16f8dd6f6e]
|
/usr/lib/galera/libgalera_smm.so(+0x22fd56)[0x7f16f8e28d56]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x156)[0x7f16f8e2bc16]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM8recv_ISTEPv+0x2b8)[0x7f16f8e3f1a8]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM22request_state_transferEPvRK10wsrep_uuidlPKvl+0x9f1)[0x7f16f8e44bb1]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM19process_conf_changeEPvRK15wsrep_view_infoiNS_10Replicator5StateEl+0x9bd)[0x7f16f8e2fc7d]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x52b)[0x7f16f8e039bb]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x76)[0x7f16f8e04cc6]
|
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x8b)[0x7f16f8e2ee0b]
|
/usr/lib/galera/libgalera_smm.so(galera_recv+0x2b)[0x7f16f8e4ab6b]
|
/usr/sbin/mysqld(+0x6ce2c4)[0x55913726a2c4]
|
/usr/sbin/mysqld(start_wsrep_THD+0x373)[0x55913725c4c3]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f1700c924a4]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f16ff3f1d0f]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f16d00143ab): UPDATE `h12_part` SET `part_json`=JSON_SET(IFNULL(NULLIF(TRIM(`part_json`), ''), '{}'), '$.core_charge', 0, '$.core_text', NULL) WHERE `part_id`=10531
|
Connection ID (thread ID): 2
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Attachments
Issue Links
- relates to
-
MDEV-17466 Virtual column value not available during purge
- Closed
-
MDEV-19775 Bug in WSREP/Galera with virtual columns (keeps crashing on startup)
- Closed
-
MDEV-22759 Failing assertion: !cursor->index->is_committed() upon update on table with HASH index
- Confirmed
-
MDEV-14643 InnoDB: Failing assertion: !cursor->index->is_committed()
- Closed
-
MDEV-20396 Server crashes after DELETE with SET NULL Foreign key and a virtual column in index
- Closed
-
MDEV-22061 InnoDB: Assertion of missing row in sec index row_start upon REPLACE on a system-versioned table
- Closed
-
MDEV-22076 InnoDB: Failing assertion: !cursor->index->is_committed() or Assertion `0' in row_purge_remove_sec_if_poss_leaf after DML/DDL with XA
- Closed
-
MDEV-27744 LPAD in vcol created in ORACLE mode makes table corrupted in non-ORACLE
- Closed