Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
10.2.9
-
None
-
Ubuntu 16.04 LTS
Description
Upgraded our internal cluster (3 nodes) to 10.2.9 (from 10.1) one day ago and this morning we found that all 3 nodes crashed at the same time. Below is a relevant log entries from one of the nodes. Please let me know if I can help with any additional information.
Oct 6 08:44:36 masha3 mysqld[7310]: 171006 8:44:36 [ERROR] mysqld got signal 11 ;
|
Oct 6 08:44:36 masha3 mysqld[7310]: This could be because you hit a bug. It is also possible that this binary
|
Oct 6 08:44:36 masha3 mysqld[7310]: or one of the libraries it was linked against is corrupt, improperly built,
|
Oct 6 08:44:36 masha3 mysqld[7310]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Oct 6 08:44:36 masha3 mysqld[7310]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Oct 6 08:44:36 masha3 mysqld[7310]: We will try our best to scrape up some info that will hopefully help
|
Oct 6 08:44:36 masha3 mysqld[7310]: diagnose the problem, but since we have already crashed,
|
Oct 6 08:44:36 masha3 mysqld[7310]: something is definitely wrong and this may fail.
|
Oct 6 08:44:36 masha3 mysqld[7310]: Server version: 10.2.9-MariaDB-10.2.9+maria~xenial-log
|
Oct 6 08:44:36 masha3 mysqld[7310]: key_buffer_size=134217728
|
Oct 6 08:44:36 masha3 mysqld[7310]: read_buffer_size=2097152
|
Oct 6 08:44:36 masha3 mysqld[7310]: max_used_connections=88
|
Oct 6 08:44:36 masha3 mysqld[7310]: max_threads=202
|
Oct 6 08:44:36 masha3 mysqld[7310]: thread_count=112
|
Oct 6 08:44:36 masha3 mysqld[7310]: It is possible that mysqld could use up to
|
Oct 6 08:44:36 masha3 mysqld[7310]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1376413 K bytes of memory
|
Oct 6 08:44:36 masha3 mysqld[7310]: Hope that's ok; if not, decrease some variables in the equation.
|
Oct 6 08:44:36 masha3 mysqld[7310]: Thread pointer: 0x7ff3040009a8
|
Oct 6 08:44:36 masha3 mysqld[7310]: Attempting backtrace. You can use the following information to find out
|
Oct 6 08:44:36 masha3 mysqld[7310]: where mysqld died. If you see no messages after this, something went
|
Oct 6 08:44:36 masha3 mysqld[7310]: terribly wrong...
|
Oct 6 08:44:36 masha3 mysqld[7310]: stack_bottom = 0x7ff78c3de918 thread_stack 0x49000
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b3d95957ee]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55b3d902e535]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7ff7ba9ef390]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(+0x402e75)[0x55b3d8ddbe75]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(+0x8bc26f)[0x55b3d929526f]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2b1a)[0x55b3d8f1cb7a]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x5f4)[0x55b3d8f67c94]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x14e7)[0x55b3d8e924f7]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a)[0x55b3d8e9a09a]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x825)[0x55b3d910fab5]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(wsrep_apply_cb+0x5b2)[0x55b3d8fd4c62]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0x106)[0x7ff7afd23316]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(+0x22e1bf)[0x7ff7afd731bf]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0xbe)[0x7ff7afd75d7e]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x13e)[0x7ff7afd7867e]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x1d8)[0x7ff7afd4f848]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x76)[0x7ff7afd51596]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x83)[0x7ff7afd78d63]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/lib/galera/libgalera_smm.so(galera_recv+0x2b)[0x7ff7afd92ddb]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(+0x5fce54)[0x55b3d8fd5e54]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /usr/sbin/mysqld(start_wsrep_THD+0x455)[0x55b3d8fc5355]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7ff7ba9e56ba]
|
Oct 6 08:44:36 masha3 mysqld[7310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff7ba0903dd]
|
Oct 6 08:44:36 masha3 mysqld[7310]: Trying to get some variables.
|
Oct 6 08:44:36 masha3 mysqld[7310]: Some pointers may be invalid and cause the dump to abort.
|
Oct 6 08:44:36 masha3 mysqld[7310]: Query (0x7ff3040963fe): ALTER TABLE `AttendDet` DROP COLUMN IF EXISTS `Counter`
|
Oct 6 08:44:36 masha3 mysqld[7310]: Connection ID (thread ID): 18
|
Oct 6 08:44:36 masha3 mysqld[7310]: Status: NOT_KILLED
|
Oct 6 08:44:36 masha3 mysqld[7310]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
|
Oct 6 08:44:36 masha3 mysqld[7310]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Oct 6 08:44:36 masha3 mysqld[7310]: information that should help you find out what is causing the crash.
|
Oct 6 08:44:36 masha3 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
|
Oct 6 08:44:36 masha3 systemd[1]: mariadb.service: Unit entered failed state.
|
Oct 6 08:44:36 masha3 systemd[1]: mariadb.service: Failed with result 'signal'.
|
Oct 6 08:44:41 masha3 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
|
Oct 6 08:44:41 masha3 systemd[1]: Stopped MariaDB database server.
|
Attachments
Issue Links
- blocks
-
MDEV-14046 Allow ALGORITHM=INPLACE for 10.1 tables that contain virtual columns
- Closed
- relates to
-
MDEV-18084 Server crashes in row_upd_changes_some_index_ord_field_binary or Assertion `pos < index->n_def' failed in dict_index_get_nth_field upon UPDATE after upgrade from 10.1/10.0
- Closed