Nirbhay Choubey (Inactive) added a comment - 2016-05-12 23:02 lubor Can you share the full error log? I would like to know if this happened during SST. What was the SST method (wsrep_sst_method)?

LuborJ added a comment - 2016-05-12 23:22 - edited This is not related to SST. First lines after start up were this crashed which I sent.

LuborJ added a comment - 2016-05-18 17:21 May 18 13:12:55 SQL2 mysqld: 160518 13:12:55 [ERROR] mysqld got signal 11 ; May 18 13:12:55 SQL2 mysqld: This could be because you hit a bug. It is also possible that this binary May 18 13:12:55 SQL2 mysqld: or one of the libraries it was linked against is corrupt, improperly built, May 18 13:12:55 SQL2 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. May 18 13:12:55 SQL2 mysqld: May 18 13:12:55 SQL2 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs May 18 13:12:55 SQL2 mysqld: May 18 13:12:55 SQL2 mysqld: We will try our best to scrape up some info that will hopefully help May 18 13:12:55 SQL2 mysqld: diagnose the problem, but since we have already crashed, May 18 13:12:55 SQL2 mysqld: something is definitely wrong and this may fail. May 18 13:12:55 SQL2 mysqld: May 18 13:12:55 SQL2 mysqld: Server version: 10.1.14-MariaDB-1~jessie May 18 13:12:55 SQL2 mysqld: key_buffer_size=67108864 May 18 13:12:55 SQL2 mysqld: read_buffer_size=1048576 May 18 13:12:55 SQL2 mysqld: max_used_connections=153 May 18 13:12:55 SQL2 mysqld: max_threads=10002 May 18 13:12:55 SQL2 mysqld: thread_count=24 May 18 13:12:55 SQL2 mysqld: It is possible that mysqld could use up to May 18 13:12:55 SQL2 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 92448832 K bytes of memory May 18 13:12:55 SQL2 mysqld: Hope that's ok; if not, decrease some variables in the equation. May 18 13:12:55 SQL2 mysqld: May 18 13:12:55 SQL2 mysqld: Thread pointer: 0x0x7eeec0bea008 May 18 13:12:55 SQL2 mysqld: Attempting backtrace. You can use the following information to find out May 18 13:12:55 SQL2 mysqld: where mysqld died. If you see no messages after this, something went May 18 13:12:55 SQL2 mysqld: terribly wrong... May 18 13:12:55 SQL2 mysqld: stack_bottom = 0x7ef69397cdf8 thread_stack 0x40000 May 18 13:13:05 SQL2 mysqld: 2016-05-18 13:13:05 139603189167872 [Warning] WSREP: last inactive check more than PT1.5S ago (PT10.2603S), skipping check May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x5649f4a6f8be] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x34d) [0x5649f45b192d] May 18 13:13:05 SQL2 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7effef50f8d0] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG21do_checkpoint_requestEm+0x98) [0x5649f4666dd8] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG20checkpoint_and_purgeEm+0x11) [0x5649f4666e01] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG23trx_group_commit_leaderEPNS_18group_commit_entryE+0x513) [0x5649f4669cb3] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG34write_transaction_to_binlog_eventsEPNS_18group_commit_entryE+0x1bb) [0x5649f466a13b] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG27write_transaction_to_binlogEP3THDP17binlog_cache_mngrP9Log_eventbbb+0xcf) [0x5649f466a36f] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(+0x65a4ef) [0x5649f466a4ef] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(+0x65a88c) [0x5649f466a88c] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(+0x65af5a) [0x5649f466af5a] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z19ha_commit_one_phaseP3THDb+0x92) [0x5649f45b4262] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x346) [0x5649f45b4a86] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x38) [0x5649f450a928] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb24) [0x5649f4427984] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26e) [0x5649f443015e] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(+0x420959) [0x5649f4430959] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x16ec) [0x5649f443267c] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x172) [0x5649f4433362] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x182) [0x5649f44fc842] May 18 13:13:05 SQL2 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40) [0x5649f44fca00] May 18 13:13:05 SQL2 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7effef5080a4] May 18 13:13:05 SQL2 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7effed6b487d] May 18 13:13:05 SQL2 mysqld: May 18 13:13:05 SQL2 mysqld: Trying to get some variables. May 18 13:13:05 SQL2 mysqld: Some pointers may be invalid and cause the dump to abort. May 18 13:13:05 SQL2 mysqld: Query (0x7eef144af020): is an invalid pointer May 18 13:13:05 SQL2 mysqld: Connection ID (thread ID): 2936470 May 18 13:13:05 SQL2 mysqld: Status: NOT_KILLED May 18 13:13:05 SQL2 mysqld: May 18 13:13:05 SQL2 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on May 18 13:13:05 SQL2 mysqld: May 18 13:13:05 SQL2 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains May 18 13:13:05 SQL2 mysqld: information that should help you find out what is causing the crash. May 18 13:13:09 SQL2 mysqld_safe: Number of processes running now: 0 May 18 13:13:09 SQL2 mysqld_safe: WSREP: not restarting wsrep node automatically May 18 13:13:09 SQL2 mysqld_safe: mysqld from pid file /data/mysql/sql2.pid ended

Nils Meyer added a comment - 2016-07-18 05:22 - edited I do have seemingly random crashes as well, I don't know if this is related since the stack trace is a bit different: 160715 14:39:44 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.14-MariaDB-1~jessie key_buffer_size=33554432 read_buffer_size=131072 max_used_connections=219 max_threads=4098 thread_count=88 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9033940 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x7fe553162008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe575ecd1f8 thread_stack 0x48400 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558232bc68be] /usr/sbin/mysqld(handle_fatal_signal+0x34d)[0x55823270892d] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fe80553b8d0] /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x7b)[0x5582327bdbbb] /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG5unlogEmy+0x2a)[0x5582327bde3a] /usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x758)[0x55823270be98] /usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x38)[0x558232661928] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb24)[0x55823257e984] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4c5)[0x55823259a505] /usr/sbin/mysqld(+0x433764)[0x55823259a764] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x176)[0x55823259ab96] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1324)[0x5582325892b4] /usr/sbin/mysqld(_Z10do_commandP3THD+0x172)[0x55823258a362] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x182)[0x558232653842] /usr/sbin/mysqld(handle_one_connection+0x40)[0x558232653a00] /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fe8055340a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe8036df87d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe558c53020): insert into `availabilities` (`application_id`, `product_variant_id`, `is_available`, `updated_at`, `created_at`) values ('139', '8526760', '1', '2016-07-15 12:39:43', '2016-07-15 12:39:43') Connection ID (thread ID): 4966284 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on   The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. There are a lot of those queries that went through without a problem, here's the table structure: MariaDB [project-x]> show create table availabilities\G *************************** 1. row *************************** Table: availabilities Create Table: CREATE TABLE `availabilities` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `application_id` int(10) unsigned NOT NULL, `product_variant_id` int(10) unsigned NOT NULL, `is_available` tinyint(1) NOT NULL, `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', PRIMARY KEY (`id`), KEY `availabilities_application_id_foreign` (`application_id`), KEY `availabilities_product_variant_id_foreign` (`product_variant_id`), CONSTRAINT `availabilities_application_id_foreign` FOREIGN KEY (`application_id`) REFERENCES `applications` (`id`), CONSTRAINT `availabilities_product_variant_id_foreign` FOREIGN KEY (`product_variant_id`) REFERENCES `product_variants` (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=232970894 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci WSRep Options: [mysqld] wsrep_on=On wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_cluster_name=$censored wsrep_sst_method=xtrabackup-v2 wsrep_cluster_address=gcomm://$censored wsrep_provider_options="gcache.size=1G" Version 10.1.14-MariaDB-1~jessie from the MariaDB repos, running on Debian Jessie, 64 bit in a Xen 4.4 PVHVM domU (I have no hypervisor access), kernel 4.6.0-0.bpo.1-amd64 #1 SMP Debian 4.6.3-1~bpo8+1, most current libraries. Is there anything I can do to help resolve this? We could potentially run a test cluster, however I had to move the production setup to a master-master type setup since we also had trouble with SST ( MDEV-9423 )

Andrei Elkin added a comment - 2017-12-13 13:07 Fixed by MDEV-9510 .