141201 04:01:21 mysqld_safe mysqld restarted 141201 04:01:21 mysqld_safe WSREP: Running position recovery with --log_error='/nmc/mysql/wsrep_recovery.Q3swip' --pid-file='/nmc/mysql/db2.nmc.ug.local-recover.pid' 141201 04:01:54 mysqld_safe WSREP: Recovered position 41b84a03-39b2-11e4-a970-aa21a185ae66:22129924 141201 4:01:54 [Note] WSREP: wsrep_start_position var submitted: '41b84a03-39b2-11e4-a970-aa21a185ae66:22129924' 141201 4:01:54 [Note] InnoDB: Using mutexes to ref count buffer pool pages 141201 4:01:54 [Note] InnoDB: The InnoDB memory heap is disabled 141201 4:01:54 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 141201 4:01:54 [Note] InnoDB: Compressed tables use zlib 1.2.3 141201 4:01:54 [Note] InnoDB: Using Linux native AIO 141201 4:01:54 [Note] InnoDB: Using CPU crc32 instructions 141201 4:01:54 [Note] InnoDB: Initializing buffer pool, size = 32.0G 141201 4:01:55 [Note] InnoDB: Completed initialization of buffer pool 141201 4:01:56 [Note] InnoDB: Highest supported file format is Barracuda. 141201 4:01:56 [Note] InnoDB: 128 rollback segment(s) are active. 141201 4:01:56 [Note] InnoDB: Waiting for purge to start 141201 4:01:56 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 594091148202 141201 4:01:56 [Note] Plugin 'FEEDBACK' is disabled. 141201 4:01:56 [Note] Server socket created on IP: '0.0.0.0'. 141201 4:01:56 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired 141201 4:01:56 [ERROR] mysqld: Table 'event' is marked as crashed and should be repaired 141201 4:01:56 [Warning] Checking table: './mysql/event' 141201 4:01:56 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly 141201 4:01:56 [Note] Event Scheduler: Loaded 1 event 141201 4:01:56 [Note] WSREP: Initial position: 41b84a03-39b2-11e4-a970-aa21a185ae66:22129924 141201 4:01:56 [Note] WSREP: wsrep_load(): loading provider library 'none' 141201 4:01:56 [Note] Reading of all Master_info entries succeded 141201 4:01:56 [Note] Added new Master_info '' to hash table 141201 4:01:56 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.0.13-MariaDB-wsrep-log' socket: '/nmc/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_25.10.r4123 141201 4:01:56 [Note] Event Scheduler: scheduler thread started with id 2 141201 9:47:57 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 141201 12:31:05 [Warning] Aborted connection 33228 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141201 15:09:23 [Warning] Aborted connection 213236 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141201 15:42:04 [Warning] Aborted connection 231512 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141201 16:07:46 [Warning] Aborted connection 34900 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141201 17:23:07 [Warning] Aborted connection 299 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141201 18:07:05 [Warning] Aborted connection 106100 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141201 23:31:11 [Warning] Aborted connection 94025 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141202 3:00:11 [Warning] Aborted connection 280939 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141202 3:29:56 [Warning] Aborted connection 1093627 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141202 4:00:04 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.13-MariaDB-wsrep-log key_buffer_size=2097152 read_buffer_size=2097152 max_used_connections=357 max_threads=514 thread_count=108 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2118249 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f08e8f6a008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f065385bce0 thread_stack 0x48000 /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb9541b] /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x744b48] /lib64/libpthread.so.0[0x379e80f710] /usr/sbin/mysqld[0x94bc7e] /usr/sbin/mysqld[0x8a25b7] /usr/sbin/mysqld(_ZN7handler13ha_index_prevEPh+0x10c)[0x74999c] /usr/sbin/mysqld[0xb701f3] /usr/sbin/mysqld(_ZN7handler13ha_index_prevEPh+0x10c)[0x74999c] /usr/sbin/mysqld(_ZN17QUICK_SELECT_DESC8get_nextEv+0x5c)[0x81aa1c] /usr/sbin/mysqld[0x8331d8] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1b2)[0x602232] /usr/sbin/mysqld[0x61931d] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa3d)[0x62c18d] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x11)[0x62e001] /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1dd)[0x62abcd] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x28d)[0x62e35d] /usr/sbin/mysqld[0x5d3329] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x543e)[0x5debee] /usr/sbin/mysqld[0x5e1752] /usr/sbin/mysqld[0x5e210b] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x16d0)[0x5e3c30] /usr/sbin/mysqld(_Z10do_commandP3THD+0x132)[0x5e4402] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x54b)[0x6a31cb] /usr/sbin/mysqld(handle_one_connection+0x42)[0x6a32c2] /lib64/libpthread.so.0[0x379e8079d1] /lib64/libc.so.6(clone+0x6d)[0x379e4e886d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f06e2ffd020): is an invalid pointer Connection ID (thread ID): 1753220 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 141202 04:00:05 mysqld_safe Number of processes running now: 0 141202 04:00:05 mysqld_safe mysqld restarted 141202 04:00:05 mysqld_safe WSREP: Running position recovery with --log_error='/nmc/mysql/wsrep_recovery.z8sXwJ' --pid-file='/nmc/mysql/db2.nmc.ug.local-recover.pid' 141202 04:00:36 mysqld_safe WSREP: Recovered position 41b84a03-39b2-11e4-a970-aa21a185ae66:22129924 141202 4:00:36 [Note] WSREP: wsrep_start_position var submitted: '41b84a03-39b2-11e4-a970-aa21a185ae66:22129924' 141202 4:00:36 [Note] InnoDB: Using mutexes to ref count buffer pool pages 141202 4:00:36 [Note] InnoDB: The InnoDB memory heap is disabled 141202 4:00:36 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 141202 4:00:36 [Note] InnoDB: Compressed tables use zlib 1.2.3 141202 4:00:36 [Note] InnoDB: Using Linux native AIO 141202 4:00:36 [Note] InnoDB: Using CPU crc32 instructions 141202 4:00:36 [Note] InnoDB: Initializing buffer pool, size = 32.0G 141202 4:00:37 [Note] InnoDB: Completed initialization of buffer pool 141202 4:00:37 [Note] InnoDB: Highest supported file format is Barracuda. 141202 4:00:38 [Note] InnoDB: 128 rollback segment(s) are active. 141202 4:00:38 [Note] InnoDB: Waiting for purge to start 141202 4:00:38 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 607673746115 141202 4:00:38 [Note] Plugin 'FEEDBACK' is disabled. 141202 4:00:38 [Note] Server socket created on IP: '0.0.0.0'. 141202 4:00:38 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired 141202 4:00:38 [ERROR] mysqld: Table 'event' is marked as crashed and should be repaired 141202 4:00:38 [Warning] Checking table: './mysql/event' 141202 4:00:38 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly 141202 4:00:38 [Note] Event Scheduler: Loaded 1 event 141202 4:00:38 [Note] WSREP: Initial position: 41b84a03-39b2-11e4-a970-aa21a185ae66:22129924 141202 4:00:38 [Note] WSREP: wsrep_load(): loading provider library 'none' 141202 4:00:38 [Note] Event Scheduler: scheduler thread started with id 2 141202 4:00:38 [Note] Reading of all Master_info entries succeded 141202 4:00:38 [Note] Added new Master_info '' to hash table 141202 4:00:38 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.0.13-MariaDB-wsrep-log' socket: '/nmc/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_25.10.r4123 141202 13:58:20 [Warning] Aborted connection 139558 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error) 141202 14:47:52 [Warning] Aborted connection 201263 to db: 'zabbix' user: 'zabbix' host: 'zabbix.nmc.ug.local' (Unknown error)