130601 17:28:59 InnoDB: Assertion failure in thread 140059692267264 in file rem0rec.c line 564 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 130601 17:28:59 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.29-MariaDB-log key_buffer_size=268435456 read_buffer_size=2097152 max_used_connections=8 max_threads=1002 thread_count=5 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4380858 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f57340009a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6230346d98 thread_stack 0x48000 ??:0(my_print_stacktrace)[0xa9d44e] ??:0(handle_fatal_signal)[0x6e3c2b] ??:0(??)[0x309ec0f500] ??:0(??)[0x366f2328a5] ??:0(??)[0x366f234085] ??:0(rec_get_offsets_func.clone.4)[0x86af3b] ??:0(wsrep_rec_get_foreign_key)[0x86b8bc] ??:0(wsrep_append_foreign_key)[0x84e432] ??:0(row_ins_foreign_check_on_constraint)[0x9764bb] ??:0(row_ins_check_foreign_constraint)[0x977065] ??:0(row_upd_check_references_constraints)[0x888025] ??:0(row_upd_clust_step)[0x8892fc] ??:0(row_upd_step)[0x88991e] ??:0(row_update_for_mysql)[0x874fd9] ??:0(ha_innobase::delete_row(unsigned char const*))[0x851448] ??:0(handler::ha_delete_row(unsigned char const*))[0x6e7f3d] ??:0(Delete_rows_log_event::do_exec_row(Relay_log_info const*))[0x7a9a6b] ??:0(Rows_log_event::do_apply_event(Relay_log_info const*))[0x7b4122] ??:0(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*))[0x520f5d] ??:0(exec_relay_log_event(THD*, Relay_log_info*))[0x5228f9] ??:0(handle_slave_sql)[0x526990] ??:0(??)[0x309ec07851] ??:0(??)[0x366f2e890d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 3563 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130601 17:29:00 mysqld_safe Number of processes running now: 0 130601 17:29:00 mysqld_safe WSREP: not restarting wsrep node automatically 130601 17:29:00 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 130601 17:44:27 mysqld_safe Starting mysqld daemon with databases from /dbdata/mysql 130601 17:44:27 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.cwarJRKH4e 130601 17:44:43 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 130601 17:44:43 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1' 130601 17:44:43 [Note] WSREP: Read nil XID from storage engines, skipping position init 130601 17:44:43 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 130601 17:44:43 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy loaded succesfully. 130601 17:44:43 [Note] WSREP: Found saved state: 097a4e0a-caae-11e2-0800-99716173b34c:-1 130601 17:44:43 [Note] WSREP: Reusing existing '/dbdata/mysql//galera.cache'. 130601 17:44:43 [Note] WSREP: Passing config to GCS: base_host = 10.168.8.237; base_port = 4567; cert.log_conflicts = no; gcache.dir = /dbdata/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /dbdata/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3 130601 17:44:43 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 130601 17:44:43 [Note] WSREP: wsrep_sst_grab() 130601 17:44:43 [Note] WSREP: Start replication 130601 17:44:43 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 130601 17:44:43 [Note] WSREP: protonet asio version 0 130601 17:44:43 [Note] WSREP: backend: asio 130601 17:44:43 [Note] WSREP: GMCast version 0 130601 17:44:43 [Note] WSREP: (cc3b6774-cac9-11e2-0800-8a1b69a89780, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 130601 17:44:43 [Note] WSREP: (cc3b6774-cac9-11e2-0800-8a1b69a89780, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 130601 17:44:43 [Note] WSREP: EVS version 0 130601 17:44:43 [Note] WSREP: PC version 0 130601 17:44:43 [Note] WSREP: gcomm: connecting to group 'PPT_Prod_Galera_Cluster', peer '' 130601 17:44:43 [Note] WSREP: Node cc3b6774-cac9-11e2-0800-8a1b69a89780 state prim 130601 17:44:43 [Note] WSREP: view(view_id(PRIM,cc3b6774-cac9-11e2-0800-8a1b69a89780,1) memb { cc3b6774-cac9-11e2-0800-8a1b69a89780, } joined { } left { } partitioned { }) 130601 17:44:43 [Note] WSREP: gcomm: connected 130601 17:44:43 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 130601 17:44:43 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 130601 17:44:43 [Note] WSREP: Opened channel 'PPT_Prod_Galera_Cluster' 130601 17:44:43 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 130601 17:44:43 [Note] WSREP: Waiting for SST to complete. 130601 17:44:43 [Note] WSREP: Starting new group from scratch: cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b 130601 17:44:43 [Note] WSREP: STATE_EXCHANGE: sent state UUID: cc3c512b-cac9-11e2-0800-28befeb0b809 130601 17:44:43 [Note] WSREP: STATE EXCHANGE: sent state msg: cc3c512b-cac9-11e2-0800-28befeb0b809 130601 17:44:43 [Note] WSREP: STATE EXCHANGE: got state msg: cc3c512b-cac9-11e2-0800-28befeb0b809 from 0 (node1) 130601 17:44:43 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf_id = 0, members = 1/1 (joined/total), act_id = 0, last_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b 130601 17:44:43 [Note] WSREP: Flow-control interval: [16, 16] 130601 17:44:43 [Note] WSREP: Restored state OPEN -> JOINED (0) 130601 17:44:43 [Note] WSREP: Member 0 (node1) synced with group. 130601 17:44:43 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 130601 17:44:43 [Note] WSREP: New cluster view: global state: cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2 130601 17:44:43 [Note] WSREP: SST complete, seqno: 0 130601 17:44:43 InnoDB: The InnoDB memory heap is disabled 130601 17:44:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130601 17:44:43 InnoDB: Compressed tables use zlib 1.2.3 130601 17:44:43 InnoDB: Using Linux native AIO 130601 17:44:43 InnoDB: Initializing buffer pool, size = 40.0G 130601 17:44:46 InnoDB: Completed initialization of buffer pool 130601 17:44:46 InnoDB: highest supported file format is Barracuda. 130601 17:44:48 InnoDB: Waiting for the background threads to start 130601 17:44:49 Percona XtraDB (http://www.percona.com) 1.1.8-29.3 started; log sequence number 1711522824641 130601 17:44:49 [Note] Plugin 'FEEDBACK' is disabled. 130601 17:44:49 [ERROR] Failed to open the relay log './mysqld-relay-bin.000002' (relay_log_pos 39397979) 130601 17:44:49 [ERROR] Could not find target log during relay log initialization 130601 17:44:49 [ERROR] Failed to initialize the master info structure 130601 17:44:49 [Note] Event Scheduler: Loaded 0 events 130601 17:44:49 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.29-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_23.7.3.rXXXX 130601 17:44:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130601 17:44:49 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2 130601 17:44:49 [Note] WSREP: Synchronized with group, ready for connections 130601 17:44:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130601 17:47:31 [Note] /usr/sbin/mysqld: Normal shutdown 130601 17:47:31 [Note] WSREP: Stop replication 130601 17:47:31 [Note] WSREP: Closing send monitor... 130601 17:47:31 [Note] WSREP: Closed send monitor. 130601 17:47:31 [Note] WSREP: gcomm: terminating thread 130601 17:47:31 [Note] WSREP: gcomm: joining thread 130601 17:47:31 [Note] WSREP: gcomm: closing backend 130601 17:47:31 [Note] WSREP: view((empty)) 130601 17:47:31 [Note] WSREP: Received self-leave message. 130601 17:47:31 [Note] WSREP: gcomm: closed 130601 17:47:31 [Note] WSREP: Flow-control interval: [0, 0] 130601 17:47:31 [Note] WSREP: Received SELF-LEAVE. Closing connection. 130601 17:47:31 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 0) 130601 17:47:31 [Note] WSREP: RECV thread exiting 0: Success 130601 17:47:31 [Note] WSREP: New cluster view: global state: cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2 130601 17:47:31 [Note] WSREP: recv_thread() joined. 130601 17:47:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130601 17:47:31 [Note] WSREP: Closing slave action queue. 130601 17:47:31 [Note] WSREP: applier thread exiting (code:0) 130601 17:47:33 [Note] WSREP: rollbacker thread exiting 130601 17:47:33 [Note] Event Scheduler: Purging the queue. 0 events 130601 17:47:33 [Note] WSREP: dtor state: CLOSED 130601 17:47:33 [Note] WSREP: apply mon: entered 0 130601 17:47:33 [Note] WSREP: apply mon: entered 0 130601 17:47:33 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0 130601 17:47:33 [Note] WSREP: cert index usage at exit 0 130601 17:47:33 [Note] WSREP: cert trx map usage at exit 0 130601 17:47:33 [Note] WSREP: deps set usage at exit 0 130601 17:47:33 [Note] WSREP: avg deps dist 0 130601 17:47:33 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0 130601 17:47:33 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 0) 130601 17:47:33 [Note] WSREP: Flushing memory map to disk... 130601 17:47:33 InnoDB: Starting shutdown... 130601 17:47:36 InnoDB: Shutdown completed; log sequence number 1711522886914 130601 17:47:36 [Note] /usr/sbin/mysqld: Shutdown complete 130601 17:47:36 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 130601 17:49:45 mysqld_safe Starting mysqld daemon with databases from /dbdata/mysql 130601 17:49:45 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.LLrnrKh5Of 130601 17:49:54 mysqld_safe WSREP: Recovered position cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0 130601 17:49:54 [Note] WSREP: wsrep_start_position var submitted: 'cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0' 130601 17:49:54 [Note] WSREP: Read nil XID from storage engines, skipping position init 130601 17:49:54 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 130601 17:49:54 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy loaded succesfully. 130601 17:49:54 [Note] WSREP: Found saved state: cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0 130601 17:49:54 [Note] WSREP: Reusing existing '/dbdata/mysql//galera.cache'. 130601 17:49:54 [Note] WSREP: Passing config to GCS: base_host = 10.168.8.237; base_port = 4567; cert.log_conflicts = no; gcache.dir = /dbdata/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /dbdata/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3 130601 17:49:54 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1 130601 17:49:54 [Note] WSREP: wsrep_sst_grab() 130601 17:49:54 [Note] WSREP: Start replication 130601 17:49:54 [Note] WSREP: Setting initial position to cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0 130601 17:49:54 [Note] WSREP: protonet asio version 0 130601 17:49:54 [Note] WSREP: backend: asio 130601 17:49:54 [Note] WSREP: GMCast version 0 130601 17:49:54 [Note] WSREP: (85606f46-caca-11e2-0800-0565e5c2434b, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 130601 17:49:54 [Note] WSREP: (85606f46-caca-11e2-0800-0565e5c2434b, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 130601 17:49:54 [Note] WSREP: EVS version 0 130601 17:49:54 [Note] WSREP: PC version 0 130601 17:49:54 [Note] WSREP: gcomm: connecting to group 'PPT_Prod_Galera_Cluster', peer '' 130601 17:49:54 [Note] WSREP: Node 85606f46-caca-11e2-0800-0565e5c2434b state prim 130601 17:49:54 [Note] WSREP: view(view_id(PRIM,85606f46-caca-11e2-0800-0565e5c2434b,1) memb { 85606f46-caca-11e2-0800-0565e5c2434b, } joined { } left { } partitioned { }) 130601 17:49:54 [Note] WSREP: gcomm: connected 130601 17:49:54 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 130601 17:49:54 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 130601 17:49:54 [Note] WSREP: Opened channel 'PPT_Prod_Galera_Cluster' 130601 17:49:54 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 130601 17:49:54 [Note] WSREP: Waiting for SST to complete. 130601 17:49:54 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 85615877-caca-11e2-0800-9ad30f3520c1 130601 17:49:54 [Note] WSREP: STATE EXCHANGE: sent state msg: 85615877-caca-11e2-0800-9ad30f3520c1 130601 17:49:54 [Note] WSREP: STATE EXCHANGE: got state msg: 85615877-caca-11e2-0800-9ad30f3520c1 from 0 (node1) 130601 17:49:54 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf_id = 0, members = 1/1 (joined/total), act_id = 0, last_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b 130601 17:49:54 [Note] WSREP: Flow-control interval: [16, 16] 130601 17:49:54 [Note] WSREP: Restored state OPEN -> JOINED (0) 130601 17:49:54 [Note] WSREP: Member 0 (node1) synced with group. 130601 17:49:54 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 130601 17:49:54 [Note] WSREP: New cluster view: global state: cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2 130601 17:49:54 [Note] WSREP: SST complete, seqno: 0 130601 17:49:54 InnoDB: The InnoDB memory heap is disabled 130601 17:49:54 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130601 17:49:54 InnoDB: Compressed tables use zlib 1.2.3 130601 17:49:54 InnoDB: Using Linux native AIO 130601 17:49:54 InnoDB: Initializing buffer pool, size = 40.0G 130601 17:49:57 InnoDB: Completed initialization of buffer pool 130601 17:49:57 InnoDB: highest supported file format is Barracuda. 130601 17:49:58 InnoDB: Waiting for the background threads to start 130601 17:49:59 Percona XtraDB (http://www.percona.com) 1.1.8-29.3 started; log sequence number 1711522886914 130601 17:49:59 [Note] Plugin 'FEEDBACK' is disabled. 130601 17:49:59 [Note] WSREP: Waiting to reach ready state 130601 17:49:59 [Note] Event Scheduler: Loaded 0 events 130601 17:49:59 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.29-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_23.7.3.rXXXX 130601 17:49:59 [Note] Slave I/O thread: connected to master 'x@10.168.8.161:3306',replication started in log 'server1-bin.000705' at position 380307307 130601 17:49:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130601 17:49:59 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2 130601 17:49:59 [Note] WSREP: Synchronized with group, ready for connections 130601 17:49:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130601 17:49:59 [Note] WSREP: ready state reached 130601 17:49:59 [Note] Slave SQL thread initialized, starting replication in log 'server1-bin.000703' at position 969197872, relay log './mysqld-relay-bin.000002' position: 39397979 130601 17:49:59 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 130601 21:19:54 [Note] Error reading relay log event: slave SQL thread was killed 130601 21:19:54 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 130601 21:19:54 [Note] Slave I/O thread killed while reading event 130601 21:19:54 [Note] Slave I/O thread exiting, read up to log 'server1-bin.000707', position 715885219 130602 4:24:33 [Note] WSREP: ready state reached 130602 4:24:33 [Note] Slave SQL thread initialized, starting replication in log 'server1-bin.000707' at position 715885219, relay log './mysqld-relay-bin.000014' position: 715885505 130602 4:24:33 [Note] Slave I/O thread: connected to master 'x@10.168.8.161:3306',replication started in log 'server1-bin.000707' at position 715885219 130602 4:24:33 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 130602 4:25:56 [Warning] Aborted connection 20 to db: 'unconnected' user: 'x' host: 'localhost' (Unknown error) 130602 4:26:02 [Warning] Aborted connection 23 to db: 'unconnected' user: 'x' host: 'localhost' (Unknown error) 130602 4:33:31 [ERROR] mysqld: Table 'backup_progress' is marked as crashed and should be repaired 130602 4:35:37 [ERROR] mysqld: Table 'backup_history' is marked as crashed and should be repaired 130602 14:37:00 [Warning] Aborted connection 26 to db: 'unconnected' user: 'x' host: 'localhost' (Unknown error) 130602 20:40:15 InnoDB: Assertion failure in thread 139934599468800 in file rem0rec.c line 564 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 130602 20:40:15 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.29-MariaDB-log key_buffer_size=268435456 read_buffer_size=2097152 max_used_connections=5 max_threads=1002 thread_count=3 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4380858 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f3a4c0008c0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f451017cd98 thread_stack 0x48000 ??:0(my_print_stacktrace)[0xa9d44e] ??:0(handle_fatal_signal)[0x6e3c2b] ??:0(??)[0x309ec0f500] ??:0(??)[0x366f2328a5] ??:0(??)[0x366f234085] ??:0(rec_get_offsets_func.clone.4)[0x86af3b] ??:0(wsrep_rec_get_foreign_key)[0x86b8bc] ??:0(wsrep_append_foreign_key)[0x84e432] ??:0(row_ins_foreign_check_on_constraint)[0x9764bb] ??:0(row_ins_check_foreign_constraint)[0x977065] ??:0(row_upd_check_references_constraints)[0x888025] ??:0(row_upd_clust_step)[0x8892fc] ??:0(row_upd_step)[0x88991e] ??:0(row_update_for_mysql)[0x874fd9] ??:0(ha_innobase::delete_row(unsigned char const*))[0x851448] ??:0(handler::ha_delete_row(unsigned char const*))[0x6e7f3d] ??:0(Delete_rows_log_event::do_exec_row(Relay_log_info const*))[0x7a9a6b] ??:0(Rows_log_event::do_apply_event(Relay_log_info const*))[0x7b4122] ??:0(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*))[0x520f5d] ??:0(exec_relay_log_event(THD*, Relay_log_info*))[0x5228f9] ??:0(handle_slave_sql)[0x526990] ??:0(??)[0x309ec07851] ??:0(??)[0x366f2e890d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130602 20:40:16 mysqld_safe Number of processes running now: 0 130602 20:40:16 mysqld_safe WSREP: not restarting wsrep node automatically 130602 20:40:16 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 130603 07:25:44 mysqld_safe Starting mysqld daemon with databases from /dbdata/mysql 130603 07:25:44 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.cQmyHf1sEe 130603 07:25:57 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 130603 7:25:57 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1' 130603 7:25:57 [Note] WSREP: Read nil XID from storage engines, skipping position init 130603 7:25:57 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 130603 7:25:57 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy loaded succesfully. 130603 7:25:57 [Note] WSREP: Found saved state: cc3c3a0c-cac9-11e2-0800-6c4fcad1c39b:-1 130603 7:25:57 [Note] WSREP: Reusing existing '/dbdata/mysql//galera.cache'. 130603 7:25:57 [Note] WSREP: Passing config to GCS: base_host = 10.168.8.237; base_port = 4567; cert.log_conflicts = no; gcache.dir = /dbdata/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /dbdata/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3 130603 7:25:57 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 130603 7:25:57 [Note] WSREP: wsrep_sst_grab() 130603 7:25:57 [Note] WSREP: Start replication 130603 7:25:57 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 130603 7:25:57 [Note] WSREP: protonet asio version 0 130603 7:25:57 [Note] WSREP: backend: asio 130603 7:25:57 [Note] WSREP: GMCast version 0 130603 7:25:57 [Note] WSREP: (aff7e71a-cc05-11e2-0800-195373eb0f08, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 130603 7:25:57 [Note] WSREP: (aff7e71a-cc05-11e2-0800-195373eb0f08, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 130603 7:25:57 [Note] WSREP: EVS version 0 130603 7:25:57 [Note] WSREP: PC version 0 130603 7:25:57 [Note] WSREP: gcomm: connecting to group 'PPT_Prod_Galera_Cluster', peer '' 130603 7:25:57 [Note] WSREP: Node aff7e71a-cc05-11e2-0800-195373eb0f08 state prim 130603 7:25:57 [Note] WSREP: view(view_id(PRIM,aff7e71a-cc05-11e2-0800-195373eb0f08,1) memb { aff7e71a-cc05-11e2-0800-195373eb0f08, } joined { } left { } partitioned { }) 130603 7:25:57 [Note] WSREP: gcomm: connected 130603 7:25:57 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 130603 7:25:57 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 130603 7:25:57 [Note] WSREP: Opened channel 'PPT_Prod_Galera_Cluster' 130603 7:25:57 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 130603 7:25:57 [Note] WSREP: Waiting for SST to complete. 130603 7:25:57 [Note] WSREP: Starting new group from scratch: aff8cfc6-cc05-11e2-0800-0ef8cb00c81b 130603 7:25:57 [Note] WSREP: STATE_EXCHANGE: sent state UUID: aff90260-cc05-11e2-0800-e73cbc70157e 130603 7:25:57 [Note] WSREP: STATE EXCHANGE: sent state msg: aff90260-cc05-11e2-0800-e73cbc70157e 130603 7:25:57 [Note] WSREP: STATE EXCHANGE: got state msg: aff90260-cc05-11e2-0800-e73cbc70157e from 0 (node1) 130603 7:25:57 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf_id = 0, members = 1/1 (joined/total), act_id = 0, last_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = aff8cfc6-cc05-11e2-0800-0ef8cb00c81b 130603 7:25:57 [Note] WSREP: Flow-control interval: [16, 16] 130603 7:25:57 [Note] WSREP: Restored state OPEN -> JOINED (0) 130603 7:25:57 [Note] WSREP: Member 0 (node1) synced with group. 130603 7:25:57 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 130603 7:25:57 [Note] WSREP: New cluster view: global state: aff8cfc6-cc05-11e2-0800-0ef8cb00c81b:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2 130603 7:25:57 [Note] WSREP: SST complete, seqno: 0 130603 7:25:57 InnoDB: The InnoDB memory heap is disabled 130603 7:25:57 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130603 7:25:57 InnoDB: Compressed tables use zlib 1.2.3 130603 7:25:57 InnoDB: Using Linux native AIO 130603 7:25:57 InnoDB: Initializing buffer pool, size = 40.0G 130603 7:26:00 InnoDB: Completed initialization of buffer pool 130603 7:26:00 InnoDB: highest supported file format is Barracuda. 130603 7:26:02 InnoDB: Waiting for the background threads to start 130603 7:26:03 Percona XtraDB (http://www.percona.com) 1.1.8-29.3 started; log sequence number 1795068749763 130603 7:26:03 [Note] Plugin 'FEEDBACK' is disabled. 130603 7:26:03 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired 130603 7:26:03 [ERROR] mysqld: Table 'user' is marked as crashed and should be repaired 130603 7:26:03 [Warning] Checking table: './mysql/user' 130603 7:26:03 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly 130603 7:26:03 [Note] WSREP: Waiting to reach ready state 130603 7:26:03 [Note] Slave I/O thread: connected to master 'x@10.168.8.161:3306',replication started in log 'server1-bin.000714' at position 233474381 130603 7:26:03 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 130603 7:26:03 [Note] Event Scheduler: Loaded 0 events 130603 7:26:03 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.29-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_23.7.3.rXXXX 130603 7:26:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130603 7:26:03 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2 130603 7:26:03 [Note] WSREP: Synchronized with group, ready for connections 130603 7:26:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130603 7:26:03 [Note] WSREP: ready state reached 130603 7:26:03 [Note] Slave SQL thread initialized, starting replication in log 'server1-bin.000714' at position 233420352, relay log './mysqld-relay-bin.000035' position: 233420638 130603 7:28:55 [Warning] Aborted connection 6 to db: 'unconnected' user: 'x' host: 'localhost' (Unknown error) 130603 7:30:02 InnoDB: Assertion failure in thread 140289610331904 in file rem0rec.c line 564 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 130603 7:30:02 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.29-MariaDB-log key_buffer_size=268435456 read_buffer_size=2097152 max_used_connections=4 max_threads=1002 thread_count=3 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4380858 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f8cf40009a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f97b863ad98 thread_stack 0x48000 ??:0(my_print_stacktrace)[0xa9d44e] ??:0(handle_fatal_signal)[0x6e3c2b] ??:0(??)[0x309ec0f500] ??:0(??)[0x366f2328a5] ??:0(??)[0x366f234085] ??:0(rec_get_offsets_func.clone.4)[0x86af3b] ??:0(wsrep_rec_get_foreign_key)[0x86b8bc] ??:0(wsrep_append_foreign_key)[0x84e432] ??:0(row_ins_foreign_check_on_constraint)[0x9764bb] ??:0(row_ins_check_foreign_constraint)[0x977065] ??:0(row_upd_check_references_constraints)[0x888025] ??:0(row_upd_clust_step)[0x8892fc] ??:0(row_upd_step)[0x88991e] ??:0(row_update_for_mysql)[0x874fd9] ??:0(ha_innobase::delete_row(unsigned char const*))[0x851448] ??:0(handler::ha_delete_row(unsigned char const*))[0x6e7f3d] ??:0(Delete_rows_log_event::do_exec_row(Relay_log_info const*))[0x7a9a6b] ??:0(Rows_log_event::do_apply_event(Relay_log_info const*))[0x7b4122] ??:0(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*))[0x520f5d] ??:0(exec_relay_log_event(THD*, Relay_log_info*))[0x5228f9] ??:0(handle_slave_sql)[0x526990] ??:0(??)[0x309ec07851] ??:0(??)[0x366f2e890d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 4 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130603 07:30:03 mysqld_safe Number of processes running now: 0 130603 07:30:03 mysqld_safe WSREP: not restarting wsrep node automatically 130603 07:30:03 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 130603 07:34:34 mysqld_safe Starting mysqld daemon with databases from /dbdata/mysql 130603 07:34:34 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.XMlV7Gk6TY 130603 07:35:13 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 130603 7:35:13 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1' 130603 7:35:13 [Note] WSREP: Read nil XID from storage engines, skipping position init 130603 7:35:13 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 130603 7:35:13 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy loaded succesfully. 130603 7:35:13 [Note] WSREP: Found saved state: aff8cfc6-cc05-11e2-0800-0ef8cb00c81b:-1 130603 7:35:13 [Note] WSREP: Reusing existing '/dbdata/mysql//galera.cache'. 130603 7:35:13 [Note] WSREP: Passing config to GCS: base_host = 10.168.8.237; base_port = 4567; cert.log_conflicts = no; gcache.dir = /dbdata/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /dbdata/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3 130603 7:35:13 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 130603 7:35:13 [Note] WSREP: wsrep_sst_grab() 130603 7:35:13 [Note] WSREP: Start replication 130603 7:35:13 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 130603 7:35:13 [Note] WSREP: protonet asio version 0 130603 7:35:13 [Note] WSREP: backend: asio 130603 7:35:13 [Note] WSREP: GMCast version 0 130603 7:35:13 [Note] WSREP: (fb65365b-cc06-11e2-0800-88dd0d31c771, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 130603 7:35:13 [Note] WSREP: (fb65365b-cc06-11e2-0800-88dd0d31c771, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 130603 7:35:13 [Note] WSREP: EVS version 0 130603 7:35:13 [Note] WSREP: PC version 0 130603 7:35:13 [Note] WSREP: gcomm: connecting to group 'PPT_Prod_Galera_Cluster', peer '' 130603 7:35:13 [Note] WSREP: Node fb65365b-cc06-11e2-0800-88dd0d31c771 state prim 130603 7:35:13 [Note] WSREP: view(view_id(PRIM,fb65365b-cc06-11e2-0800-88dd0d31c771,1) memb { fb65365b-cc06-11e2-0800-88dd0d31c771, } joined { } left { } partitioned { }) 130603 7:35:13 [Note] WSREP: gcomm: connected 130603 7:35:13 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 130603 7:35:13 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 130603 7:35:13 [Note] WSREP: Opened channel 'PPT_Prod_Galera_Cluster' 130603 7:35:13 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 130603 7:35:13 [Note] WSREP: Waiting for SST to complete. 130603 7:35:13 [Note] WSREP: Starting new group from scratch: fb660978-cc06-11e2-0800-04612f7800bd 130603 7:35:13 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fb661f53-cc06-11e2-0800-49ca22b306d1 130603 7:35:13 [Note] WSREP: STATE EXCHANGE: sent state msg: fb661f53-cc06-11e2-0800-49ca22b306d1 130603 7:35:13 [Note] WSREP: STATE EXCHANGE: got state msg: fb661f53-cc06-11e2-0800-49ca22b306d1 from 0 (node1) 130603 7:35:13 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf_id = 0, members = 1/1 (joined/total), act_id = 0, last_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = fb660978-cc06-11e2-0800-04612f7800bd 130603 7:35:13 [Note] WSREP: Flow-control interval: [16, 16] 130603 7:35:13 [Note] WSREP: Restored state OPEN -> JOINED (0) 130603 7:35:13 [Note] WSREP: Member 0 (node1) synced with group. 130603 7:35:13 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 130603 7:35:13 [Note] WSREP: New cluster view: global state: fb660978-cc06-11e2-0800-04612f7800bd:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2 130603 7:35:13 [Note] WSREP: SST complete, seqno: 0 130603 7:35:13 InnoDB: The InnoDB memory heap is disabled 130603 7:35:13 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130603 7:35:13 InnoDB: Compressed tables use zlib 1.2.3 130603 7:35:13 InnoDB: Using Linux native AIO 130603 7:35:13 InnoDB: Initializing buffer pool, size = 40.0G 130603 7:35:16 InnoDB: Completed initialization of buffer pool 130603 7:35:16 InnoDB: highest supported file format is Barracuda. 130603 7:35:18 InnoDB: Waiting for the background threads to start 130603 7:35:19 Percona XtraDB (http://www.percona.com) 1.1.8-29.3 started; log sequence number 1795885274771 130603 7:35:19 [Note] Plugin 'FEEDBACK' is disabled. 130603 7:35:19 [Note] WSREP: Waiting to reach ready state 130603 7:35:19 [Note] Event Scheduler: Loaded 0 events 130603 7:35:19 [Note] Slave I/O thread: connected to master 'x@10.168.8.161:3306',replication started in log 'server1-bin.000716' at position 868089428 130603 7:35:19 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.29-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_23.7.3.rXXXX 130603 7:35:19 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130603 7:35:19 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2 130603 7:35:19 [Note] WSREP: Synchronized with group, ready for connections 130603 7:35:19 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130603 7:35:19 [Note] WSREP: ready state reached 130603 7:35:19 [Note] Slave SQL thread initialized, starting replication in log 'server1-bin.000714' at position 1033833853, relay log './mysqld-relay-bin.000037' position: 800359864 130603 7:35:19 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 130603 7:35:19 InnoDB: Assertion failure in thread 139851920447232 in file rem0rec.c line 564 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 130603 7:35:19 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.29-MariaDB-log key_buffer_size=268435456 read_buffer_size=2097152 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10537146 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f26e00009a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f31d00a0d98 thread_stack 0x48000 ??:0(my_print_stacktrace)[0xa9d44e] ??:0(handle_fatal_signal)[0x6e3c2b] ??:0(??)[0x309ec0f500] ??:0(??)[0x366f2328a5] ??:0(??)[0x366f234085] ??:0(rec_get_offsets_func.clone.4)[0x86af3b] ??:0(wsrep_rec_get_foreign_key)[0x86b8bc] ??:0(wsrep_append_foreign_key)[0x84e432] ??:0(row_ins_foreign_check_on_constraint)[0x9764bb] ??:0(row_ins_check_foreign_constraint)[0x977065] ??:0(row_upd_check_references_constraints)[0x888025] ??:0(row_upd_clust_step)[0x8892fc] ??:0(row_upd_step)[0x88991e] ??:0(row_update_for_mysql)[0x874fd9] ??:0(ha_innobase::delete_row(unsigned char const*))[0x851448] ??:0(handler::ha_delete_row(unsigned char const*))[0x6e7f3d] ??:0(Delete_rows_log_event::do_exec_row(Relay_log_info const*))[0x7a9a6b] ??:0(Rows_log_event::do_apply_event(Relay_log_info const*))[0x7b4122] ??:0(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*))[0x520f5d] ??:0(exec_relay_log_event(THD*, Relay_log_info*))[0x5228f9] ??:0(handle_slave_sql)[0x526990] ??:0(??)[0x309ec07851] ??:0(??)[0x366f2e890d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 4 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130603 07:35:19 mysqld_safe Number of processes running now: 0 130603 07:35:19 mysqld_safe WSREP: not restarting wsrep node automatically 130603 07:35:19 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended 130603 07:42:39 mysqld_safe Starting mysqld daemon with databases from /dbdata/mysql 130603 07:42:39 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.ihExJHluUM 130603 07:42:51 mysqld_safe WSREP: Recovered position fb660978-cc06-11e2-0800-04612f7800bd:0 130603 7:42:51 [Note] WSREP: wsrep_start_position var submitted: 'fb660978-cc06-11e2-0800-04612f7800bd:0' 130603 7:42:51 [Note] WSREP: Read nil XID from storage engines, skipping position init 130603 7:42:51 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 130603 7:42:51 [Note] WSREP: wsrep_load(): Galera 23.2.4(r147) by Codership Oy loaded succesfully. 130603 7:42:51 [Note] WSREP: Found saved state: fb660978-cc06-11e2-0800-04612f7800bd:-1 130603 7:42:51 [Note] WSREP: Reusing existing '/dbdata/mysql//galera.cache'. 130603 7:42:51 [Note] WSREP: Passing config to GCS: base_host = 10.168.8.237; base_port = 4567; cert.log_conflicts = no; gcache.dir = /dbdata/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /dbdata/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3 130603 7:42:51 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1 130603 7:42:51 [Note] WSREP: wsrep_sst_grab() 130603 7:42:51 [Note] WSREP: Start replication 130603 7:42:51 [Note] WSREP: Setting initial position to fb660978-cc06-11e2-0800-04612f7800bd:0 130603 7:42:51 [Note] WSREP: protonet asio version 0 130603 7:42:51 [Note] WSREP: backend: asio 130603 7:42:51 [Note] WSREP: GMCast version 0 130603 7:42:51 [Note] WSREP: (0c599062-cc08-11e2-0800-a2ece11ead9c, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 130603 7:42:51 [Note] WSREP: (0c599062-cc08-11e2-0800-a2ece11ead9c, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 130603 7:42:51 [Note] WSREP: EVS version 0 130603 7:42:51 [Note] WSREP: PC version 0 130603 7:42:51 [Note] WSREP: gcomm: connecting to group 'PPT_Prod_Galera_Cluster', peer '' 130603 7:42:51 [Note] WSREP: Node 0c599062-cc08-11e2-0800-a2ece11ead9c state prim 130603 7:42:51 [Note] WSREP: view(view_id(PRIM,0c599062-cc08-11e2-0800-a2ece11ead9c,1) memb { 0c599062-cc08-11e2-0800-a2ece11ead9c, } joined { } left { } partitioned { }) 130603 7:42:51 [Note] WSREP: gcomm: connected 130603 7:42:51 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 130603 7:42:51 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 130603 7:42:51 [Note] WSREP: Opened channel 'PPT_Prod_Galera_Cluster' 130603 7:42:51 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 130603 7:42:51 [Note] WSREP: Waiting for SST to complete. 130603 7:42:51 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0c5a5ef4-cc08-11e2-0800-c293d114708a 130603 7:42:51 [Note] WSREP: STATE EXCHANGE: sent state msg: 0c5a5ef4-cc08-11e2-0800-c293d114708a 130603 7:42:51 [Note] WSREP: STATE EXCHANGE: got state msg: 0c5a5ef4-cc08-11e2-0800-c293d114708a from 0 (node1) 130603 7:42:51 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf_id = 0, members = 1/1 (joined/total), act_id = 0, last_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = fb660978-cc06-11e2-0800-04612f7800bd 130603 7:42:51 [Note] WSREP: Flow-control interval: [16, 16] 130603 7:42:51 [Note] WSREP: Restored state OPEN -> JOINED (0) 130603 7:42:51 [Note] WSREP: Member 0 (node1) synced with group. 130603 7:42:51 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 130603 7:42:51 [Note] WSREP: New cluster view: global state: fb660978-cc06-11e2-0800-04612f7800bd:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2 130603 7:42:51 [Note] WSREP: SST complete, seqno: 0 130603 7:42:51 InnoDB: The InnoDB memory heap is disabled 130603 7:42:51 InnoDB: Mutexes and rw_locks use GCC atomic builtins 130603 7:42:51 InnoDB: Compressed tables use zlib 1.2.3 130603 7:42:51 InnoDB: Using Linux native AIO 130603 7:42:51 InnoDB: Initializing buffer pool, size = 40.0G 130603 7:42:54 InnoDB: Completed initialization of buffer pool 130603 7:42:54 InnoDB: highest supported file format is Barracuda. 130603 7:42:55 InnoDB: Waiting for the background threads to start 130603 7:42:56 Percona XtraDB (http://www.percona.com) 1.1.8-29.3 started; log sequence number 1795885389126 130603 7:42:56 [Note] Plugin 'FEEDBACK' is disabled. 130603 7:42:56 [Note] WSREP: Waiting to reach ready state 130603 7:42:56 [Note] Event Scheduler: Loaded 0 events 130603 7:42:56 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.29-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_23.7.3.rXXXX 130603 7:42:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130603 7:42:56 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2 130603 7:42:56 [Note] WSREP: Synchronized with group, ready for connections 130603 7:42:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 130603 7:42:56 [Note] WSREP: ready state reached 130603 7:42:56 [Note] Slave SQL thread initialized, starting replication in log 'server1-bin.000714' at position 1033833853, relay log './mysqld-relay-bin.000037' position: 800359864 130603 7:42:56 [Note] Slave I/O thread: connected to master 'x@10.168.8.161:3306',replication started in log 'server1-bin.000716' at position 872047406 130603 7:42:56 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 130603 7:42:56 InnoDB: Assertion failure in thread 140097338754816 in file rem0rec.c line 564 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 130603 7:42:56 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.29-MariaDB-log key_buffer_size=268435456 read_buffer_size=2097152 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4380858 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f60100009a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6af41c3d98 thread_stack 0x48000 ??:0(my_print_stacktrace)[0xa9d44e] ??:0(handle_fatal_signal)[0x6e3c2b] ??:0(??)[0x309ec0f500] ??:0(??)[0x366f2328a5] ??:0(??)[0x366f234085] ??:0(rec_get_offsets_func.clone.4)[0x86af3b] ??:0(wsrep_rec_get_foreign_key)[0x86b8bc] ??:0(wsrep_append_foreign_key)[0x84e432] ??:0(row_ins_foreign_check_on_constraint)[0x9764bb] ??:0(row_ins_check_foreign_constraint)[0x977065] ??:0(row_upd_check_references_constraints)[0x888025] ??:0(row_upd_clust_step)[0x8892fc] ??:0(row_upd_step)[0x88991e] ??:0(row_update_for_mysql)[0x874fd9] ??:0(ha_innobase::delete_row(unsigned char const*))[0x851448] ??:0(handler::ha_delete_row(unsigned char const*))[0x6e7f3d] ??:0(Delete_rows_log_event::do_exec_row(Relay_log_info const*))[0x7a9a6b] ??:0(Rows_log_event::do_apply_event(Relay_log_info const*))[0x7b4122] ??:0(apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*))[0x520f5d] ??:0(exec_relay_log_event(THD*, Relay_log_info*))[0x5228f9] ??:0(handle_slave_sql)[0x526990] ??:0(??)[0x309ec07851] ??:0(??)[0x366f2e890d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 4 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130603 07:42:57 mysqld_safe Number of processes running now: 0 130603 07:42:57 mysqld_safe WSREP: not restarting wsrep node automatically 130603 07:42:57 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended