170328 09:02:54 mysqld_safe Starting mysqld daemon with databases from /var/persistent 170328 09:02:54 mysqld_safe WSREP: Running position recovery with --log_error='/var/persistent/wsrep_recovery.plR9Kg' --pid-file='/var/persistent/vmnds23node1-recover.pid' 2017-03-28 9:02:54 139916663138112 [Note] /usr/local/mariadb/mariadb_base/bin/mysqld (mysqld 10.1.21-MariaDB) starting as process 23390 ... 170328 09:02:57 mysqld_safe WSREP: Recovered position d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21:161 2017-03-28 9:02:57 139962560333632 [Note] /usr/local/mariadb/mariadb_base/bin/mysqld (mysqld 10.1.21-MariaDB) starting as process 23482 ... 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Read nil XID from storage engines, skipping position init 2017-03-28 9:02:57 139962560333632 [Note] WSREP: wsrep_load(): loading provider library '/lib/libgalera_smm.so' 2017-03-28 9:02:57 139962560333632 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy loaded successfully. 2017-03-28 9:02:57 139962560333632 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm. 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Found saved state: d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21:-1, safe_to_bootsrap: 0 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Passing config to GCS: base_dir = /var/persistent/; base_host = 10.9.29.31; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT2.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 512; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 256; evs.view_forget_timeout = PT24H; gcache.dir = /var/persistent/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/persistent//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2410151936; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://0.0.0.0:4567; gmcast.segment = 0; gmc 2017-03-28 9:02:57 139962560333632 [Note] WSREP: GCache history reset: old(d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21:0) -> new(d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21:161) 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Assign initial position for certification: 161, protocol version: -1 2017-03-28 9:02:57 139962560333632 [Note] WSREP: wsrep_sst_grab() 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Start replication 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Setting initial position to d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21:161 2017-03-28 9:02:57 139962560333632 [Note] WSREP: protonet asio version 0 2017-03-28 9:02:57 139962560333632 [Note] WSREP: Using CRC-32C for message checksums. 2017-03-28 9:02:57 139962560333632 [Note] WSREP: backend: asio 2017-03-28 9:02:57 139962560333632 [Note] WSREP: gcomm thread scheduling priority set to other:0 2017-03-28 9:02:57 139962560333632 [Warning] WSREP: access file(/var/persistent//gvwstate.dat) failed(No such file or directory) 2017-03-28 9:02:57 139962560333632 [Note] WSREP: restore pc from disk failed 2017-03-28 9:02:57 139962560333632 [Note] WSREP: GMCast version 0 2017-03-28 9:02:57 139962560333632 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2017-03-28 9:02:57 139962560333632 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2017-03-28 9:02:57 139962560333632 [Note] WSREP: EVS version 0 2017-03-28 9:02:57 139962560333632 [Note] WSREP: gcomm: connecting to group 'mdbcluster', peer '10.9.29.31:4567,10.9.29.32:4567' 2017-03-28 9:02:57 139962560333632 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') connection established to dd871779 tcp://10.9.29.31:4567 2017-03-28 9:02:57 139962560333632 [Warning] WSREP: (dd871779, 'tcp://0.0.0.0:4567') address 'tcp://10.9.29.31:4567' points to own listening address, blacklisting 2017-03-28 9:02:57 139962560333632 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') connection established to ca4b3f9b tcp://10.9.29.32:4567 2017-03-28 9:02:57 139962560333632 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2017-03-28 9:02:58 139962560333632 [Note] WSREP: declaring ca4b3f9b at tcp://10.9.29.32:4567 stable 2017-03-28 9:02:58 139962560333632 [Note] WSREP: Node ca4b3f9b state prim 2017-03-28 9:02:58 139962560333632 [Note] WSREP: view(view_id(PRIM,ca4b3f9b,2) memb { ca4b3f9b,0 dd871779,0 } joined { } left { } partitioned { }) 2017-03-28 9:02:58 139962560333632 [Note] WSREP: save pc into disk 2017-03-28 9:02:58 139962560333632 [Note] WSREP: gcomm: connected 2017-03-28 9:02:58 139962560333632 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2017-03-28 9:02:58 139962560333632 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2017-03-28 9:02:58 139962560333632 [Note] WSREP: Opened channel 'mdbcluster' 2017-03-28 9:02:58 139962560333632 [Note] WSREP: Waiting for SST to complete. 2017-03-28 9:02:58 139962229966592 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2017-03-28 9:02:58 139962229966592 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2017-03-28 9:02:58 139962229966592 [Note] WSREP: STATE EXCHANGE: sent state msg: ddd75464-13b6-11e7-99c7-6ee1c9392090 2017-03-28 9:02:58 139962229966592 [Note] WSREP: STATE EXCHANGE: got state msg: ddd75464-13b6-11e7-99c7-6ee1c9392090 from 0 (10.9.29.32) 2017-03-28 9:02:58 139962229966592 [Note] WSREP: STATE EXCHANGE: got state msg: ddd75464-13b6-11e7-99c7-6ee1c9392090 from 1 (10.9.29.31) 2017-03-28 9:02:58 139962229966592 [Note] WSREP: Quorum results: version = 4, component = PRIMARY, conf_id = 1, members = 2/2 (joined/total), act_id = 161, last_appl. = -1, protocols = 0/7/3 (gcs/repl/appl), group UUID = d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21 2017-03-28 9:02:58 139962229966592 [Note] WSREP: Flow-control interval: [23, 23] 2017-03-28 9:02:58 139962229966592 [Note] WSREP: Restored state OPEN -> JOINED (161) 2017-03-28 9:02:58 139962558851840 [Note] WSREP: New cluster view: global state: d4f1c312-0a7c-11e7-9cb3-cf3bd2177a21:161, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3 2017-03-28 9:02:58 139962560333632 [Note] WSREP: SST complete, seqno: 161 2017-03-28 9:02:58 139962229966592 [Note] WSREP: Member 1.0 (10.9.29.31) synced with group. 2017-03-28 9:02:58 139962229966592 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 161) 2017-3-28 9:2:58 BW Encrypt plug [Note] Initializing mdb plugin... 2017-3-28 9:2:58 BW Encrypt plug [Note] Plugin properly initialized 2017-3-28 9:2:58 BW Encrypt plug [Note] Load keystore request 2017-3-28 9:2:58 BW Encrypt plug [Note] Latest key version = 2 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: The InnoDB memory heap is disabled 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Compressed tables use zlib 1.2.3 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Using Linux native AIO 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Using generic crc32 instructions 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Initializing buffer pool, size = 919.0M 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Completed initialization of buffer pool 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Highest supported file format is Barracuda. 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: 128 rollback segment(s) are active. 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Waiting for purge to start 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1742871 2017-03-28 9:02:59 139960325744384 [Note] InnoDB: Dumping buffer pool(s) not yet started 2017-03-28 09:02:59 7f4b0d7fa700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool 2017-03-28 9:02:59 139962560333632 [Note] InnoDB: Creating #1 thread id 139960300566272 total threads 1. 2017-03-28 9:02:59 139962560333632 [Note] Plugin 'FEEDBACK' is disabled. 170328 9:02:59 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED. 170328 9:02:59 server_audit: logging started to the file /var/logs/auditLog. 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-03-28 9:02:59 139962560333632 [Note] Server socket created on IP: '::'. 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:2:59 BW Encrypt plug [Note] Key version 2 found 2017-03-28 09:02:59 7f4b0d7fa700 InnoDB: Buffer pool(s) load completed at 170328 9:02:59 2017-03-28 9:02:59 139962560333632 [Note] /usr/local/mariadb/mariadb_base/bin/mysqld: ready for connections. Version: '10.1.21-MariaDB' socket: '/var/run/mysql.sock' port: 3306 Source distribution 2017-03-28 9:02:59 139962558851840 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-03-28 9:02:59 139962558851840 [Note] WSREP: REPL Protocols: 7 (3, 2) 2017-03-28 9:02:59 139962558851840 [Note] WSREP: Assign initial position for certification: 161, protocol version: 3 2017-03-28 9:02:59 139962476943104 [Note] WSREP: Service thread queue flushed. 2017-03-28 9:02:59 139962558851840 [Note] WSREP: Synchronized with group, ready for connections 2017-03-28 9:02:59 139962558851840 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-03-28 9:03:00 139962238359296 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') connection to peer dd871779 with addr tcp://10.9.29.31:4567 timed out, no messages seen in PT3S 2017-03-28 9:03:00 139962238359296 [Note] WSREP: (dd871779, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-3-28 9:3:1 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:16:34 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:20:3 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:23:17 BW Encrypt plug [Note] Key version 2 found 2017-3-28 9:23:17 BW Encrypt plug [Note] Key version 2 found 2017-03-28 9:31:07 139962181204736 [Warning] IP address '10.9.29.32' could not be resolved: Name or service not known 2017-03-28 9:46:07 139962181204736 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table jfd.profile_building_subscriber; Cannot delete or update a parent row: a foreign key constraint fails (`jfd`.`profile_building_task`, CONSTRAINT `profile_building_task_ibfk_1` FOREIGN KEY (`profile_building_subscriber_uid`) REFERENCES `profile_building_subscriber` (`profile_building_subscriber_), Error_code: 1451; handler error HA_ERR_ROW_IS_REFERENCED; the event's master log FIRST, end_log_pos 177, Internal MariaDB error code: 1451 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: RBR event 2 Delete_rows_v1 apply warning: 152, 6947 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: Failed to apply app buffer: seqno: 6947, status: 1 at galera/src/trx_handle.cpp:apply():351 Retrying 2th time 2017-03-28 9:46:07 139962181204736 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table jfd.profile_building_subscriber; Cannot delete or update a parent row: a foreign key constraint fails (`jfd`.`profile_building_task`, CONSTRAINT `profile_building_task_ibfk_1` FOREIGN KEY (`profile_building_subscriber_uid`) REFERENCES `profile_building_subscriber` (`profile_building_subscriber_), Error_code: 1451; handler error HA_ERR_ROW_IS_REFERENCED; the event's master log FIRST, end_log_pos 177, Internal MariaDB error code: 1451 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: RBR event 2 Delete_rows_v1 apply warning: 152, 6947 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: Failed to apply app buffer: seqno: 6947, status: 1 at galera/src/trx_handle.cpp:apply():351 Retrying 3th time 2017-03-28 9:46:07 139962181204736 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table jfd.profile_building_subscriber; Cannot delete or update a parent row: a foreign key constraint fails (`jfd`.`profile_building_task`, CONSTRAINT `profile_building_task_ibfk_1` FOREIGN KEY (`profile_building_subscriber_uid`) REFERENCES `profile_building_subscriber` (`profile_building_subscriber_), Error_code: 1451; handler error HA_ERR_ROW_IS_REFERENCED; the event's master log FIRST, end_log_pos 177, Internal MariaDB error code: 1451 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: RBR event 2 Delete_rows_v1 apply warning: 152, 6947 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: Failed to apply app buffer: seqno: 6947, status: 1 at galera/src/trx_handle.cpp:apply():351 Retrying 4th time 2017-03-28 9:46:07 139962181204736 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table jfd.profile_building_subscriber; Cannot delete or update a parent row: a foreign key constraint fails (`jfd`.`profile_building_task`, CONSTRAINT `profile_building_task_ibfk_1` FOREIGN KEY (`profile_building_subscriber_uid`) REFERENCES `profile_building_subscriber` (`profile_building_subscriber_), Error_code: 1451; handler error HA_ERR_ROW_IS_REFERENCED; the event's master log FIRST, end_log_pos 177, Internal MariaDB error code: 1451 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: RBR event 2 Delete_rows_v1 apply warning: 152, 6947 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: failed to replay trx: source: dd871779-13b6-11e7-831f-5a24b2d09c67 version: 3 local: 1 state: REPLAYING flags: 1 conn_id: 10316 trx_id: 26288 seqnos (l: 6840, g: 6947, s: 6945, d: 6946, ts: 3092915556995) 2017-03-28 9:46:07 139962181204736 [Warning] WSREP: Failed to apply trx 6947 4 times 2017-03-28 9:46:07 139962181204736 [ERROR] WSREP: trx_replay failed for: 6, schema: (null), query: void 2017-03-28 9:46:07 139962181204736 [ERROR] Aborting 2017-03-28 9:46:10 139960325744384 [Warning] InnoDB: Dumping buffer pool(s) to .//ib_buffer_pool 2017-03-28 9:46:10 139960325744384 [Warning] InnoDB: Buffer pool(s) dump completed at 170328 9:46:10 170328 9:46:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.21-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=2 max_threads=1502 thread_count=3 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3430253 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x38f2d48 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4b929ec2c8 thread_stack 0x48400 /usr/local/mariadb/mariadb_base/bin/mysqld(my_print_stacktrace+0x24)[0xb506d4] /usr/local/mariadb/mariadb_base/bin/mysqld(handle_fatal_signal+0x4cc)[0x727a7c] /usr/local/mariadb/mariadb_base/lib64/libpthread.so.0(+0xf370)[0x7f4b926e9370] /usr/local/mariadb/mariadb_base/bin/mysqld[0x54c8ea] /usr/local/mariadb/mariadb_base/bin/mysqld[0x556762] /usr/local/mariadb/mariadb_base/bin/mysqld[0x541faa] /usr/local/mariadb/mariadb_base/bin/mysqld[0x542a2b] /usr/local/mariadb/mariadb_base/bin/mysqld(_Z16acl_authenticateP3THDj+0xbf)[0x550d2f] /usr/local/mariadb/mariadb_base/bin/mysqld(_Z16login_connectionP3THD+0x132)[0x668fc2] /usr/local/mariadb/mariadb_base/bin/mysqld(_Z22thd_prepare_connectionP3THD+0x11)[0x6694c1] /usr/local/mariadb/mariadb_base/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x11f)[0x669b9f] /usr/local/mariadb/mariadb_base/bin/mysqld(handle_one_connection+0x40)[0x669d90] /usr/local/mariadb/mariadb_base/lib64/libpthread.so.0(+0x7dc5)[0x7f4b926e1dc5] /usr/local/mariadb/mariadb_base/lib64/libc.so.6(clone+0x6d)[0x7f4b908f173d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 10317 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. We think the query pointer is invalid, but we will try to print it anyway. Query: 170328 09:46:15 mysqld_safe Number of processes running now: 0 170328 09:46:15 mysqld_safe WSREP: not restarting wsrep node automatically 170328 09:46:15 mysqld_safe mysqld from pid file /var/run/mysql.pid ended