Crash - null pointer exception

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: We intentionally generate a memory trap.

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: If you get repeated assertion failures or crashes, even

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: immediately after the mysqld startup, there may be

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

Sep 28 23:11:54 db-master-prod-b mysqld: InnoDB: about forcing recovery.

Sep 28 23:11:54 db-master-prod-b mysqld: 150928 23:11:54 [ERROR] mysqld got signal 6 ;

Sep 28 23:11:54 db-master-prod-b mysqld: This could be because you hit a bug. It is also possible that this binary

Sep 28 23:11:54 db-master-prod-b mysqld: or one of the libraries it was linked against is corrupt, improperly built,

Sep 28 23:11:54 db-master-prod-b mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: We will try our best to scrape up some info that will hopefully help

Sep 28 23:11:54 db-master-prod-b mysqld: diagnose the problem, but since we have already crashed, 

Sep 28 23:11:54 db-master-prod-b mysqld: something is definitely wrong and this may fail.

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: Server version: 10.0.21-MariaDB-1~wheezy-wsrep-log

Sep 28 23:11:54 db-master-prod-b mysqld: key_buffer_size=134217728

Sep 28 23:11:54 db-master-prod-b mysqld: read_buffer_size=2097152

Sep 28 23:11:54 db-master-prod-b mysqld: max_used_connections=30

Sep 28 23:11:54 db-master-prod-b mysqld: max_threads=102

Sep 28 23:11:54 db-master-prod-b mysqld: thread_count=19

Sep 28 23:11:54 db-master-prod-b mysqld: It is possible that mysqld could use up to 

Sep 28 23:11:54 db-master-prod-b mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2013173 K  bytes of memory

Sep 28 23:11:54 db-master-prod-b mysqld: Hope that's ok; if not, decrease some variables in the equation.

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: Thread pointer: 0x0x7f1eb4077008

Sep 28 23:11:54 db-master-prod-b mysqld: Attempting backtrace. You can use the following information to find out

Sep 28 23:11:54 db-master-prod-b mysqld: where mysqld died. If you see no messages after this, something went

Sep 28 23:11:54 db-master-prod-b mysqld: terribly wrong...

Sep 28 23:11:54 db-master-prod-b mysqld: stack_bottom = 0x7f21c4bfdd5f thread_stack 0x48000

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f21dbc6b44b]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x422)[0x7f21db7e77e2]

Sep 28 23:11:54 db-master-prod-b mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7f21dae080a0]

Sep 28 23:11:54 db-master-prod-b mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f21d9462165]

Sep 28 23:11:54 db-master-prod-b mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7f21d94653e0]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(+0x83cda5)[0x7f21dba73da5]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(+0x7473c2)[0x7f21db97e3c2]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(+0x5b2016)[0x7f21db7e9016]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x286)[0x7f21db7eac66]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x29)[0x7f21db74e139]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x425)[0x7f21db66fb35]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1e)[0x7f21db8e622e]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x8e)[0x7f21db8eb9ee]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x15d)[0x7f21db8ebedd]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x739)[0x7f21db8e95e9]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x626)[0x7f21db8ea8d6]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN14Event_job_data7executeEP3THDb+0x56f)[0x7f21db72aaaf]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(_ZN19Event_worker_thread3runEP3THDP28Event_queue_element_for_exec+0x119)[0x7f21db91bda9]

Sep 28 23:11:54 db-master-prod-b mysqld: /usr/sbin/mysqld(event_worker_thread+0x56)[0x7f21db91be46]

Sep 28 23:11:54 db-master-prod-b mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f21dadffb50]

Sep 28 23:11:54 db-master-prod-b mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21d950b95d]

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: Trying to get some variables.

Sep 28 23:11:54 db-master-prod-b mysqld: Some pointers may be invalid and cause the dump to abort.

Sep 28 23:11:54 db-master-prod-b mysqld: Query (0x7f1eb9121020): is an invalid pointer

Sep 28 23:11:54 db-master-prod-b mysqld: Connection ID (thread ID): 369097

Sep 28 23:11:54 db-master-prod-b mysqld: Status: NOT_KILLED

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

Sep 28 23:11:54 db-master-prod-b mysqld: 

Sep 28 23:11:54 db-master-prod-b mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

Sep 28 23:11:54 db-master-prod-b mysqld: information that should help you find out what is causing the crash.

Sep 28 23:11:58 db-master-prod-b mysqld_safe: Number of processes running now: 0

Sep 28 23:11:58 db-master-prod-b mysqld_safe: WSREP: not restarting wsrep node automatically

Sep 28 23:11:58 db-master-prod-b mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

Sep 28 23:12:50 db-master-prod-b collectd[3005]: mysql plugin: Lost connection to instance "tock_prod": MySQL server has gone away

Sep 28 23:12:50 db-master-prod-b collectd[3005]: mysql plugin: Failed to connect to database <none> at server localhost: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

Sep 28 23:12:50 db-master-prod-b collectd[3005]: read-function of plugin `mysql-tock_prod' failed. Will suspend it for 120.000 seconds.

Sep 28 23:14:50 db-master-prod-b collectd[3005]: mysql plugin: Failed to connect to database <none> at server localhost: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

Sep 28 23:14:50 db-master-prod-b collectd[3005]: read-function of plugin `mysql-tock_prod' failed. Will suspend it for 240.000 seconds.

Sep 28 23:17:01 db-master-prod-b /USR/SBIN/CRON[3693]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Sep 28 23:18:50 db-master-prod-b collectd[3005]: mysql plugin: Failed to connect to database <none> at server localhost: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

Sep 28 23:18:50 db-master-prod-b collectd[3005]: read-function of plugin `mysql-tock_prod' failed. Will suspend it for 480.000 seconds.

Sep 28 23:26:50 db-master-prod-b collectd[3005]: mysql plugin: Failed to connect to database <none> at server localhost: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

Sep 28 23:26:50 db-master-prod-b collectd[3005]: read-function of plugin `mysql-tock_prod' failed. Will suspend it for 960.000 seconds.

Sep 28 23:30:30 db-master-prod-b mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql

Sep 28 23:30:30 db-master-prod-b mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.TW6WRH' --pid-file='/var/lib/mysql/db-master-prod-b-recover.pid'

Sep 28 23:30:30 db-master-prod-b mysqld: 150928 23:30:30 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-1~wheezy-wsrep-log) starting as process 4333 ...

Sep 28 23:30:39 db-master-prod-b mysqld_safe: WSREP: Recovered position a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144078

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-1~wheezy-wsrep-log) starting as process 4514 ...

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Read nil XID from storage engines, skipping position init

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: wsrep_load(): Galera 3.9(rXXXX) by Codership Oy <info@codership.com> loaded successfully.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: CRC-32C: using hardware acceleration.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Found saved state: a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:-1

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Passing config to GCS: base_host = 10.240.115.41; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.

Sep 28 23:30:39 db-master-prod-b mysqld: npvo = false; pc.recove

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Service thread queue flushed.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Assign initial position for certification: 2144078, protocol version: -1

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: wsrep_sst_grab()

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Start replication

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Setting initial position to a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144078

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: protonet asio version 0

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Using CRC-32C for message checksums.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: backend: asio

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: restore pc from disk successfully

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: GMCast version 0

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') multicast: , ttl: 1

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: EVS version 0

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: gcomm: connecting to group 'tock_db_cluster_prod', peer 'db-master-prod-a:,db-master-prod-b:,db-master-prod-c:'

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Warning] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') address 'tcp://10.240.115.41:4567' points to own listening address, blacklisting

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') address 'tcp://10.240.115.41:4567' pointing to uuid 0b544c33 is blacklisted, skipping

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') address 'tcp://10.240.115.41:4567' pointing to uuid 0b544c33 is blacklisted, skipping

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') address 'tcp://10.240.115.41:4567' pointing to uuid 0b544c33 is blacklisted, skipping

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') address 'tcp://10.240.115.41:4567' pointing to uuid 0b544c33 is blacklisted, skipping

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: declaring 3e8a20de at tcp://10.240.12.54:4567 stable

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: declaring 7c11e4e6 at tcp://10.240.173.81:4567 stable

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Node 3e8a20de state prim

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: view(view_id(PRIM,0b544c33,111) memb {

Sep 28 23:30:39 db-master-prod-b mysqld: #0110b544c33,0

Sep 28 23:30:39 db-master-prod-b mysqld: #0113e8a20de,0

Sep 28 23:30:39 db-master-prod-b mysqld: #0117c11e4e6,0

Sep 28 23:30:39 db-master-prod-b mysqld: } joined {

Sep 28 23:30:39 db-master-prod-b mysqld: } left {

Sep 28 23:30:39 db-master-prod-b mysqld: } partitioned {

Sep 28 23:30:39 db-master-prod-b mysqld: })

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: save pc into disk

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: clear restored view

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: gcomm: connected

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Opened channel 'tock_db_cluster_prod'

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Waiting for SST to complete.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: STATE_EXCHANGE: sent state UUID: edc923c9-6638-11e5-b27c-2ff4c624316d

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: STATE EXCHANGE: sent state msg: edc923c9-6638-11e5-b27c-2ff4c624316d

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: STATE EXCHANGE: got state msg: edc923c9-6638-11e5-b27c-2ff4c624316d from 0 (db-master-prod-b)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: STATE EXCHANGE: got state msg: edc923c9-6638-11e5-b27c-2ff4c624316d from 1 (db-master-prod-c)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: STATE EXCHANGE: got state msg: edc923c9-6638-11e5-b27c-2ff4c624316d from 2 (db-master-prod-a)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Quorum results:

Sep 28 23:30:39 db-master-prod-b mysqld: #011version    = 3,

Sep 28 23:30:39 db-master-prod-b mysqld: #011component  = PRIMARY,

Sep 28 23:30:39 db-master-prod-b mysqld: #011conf_id    = 106,

Sep 28 23:30:39 db-master-prod-b mysqld: #011members    = 2/3 (joined/total),

Sep 28 23:30:39 db-master-prod-b mysqld: #011act_id     = 2144109,

Sep 28 23:30:39 db-master-prod-b mysqld: #011last_appl. = -1,

Sep 28 23:30:39 db-master-prod-b mysqld: #011protocols  = 0/7/3 (gcs/repl/appl),

Sep 28 23:30:39 db-master-prod-b mysqld: #011group UUID = a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Flow-control interval: [28, 28]

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2144109)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: State transfer required: 

Sep 28 23:30:39 db-master-prod-b mysqld: #011Group state: a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144109

Sep 28 23:30:39 db-master-prod-b mysqld: #011Local state: a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144078

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: New cluster view: global state: a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144109, view# 107: Primary, number of nodes: 3, my index: 0, protocol version 3

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Warning] WSREP: Gap in state sequence. Need state transfer.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.240.115.41' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '4514' --binlog '/var/log/mysql/mariadb-bin' '

Sep 28 23:30:39 db-master-prod-b rsyncd[4557]: rsyncd version 3.0.9 starting, listening on port 4444

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Prepared SST request: rsync|10.240.115.41:4444/rsync_sst

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: REPL Protocols: 7 (3, 2)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Service thread queue flushed.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Assign initial position for certification: 2144109, protocol version: 3

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Service thread queue flushed.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.240.115.41:4568

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Member 0.0 (db-master-prod-b) requested state transfer from '*any*'. Selected 1.0 (db-master-prod-c)(SYNCED) as donor.

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2144109)

Sep 28 23:30:39 db-master-prod-b mysqld: 150928 23:30:39 [Note] WSREP: Requesting state transfer: success, donor: 1

Sep 28 23:30:39 db-master-prod-b rsyncd[4573]: connect from db-master-prod-c.c.rapid-depot-817.internal (10.240.12.54)

Sep 28 23:30:39 db-master-prod-b rsyncd[4573]: rsync to rsync_sst/ from db-master-prod-c.c.rapid-depot-817.internal (10.240.12.54)

Sep 28 23:30:39 db-master-prod-b rsyncd[4573]: receiving file list

Sep 28 23:30:40 db-master-prod-b mysqld: 150928 23:30:40 [Note] WSREP: 1.0 (db-master-prod-c): State transfer to 0.0 (db-master-prod-b) complete.

Sep 28 23:30:40 db-master-prod-b mysqld: 150928 23:30:40 [Note] WSREP: Member 1.0 (db-master-prod-c) synced with group.

Sep 28 23:30:40 db-master-prod-b rsyncd[4573]: sent 54 bytes  received 180 bytes  total size 45

Sep 28 23:30:40 db-master-prod-b mysqld: WSREP_SST: [INFO] Joiner cleanup. (20150928 23:30:40.876)

Sep 28 23:30:41 db-master-prod-b rsyncd[4557]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]

Sep 28 23:30:41 db-master-prod-b mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20150928 23:30:41.381)

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] WSREP: SST complete, seqno: 2144078

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Using mutexes to ref count buffer pool pages

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: The InnoDB memory heap is disabled

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Memory barrier is not used

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Compressed tables use zlib 1.2.7

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Using Linux native AIO

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Using CPU crc32 instructions

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Initializing buffer pool, size = 10.0G

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Completed initialization of buffer pool

Sep 28 23:30:41 db-master-prod-b mysqld: 150928 23:30:41 [Note] InnoDB: Highest supported file format is Barracuda.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] InnoDB: 128 rollback segment(s) are active.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] InnoDB: Waiting for purge to start

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 3290481818

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] Plugin 'FEEDBACK' is disabled.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] Server socket created on IP: '0.0.0.0'.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [ERROR] mysqld: Table 'event' is marked as crashed and should be repaired

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Warning] Checking table:   './mysql/event'

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] Event Scheduler: Loaded 1 event

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: Signalling provider to continue.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: SST received: a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144078

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: Receiving IST: 31 writesets, seqnos 2144078-2144109

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] Event Scheduler: scheduler thread started with id 4

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] Reading of all Master_info entries succeded

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] Added new Master_info '' to hash table

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] /usr/sbin/mysqld: ready for connections.

Sep 28 23:30:42 db-master-prod-b mysqld: Version: '10.0.21-MariaDB-1~wheezy-wsrep-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution, wsrep_25.10.r4144

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: IST received: a54b1cb6-dd60-11e4-9e2f-fa9dc559c84a:2144109

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: 0.0 (db-master-prod-b): State transfer from 1.0 (db-master-prod-c) complete.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: Shifting JOINER -> JOINED (TO: 2144109)

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: Member 0.0 (db-master-prod-b) synced with group.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 2144109)

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: Synchronized with group, ready for connections

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Sep 28 23:30:42 db-master-prod-b mysqld: 150928 23:30:42 [Note] WSREP: (0b544c33, 'tcp://0.0.0.0:4567') turning message relay requesting off

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4808]: Upgrading MySQL tables if necessary.

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4812]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4812]: Looking for 'mysql' as: /usr/bin/mysql

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4812]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4812]: This installation of MySQL is already upgraded to 10.0.21-MariaDB, use --force if you still need to run mysql_upgrade

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4823]: Checking for insecure root accounts.

Sep 28 23:30:43 db-master-prod-b /etc/mysql/debian-start[4827]: Triggering myisam-recover for all MyISAM tables

Sep 28 23:30:52 db-master-prod-b mysqld: 150928 23:30:52 [Warning] IP address '10.52.2.3' could not be resolved: Name or service not known

Sep 28 23:30:54 db-master-prod-b mysqld: 150928 23:30:54 [Warning] IP address '10.68.2.3' could not be resolved: Name or service not known

Sep 28 23:30:56 db-master-prod-b mysqld: 150928 23:30:56 [Warning] IP address '10.68.0.3' could not be resolved: Name or service not known

Sep 28 23:30:57 db-master-prod-b mysqld: 150928 23:30:57 [Warning] IP address '10.68.1.3' could not be resolved: Name or service not known

Sep 28 23:31:25 db-master-prod-b mysqld: 150928 23:31:25 [Warning] IP address '10.52.0.3' could not be resolved: Name or service not known

Sep 28 23:31:28 db-master-prod-b mysqld: 150928 23:31:28 [Warning] IP address '10.52.1.3' could not be resolved: Name or service not known

Sep 28 23:42:50 db-master-prod-b collectd[3005]: mysql plugin: Successfully connected to database <none> at server Localhost via UNIX socket (server version: 10.0.21-MariaDB-1~wheezy-wsrep-log, protocol version: 10)