Oct 21 15:51:58 MEDX-C1 mysqld: 2016-10-21 15:51:58 7f3905a40b00 InnoDB: Assertion failure in thread 139882884500224 in file rem0rec.cc line 580 Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: We intentionally generate a memory trap. Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: If you get repeated assertion failures or crashes, even Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: immediately after the mysqld startup, there may be Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html Oct 21 15:51:58 MEDX-C1 mysqld: InnoDB: about forcing recovery. Oct 21 15:51:58 MEDX-C1 mysqld: 161021 15:51:58 [ERROR] mysqld got signal 6 ; Oct 21 15:51:58 MEDX-C1 mysqld: This could be because you hit a bug. It is also possible that this binary Oct 21 15:51:58 MEDX-C1 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Oct 21 15:51:58 MEDX-C1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Oct 21 15:51:58 MEDX-C1 mysqld: Oct 21 15:51:58 MEDX-C1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Oct 21 15:51:58 MEDX-C1 mysqld: Oct 21 15:51:58 MEDX-C1 mysqld: We will try our best to scrape up some info that will hopefully help Oct 21 15:51:58 MEDX-C1 mysqld: diagnose the problem, but since we have already crashed, Oct 21 15:51:58 MEDX-C1 mysqld: something is definitely wrong and this may fail. Oct 21 15:51:58 MEDX-C1 mysqld: Oct 21 15:51:58 MEDX-C1 mysqld: Server version: 10.1.14-MariaDB-1~trusty Oct 21 15:51:58 MEDX-C1 mysqld: key_buffer_size=134217728 Oct 21 15:51:58 MEDX-C1 mysqld: read_buffer_size=2097152 Oct 21 15:51:58 MEDX-C1 mysqld: max_used_connections=393 Oct 21 15:51:58 MEDX-C1 mysqld: max_threads=512 Oct 21 15:51:58 MEDX-C1 mysqld: thread_count=149 Oct 21 15:51:58 MEDX-C1 mysqld: It is possible that mysqld could use up to Oct 21 15:51:58 MEDX-C1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9578723 K bytes of memory Oct 21 15:51:58 MEDX-C1 mysqld: Hope that's ok; if not, decrease some variables in the equation. Oct 21 15:51:58 MEDX-C1 mysqld: Oct 21 15:51:58 MEDX-C1 mysqld: Thread pointer: 0x0x7f2f57e8c008 Oct 21 15:51:58 MEDX-C1 mysqld: Attempting backtrace. You can use the following information to find out Oct 21 15:51:58 MEDX-C1 mysqld: where mysqld died. If you see no messages after this, something went Oct 21 15:51:58 MEDX-C1 mysqld: terribly wrong... Oct 21 15:51:58 MEDX-C1 mysqld: stack_bottom = 0x7f3905a401f0 thread_stack 0x48400 Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f394f333b5e] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7f394ee5fabd] Oct 21 15:51:58 MEDX-C1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f394d3b2340] Oct 21 15:51:58 MEDX-C1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f394c805cc9] Oct 21 15:51:58 MEDX-C1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f394c8090d8] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x88495b)[0x7f394f13395b] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x886f6f)[0x7f394f135f6f] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x7f224e)[0x7f394f0a124e] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x8959f9)[0x7f394f1449f9] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x8980a4)[0x7f394f1470a4] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x8c45c8)[0x7f394f1735c8] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x8c8d5a)[0x7f394f177d5a] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x8c9647)[0x7f394f178647] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x8a9675)[0x7f394f158675] Oct 21 15:51:58 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x7f42d7)[0x7f394f0a32d7] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x2df)[0x7f394ee6a0ef] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyyP13select_result+0x1292)[0x7f394ef85a32] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3dcd)[0x7f394ece162d] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b9)[0x7f394ecfa4e9] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(+0x44b633)[0x7f394ecfa633] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x17e)[0x7f394ecfab2e] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xfbf)[0x7f394ece8f2f] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7f394eceab89] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f394edae24a] Oct 21 15:51:59 MEDX-C1 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f394edae420] Oct 21 15:51:59 MEDX-C1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f394d3aa182] Oct 21 15:51:59 MEDX-C1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f394c8c947d] Oct 21 15:51:59 MEDX-C1 mysqld: Oct 21 15:51:59 MEDX-C1 mysqld: Trying to get some variables. Oct 21 15:51:59 MEDX-C1 mysqld: Some pointers may be invalid and cause the dump to abort. Oct 21 15:51:59 MEDX-C1 mysqld: Query (0x7f2f3ad4e020): is an invalid pointer Oct 21 15:51:59 MEDX-C1 mysqld: Connection ID (thread ID): 3856570 Oct 21 15:51:59 MEDX-C1 mysqld: Status: NOT_KILLED Oct 21 15:51:59 MEDX-C1 mysqld: Oct 21 15:51:59 MEDX-C1 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on Oct 21 15:51:59 MEDX-C1 mysqld: Oct 21 15:51:59 MEDX-C1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Oct 21 15:51:59 MEDX-C1 mysqld: information that should help you find out what is causing the crash. Oct 21 17:26:41 MEDX-C1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql Oct 21 17:26:41 MEDX-C1 mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.fQbvC2' --pid-file='/var/lib/mysql/MEDX-C1-recover.pid' Oct 21 17:26:41 MEDX-C1 mysqld: 2016-10-21 17:26:41 140282973906880 [Note] /usr/sbin/mysqld (mysqld 10.1.14-MariaDB-1~trusty) starting as process 28581 ... Oct 21 17:26:53 MEDX-C1 mysqld_safe: WSREP: Recovered position 2890059e-2cc6-11e6-b4ce-7e25702c017a:80630955 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] /usr/sbin/mysqld (mysqld 10.1.14-MariaDB-1~trusty) starting as process 28618 ... Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Read nil XID from storage engines, skipping position init Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: wsrep_load(): Galera 25.3.15(r3578) by Codership Oy loaded successfully. Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: CRC-32C: using hardware acceleration. Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Found saved state: 2890059e-2cc6-11e6-b4ce-7e25702c017a:-1 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.3.20.11; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/log/mariadb_bin; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/log/mariadb_bin/galera.cache; gcache.page_size = 1G; gcache.size = 1G; gcs.fc_debug = 0; gcs.fc_factor = 0.75; gcs.fc_limit = 32; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; Oct 21 17:26:53 MEDX-C1 mysqld: pc.ignore_quorum = false; pc.ignore_sb = f Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139953786509056 [Note] WSREP: Service thread queue flushed. Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Assign initial position for certification: 80630955, protocol version: -1 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: wsrep_sst_grab() Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Start replication Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Setting initial position to 2890059e-2cc6-11e6-b4ce-7e25702c017a:80630955 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: protonet asio version 0 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: Using CRC-32C for message checksums. Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: backend: asio Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: restore pc from disk successfully Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: GMCast version 0 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: (9a7f755a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: (9a7f755a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: EVS version 0 Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: gcomm: connecting to group 'TestSystem', peer 'MEDX-C3:,MEDX-C1:,MEDX-C2:' Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Warning] WSREP: (9a7f755a, 'tcp://0.0.0.0:4567') address 'tcp://10.3.20.11:4567' points to own listening address, blacklisting Oct 21 17:26:53 MEDX-C1 mysqld: 2016-10-21 17:26:53 139954954373056 [Note] WSREP: (9a7f755a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: declaring a96d0388 at tcp://10.3.20.13:4567 stable Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: declaring b60dd028 at tcp://10.3.20.12:4567 stable Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: Node a96d0388 state prim Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: view(view_id(PRIM,9a7f755a,5) memb { Oct 21 17:26:54 MEDX-C1 mysqld: #0119a7f755a,0 Oct 21 17:26:54 MEDX-C1 mysqld: #011a96d0388,0 Oct 21 17:26:54 MEDX-C1 mysqld: #011b60dd028,0 Oct 21 17:26:54 MEDX-C1 mysqld: } joined { Oct 21 17:26:54 MEDX-C1 mysqld: } left { Oct 21 17:26:54 MEDX-C1 mysqld: } partitioned { Oct 21 17:26:54 MEDX-C1 mysqld: }) Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: save pc into disk Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: clear restored view Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: gcomm: connected Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: Opened channel 'TestSystem' Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954373056 [Note] WSREP: Waiting for SST to complete. Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 8f918060-97b3-11e6-b45d-9b08192121e3 Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: STATE EXCHANGE: sent state msg: 8f918060-97b3-11e6-b45d-9b08192121e3 Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: STATE EXCHANGE: got state msg: 8f918060-97b3-11e6-b45d-9b08192121e3 from 0 (MEDX-C1) Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: STATE EXCHANGE: got state msg: 8f918060-97b3-11e6-b45d-9b08192121e3 from 1 (MEDX-C3) Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: STATE EXCHANGE: got state msg: 8f918060-97b3-11e6-b45d-9b08192121e3 from 2 (MEDX-C2) Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: Quorum results: Oct 21 17:26:54 MEDX-C1 mysqld: #011version = 3, Oct 21 17:26:54 MEDX-C1 mysqld: #011component = PRIMARY, Oct 21 17:26:54 MEDX-C1 mysqld: #011conf_id = 4, Oct 21 17:26:54 MEDX-C1 mysqld: #011members = 2/3 (joined/total), Oct 21 17:26:54 MEDX-C1 mysqld: #011act_id = 80682484, Oct 21 17:26:54 MEDX-C1 mysqld: #011last_appl. = -1, Oct 21 17:26:54 MEDX-C1 mysqld: #011protocols = 0/7/3 (gcs/repl/appl), Oct 21 17:26:54 MEDX-C1 mysqld: #011group UUID = 2890059e-2cc6-11e6-b4ce-7e25702c017a Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: Flow-control interval: [41, 55] Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953723926272 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 80682484) Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954058496 [Note] WSREP: State transfer required: Oct 21 17:26:54 MEDX-C1 mysqld: #011Group state: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80682484 Oct 21 17:26:54 MEDX-C1 mysqld: #011Local state: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80630955 Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954058496 [Note] WSREP: New cluster view: global state: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80682484, view# 5: Primary, number of nodes: 3, my index: 0, protocol version 3 Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139954954058496 [Warning] WSREP: Gap in state sequence. Need state transfer. Oct 21 17:26:54 MEDX-C1 mysqld: 2016-10-21 17:26:54 139953694570240 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.3.20.11' --datadir '/var/lib/mysql/' --parent '28618' --binlog '/var/log/mariadb_bin/mariadb-bin' ' Oct 21 17:26:54 MEDX-C1 mysqld: WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20161021 17:26:54.704) Oct 21 17:26:54 MEDX-C1 -wsrep-sst-joiner: Streaming with xbstream Oct 21 17:26:54 MEDX-C1 -wsrep-sst-joiner: Using socat as streamer Oct 21 17:26:54 MEDX-C1 -wsrep-sst-joiner: Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954058496 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.3.20.11:4444/xtrabackup_sst//1 Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954058496 [Note] WSREP: REPL Protocols: 7 (3, 2) Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139953786509056 [Note] WSREP: Service thread queue flushed. Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954058496 [Note] WSREP: Assign initial position for certification: 80682484, protocol version: 3 Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139953786509056 [Note] WSREP: Service thread queue flushed. Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954058496 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.3.20.11:4568 Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139953723926272 [Note] WSREP: Member 0.0 (MEDX-C1) requested state transfer from 'MEDX-C2'. Selected 2.0 (MEDX-C2)(SYNCED) as donor. Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139953723926272 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 80682497) Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954058496 [Note] WSREP: Requesting state transfer: success, donor: 2 Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139953723926272 [Note] WSREP: 2.0 (MEDX-C2): State transfer to 0.0 (MEDX-C1) complete. Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139953723926272 [Note] WSREP: Member 2.0 (MEDX-C2) synced with group. Oct 21 17:26:55 MEDX-C1 -wsrep-sst-joiner: xtrabackup_ist received from donor: Running IST Oct 21 17:26:55 MEDX-C1 -wsrep-sst-joiner: Galera co-ords from recovery: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80630955 0 Oct 21 17:26:55 MEDX-C1 -wsrep-sst-joiner: Total time on joiner: 0 seconds Oct 21 17:26:55 MEDX-C1 -wsrep-sst-joiner: Removing the sst_in_progress file Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] WSREP: SST complete, seqno: 80630955 Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Using mutexes to ref count buffer pool pages Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: The InnoDB memory heap is disabled Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Memory barrier is not used Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Compressed tables use zlib 1.2.8 Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Using Linux native AIO Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Using SSE crc32 instructions Oct 21 17:26:55 MEDX-C1 mysqld: 2016-10-21 17:26:55 139954954373056 [Note] InnoDB: Initializing buffer pool, size = 35.2G Oct 21 17:26:56 MEDX-C1 mysqld: 2016-10-21 17:26:56 139954954373056 [Note] InnoDB: Completed initialization of buffer pool Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] InnoDB: Highest supported file format is Barracuda. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953732318976 [Note] WSREP: (9a7f755a, 'tcp://0.0.0.0:4567') turning message relay requesting off Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] InnoDB: 128 rollback segment(s) are active. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] InnoDB: Waiting for purge to start Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 589275158653 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139911764084480 [Note] InnoDB: Dumping buffer pool(s) not yet started Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] Plugin 'FEEDBACK' is disabled. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] Server socket created on IP: '0.0.0.0'. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] WSREP: Signalling provider to continue. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] WSREP: SST received: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80630955 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: Receiving IST: 51529 writesets, seqnos 80630955-80682484 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] Reading of all Master_info entries succeded Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] Added new Master_info '' to hash table Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954373056 [Note] /usr/sbin/mysqld: ready for connections. Oct 21 17:26:57 MEDX-C1 mysqld: Version: '10.1.14-MariaDB-1~trusty' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table mediator.HIST_SEQUENCE; Duplicate entry '3898456' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 2047, Internal MariaDB error code: 1062 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: RBR event 2 Write_rows_v1 apply warning: 121, 80630956 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: Failed to apply app buffer: seqno: 80630956, status: 1 Oct 21 17:26:57 MEDX-C1 mysqld: #011 at galera/src/trx_handle.cpp:apply():351 Oct 21 17:26:57 MEDX-C1 mysqld: Retrying 2th time Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table mediator.HIST_SEQUENCE; Duplicate entry '3898456' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 2047, Internal MariaDB error code: 1062 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: RBR event 2 Write_rows_v1 apply warning: 121, 80630956 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: Failed to apply app buffer: seqno: 80630956, status: 1 Oct 21 17:26:57 MEDX-C1 mysqld: #011 at galera/src/trx_handle.cpp:apply():351 Oct 21 17:26:57 MEDX-C1 mysqld: Retrying 3th time Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table mediator.HIST_SEQUENCE; Duplicate entry '3898456' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 2047, Internal MariaDB error code: 1062 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: RBR event 2 Write_rows_v1 apply warning: 121, 80630956 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: Failed to apply app buffer: seqno: 80630956, status: 1 Oct 21 17:26:57 MEDX-C1 mysqld: #011 at galera/src/trx_handle.cpp:apply():351 Oct 21 17:26:57 MEDX-C1 mysqld: Retrying 4th time Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table mediator.HIST_SEQUENCE; Duplicate entry '3898456' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 2047, Internal MariaDB error code: 1062 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Warning] WSREP: RBR event 2 Write_rows_v1 apply warning: 121, 80630956 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [ERROR] WSREP: receiving IST failed, node restart required: Failed to apply trx 80630956 4 times Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [ERROR] WSREP: failed trx: source: 9a7f755a-8bde-11e6-b07f-5a9103c7c761 version: 3 local: 0 state: APPLYING flags: 257 conn_id: 3855327 trx_id: 805050408 seqnos (l: -1, g: 80630956, s: 80630955, d: 80630952, ts: 1478679527680910) Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: Closing send monitor... Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: Closed send monitor. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: gcomm: terminating thread Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: gcomm: joining thread Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: gcomm: closing backend Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: view(view_id(NON_PRIM,9a7f755a,5) memb { Oct 21 17:26:57 MEDX-C1 mysqld: #0119a7f755a,0 Oct 21 17:26:57 MEDX-C1 mysqld: } joined { Oct 21 17:26:57 MEDX-C1 mysqld: } left { Oct 21 17:26:57 MEDX-C1 mysqld: } partitioned { Oct 21 17:26:57 MEDX-C1 mysqld: #011a96d0388,0 Oct 21 17:26:57 MEDX-C1 mysqld: #011b60dd028,0 Oct 21 17:26:57 MEDX-C1 mysqld: }) Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: view((empty)) Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: gcomm: closed Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Flow-control interval: [24, 32] Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Received NON-PRIMARY. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Shifting JOINER -> OPEN (TO: 80682525) Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Received self-leave message. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Flow-control interval: [0, 0] Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Received SELF-LEAVE. Closing connection. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 80682525) Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139953723926272 [Note] WSREP: RECV thread exiting 0: Success Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: recv_thread() joined. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: Closing replication queue. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: Closing slave action queue. Oct 21 17:26:57 MEDX-C1 mysqld: 2016-10-21 17:26:57 139954954058496 [Note] WSREP: /usr/sbin/mysqld: Terminated. Oct 21 17:32:43 MEDX-C1 mysqld_safe: Number of processes running now: 0 Oct 21 17:32:43 MEDX-C1 mysqld_safe: WSREP: not restarting wsrep node automatically Oct 21 17:32:43 MEDX-C1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended Oct 21 17:40:16 MEDX-C1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql Oct 21 17:40:16 MEDX-C1 mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.f3KNyn' --pid-file='/var/lib/mysql/MEDX-C1-recover.pid' Oct 21 17:40:16 MEDX-C1 mysqld: 2016-10-21 17:40:16 139631584999360 [Note] /usr/sbin/mysqld (mysqld 10.1.14-MariaDB-1~trusty) starting as process 30692 ... Oct 21 17:40:24 MEDX-C1 mysqld_safe: WSREP: Recovered position 2890059e-2cc6-11e6-b4ce-7e25702c017a:80630955 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] /usr/sbin/mysqld (mysqld 10.1.14-MariaDB-1~trusty) starting as process 30727 ... Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Read nil XID from storage engines, skipping position init Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: wsrep_load(): Galera 25.3.15(r3578) by Codership Oy loaded successfully. Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: CRC-32C: using hardware acceleration. Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.3.20.11; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/log/mariadb_bin; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/log/mariadb_bin/galera.cache; gcache.page_size = 1G; gcache.size = 1G; gcs.fc_debug = 0; gcs.fc_factor = 0.75; gcs.fc_limit = 32; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; Oct 21 17:40:24 MEDX-C1 mysqld: pc.ignore_quorum = false; pc.ignore_sb = f Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322797180672 [Note] WSREP: Service thread queue flushed. Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: wsrep_sst_grab() Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Start replication Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: protonet asio version 0 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Using CRC-32C for message checksums. Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: backend: asio Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: restore pc from disk failed Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: GMCast version 0 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: (723d4ad6, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: (723d4ad6, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: EVS version 0 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: gcomm: connecting to group 'TestSystem', peer 'MEDX-C3:,MEDX-C1:,MEDX-C2:' Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Warning] WSREP: (723d4ad6, 'tcp://0.0.0.0:4567') address 'tcp://10.3.20.11:4567' points to own listening address, blacklisting Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: (723d4ad6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: declaring a96d0388 at tcp://10.3.20.13:4567 stable Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: declaring b60dd028 at tcp://10.3.20.12:4567 stable Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Node a96d0388 state prim Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: view(view_id(PRIM,723d4ad6,7) memb { Oct 21 17:40:24 MEDX-C1 mysqld: #011723d4ad6,0 Oct 21 17:40:24 MEDX-C1 mysqld: #011a96d0388,0 Oct 21 17:40:24 MEDX-C1 mysqld: #011b60dd028,0 Oct 21 17:40:24 MEDX-C1 mysqld: } joined { Oct 21 17:40:24 MEDX-C1 mysqld: } left { Oct 21 17:40:24 MEDX-C1 mysqld: } partitioned { Oct 21 17:40:24 MEDX-C1 mysqld: }) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: save pc into disk Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: gcomm: connected Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Opened channel 'TestSystem' Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964655552 [Note] WSREP: Waiting for SST to complete. Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 728a33e2-97b5-11e6-9e60-7a7c8b6dc3fd Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: STATE EXCHANGE: sent state msg: 728a33e2-97b5-11e6-9e60-7a7c8b6dc3fd Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: STATE EXCHANGE: got state msg: 728a33e2-97b5-11e6-9e60-7a7c8b6dc3fd from 1 (MEDX-C3) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: STATE EXCHANGE: got state msg: 728a33e2-97b5-11e6-9e60-7a7c8b6dc3fd from 2 (MEDX-C2) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: STATE EXCHANGE: got state msg: 728a33e2-97b5-11e6-9e60-7a7c8b6dc3fd from 0 (MEDX-C1) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: Quorum results: Oct 21 17:40:24 MEDX-C1 mysqld: #011version = 3, Oct 21 17:40:24 MEDX-C1 mysqld: #011component = PRIMARY, Oct 21 17:40:24 MEDX-C1 mysqld: #011conf_id = 6, Oct 21 17:40:24 MEDX-C1 mysqld: #011members = 2/3 (joined/total), Oct 21 17:40:24 MEDX-C1 mysqld: #011act_id = 80685578, Oct 21 17:40:24 MEDX-C1 mysqld: #011last_appl. = -1, Oct 21 17:40:24 MEDX-C1 mysqld: #011protocols = 0/7/3 (gcs/repl/appl), Oct 21 17:40:24 MEDX-C1 mysqld: #011group UUID = 2890059e-2cc6-11e6-b4ce-7e25702c017a Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: Flow-control interval: [41, 55] Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322730403584 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 80685578) Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964340992 [Note] WSREP: State transfer required: Oct 21 17:40:24 MEDX-C1 mysqld: #011Group state: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80685578 Oct 21 17:40:24 MEDX-C1 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964340992 [Note] WSREP: New cluster view: global state: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80685578, view# 7: Primary, number of nodes: 3, my index: 0, protocol version 3 Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140323964340992 [Warning] WSREP: Gap in state sequence. Need state transfer. Oct 21 17:40:24 MEDX-C1 mysqld: 2016-10-21 17:40:24 140322701047552 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.3.20.11' --datadir '/var/lib/mysql/' --parent '30727' --binlog '/var/log/mariadb_bin/mariadb-bin' ' Oct 21 17:40:24 MEDX-C1 mysqld: WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20161021 17:40:24.962) Oct 21 17:40:24 MEDX-C1 -wsrep-sst-joiner: Streaming with xbstream Oct 21 17:40:24 MEDX-C1 -wsrep-sst-joiner: Using socat as streamer Oct 21 17:40:25 MEDX-C1 -wsrep-sst-joiner: Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140323964340992 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.3.20.11:4444/xtrabackup_sst//1 Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140323964340992 [Note] WSREP: REPL Protocols: 7 (3, 2) Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140322797180672 [Note] WSREP: Service thread queue flushed. Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140323964340992 [Note] WSREP: Assign initial position for certification: 80685578, protocol version: 3 Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140322797180672 [Note] WSREP: Service thread queue flushed. Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140323964340992 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (2890059e-2cc6-11e6-b4ce-7e25702c017a): 1 (Operation not permitted) Oct 21 17:40:25 MEDX-C1 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable. Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140322730403584 [Note] WSREP: Member 0.0 (MEDX-C1) requested state transfer from 'MEDX-C2'. Selected 2.0 (MEDX-C2)(SYNCED) as donor. Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140322730403584 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 80685579) Oct 21 17:40:25 MEDX-C1 mysqld: 2016-10-21 17:40:25 140323964340992 [Note] WSREP: Requesting state transfer: success, donor: 2 Oct 21 17:40:25 MEDX-C1 -wsrep-sst-joiner: Proceeding with SST Oct 21 17:40:25 MEDX-C1 -wsrep-sst-joiner: Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) Oct 21 17:40:25 MEDX-C1 -wsrep-sst-joiner: Cleaning the existing datadir and innodb-data/log directories Oct 21 17:40:26 MEDX-C1 -wsrep-sst-joiner: removed ‘/var/lib/mysql/lost@002bfound/db.opt’ ... REMOVE LOTS FROM DATA DIR (PRESUMABLY EVERYTHING) Oct 21 17:40:26 MEDX-C1 -wsrep-sst-joiner: removed ‘/var/lib/mysql/ib_logfile0’ Oct 21 17:40:26 MEDX-C1 -wsrep-sst-joiner: Cleaning the binlog directory /var/log/mariadb_bin as well Oct 21 17:40:27 MEDX-C1 mysqld: 2016-10-21 17:40:27 140322742990592 [Note] WSREP: (723d4ad6, 'tcp://0.0.0.0:4567') turning message relay requesting off Oct 21 17:40:28 MEDX-C1 -wsrep-sst-joiner: removed ‘/var/log/mariadb_bin/mariadb-bin.000259’ ... REMOVE LOTS OF BINARY LOGS Oct 21 17:40:32 MEDX-C1 -wsrep-sst-joiner: removed ‘/var/log/mariadb_bin/mariadb-bin.000239’ Oct 21 17:40:32 MEDX-C1 -wsrep-sst-joiner: Waiting for SST streaming to complete! Oct 21 17:43:43 MEDX-C1 mysqld: 2016-10-21 17:43:43 140322730403584 [Note] WSREP: 2.0 (MEDX-C2): State transfer to 0.0 (MEDX-C1) complete. Oct 21 17:43:43 MEDX-C1 mysqld: 2016-10-21 17:43:43 140322730403584 [Note] WSREP: Member 2.0 (MEDX-C2) synced with group. Oct 21 17:43:43 MEDX-C1 -wsrep-sst-joiner: ls: cannot access mariadb-bin.[0-9]*: No such file or directory Oct 21 17:43:43 MEDX-C1 -wsrep-sst-joiner: Preparing the backup at /var/lib/mysql//.sst Oct 21 17:43:43 MEDX-C1 -wsrep-sst-joiner: Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} 2>&1 | logger -p daemon.err -t -innobackupex-apply Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: 161021 17:43:43 innobackupex: Starting the apply-log operation Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: IMPORTANT: Please check that the apply-log run completes successfully. Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: At the end of a successful apply-log run innobackupex Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: prints "completed OK!". Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: innobackupex version 2.4.3 based on MySQL server 5.7.11 Linux (x86_64) (revision id: 6a46905) Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: cd to /var/lib/mysql/.sst Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: This target seems to be not prepared yet. Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Number of pools: 1 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: xtrabackup_logfile detected: size=43515904, start_lsn=(587994882209) Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: using the following InnoDB configuration for recovery: Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_data_home_dir = . Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_group_home_dir = . Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_files_in_group = 1 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_file_size = 43515904 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: using the following InnoDB configuration for recovery: Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_data_home_dir = . Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_group_home_dir = . Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_files_in_group = 1 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_file_size = 43515904 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: Starting InnoDB instance for recovery. Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: PUNCH HOLE support available Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Mutexes and rw_locks use GCC atomic builtins Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Uses event mutexes Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Compressed tables use zlib 1.2.8 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Number of pools: 1 Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Using CPU crc32 instructions Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Completed initialization of buffer pool Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Oct 21 17:43:43 MEDX-C1 -innobackupex-apply: InnoDB: Highest supported file format is Barracuda. Oct 21 17:43:44 MEDX-C1 -innobackupex-apply: InnoDB: Log scan progressed past the checkpoint lsn 587994882209 Oct 21 17:43:44 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588000124928 (13%) Oct 21 17:43:44 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588005367808 (27%) Oct 21 17:43:44 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588010610688 (40%) Oct 21 17:43:44 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588015853568 (54%) Oct 21 17:43:45 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588021096448 (67%) Oct 21 17:43:45 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588026339328 (81%) Oct 21 17:43:45 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588031582208 (94%) Oct 21 17:43:45 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588033579829 (100%) Oct 21 17:43:45 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 587998093312 (8%) Oct 21 17:43:46 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588003336192 (21%) Oct 21 17:43:46 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588008579072 (35%) Oct 21 17:43:46 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588013821952 (48%) Oct 21 17:43:46 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588019064832 (62%) Oct 21 17:43:46 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588024307712 (76%) Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588029550592 (89%) Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588033579829 (100%) Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Database was not shutdown normally! Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Starting crash recovery. Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 587996061696 (3%) Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588001304576 (16%) Oct 21 17:43:47 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588006547456 (30%) Oct 21 17:43:48 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588011790336 (43%) Oct 21 17:43:48 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588017033216 (57%) Oct 21 17:43:48 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588022276096 (70%) Oct 21 17:43:48 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588027518976 (84%) Oct 21 17:43:48 MEDX-C1 -innobackupex-apply: InnoDB: Starting an apply batch of log records to the database... Oct 21 17:43:55 MEDX-C1 -innobackupex-apply: InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Oct 21 17:43:55 MEDX-C1 -innobackupex-apply: InnoDB: Apply batch completed Oct 21 17:43:55 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588032761856 (97%) Oct 21 17:43:55 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588033579829 (100%) Oct 21 17:43:55 MEDX-C1 -innobackupex-apply: InnoDB: Starting an apply batch of log records to the database... Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: Apply batch completed Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: xtrabackup: Last MySQL binlog file position 4420093, file name /var/log/mysql/mariadb-bin.000742 Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: Creating shared tablespace for temporary tables Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: File './ibtmp1' size is now 12 MB. Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active. Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: 32 non-redo rollback segment(s) are active. Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: 5.7.11 started; log sequence number 588033579829 Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: page_cleaner: 1000ms intended loop took 12962ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.) Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: xtrabackup: Last MySQL binlog file position 4420093, file name /var/log/mysql/mariadb-bin.000742 Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: xtrabackup: starting shutdown with innodb_fast_shutdown = 1 Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: FTS optimize thread exiting. Oct 21 17:43:56 MEDX-C1 -innobackupex-apply: InnoDB: Starting shutdown... Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Shutdown completed; log sequence number 588033580419 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Number of pools: 1 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: using the following InnoDB configuration for recovery: Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_data_home_dir = . Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_group_home_dir = . Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_files_in_group = 2 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: innodb_log_file_size = 104857600 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: PUNCH HOLE support available Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Mutexes and rw_locks use GCC atomic builtins Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Uses event mutexes Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Compressed tables use zlib 1.2.8 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Number of pools: 1 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Using CPU crc32 instructions Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Completed initialization of buffer pool Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Setting log file ./ib_logfile101 size to 100 MB Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Progress in MB: Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: 100 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Setting log file ./ib_logfile1 size to 100 MB Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Progress in MB: Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: 100 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: New log files created, LSN=588033580419 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Highest supported file format is Barracuda. Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Log scan progressed past the checkpoint lsn 588033580556 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588033580565 (0%) Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Doing recovery: scanned up to log sequence number 588033580565 (0%) Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Database was not shutdown normally! Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Starting crash recovery. Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: xtrabackup: Last MySQL binlog file position 4420093, file name /var/log/mysql/mariadb-bin.000742 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Removed temporary tablespace data file: "ibtmp1" Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Creating shared tablespace for temporary tables Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: File './ibtmp1' size is now 12 MB. Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active. Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: 32 non-redo rollback segment(s) are active. Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: 5.7.11 started; log sequence number 588033580565 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: xtrabackup: starting shutdown with innodb_fast_shutdown = 1 Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: FTS optimize thread exiting. Oct 21 17:43:58 MEDX-C1 -innobackupex-apply: InnoDB: Starting shutdown... Oct 21 17:44:00 MEDX-C1 -innobackupex-apply: InnoDB: Shutdown completed; log sequence number 588033581269 Oct 21 17:44:00 MEDX-C1 -innobackupex-apply: 161021 17:44:00 completed OK! Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: rm: cannot remove ‘/var/lib/mysql//innobackup.prepare.log’: No such file or directory Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: rm: cannot remove ‘/var/lib/mysql//innobackup.move.log’: No such file or directory Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: Moving the backup to /var/lib/mysql/ Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: Evaluating innobackupex --no-version-check --move-back --force-non-empty-directories ${DATA} 2>&1 | logger -p daemon.err -t -innobackupex-move Oct 21 17:44:00 MEDX-C1 -innobackupex-move: 161021 17:44:00 innobackupex: Starting the move-back operation Oct 21 17:44:00 MEDX-C1 -innobackupex-move: Oct 21 17:44:00 MEDX-C1 -innobackupex-move: IMPORTANT: Please check that the move-back run completes successfully. Oct 21 17:44:00 MEDX-C1 -innobackupex-move: At the end of a successful move-back run innobackupex Oct 21 17:44:00 MEDX-C1 -innobackupex-move: prints "completed OK!". Oct 21 17:44:00 MEDX-C1 -innobackupex-move: Oct 21 17:44:00 MEDX-C1 -innobackupex-move: innobackupex version 2.4.3 based on MySQL server 5.7.11 Linux (x86_64) (revision id: 6a46905) Oct 21 17:44:00 MEDX-C1 -innobackupex-move: 161021 17:44:00 [01] Moving ib_logfile0 to /var/lib/mysql/ib_logfile0 Oct 21 17:44:00 MEDX-C1 -innobackupex-move: 161021 17:44:00 [01] ...done ......... MOVE LOTS OF FILES FOR ALL DATABASES Oct 21 17:44:00 MEDX-C1 -innobackupex-move: 161021 17:44:00 [01] Moving ./performance_schema/db.opt to /var/lib/mysql/performance_schema/db.opt Oct 21 17:44:00 MEDX-C1 -innobackupex-move: 161021 17:44:00 [01] ...done Oct 21 17:44:00 MEDX-C1 -innobackupex-move: 161021 17:44:00 completed OK! Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: Move successful, removing /var/lib/mysql//.sst Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: Galera co-ords from recovery: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80686824 Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: Total time on joiner: 0 seconds Oct 21 17:44:00 MEDX-C1 -wsrep-sst-joiner: Removing the sst_in_progress file Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] WSREP: SST complete, seqno: 80686824 Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Using mutexes to ref count buffer pool pages Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: The InnoDB memory heap is disabled Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Memory barrier is not used Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Compressed tables use zlib 1.2.8 Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Using Linux native AIO Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Using SSE crc32 instructions Oct 21 17:44:00 MEDX-C1 mysqld: 2016-10-21 17:44:00 140323964655552 [Note] InnoDB: Initializing buffer pool, size = 35.2G Oct 21 17:44:01 MEDX-C1 mysqld: 2016-10-21 17:44:01 140323964655552 [Note] InnoDB: Completed initialization of buffer pool Oct 21 17:44:02 MEDX-C1 mysqld: 2016-10-21 17:44:02 140323964655552 [Note] InnoDB: Highest supported file format is Barracuda. Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] InnoDB: 128 rollback segment(s) are active. Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] InnoDB: Waiting for purge to start Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 588033581269 Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140280787339008 [Note] InnoDB: Dumping buffer pool(s) not yet started Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] Plugin 'FEEDBACK' is disabled. Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] Server socket created on IP: '0.0.0.0'. Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] WSREP: Signalling provider to continue. Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] WSREP: SST received: 2890059e-2cc6-11e6-b4ce-7e25702c017a:80686824 Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140322730403584 [Note] WSREP: 0.0 (MEDX-C1): State transfer from 2.0 (MEDX-C2) complete. Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140322730403584 [Note] WSREP: Shifting JOINER -> JOINED (TO: 80686876) Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] Reading of all Master_info entries succeded Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] Added new Master_info '' to hash table Oct 21 17:44:03 MEDX-C1 mysqld: 2016-10-21 17:44:03 140323964655552 [Note] /usr/sbin/mysqld: ready for connections. Oct 21 17:44:03 MEDX-C1 mysqld: Version: '10.1.14-MariaDB-1~trusty' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution Oct 21 17:44:04 MEDX-C1 mysqld: 2016-10-21 17:44:04 140322730403584 [Note] WSREP: Member 0.0 (MEDX-C1) synced with group. Oct 21 17:44:04 MEDX-C1 mysqld: 2016-10-21 17:44:04 140322730403584 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 80686876) Oct 21 17:44:04 MEDX-C1 mysqld: 2016-10-21 17:44:04 140323964340992 [Note] WSREP: Synchronized with group, ready for connections