Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.2.10, 10.1.34, 10.1.35
Description
Ran pt-archiver to archive records and all the nodes went down with the error below.
2018-01-04 11:36:27 140527176017664 [ERROR] mysqld: Can't find record in 'log_system'
|
2018-01-04 11:36:28 140527176017664 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table dataworks.log_system; Can't find record in 'log_system', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 541, Internal MariaDB error code: 1032
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: RBR event 3 Delete_rows_v1 apply warning: 120, 397232798
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: Failed to apply app buffer: seqno: 397232798, status: 1
|
at galera/src/trx_handle.cpp:apply():351
|
Retrying 2th time
|
2018-01-04 11:36:28 140527176017664 [ERROR] mysqld: Can't find record in 'log_system'
|
2018-01-04 11:36:28 140527176017664 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table dataworks.log_system; Can't find record in 'log_system', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 541, Internal MariaDB error code: 1032
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: RBR event 3 Delete_rows_v1 apply warning: 120, 397232798
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: Failed to apply app buffer: seqno: 397232798, status: 1
|
at galera/src/trx_handle.cpp:apply():351
|
Retrying 3th time
|
2018-01-04 11:36:28 140527176017664 [ERROR] mysqld: Can't find record in 'log_system'
|
2018-01-04 11:36:28 140527176017664 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table dataworks.log_system; Can't find record in 'log_system', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 541, Internal MariaDB error code: 1032
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: RBR event 3 Delete_rows_v1 apply warning: 120, 397232798
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: Failed to apply app buffer: seqno: 397232798, status: 1
|
at galera/src/trx_handle.cpp:apply():351
|
Retrying 4th time
|
2018-01-04 11:36:28 140527176017664 [ERROR] mysqld: Can't find record in 'log_system'
|
2018-01-04 11:36:28 140527176017664 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table dataworks.log_system; Can't find record in 'log_system', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 541, Internal MariaDB error code: 1032
|
2018-01-04 11:36:28 140527176017664 [Warning] WSREP: RBR event 3 Delete_rows_v1 apply warning: 120, 397232798
|
2018-01-04 11:36:28 140527176017664 [ERROR] WSREP: Failed to apply trx: source: a16066e0-e4f9-11e7-ae4d-665dac15733d version: 3 local: 0 state: APPLYING flags: 1 conn_id: 3358326 trx_id: 822622245 seqnos (l: 5873410, g: 397232798, s: 397232797, d: 397232772, ts: 3278302040065719)
|
2018-01-04 11:36:28 140527176017664 [ERROR] WSREP: Failed to apply trx 397232798 4 times
|
2018-01-04 11:36:28 140527176017664 [ERROR] WSREP: Node consistency compromized, aborting...
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: Closing send monitor...
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: Closed send monitor.
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: gcomm: terminating thread
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: gcomm: joining thread
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: gcomm: closing backend
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: view(view_id(NON_PRIM,17fca7cf,203) memb {
|
877cbcec,0
|
} joined {
|
} left {
|
} partitioned {
|
17fca7cf,0
|
6ec5c1cb,0
|
a16066e0,0
|
cdc1bdfd,0
|
})
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: view((empty))
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: gcomm: closed
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Flow-control interval: [64, 80]
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Received NON-PRIMARY.
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 397232798)
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Received self-leave message.
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Flow-control interval: [0, 0]
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Received SELF-LEAVE. Closing connection.
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 397232798)
|
2018-01-04 11:36:28 140553635206912 [Note] WSREP: RECV thread exiting 0: Success
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: recv_thread() joined.
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: Closing replication queue.
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: Closing slave action queue.
|
2018-01-04 11:36:28 140527176017664 [Note] WSREP: /usr/sbin/mysqld: Terminated.
|
2018-01-04 11:36:42 140645890877568 [Note] WSREP: Read nil XID from storage engines, skipping position init
|
2018-01-04 11:36:42 140645890877568 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
|
2018-01-04 11:36:42 140645890877568 [Note] WSREP: wsrep_load(): Galera 25.3.20(r3703) by Codership Oy <info@codership.com> loaded successfully.
|