[MDEV-19448] mariadb node replication crash on HA_ERR_FOUND_DUPP_KEY Created: 2019-05-13  Updated: 2020-01-13  Resolved: 2020-01-12

Status: Closed
Project: MariaDB Server
Component/s: Galera, Storage Engine - InnoDB
Affects Version/s: 10.2.22
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: David Delbecq Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: crash, galera, need_feedback, replication
Environment:

centos 7

Attachments: File mariadb-crash.tar.gz    

 Description   

since upgrade of our galera servers to 10.2.x, we experience global crash of nodes due to database inconsistency propagated through galera. This happens about 3 times a month and put our cluster in a very unstable situation as it requires a full SST that is time and IO consuming.

It always has the same scenario:
a client on one node (here nodesdb2) is inserting data that triggers unique key constraint failure (duplicate), leading to client rollbacking it's operations.
However this insert is propagated to other node (here nodesdb1) as if it was approved, and the secondary node crashes due to inconsistency of content.

After this, mariadb stops on the affected node and, on restart, triggers a full SST.

Note that we currently do all insert / select / delete operation on a single node (nodesdb2) and the other nodes (nodesdb1, the crashing node) is currently used only for redundancy. 

May 11 04:40:39 nodesdb1.myhost.com mysqld[412786]: 2019-05-11  4:40:39 139660305315584 [Warning] WSREP: Failed to report last committed 205740288, -110 (Connection timed out)
 
May 11 04:40:41 nodesdb1.myhost.com mysqld[412786]: 2019-05-11  4:40:41 139660305315584 [Warning] WSREP: Failed to report last committed 205740332, -110 (Connection timed out)
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error co
 
de: 1062
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: Failed to apply app buffer: seqno: 209185646, status: 1
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: at galera/src/trx_handle.cpp:apply():353
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: Retrying 2th time
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: Failed to apply app buffer: seqno: 209185646, status: 1
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: at galera/src/trx_handle.cpp:apply():353
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: Retrying 3th time
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: Failed to apply app buffer: seqno: 209185646, status: 1
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: at galera/src/trx_handle.cpp:apply():353
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: Retrying 4th time
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] WSREP: Failed to apply trx: source: e5c9c2b5-400f-11e9-ba3d-8f8054a48006 version: 4 local: 0 state: APPLYING flags: 1 conn_id: 10193487 trx_id: 20898181434 seqnos (l: 34565981, g: 209185646, s: 209185645, d: 209185626, ts: 14759991015976877)
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] WSREP: Failed to apply trx 209185646 4 times
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] WSREP: Node consistency compromised, aborting...
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Note] WSREP: Closing send monitor...
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Note] WSREP: Closed send monitor.
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Note] WSREP: gcomm: terminating thread
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Note] WSREP: gcomm: joining thread
 
May 12 07:32:42 nodesdb1.myhost.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Note] WSREP: gcomm: closing backend

This always happen on the same table "tags_history" which is our biggest table: about 1 445 158 186 rows, 500G storage size and has an estimated 1200 rows inserted and deleted per seconds .

journalctl -u mariadb | grep Write_rows 
......
 
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate ent
 
ry '276267-2019-04-04 20:46:58' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3615, Internal MariaDB error cod
 
e: 1062                                                                                                                                                                                        
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 88177989                                 
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate ent
 
ry '276267-2019-04-04 20:46:58' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3615, Internal MariaDB error cod
 
e: 1062                                                                                                                                                                                        
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 88177989                                 
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate ent
 
ry '276267-2019-04-04 20:46:58' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3615, Internal MariaDB error cod
 
e: 1062                                                                                                                                                                                        
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 88177989                                 
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate ent
 
ry '276267-2019-04-04 20:46:58' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3615, Internal MariaDB error cod
 
e: 1062                                                                                                                                                                                        
Apr 04 20:35:36 nodesdb1.talk2m.com mysqld[45259]: 2019-04-04 20:35:36 140556698482432 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 88177989                                 
Apr 15 20:16:41 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:16:41 139793670989568 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '279006-2019-04-15 20:37:26' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3579, Internal MariaDB error co
 
de: 1062                                                                                                                                                                                       
Apr 15 20:16:41 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:16:41 139793670989568 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 123048539                               
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '279006-2019-04-15 20:46:26' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3579, Internal MariaDB error co
 
de: 1062                                                                                                                                                                                       
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 123067718                               
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '279006-2019-04-15 20:46:26' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3579, Internal MariaDB error co
 
de: 1062                                                                                                                                                                                       
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 123067718                               
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '279006-2019-04-15 20:46:26' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3579, Internal MariaDB error co
 
de: 1062                                                                                                                                                                                       
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 123067718                               
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '279006-2019-04-15 20:46:26' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3579, Internal MariaDB error co
 
de: 1062                                                                                                                                                                                       
Apr 15 20:25:42 nodesdb1.talk2m.com mysqld[104042]: 2019-04-15 20:25:42 139793670682368 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 123067718                               
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '1058484-2019-04-19 11:09:09' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 6349, Internal MariaDB error c
 
ode: 1062                                                                                                                                                                                      
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 134929844                               
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '1058484-2019-04-19 11:09:09' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 6349, Internal MariaDB error c
 
ode: 1062                                                                                                                                                                                      
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 134929844                               
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '1058484-2019-04-19 11:09:09' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 6349, Internal MariaDB error c
 
ode: 1062                                                                                                                                                                                      
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 134929844                               
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate en
 
try '1058484-2019-04-19 11:09:09' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 6349, Internal MariaDB error code: 1062
 
Apr 19 12:14:27 nodesdb1.talk2m.com mysqld[331647]: 2019-04-19 12:14:27 140351886702336 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 134929844                              
Apr 23 13:04:24 nodesdb1.talk2m.com mysqld[192756]: 2019-04-23 13:04:24 140163747346176 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '759823-2019-04-23 11:03:50' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 2231, Internal MariaDB error code: 1062
 
Apr 23 13:04:24 nodesdb1.talk2m.com mysqld[192756]: 2019-04-23 13:04:24 140163747346176 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 147848347                              
Apr 25 02:36:07 nodesdb1.talk2m.com mysqld[192756]: 2019-04-25  2:36:07 140163747960576 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '1101784-2019-04-25 02:34:25' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 3502, Internal MariaDB error code: 1062
 
Apr 25 02:36:07 nodesdb1.talk2m.com mysqld[192756]: 2019-04-25  2:36:07 140163747960576 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 152665180                              
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '620709-2019-05-02 07:34:01' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 1007, Internal MariaDB error code: 1062
 
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 176310335                              
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '620709-2019-05-02 07:34:01' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 1007, Internal MariaDB error code: 1062
 
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 176310335                              
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '620709-2019-05-02 07:34:01' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 1007, Internal MariaDB error code: 1062
 
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 176310335                              
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '620709-2019-05-02 07:34:01' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 1007, Internal MariaDB error code: 1062
 
May 02 08:04:00 nodesdb1.talk2m.com mysqld[192756]: 2019-05-02  8:04:00 140163747346176 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 176310335                              
May 04 09:57:27 nodesdb1.talk2m.com mysqld[412786]: 2019-05-04  9:57:27 139606033254144 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '999712-2019-05-04 09:51:45' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 72003, Internal MariaDB error code: 1062
 
May 04 09:57:27 nodesdb1.talk2m.com mysqld[412786]: 2019-05-04  9:57:27 139606033254144 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 183403214                              
May 05 13:44:55 nodesdb1.talk2m.com mysqld[412786]: 2019-05-05 13:44:55 139533897619200 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '355512-2019-05-05 06:56:08' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 1007, Internal MariaDB error code: 1062
 
May 05 13:44:55 nodesdb1.talk2m.com mysqld[412786]: 2019-05-05 13:44:55 139533897619200 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 187523527                              
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646                              
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646                              
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646                              
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table MyHostData.tags_history; Duplicate entry '276267-2019-05-12 07:38:56' for key 'tagshistory_unique', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 9801, Internal MariaDB error code: 1062
 
May 12 07:32:42 nodesdb1.talk2m.com mysqld[412786]: 2019-05-12  7:32:42 139606033868544 [Warning] WSREP: RBR event 6 Write_rows_v1 apply warning: 121, 209185646

Attached you will find a full log of donor and joiner at time of error.

The failing node show no other message prior to crash, the previous message in logs was days ago.

 Comments   
Comment by Jan Lindström (Inactive) [ 2019-12-12 ]

Can you provide show create table from problematic table ?

Comment by David Delbecq [ 2020-01-13 ]

Sorry, I just came back from holiday.

Here is the requested information

MariaDB [MyHostData]> describe tags_history;
 
+----------------+---------------------+------+-----+---------+----------------+
 
| Field          | Type                | Null | Key | Default | Extra          |
 
+----------------+---------------------+------+-----+---------+----------------+
 
| id             | bigint(20) unsigned | NO   | PRI | NULL    | auto_increment |
 
| tag_id         | int(11)             | NO   | MUL | NULL    |                |
 
| history_set_id | int(11)             | NO   | MUL | NULL    |                |
 
| date           | datetime            | NO   | MUL | NULL    |                |
 
| value          | double              | YES  |     | NULL    |                |
 
| value_string   | varchar(500)        | YES  |     | NULL    |                |
 
| quality        | int(11)             | NO   |     | 0       |                |
 
+----------------+---------------------+------+-----+---------+----------------+
 
7 rows in set (0.00 sec)
 
 
 
MariaDB [MyHostData]> show create table tags_history;
 
+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
| Table        | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 
+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
| tags_history | CREATE TABLE `tags_history` (
 
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
 
  `tag_id` int(11) NOT NULL,
 
  `history_set_id` int(11) NOT NULL,
 
  `date` datetime NOT NULL,
 
  `value` double DEFAULT NULL,
 
  `value_string` varchar(500) DEFAULT NULL,
 
  `quality` int(11) NOT NULL DEFAULT 0,
 
  PRIMARY KEY (`id`),
 
  UNIQUE KEY `tagshistory_unique` (`tag_id`,`date`),
 
  KEY `tagshistory_date` (`date`),
 
  KEY `tagshistory_set_tag_date` (`tag_id`,`history_set_id`,`date`),
 
  KEY `tagshistory_set_date` (`history_set_id`,`date`),
 
  CONSTRAINT `__tagshistory_history_set` FOREIGN KEY (`history_set_id`) REFERENCES `history_sets` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
 
) ENGINE=InnoDB AUTO_INCREMENT=233680008265 DEFAULT CHARSET=utf8 |
 
+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Generated at Thu Feb 08 08:51:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.