[MDEV-9309] Galera node crashed with error code 1062 Created: 2015-12-21  Updated: 2018-07-16  Resolved: 2018-07-16

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.0.21-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Fustini Assignee: Jan Lindström (Inactive)
Resolution: Not a Bug Votes: 0
Labels: galera
Environment:

3 nodes 10.0.21-MariaDB-wsrep-log MariaDB Server, wsrep_25.10.r4144
on CentOS release 6.6 (Final)


 Description   

Hi,
I have a recurrent node crash (random on different node) on this table with this log :

151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
 
event on table cluebringer.quotas_tracking; Duplicate entry
 
'17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
 
Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
 
master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
 
151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
 
warning: 121, 153937000
 
151220 22:52:53 [Warning] WSREP: Failed to apply app buffer: seqno:
 
153937000, status: 1
 
         at galera/src/trx_handle.cpp:apply():351
 
Retrying 2th time
 
151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
 
event on table cluebringer.quotas_tracking; Duplicate entry
 
'17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
 
Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
 
master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
 
151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
 
warning: 121, 153937000
 
151220 22:52:53 [Warning] WSREP: Failed to apply app buffer: seqno:
 
153937000, status: 1
 
         at galera/src/trx_handle.cpp:apply():351
 
Retrying 3th time
 
151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
 
event on table cluebringer.quotas_tracking; Duplicate entry
 
'17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
 
Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
 
master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
 
151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
 
warning: 121, 153937000
 
151220 22:52:53 [Warning] WSREP: Failed to apply app buffer: seqno:
 
153937000, status: 1
 
         at galera/src/trx_handle.cpp:apply():351
 
Retrying 4th time
 
151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
 
event on table cluebringer.quotas_tracking; Duplicate entry
 
'17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
 
Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
 
master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
 
151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
 
warning: 121, 153937000
 
151220 22:52:53 [ERROR] WSREP: Failed to apply trx: source:
 
9a6eb3e5-96b0-11e5-a574-aa0cb56217d8 version: 3 local: 0 state: APPLYING
 
flags: 1 conn_id: 6201309 trx_id: 588064878 seqnos (l: 74892831, g:
 
153937000, s: 153936999, d: 153936976, ts: 5636331055685173)
 
151220 22:52:53 [ERROR] WSREP: Failed to apply trx 153937000 4 times
 
151220 22:52:53 [ERROR] WSREP: Node consistency compromized, aborting...

Then I restart the node and after resynch all works.
This is the table structure :

Table: quotas_tracking

Create Table: CREATE TABLE `quotas_tracking` (
 
  `QuotasLimitsID` bigint(20) unsigned DEFAULT NULL,
 
  `TrackKey` varchar(512) COLLATE latin1_bin DEFAULT NULL,
 
  `LastUpdate` bigint(20) unsigned DEFAULT NULL,
 
  `Counter` decimal(10,4) DEFAULT NULL,
 
  UNIQUE KEY `QuotasLimitsID` (`QuotasLimitsID`,`TrackKey`),
 
  KEY `quotas_tracking_idx1` (`LastUpdate`),
 
  KEY `quotas_tracking_trackkey` (`TrackKey`),
 
  CONSTRAINT `quotas_tracking_ibfk_1` FOREIGN KEY (`QuotasLimitsID`)
 
REFERENCES `quotas_limits` (`ID`)
 
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin;

This are Galera settings :

| wsrep_auto_increment_control  | ON 
| wsrep_causal_reads              | OFF 
| wsrep_certify_nonpk             | ON
 
| wsrep_cluster_address           | gcomm://10.0.9.34,10.0.9.35,10.0.9.36
 
| wsrep_cluster_name              | ops_cluster 
| wsrep_convert_lock_to_trx       | OFF 
| wsrep_data_home_dir             | /var/lib/mysql/ 
| wsrep_dbug_option               | 
| wsrep_debug                     | OFF 
| wsrep_desync                    | OFF 
| wsrep_dirty_reads               | OFF
 
| wsrep_drupal_282555_workaround  | OFF 
| wsrep_forced_binlog_format      | NONE
 
| wsrep_load_data_splitting     | ON                                                                                                                                                                                                                                                                                                | wsrep_log_conflicts             | ON 
| wsrep_max_ws_rows               | 131072 
| wsrep_max_ws_size               | 1073741824 
| wsrep_mysql_replication_bundle  | 0
 
| wsrep_node_address              | 
| wsrep_node_incoming_address     | TO                                                                                                                                                                                                                                                                                                 wsrep_node_name                 | opsgal03                                                                                                                                                                                                                                                                                       | wsrep_notify_cmd                                                                                                                     |
 
| wsrep_on                        | ON
 
| wsrep_osu_method                | TOI
 
| wsrep_provider                  | /usr/lib64/galera/libgalera_smm.so
 
| wsrep_provider_options          | base_host = 10.0.9.36; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 0; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 10.0.9.36; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 7; socket.checksum = 2; 
| wsrep_recover                   | OFF 
| wsrep_replicate_myisam          | OFF
 
| wsrep_restart_slave             | OFF
 
| wsrep_retry_autocommit          | 1                                                                                                                                                                                                                                                                                                   | wsrep_slave_fk_checks           | ON                                                                                                                                                                                                                                                                                                | wsrep_slave_threads             | 20                                                                                                                                                |
 
| wsrep_slave_uk_checks           | OFF 
| wsrep_sst_auth                  |  
| wsrep_sst_donor                 | 
| wsrep_sst_donor_rejects_queries | OFF
 
| wsrep_sst_method                | rsync                                                                                                                                                                                                                                                                                             | wsrep_sst_receive_address       | AUTO                                                                                                                                                                                                                                                                                            | wsrep_start_position            | 9e908a2d-d234-11e4-9225-07f647687ad8:131655725                                                                                                                                                                                                                                | wsrep_sync_wait                 | 0

Thank you very much

 Comments   
Comment by Jan Lindström (Inactive) [ 2018-07-16 ]

Server founds a duplicate key on other nodes and intentionally shutdowns as cluster consistency is compromised. At these situations administrator needs to made decision who is correct and correct the situation. In Galera 4 there is automatic voting on these situations.

Comment by Jan Lindström (Inactive) [ 2018-07-16 ]

Please upgrade to more recent release and if you can repeat the issue please provide full error log (use wsrep-debug=ON configuration), configuration files and instructions how to repeat.

Generated at Thu Feb 08 07:33:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.