Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9309

Galera node crashed with error code 1062

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not a Bug
    • Affects Version/s: 10.0.21-galera
    • Fix Version/s: N/A
    • Component/s: Galera
    • Labels:
    • Environment:
      3 nodes 10.0.21-MariaDB-wsrep-log MariaDB Server, wsrep_25.10.r4144
      on CentOS release 6.6 (Final)

      Description

      Hi,
      I have a recurrent node crash (random on different node) on this table with this log :

      151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
      event on table cluebringer.quotas_tracking; Duplicate entry
      '17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
      Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
      master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
      151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
      warning: 121, 153937000
      151220 22:52:53 [Warning] WSREP: Failed to apply app buffer: seqno:
      153937000, status: 1
               at galera/src/trx_handle.cpp:apply():351
      Retrying 2th time
      151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
      event on table cluebringer.quotas_tracking; Duplicate entry
      '17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
      Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
      master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
      151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
      warning: 121, 153937000
      151220 22:52:53 [Warning] WSREP: Failed to apply app buffer: seqno:
      153937000, status: 1
               at galera/src/trx_handle.cpp:apply():351
      Retrying 3th time
      151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
      event on table cluebringer.quotas_tracking; Duplicate entry
      '17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
      Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
      master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
      151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
      warning: 121, 153937000
      151220 22:52:53 [Warning] WSREP: Failed to apply app buffer: seqno:
      153937000, status: 1
               at galera/src/trx_handle.cpp:apply():351
      Retrying 4th time
      151220 22:52:53 [ERROR] Slave SQL: Could not execute Update_rows_v1
      event on table cluebringer.quotas_tracking; Duplicate entry
      '17-Recipient:xxxx@pippo.it' for key 'QuotasLimitsID',
      Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's
      master log FIRST, end_log_pos 222, Internal MariaDB error code: 1062
      151220 22:52:53 [Warning] WSREP: RBR event 2 Update_rows_v1 apply
      warning: 121, 153937000
      151220 22:52:53 [ERROR] WSREP: Failed to apply trx: source:
      9a6eb3e5-96b0-11e5-a574-aa0cb56217d8 version: 3 local: 0 state: APPLYING
      flags: 1 conn_id: 6201309 trx_id: 588064878 seqnos (l: 74892831, g:
      153937000, s: 153936999, d: 153936976, ts: 5636331055685173)
      151220 22:52:53 [ERROR] WSREP: Failed to apply trx 153937000 4 times
      151220 22:52:53 [ERROR] WSREP: Node consistency compromized, aborting...

      Then I restart the node and after resynch all works.
      This is the table structure :

      Table: quotas_tracking

      Create Table: CREATE TABLE `quotas_tracking` (
        `QuotasLimitsID` bigint(20) unsigned DEFAULT NULL,
        `TrackKey` varchar(512) COLLATE latin1_bin DEFAULT NULL,
        `LastUpdate` bigint(20) unsigned DEFAULT NULL,
        `Counter` decimal(10,4) DEFAULT NULL,
        UNIQUE KEY `QuotasLimitsID` (`QuotasLimitsID`,`TrackKey`),
        KEY `quotas_tracking_idx1` (`LastUpdate`),
        KEY `quotas_tracking_trackkey` (`TrackKey`),
        CONSTRAINT `quotas_tracking_ibfk_1` FOREIGN KEY (`QuotasLimitsID`)
      REFERENCES `quotas_limits` (`ID`)
      ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin;

      This are Galera settings :

      | wsrep_auto_increment_control  | ON 
      | wsrep_causal_reads              | OFF 
      | wsrep_certify_nonpk             | ON
      | wsrep_cluster_address           | gcomm://10.0.9.34,10.0.9.35,10.0.9.36
      | wsrep_cluster_name              | ops_cluster 
      | wsrep_convert_lock_to_trx       | OFF 
      | wsrep_data_home_dir             | /var/lib/mysql/ 
      | wsrep_dbug_option               | 
      | wsrep_debug                     | OFF 
      | wsrep_desync                    | OFF 
      | wsrep_dirty_reads               | OFF
      | wsrep_drupal_282555_workaround  | OFF 
      | wsrep_forced_binlog_format      | NONE
      | wsrep_load_data_splitting     | ON                                                                                                                                                                                                                                                                                                | wsrep_log_conflicts             | ON 
      | wsrep_max_ws_rows               | 131072 
      | wsrep_max_ws_size               | 1073741824 
      | wsrep_mysql_replication_bundle  | 0
      | wsrep_node_address              | 
      | wsrep_node_incoming_address     | TO                                                                                                                                                                                                                                                                                                 wsrep_node_name                 | opsgal03                                                                                                                                                                                                                                                                                       | wsrep_notify_cmd                                                                                                                     |
      | wsrep_on                        | ON
      | wsrep_osu_method                | TOI
      | wsrep_provider                  | /usr/lib64/galera/libgalera_smm.so
      | wsrep_provider_options          | base_host = 10.0.9.36; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 0; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 10.0.9.36; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 7; socket.checksum = 2; 
      | wsrep_recover                   | OFF 
      | wsrep_replicate_myisam          | OFF
      | wsrep_restart_slave             | OFF
      | wsrep_retry_autocommit          | 1                                                                                                                                                                                                                                                                                                   | wsrep_slave_fk_checks           | ON                                                                                                                                                                                                                                                                                                | wsrep_slave_threads             | 20                                                                                                                                                |
      | wsrep_slave_uk_checks           | OFF 
      | wsrep_sst_auth                  |  
      | wsrep_sst_donor                 | 
      | wsrep_sst_donor_rejects_queries | OFF
      | wsrep_sst_method                | rsync                                                                                                                                                                                                                                                                                             | wsrep_sst_receive_address       | AUTO                                                                                                                                                                                                                                                                                            | wsrep_start_position            | 9e908a2d-d234-11e4-9225-07f647687ad8:131655725                                                                                                                                                                                                                                | wsrep_sync_wait                 | 0

      Thank you very much

        Attachments

          Activity

            People

            Assignee:
            jplindst Jan Lindström
            Reporter:
            Alessandro Fustini
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: