Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17348

Replication stops with error 1062

    XMLWordPrintable

Details

    Description

      Sep 03 07:30:17 svsdb02b mysqld[10408]: 2018-09-03  7:30:17 11 [ERROR] Slave SQL: Error 'Duplicate entry '383' for key 'PRIMARY'' on query. Default database: 'svs'. Query: 'INSERT INTO `invoice_customer_snapshots` (year, month) VALUES (2018, 8)', Gtid 0-23278185-3223146, Internal MariaDB error code: 1062
      Sep 03 07:30:17 svsdb02b mysqld[10408]: 2018-09-03  7:30:17 11 [Warning] Slave: Duplicate entry '383' for key 'PRIMARY' Error_code: 1062
      Sep 03 07:30:17 svsdb02b mysqld[10408]: 2018-09-03  7:30:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000050' position 38275946; GTID position '0-23278185-3223145'
      Sep 03 07:30:17 svsdb02b mysqld[10408]: 2018-09-03  7:30:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin.000050' at position 38275946; GTID position '0-23278185-3223145'
      Sep 03 07:55:48 svsdb02b mysqld[10408]: 2018-09-03  7:55:48 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013)
      Sep 03 07:55:48 svsdb02b mysqld[10408]: 2018-09-03  7:55:48 10 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000050', position 78892171; GTID position 0-23278185-3231670
      

      Oct  1 04:49:32 svsdb02b mysqld[10408]: 2018-10-01  4:49:32 83771 [ERROR] Slave SQL: Error 'Duplicate entry '384' for key 'PRIMARY'' on query. Default database: 'svs'. Query: 'INSERT INTO `invoice_customer_snapsh
      ots` (year, month) VALUES (2018, 9)', Gtid 0-23278185-9619862, Internal MariaDB error code: 1062
      Oct  1 04:49:32 svsdb02b mysqld[10408]: 2018-10-01  4:49:32 83771 [Warning] Slave: Duplicate entry '384' for key 'PRIMARY' Error_code: 1062
      Oct  1 04:49:32 svsdb02b mysqld[10408]: 2018-10-01  4:49:32 83771 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'my
      sql-bin.000103' position 64738944; GTID position '0-23278185-9619861'
      Oct  1 04:49:32 svsdb02b mysqld[10408]: 2018-10-01  4:49:32 83771 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin.000103' at position 64738944; GTID position '0-23278185-9619861'
      

      CREATE TABLE `invoice_customer_snapshots` (
        `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
        `year` mediumint(8) unsigned NOT NULL,
        `month` tinyint(3) unsigned NOT NULL,
        `creation_date` timestamp NOT NULL DEFAULT current_timestamp(),
        PRIMARY KEY (`id`),
        UNIQUE KEY `id` (`id`,`year`,`month`)
      ) ENGINE=InnoDB AUTO_INCREMENT=385 DEFAULT CHARSET=utf8
      

      When we connected manually to the slave, the row didn't exist but the auto_increment was incremented. (So 384 was missing on slave, but counter was already at 385)

      We checked the master, the insert was only once in the binlog.
      We migrated from version 10.0.30 to 10.3.8. After the migration this error happens. (Edit: We migrated via mysqldump on a complete new host)
      This script runs on the 1st workday each month, and it crashed both times so far.
      We run binlogformat mixed.
      Because we couldn't find the bug we resynced the whole slave, so we killed the error state.

      I can reproduce this in our staging environment, so if you have questions I should be able to produce this error again
      This script runs 20 times in parallel, and each script tries to insert the period if it doesn't already exist. This likely relate to the error and explains why we only see this error in this particular table, because usually we don't do such crazy stunts.
      When I run this serial (instead of paralell) in dev the replications works fine.

      We plan to rework the architecture until next month, so that this only runs once, to lessen the urgency for us.

      possibly the same as: https://jira.mariadb.org/browse/MDEV-14566

      Attachments

        1. master_status.txt
          12 kB
        2. master_variables.txt
          12 kB
        3. masterconf.zip
          6 kB
        4. slave_status.txt
          13 kB
        5. slave_variables.txt
          18 kB
        6. slaveconf.zip
          6 kB

        Activity

          People

            sujatha.sivakumar Sujatha Sivakumar (Inactive)
            Benjamin Setzer Benjamin Setzer
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.