Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-39066

Failed UPDATE on MyISAM table causes discrepancy between primary and replica, replication aborts

    XMLWordPrintable

Details

    Description

      Most of the contents in the test case below seems redundant, but it creates a pre-condition for UPDATE going wrong and breaking things. I hope that when the problem is analyzed, the same condition will be created in a more meaningful manner.

      I am not sure whether the test case will work on all builds and versions, systems and architectures. It works on various builds and versions that I have on my machine, but there is a possible non-determinism under the hood, details below.

      --source include/have_binlog_format_mixed.inc
      --source include/master-slave.inc
       
      CREATE TABLE lineitem (l_orderkey INT, l_linenumber INT, PRIMARY KEY (l_orderkey,l_linenumber)) ENGINE=MyISAM;
       
      INSERT INTO lineitem VALUES (451,4),(452,1);
      DELETE FROM lineitem LIMIT 2;
      CREATE TRIGGER tr AFTER INSERT ON lineitem FOR EACH ROW DELETE FROM lineitem;
      --error ER_CANT_UPDATE_USED_TABLE_IN_SF_OR_TRG
      INSERT INTO lineitem VALUES (61, 3);
      --error ER_CANT_UPDATE_USED_TABLE_IN_SF_OR_TRG
      INSERT INTO lineitem VALUES (249, 3);
      --error ER_CANT_UPDATE_USED_TABLE_IN_SF_OR_TRG
      INSERT INTO lineitem SELECT 249, 0 FROM lineitem LIMIT 1;
       
      --error ER_DUP_ENTRY
      UPDATE lineitem SET l_linenumber = 7;
       
      DELETE FROM lineitem LIMIT 3;
       
      --sync_slave_with_master
       
      --connection master
      DROP TABLE lineitem;
      --source include/rpl_end.inc
      

      All INSERTs and DELETEs are logically deterministic:

      • first INSERT simply inserts 2 rows;
      • first DELETE removes them both – LIMIT 2 is one of seemingly redundant pieces of logic, but it leads to DELETE being binlogged in row format, which appears to be important for the test case. The same can be achieved by setting row_format to ROW before DELETE and back to MIXED after, but in my opinion it makes the test case far more artificial;
      • the following two failing INSERTs in fact insert 1 row each into the table; the AFTER INSERT trigger fails afterwards, but since it's MyISAM, the rows remain inserted. It is all quite normal, INSERTs are still written into the binary log with an error code, the slave executes them, gets a matching code, all good. I have no logical explanation why the reproduction requires an attempt to open the table within the trigger, but apparently it contributes to the outcome somehow, as not only could I not get rid of the trigger, but also couldn't change it so that it would return ER_NO_SUCH_TABLE or alike;
      • the last of three INSERTs is INSERT .. SELECT .. LIMIT rather than INSERT – it doesn't make the result non-deterministic as it inserts constant values anyway, but INSERT .. SELECT is binlogged in row format, which again appears to be important. This row event also executes on the slave just fine.

      Now, after these three INSERTs, we have the above-mentioned "pre-condition".
      Both master and slave have the same table contents:

      SELECT * FROM lineitem;
      l_orderkey      l_linenumber
      61      3
      249     0
      249     3
      connection slave;
      SELECT * FROM lineitem;
      l_orderkey      l_linenumber
      61      3
      249     0
      249     3
      

      and if we attempt to execute our UPDATE on each server (even separately, without replication involved), it will hit ER_DUP_ENTRY on both servers as expected; but before it does, it will update some rows, and they will be different rows on master and slave:

      connection slave;
      UPDATE lineitem SET l_linenumber = 7;
      ERROR 23000: Duplicate entry '249-7' for key 'PRIMARY'
      SELECT * FROM lineitem;
      l_orderkey      l_linenumber
      61      7
      249     3
      249     7
      connection master;
      UPDATE lineitem SET l_linenumber = 7;
      ERROR 23000: Duplicate entry '249-7' for key 'PRIMARY'
      SELECT * FROM lineitem;
      l_orderkey      l_linenumber
      61      7
      249     0
      249     7
      

      This is not extremely surprising, as this UPDATE is rather non-deterministic, even though in a vast majority of cases we don't see it. That's why our "pre-condition" is important, somehow previous actions create an internal state (index structure?) which is different on master and slave.
      In fact, in some versions of the test case, even SELECT was already showing it by returning the rows on master and slave in a different order. It doesn't happen in this testcase for SELECT, but it does for UPDATE, so it updates rows in a different order and ends up with a different outcome.

      Same happens in our test when UPDATE is executed on the master and is replicated, because UPDATE is binlogged in the statement format. I find it questionable, since the previous INSERT was in the row format, and the update itself isn't exactly safe.

      So, after this partially failed UPDATE, we have different data on master and slave, which of course quickly leads to a replication abort. The last DELETE is just one of many ways to demonstrate it. It attempts to delete all rows, LIMIT 3 makes it be written in a row format, and it expectedly hits

      Last_SQL_Error  Could not execute Delete_rows_v1 event on table test.lineitem; Can't find record in 'lineitem', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log master-bin.000001, end_log_pos 2489
      

      I suppose if there is a solution here, then it would be binlogging the UPDATE in row format.

      Attachments

        Activity

          People

            bnestere Brandon Nesterenko
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.