Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26803

Galera crash - Assertion. Possible parallel writeset problem

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Confirmed (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 10.5, 10.6, 10.4.21, 10.4.22
    • Fix Version/s: 10.4, 10.5, 10.6
    • Component/s: Galera
    • Labels:
      None
    • Environment:
      Ubuntu Bionic, using community packages from MariaDB repo.
      Also reproduced with build_43208 of 10.4.22

      Description

      We are experiencing a crash of all galera nodes receiving write sets. The operation is a "last resort" clean up stored procedure, that deletes many rows from the same set of related tables. It generally takes 4-5 minutes to run based on our data size, but is crashing within 10-20 seconds if it is going to go wrong.

      We have been using this stored procedure, reasonably regularly, without problem on 10.1 for several years. As suggested by Enterprise support, I have also tried this on the latest 10.4 build, which they provided me with a URL to. This also exhibits the problem.

      Unfortunately, I have been unable to replicate either simplified reproduction steps, or from a different system of ours. However, I have been able to take a "mariabackup" i.e. physical backup, and reproduce the fault on 2 other clusters. The original, and first replication were on VMware machines. The third system, is an AWS EC2 setup. All 3 have the same MariaDB configuration. I suspect the problem is exposed due to the particular on disk data.

      Attached is the log of one of the nodes receiving the writeset.

      First round of testing, I found that autocommit needs to be ON.

      Due to suspecting the data, and knowing that our QA team were trying to delete rows - I started my test again and used "OPTIMIZE TABLE" on the tables that are touched. This caused

      [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table mediator.SEQUENCE; Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos 276, Internal MariaDB error code: 1213
      

      to appear in the log, at an unusual point in the crash logging.

      Because of finding that info, I have now set wsrep_slave_thread = 1, and this completes successfully. Previously the value was 12. I have also tested = 4, which also crashed.

      Therefore with this additional knowledge, I am presuming that something in Galera is presuming it can apply certain writesets in parallel when it cannot.

        Attachments

        1. gdb.txt
          205 kB
        2. mysql-receivingNode.log
          6 kB
        3. second-of-crash.combined.log
          110 kB
        4. storedProcedure.sql
          1.0 kB
        5. table-structure.sql
          2 kB
        6. unable-to-read-page.fatal.log
          4.96 MB

          Issue Links

            Activity

              People

              Assignee:
              seppo Seppo Jaakola
              Reporter:
              brendon Brendon Abbott
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration