Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29492

WSREP: Failed to apply write set (deadlock with mysqldump)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.4.22
    • Fix Version/s: 10.4
    • Component/s: Galera, wsrep
    • Labels:
      None
    • Environment:
      MariaDB 10.4.22 / Galera 26.4.9 on Ubuntu 20.04

      Description

      Three-node Galera cluster running in master/slave mode. Intermittently on the slaves we observe wsrep fail to apply a writeset due to a deadlock at the moment a mysqldump backup kicks off. It is important to note we run mysqldump --single-transaction, hence the global read lock obtained by mysqldump exists only briefly. Restarting the node always results in a SST.

      Here are the logs from one such failure:

      2022-09-01  3:00:01 10 [Note] WSREP: MDL conflict db=zabbix table=history ticket=6 solved by abort
      2022-09-01  3:00:01 11109709 [Note] WSREP: MDL conflict db=zabbix table=history ticket=6 solved by abort
      2022-09-01  3:00:01 11109709 [Note] WSREP: MDL conflict db=zabbix table=history ticket=6 solved by abort
      2022-09-01  3:00:01 11109709 [Note] WSREP: MDL conflict db=zabbix table=history ticket=6 solved by abort
      2022-09-02  3:00:01 11198269 [Note] WSREP: MDL conflict db=zabbix table=history ticket=4 solved by abort
      2022-09-02  3:00:01 10 [Note] WSREP: MDL conflict db=zabbix table=item_discovery ticket=6 solved by abort
      2022-09-02  3:00:01 41 [Note] WSREP: MDL conflict db=zabbix table=item_discovery ticket=6 solved by abort
      2022-09-02  3:00:01 11198269 [Note] WSREP: MDL conflict db=zabbix table=item_discovery ticket=6 solved by abort
      2022-09-02  3:00:01 11198269 [Note] WSREP: MDL conflict db=zabbix table=item_discovery ticket=6 solved by abort
      2022-09-02  3:00:01 11198269 [Note] WSREP: MDL conflict db=zabbix table=item_discovery ticket=6 solved by abort
      2022-09-03  3:00:01 11286828 [Note] WSREP: MDL conflict db=zabbix table=history ticket=4 solved by abort
      2022-09-03  3:00:01 11286828 [Note] WSREP: MDL conflict db=zabbix table=history ticket=4 solved by abort
      2022-09-03  3:00:01 18 [Note] WSREP: MDL conflict db=zabbix table=history ticket=4 solved by abort
      2022-09-03  3:00:01 50 [Note] WSREP: MDL conflict db=zabbix table=history ticket=6 solved by abort
      2022-09-03  3:00:01 11286828 [Note] WSREP: MDL conflict db=zabbix table=history ticket=4 solved by abort
      2022-09-03  3:00:01 11286828 [Note] WSREP: MDL conflict db=zabbix table=trends_uint ticket=4 solved by abort
      2022-09-03  3:00:01 41 [Note] WSREP: MDL conflict db=zabbix table=trends ticket=6 solved by abort
      2022-09-03  3:00:01 41 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1213, fatal: 0 wsrep = (exec_mode: 2 conflict_state: 0 seqno: 8250807862)
      2022-09-03  3:00:01 41 [ERROR] Slave SQL: Error executing row event: 'Deadlock found when trying to get lock; try restarting transaction', Internal MariaDB error code: 1213
      2022-09-03  3:00:01 41 [Warning] WSREP: Event 7 Write_rows_v1 apply failed: 1213, seqno 8250807862
      2022-09-03  3:00:01 41 [ERROR] WSREP: Failed to apply write set: gtid: 810a5d0c-231c-11e9-b756-dbb7899d7dd0:8250807862 server_id: 9fc16e18-78a4-11ec-8921-daf10d0b3600 client_id: 71633335 trx_id: 47048095835 flags: 3 (start_transaction | commit)
      2022-09-03  3:00:01 41 [Note] WSREP: Closing send monitor...
      2022-09-03  3:00:01 41 [Note] WSREP: Closed send monitor.
      2022-09-03  3:00:01 41 [Note] WSREP: gcomm: terminating thread
      2022-09-03  3:00:01 41 [Note] WSREP: gcomm: joining thread
      2022-09-03  3:00:01 41 [Note] WSREP: gcomm: closing backend
      2022-09-03  3:00:01 41 [Note] WSREP: view(view_id(NON_PRIM,60f1bcc9-8702,254) memb {
      60f1bcc9-8702,0
      } joined {
      } left {
      } partitioned {
      9fc16e18-8921,0
      f6d62ce9-8fe8,0
      })
      2022-09-03  3:00:01 41 [Note] WSREP: PC protocol downgrade 1 -> 0
      2022-09-03  3:00:01 41 [Note] WSREP: view((empty))
      2022-09-03  3:00:01 41 [Note] WSREP: gcomm: closed
      2022-09-03  3:00:01 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2022-09-03  3:00:01 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
      2022-09-03  3:00:01 0 [Note] WSREP: Received NON-PRIMARY.
      2022-09-03  3:00:01 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8250807872)
      2022-09-03  3:00:01 0 [Note] WSREP: New SELF-LEAVE.
      2022-09-03  3:00:01 0 [Note] WSREP: Flow-control interval: [2000000, 2000000]
      2022-09-03  3:00:01 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      2022-09-03  3:00:01 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 8250807872)
      2022-09-03  3:00:01 0 [Note] WSREP: RECV thread exiting 0: Success
      

      and so on, with other message indicating the node is leaving the cluster.

        Attachments

          Activity

            People

            Assignee:
            seppo Seppo Jaakola
            Reporter:
            mreibert Mark Reibert
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.