Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29293

MariaDB stuck on starting commit state (waiting on commit order critical section)

Details

    Description

      In an environment running Galera Cluster with 6 MariaDB nodes, 1 arbitrator node, some replicas and a ProxySQL, after a network issue that triggered a state transfer on two nodes,
      for some reason, almost all the transactions hang in:

      • “starting” state on the commit statement or on "".
      • "acquiring total order isolation" on the "KILL CONNECTION" statement (The "KILL CONNECTION" was requested by the ProxySQL)
        We tried to restart the service but it hangs on stopping, ProxySQL detected this node as down and switched the traffic to another node.

      By looking at the backtrace it seems that we have a kind of "pthread_cond_wait() deadlock" executed by lock.wait() on the enter() function on the commit monitor during the commit order critical section.

      Unfortunately, we didn't find a way to reproduce the problem

      Attachments

        1. backtraces.txt
          315 kB
        2. innodb_status.txt
          67 kB
        3. process_list.txt
          467 kB
        4. processlist.png
          processlist.png
          701 kB
        5. process-list-sample.txt
          2 kB

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30963 Assertion failure !lock.was_chosen_as_deadlock_victim in trx0trx.h:1065

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29346 update_rows_log_event hung causing galera cluster failure

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30372 Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed

          • Major - Major loss of function.
          • Closed
          includes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31075 KILL QUERY maintains nodes data consistency but breaks GTID sequence

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28472 BF lock wait long for trx - Assertion `mode_ == m_local || transaction_.is_streaming()' failed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29323 Galera ha_abort_transaction is not honored if there are no InnoDB lock conflicts

          • Major - Major loss of function.
          • Open

          Activity

            Transition Time In Source Status Execution Times
            Seppo Jaakola made transition -
            Open In Progress
            		178d 21h 59m 1
            Julien Fritsch made transition -
            Open Needs Feedback
            		1h 1m 1
            Julien Fritsch made transition -
            Needs Feedback Open
            		22d 21h 59m 2
            Julien Fritsch made transition -
            Open Confirmed
            		5s 1
            Seppo Jaakola made transition -
            Confirmed In Progress
            		16d 1h 54m 1
            Julien Fritsch made transition -
            In Progress In Review
            		26d 51m 1
            Teemu Ollakka made transition -
            In Progress Needs Feedback
            		5d 19h 8m 2
            Teemu Ollakka made transition -
            Needs Feedback Closed
            		1h 3m 1
            Teemu Ollakka made transition -
            Closed Stalled
            		24s 1
            Jan Lindström made transition -
            Stalled In Review
            		11d 3h 58m 3
            Oleksandr Byelkin made transition -
            In Review Stalled
            		15d 19h 54m 4
            Julius Goryavsky made transition -
            Stalled In Progress
            		1d 4h 33m 2
            Julius Goryavsky made transition -
            In Progress Closed
            		5d 14h 8m 1

            People

              sysprg Julius Goryavsky
              williamwelter William Welter
              Votes:
              5 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.