Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29293

MariaDB stuck on starting commit state (waiting on commit order critical section)

    XMLWordPrintable

Details

    Description

      In an environment running Galera Cluster with 6 MariaDB nodes, 1 arbitrator node, some replicas and a ProxySQL, after a network issue that triggered a state transfer on two nodes,
      for some reason, almost all the transactions hang in:

      • “starting” state on the commit statement or on "".
      • "acquiring total order isolation" on the "KILL CONNECTION" statement (The "KILL CONNECTION" was requested by the ProxySQL)
        We tried to restart the service but it hangs on stopping, ProxySQL detected this node as down and switched the traffic to another node.

      By looking at the backtrace it seems that we have a kind of "pthread_cond_wait() deadlock" executed by lock.wait() on the enter() function on the commit monitor during the commit order critical section.

      Unfortunately, we didn't find a way to reproduce the problem

      Attachments

        1. process-list-sample.txt
          2 kB
        2. processlist.png
          processlist.png
          701 kB
        3. process_list.txt
          467 kB
        4. innodb_status.txt
          67 kB
        5. backtraces.txt
          315 kB

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30963 Assertion failure !lock.was_chosen_as_deadlock_victim in trx0trx.h:1065

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29346 update_rows_log_event hung causing galera cluster failure

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30372 Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed

          • Major - Major loss of function.
          • Closed
          includes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31075 KILL QUERY maintains nodes data consistency but breaks GTID sequence

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28472 BF lock wait long for trx - Assertion `mode_ == m_local || transaction_.is_streaming()' failed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29323 Galera ha_abort_transaction is not honored if there are no InnoDB lock conflicts

          • Major - Major loss of function.
          • Open

          Activity

            People

              sysprg Julius Goryavsky
              williamwelter William Welter
              Votes:
              5 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.