Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23064

Crashed node under Galera




      This morning our newly spanned Galera 3 node cluster crashed. It has been running on production for something like 6-7 hours.

      What happened is that 2 of the nodes went into an undefined state. The third one was OK but for some reason a master node could not be selected by Maxscale:

      2020-07-01 06:41:47 notice : Server changed state: server3[]: slave_down. [Slave, Synced, Running] -> [Down]
      2020-07-01 06:41:49 notice : Server changed state: server1[]: slave_down. [Slave, Synced, Running] -> [Down]
      2020-07-01 06:41:55 error : [galeramon] There are no cluster members
      2020-07-01 06:41:55 notice : Server changed state: server2[]: lost_master. [Master, Synced, Running] -> [Running]
      2020-07-01 06:56:35 error : (9) [readwritesplit] Couldn't find suitable Master from 3 candidates.
      2020-07-01 06:56:35 error : (9) Failed to create new router session for service 'Galera-Service'. See previous errors for more details.

      After a browsed on the nodes themselves i found that actually mysql had crashed with an assertion:

      mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.13/wsrep-lib/include/wsrep/client_state.hpp:603: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.

      Attaching the full crash log below.

      Also attaching the configs of one of the machines. Please note this specific one also has binary logging enabled as it is used by an external slave.

      Is there a way to prevent this?


        1. 07-02-2020-node1.txt
          35 kB
        2. 07-02-2020-node2.txt
          42 kB
        3. 07-02-2020-node3.txt
          15 kB
        4. binlog.cfg
          0.2 kB
        5. galera.cfg
          0.8 kB
        6. Maria crash.txt
          7 kB
        7. server.cfg
          3 kB
        8. table structure.txt
          2 kB

        Issue Links



              jplindst Jan Lindström (Inactive)
              mkovachev Martin Kovachev
              0 Vote for this issue
              3 Start watching this issue



                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.