Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20047

Nodes are randomly crashing

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.4.6
    • 10.4.22
    • Galera
    • None
    • Debian Stretch

    Description

      Hello,

      we have cluster with 3 nodes and during last days we did upgrade from latest 10.3 to 10.4.6 After this upgrade we have problem with randomly crashing nodes:

      {{Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: cluster conflict due to high priority abort for threads:
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: Winning thread:
      Jul 11 14:00:04 SQL1 mysqld: THD: 668932, mode: toi, state: exec, conflict: aborted, seqno: 4703314
      Jul 11 14:00:04 SQL1 mysqld: SQL: TRUNCATE [XXXXXXXXX]
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: Victim thread:
      Jul 11 14:00:04 SQL1 mysqld: THD: 668959, mode: local, state: exec, conflict: executing, seqno: -1
      Jul 11 14:00:04 SQL1 mysqld: SQL: INSERT INTO [XXXXXXXXX]
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: context: /home/buildbot/buildbot/build/mariadb-10.4.6/storage/innobase/handler/ha_innodb.cc:18658
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: MDL conflict db=[XXXXXXXXX] table=[XXXXXXXXX] ticket=3 solved by abort
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: cluster conflict due to high priority abort for threads:
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: Winning thread:
      Jul 11 14:00:04 SQL1 mysqld: THD: 668932, mode: toi, state: exec, conflict: aborted, seqno: 4703314
      Jul 11 14:00:04 SQL1 mysqld: SQL: TRUNCATE [XXXXXXXXX]
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: Victim thread:
      Jul 11 14:00:04 SQL1 mysqld: THD: 668962, mode: local, state: exec, conflict: executing, seqno: -1
      Jul 11 14:00:04 SQL1 mysqld: SQL: INSERT INTO [XXXXXXXXX]
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: context: /home/buildbot/buildbot/build/mariadb-10.4.6/storage/innobase/handler/ha_innodb.cc:18658
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668932 [Note] WSREP: MDL conflict db=[XXXXXXXXX] table=[XXXXXXXXX] ticket=3 solved by abort
      Jul 11 14:00:04 SQL1 mysqld: 2019-07-11 14:00:04 668959 [Note] WSREP: MDL conflict db=[XXXXXXXXX] table=[XXXXXXXXX] ticket=3 solved by abort
      Jul 11 14:00:04 SQL1 mysqld: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.6/wsrep-lib/src/client_state.cpp:232: int wsrep::client_state::after_statement(): Assertion `transaction_.state() == wsrep::transaction::s_aborted' failed.
      Jul 11 14:00:04 SQL1 mysqld: 190711 14:00:04 [ERROR] mysqld got signal 6 ;
      Jul 11 14:00:04 SQL1 mysqld: This could be because you hit a bug. It is also possible that this binary
      Jul 11 14:00:04 SQL1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Jul 11 14:00:04 SQL1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Jul 11 14:00:04 SQL1 mysqld:
      Jul 11 14:00:04 SQL1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Jul 11 14:00:04 SQL1 mysqld:
      Jul 11 14:00:04 SQL1 mysqld: We will try our best to scrape up some info that will hopefully help
      Jul 11 14:00:04 SQL1 mysqld: diagnose the problem, but since we have already crashed,
      Jul 11 14:00:04 SQL1 mysqld: something is definitely wrong and this may fail.
      Jul 11 14:00:04 SQL1 mysqld:
      Jul 11 14:00:04 SQL1 mysqld: Server version: 10.4.6-MariaDB-1:10.4.6+maria~stretch-log
      Jul 11 14:00:04 SQL1 mysqld: key_buffer_size=67108864
      Jul 11 14:00:04 SQL1 mysqld: read_buffer_size=1048576
      Jul 11 14:00:04 SQL1 mysqld: max_used_connections=198
      Jul 11 14:00:04 SQL1 mysqld: max_threads=5002
      Jul 11 14:00:04 SQL1 mysqld: thread_count=223
      Jul 11 14:00:04 SQL1 mysqld: It is possible that mysqld could use up to
      Jul 11 14:00:04 SQL1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 66774957 K bytes of memory
      Jul 11 14:00:04 SQL1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Jul 11 14:00:04 SQL1 mysqld:
      Jul 11 14:00:04 SQL1 mysqld: Thread pointer: 0x7fbfec0009a8
      Jul 11 14:00:04 SQL1 mysqld: Attempting backtrace. You can use the following information to find out
      Jul 11 14:00:04 SQL1 mysqld: where mysqld died. If you see no messages after this, something went
      Jul 11 14:00:04 SQL1 mysqld: terribly wrong...
      Jul 11 14:00:04 SQL1 mysqld: mysqld: /home/buildbot/buildbot/build/mariadb-10.4.6/wsrep-lib/src/client_state.cpp:232: int wsrep::client_state::after_statement(): Assertion `transaction_.state() == wsrep::transaction::s_aborted' failed.
      Jul 11 14:00:07 SQL1 mysqld_safe: Number of processes running now: 0
      Jul 11 14:00:07 SQL1 mysqld_safe: WSREP: not restarting wsrep node automatically
      Jul 11 14:00:07 SQL1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
      }}

      Attachments

        Issue Links

          Activity

            People

              jplindst Jan Lindström (Inactive)
              lubor LuborJ
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.