Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23025

int wsrep::transaction::before_rollback(): Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed signal 6

    XMLWordPrintable

    Details

      Description

      Occsionally a node crash with

      transaction.cpp:632: int wsrep::transaction::before_rollback(): Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed.
      

      Maybe related to MDEV-22222 / MDEV-22223 / MDEV-18935

      some stacktraces:

      2020-06-23  8:15:02 37433 [Warning] Aborted connection 37433 to db: 'information_schema' user: 'pam_siha' host: '10.109.113.79' (Got an error reading communication packets)
      mysqld: /home/buildbot/buildbot/build/mariadb-10.4.13/wsrep-lib/src/transaction.cpp:632: int wsrep::transaction::before_rollback(): Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed.
      200623  8:50:13 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic-log
      key_buffer_size=134217728
      read_buffer_size=16777216
      max_used_connections=257
      max_threads=6002
      thread_count=394
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 123200322 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f27a8060dc8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fb26eeceda8 thread_stack 0x30000
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5610cdfa2e8e]
      

      mysqld: /home/buildbot/buildbot/build/mariadb-10.4.13/wsrep-lib/src/transaction.cpp:632: int wsrep::transaction::before_rollback(): Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed.
      200622  9:40:16 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic-log
      key_buffer_size=134217728
      read_buffer_size=16777216
      max_used_connections=306
      max_threads=6002
      thread_count=446
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 123200322 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f65a4375ec8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f6ba398cda8 thread_stack 0x30000
      [... some aborted connections]
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f56b16ce8e]
      /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55f56abe8915]
      

      2020-06-22  9:49:37 0 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '10.4.13-MariaDB-1:10.4.13+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
      2020-06-22  9:49:37 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 1095767230
      2020-06-22  9:49:37 1 [Note] WSREP: Min available from gcache for CC from group: 1093637757
      2020-06-22  9:49:37 1 [Note] WSREP: Server S21006 synced with group
      2020-06-22  9:49:37 1 [Note] WSREP: Server status change joined -> synced
      2020-06-22  9:49:37 1 [Note] WSREP: Synchronized with group, ready for connections
      2020-06-22  9:49:48 0 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000000 of size 1073741824 bytes
      mysqld: /home/buildbot/buildbot/build/mariadb-10.4.13/wsrep-lib/src/transaction.cpp:632: int wsrep::transaction::before_rollback(): Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed.
      200622  9:49:53 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic-log
      key_buffer_size=134217728
      read_buffer_size=16777216
      max_used_connections=1017
      max_threads=6002
      thread_count=1157
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 123200322 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7ebecc01d0b8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7ebc6ad3fda8 thread_stack 0x30000
      *** buffer overflow detected ***: /usr/sbin/mysqld terminated
      2020-06-22  9:49:54 2098 [Warning] Aborted connection 2098 to db: 'unconnected' user: 'unauthenticated' host: '10.109.113.79' (This connection closed normally without authentication)
      2020-06-22  9:52:22 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 7f40ce9f-916a-11ea-a326-97aa42b5043e:1095767404
      wsrep loader: [INFO] wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
      wsrep loader: [INFO] wsrep_load(): Galera 26.4.4(r4599) by Codership Oy <info@codership.com> loaded successfully.
      2020-06-22  9:52:22 0 [Note] WSREP: CRC-32C: using hardware acceleration.
      2020-06-22  9:52:22 0 [Note] WSREP: Found saved state: 7f40ce9f-916a-11ea-a326-97aa42b5043e:-1, safe_to_bootstrap: 1
      2020-06-22  9:52:22 0 [Note] WSREP: GCache DEBUG: opened preamble:
      Version: 2
      UUID: 7f40ce9f-916a-11ea-a326-97aa42b5043e
      Seqno: -1 - -1
      Offset: -1
      Synced: 0
      2020-06-22  9:52:22 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 7f40ce9f-916a-11ea-a326-97aa42b5043e, offset: -1
      2020-06-22  9:52:22 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (         0/3221225496 bytes) complete.
      2020-06-22  9:52:23 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (3221225496/3221225496 bytes) complete.
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              sysprg Julius Goryavsky
              Reporter:
              Richard Richard Stracke
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: