Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28472

BF lock wait long for trx - Assertion `mode_ == m_local || transaction_.is_streaming()' failed

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.6.7
    • Fix Version/s: 10.6
    • Component/s: Galera
    • Labels:
      None
    • Environment:
      CentOS Stream 8 (latest patches)
      VMware VSAN 7.x
      4 node Galera cluster
      ProxySQL splitting 1 writer with 3 backup readers

      Description

      We've been running this environment for awhile now. This is the first time we've seen this particular issue. This database/table has been unaltered in years, and has millions of writes to it daily.

      2022-05-04 2:09:15 16 [Note] InnoDB: WSREP: BF lock wait long for trx:0xe3d26236f query: insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)

      mariadbd: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.7/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.

      Server version: 10.6.7-MariaDB-log
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=47
      max_threads=65537
      thread_count=51

      Once the above happens, a simple restart doesn't work. We have to purge the data from the node, and resync everything back over to get everything to start up.

      Here is the relevant config information:
      [mariadb]
      tmpdir=/var/tmp/
      binlog_format=row
      default_storage_engine=InnoDB
      innodb_autoinc_lock_mode=2
      innodb_status_output=off
      innodb_flush_log_at_trx_commit=2
      innodb_buffer_pool_size=4G
      innodb_log_file_size=4G
      innodb_io_capacity=8000
      innodb_lock_wait_timeout=1
      innodb_monitor_enable=all
      thread_handling = pool-of-threads
      thread_stack = 192000
      thread_cache_size = 4
      thread_pool_size = 8
      thread_pool_oversubscribe = 3
      max_connections=500
      max_allowed_packet=512M
      performance_schema=ON
      performance_schema_session_connect_attrs_size = 1024
      userstat = 1
      slow_query_log
      slow_query_log_file=/var/log/mariadb/slow_query.log
      log_error=/var/log/mariadb/mariadb.log

      [galera]
      wsrep_on=ON
      wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
      wsrep_sst_method=rsync
      wsrep_slave_threads=16
      wsrep_provider_options="gmcast.segment=0; gcache.size=4G; gcache.page_size=4G;"

        Attachments

        1. alter_v1.yy
          0.8 kB
        2. node1.err
          1.46 MB
        3. node2.err
          691 kB

          Activity

            People

            Assignee:
            seppo Seppo Jaakola
            Reporter:
            dbray_sd Daniel Bray
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.