Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.6.7
-
None
-
CentOS Stream 8 (latest patches)
VMware VSAN 7.x
4 node Galera cluster
ProxySQL splitting 1 writer with 3 backup readers
Description
We've been running this environment for awhile now. This is the first time we've seen this particular issue. This database/table has been unaltered in years, and has millions of writes to it daily.
2022-05-04 2:09:15 16 [Note] InnoDB: WSREP: BF lock wait long for trx:0xe3d26236f query: insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
mariadbd: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.7/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
Server version: 10.6.7-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=47
max_threads=65537
thread_count=51
Once the above happens, a simple restart doesn't work. We have to purge the data from the node, and resync everything back over to get everything to start up.
Here is the relevant config information:
[mariadb]
tmpdir=/var/tmp/
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_status_output=off
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size=4G
innodb_log_file_size=4G
innodb_io_capacity=8000
innodb_lock_wait_timeout=1
innodb_monitor_enable=all
thread_handling = pool-of-threads
thread_stack = 192000
thread_cache_size = 4
thread_pool_size = 8
thread_pool_oversubscribe = 3
max_connections=500
max_allowed_packet=512M
performance_schema=ON
performance_schema_session_connect_attrs_size = 1024
userstat = 1
slow_query_log
slow_query_log_file=/var/log/mariadb/slow_query.log
log_error=/var/log/mariadb/mariadb.log
[galera]
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
wsrep_sst_method=rsync
wsrep_slave_threads=16
wsrep_provider_options="gmcast.segment=0; gcache.size=4G; gcache.page_size=4G;"
Attachments
Issue Links
- relates to
-
MDEV-29293 MariaDB stuck on starting commit state (waiting on commit order critical section)
- Closed