We've been running this environment for awhile now. This is the first time we've seen this particular issue. This database/table has been unaltered in years, and has millions of writes to it daily.
2022-05-04 2:09:15 16 [Note] InnoDB: WSREP: BF lock wait long for trx:0xe3d26236f query: insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
mariadbd: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.7/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
Server version: 10.6.7-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=47
max_threads=65537
thread_count=51
Once the above happens, a simple restart doesn't work. We have to purge the data from the node, and resync everything back over to get everything to start up.
Here is the relevant config information:
[mariadb]
tmpdir=/var/tmp/
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_status_output=off
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size=4G
innodb_log_file_size=4G
innodb_io_capacity=8000
innodb_lock_wait_timeout=1
innodb_monitor_enable=all
thread_handling = pool-of-threads
thread_stack = 192000
thread_cache_size = 4
thread_pool_size = 8
thread_pool_oversubscribe = 3
max_connections=500
max_allowed_packet=512M
performance_schema=ON
performance_schema_session_connect_attrs_size = 1024
userstat = 1
slow_query_log
slow_query_log_file=/var/log/mariadb/slow_query.log
log_error=/var/log/mariadb/mariadb.log
[galera]
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
wsrep_sst_method=rsync
wsrep_slave_threads=16
wsrep_provider_options="gmcast.segment=0; gcache.size=4G; gcache.page_size=4G;"
relates to
MDEV-29293
MariaDB stuck on starting commit state (waiting on commit order critical section)
Closed
Jan Lindström (Inactive)
made changes -
2022-06-15 06:09
Field
Original Value
New Value
Assignee
Jan Lindström
[ jplindst
]
Elena Stepanova
made changes -
2022-07-18 10:47
Fix Version/s
10.6
[ 24028
]
Seppo Jaakola
made changes -
2023-04-20 06:53
Status
Open
[ 1
]
In Progress
[ 3
]
Seppo Jaakola
made changes -
2023-04-20 07:07
Status
In Progress
[ 3
]
In Testing
[ 10301
]
Seppo Jaakola
made changes -
2023-04-20 07:07
Assignee
Seppo Jaakola
[ seppo
]
Ramesh Sivaraman
[ JIRAUSER48189
]
Ramesh Sivaraman
made changes -
2023-04-21 06:28
Assignee
Ramesh Sivaraman
[ JIRAUSER48189
]
Seppo Jaakola
[ seppo
]
Status
In Testing
[ 10301
]
Stalled
[ 10000
]
Jan Lindström
made changes -
2023-08-07 05:43
Fix Version/s
10.6.15
[ 29013
]
Fix Version/s
10.6
[ 24028
]
Resolution
Fixed
[ 1
]
Status
Stalled
[ 10000
]
Closed
[ 6
]
{"report":{"fcp":1430.8000001907349,"ttfb":293.90000009536743,"pageVisibility":"visible","entityId":110566,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"f59efd3d-e83a-4bc0-903d-452dd1b3f19c","navigationType":0,"readyForUser":1515.4000000953674,"redirectCount":0,"resourceLoadedEnd":1923,"resourceLoadedStart":301.40000009536743,"resourceTiming":[{"duration":470.69999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":301.40000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":301.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":772.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":470.80000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":301.59999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":301.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":772.4000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":477,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":301.80000019073486,"connectEnd":301.80000019073486,"connectStart":301.80000019073486,"domainLookupEnd":301.80000019073486,"domainLookupStart":301.80000019073486,"fetchStart":301.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":301.80000019073486,"responseEnd":778.8000001907349,"responseStart":778.8000001907349,"secureConnectionStart":301.80000019073486},{"duration":562.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":302.09999990463257,"connectEnd":302.09999990463257,"connectStart":302.09999990463257,"domainLookupEnd":302.09999990463257,"domainLookupStart":302.09999990463257,"fetchStart":302.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":302.09999990463257,"responseEnd":864.5999999046326,"responseStart":864.5999999046326,"secureConnectionStart":302.09999990463257},{"duration":566.3000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":302.19999980926514,"connectEnd":302.19999980926514,"connectStart":302.19999980926514,"domainLookupEnd":302.19999980926514,"domainLookupStart":302.19999980926514,"fetchStart":302.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":302.19999980926514,"responseEnd":868.5,"responseStart":868.5,"secureConnectionStart":302.19999980926514},{"duration":566.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":302.5,"connectEnd":302.5,"connectStart":302.5,"domainLookupEnd":302.5,"domainLookupStart":302.5,"fetchStart":302.5,"redirectEnd":0,"redirectStart":0,"requestStart":302.5,"responseEnd":869,"responseStart":869,"secureConnectionStart":302.5},{"duration":567.1000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":302.69999980926514,"connectEnd":302.69999980926514,"connectStart":302.69999980926514,"domainLookupEnd":302.69999980926514,"domainLookupStart":302.69999980926514,"fetchStart":302.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":302.69999980926514,"responseEnd":869.8000001907349,"responseStart":869.8000001907349,"secureConnectionStart":302.69999980926514},{"duration":646.0999999046326,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":302.80000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":302.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":948.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":567.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":303,"connectEnd":303,"connectStart":303,"domainLookupEnd":303,"domainLookupStart":303,"fetchStart":303,"redirectEnd":0,"redirectStart":0,"requestStart":303,"responseEnd":870.4000000953674,"responseStart":870.4000000953674,"secureConnectionStart":303},{"duration":645.8000001907349,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":303.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":303.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":949,"responseStart":0,"secureConnectionStart":0},{"duration":567.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":303.40000009536743,"connectEnd":303.40000009536743,"connectStart":303.40000009536743,"domainLookupEnd":303.40000009536743,"domainLookupStart":303.40000009536743,"fetchStart":303.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":303.40000009536743,"responseEnd":871,"responseStart":871,"secureConnectionStart":303.40000009536743},{"duration":816.3000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":311.69999980926514,"connectEnd":311.69999980926514,"connectStart":311.69999980926514,"domainLookupEnd":311.69999980926514,"domainLookupStart":311.69999980926514,"fetchStart":311.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":311.69999980926514,"responseEnd":1128,"responseStart":1128,"secureConnectionStart":311.69999980926514},{"duration":1589.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":322.90000009536743,"connectEnd":322.90000009536743,"connectStart":322.90000009536743,"domainLookupEnd":322.90000009536743,"domainLookupStart":322.90000009536743,"fetchStart":322.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":322.90000009536743,"responseEnd":1912.0999999046326,"responseStart":1912.0999999046326,"secureConnectionStart":322.90000009536743},{"duration":135.90000009536743,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":993.6999998092651,"connectEnd":993.6999998092651,"connectStart":993.6999998092651,"domainLookupEnd":993.6999998092651,"domainLookupStart":993.6999998092651,"fetchStart":993.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":993.6999998092651,"responseEnd":1129.5999999046326,"responseStart":1129.5999999046326,"secureConnectionStart":993.6999998092651},{"duration":557.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1365.9000000953674,"connectEnd":1365.9000000953674,"connectStart":1365.9000000953674,"domainLookupEnd":1365.9000000953674,"domainLookupStart":1365.9000000953674,"fetchStart":1365.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":1365.9000000953674,"responseEnd":1923,"responseStart":1923,"secureConnectionStart":1365.9000000953674}],"fetchStart":1,"domainLookupStart":1,"domainLookupEnd":1,"connectStart":1,"connectEnd":1,"requestStart":88,"responseStart":294,"responseEnd":323,"domLoading":300,"domInteractive":1960,"domContentLoadedEventStart":1960,"domContentLoadedEventEnd":2023,"domComplete":2911,"loadEventStart":2911,"loadEventEnd":2911,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1914.1999998092651},{"name":"bigPipe.sidebar-id.end","time":1915.1999998092651},{"name":"bigPipe.activity-panel-pipe-id.start","time":1915.3000001907349},{"name":"bigPipe.activity-panel-pipe-id.end","time":1919},{"name":"activityTabFullyLoaded","time":2044.6999998092651}],"measures":[],"correlationId":"1aea046b348f91","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":147,"dbReadsTimeInMs":29,"dbConnsTimeInMs":39,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}