After update to 10.4.17 we noticed that the replication suddenly stops for N minutes.
The response of SHOW ENGINE INNODB STATUS shows
SEMAPHORES
|
----------
|
OS WAIT ARRAY INFO: reservation count 440929294
|
--Thread 139722467284736 has waited at btr0cur.cc line 1492 for 60.00 seconds the semaphore:
|
SX-lock on RW-latch at 0x7ef2b8011158 created in file dict0dict.cc line 1954
|
a writer (thread id 139581332059904) has reserved it in mode SX
|
number of readers 0, waiters flag 1, lock_word: 10000000
|
Last time write locked in file dict0stats.cc line 1968
|
OS WAIT ARRAY INFO: signal count 853851161
|
RW-shared spins 1894185518, rounds 11977859561, OS waits 89917364
|
RW-excl spins 928170891, rounds 6504095554, OS waits 24124968
|
RW-sx spins 18253227, rounds 252368118, OS waits 3117148
|
Spin rounds per wait: 6.32 RW-shared, 7.01 RW-excl, 13.83 RW-sx
|
------------
|
TRANSACTIONS
|
------------
|
Trx id counter 60395546748
|
Purge done for trx's n:o < 60395546744 undo n:o < 0 state: running but idle
|
History list length 67
|
LIST OF TRANSACTIONS FOR EACH SESSION:
|
---TRANSACTION 421197791234584, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
---TRANSACTION 421197791230328, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
---TRANSACTION 421197791226072, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
---TRANSACTION 421197791179256, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
---TRANSACTION 421197791217560, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
---TRANSACTION 60395546745, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 12, OS thread handle 139722802640640, query id 9925262534 Waiting for prior transaction to commit
|
---TRANSACTION 60395546740, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 14, OS thread handle 139722468513536, query id 9925262528 Waiting for prior transaction to commit
|
---TRANSACTION 60395546737, ACTIVE 60 sec updating or deleting
|
mysql tables in use 1, locked 1
|
2 lock struct(s), heap size 1128, 1 row lock(s)
|
MySQL thread id 18, OS thread handle 139722467284736, query id 9925262524 Update_rows_log_event::ha_update_row(-1)
|
UPDATE tableName
|
SET `column` = '[1,2,3,4,5]', `updatedAt` = '2021-02-03T14:54:28Z'
|
WHERE id = 1234
|
---TRANSACTION 60395546747, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 19, OS thread handle 139581271136000, query id 9925262538 Waiting for prior transaction to commit
|
---TRANSACTION 60395546746, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 13, OS thread handle 139722468820736, query id 9925262537 Waiting for prior transaction to commit
|
---TRANSACTION 60395546741, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 17, OS thread handle 139722467591936, query id 9925262530 Waiting for prior transaction to commit
|
---TRANSACTION 60395546744, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 15, OS thread handle 139722468206336, query id 9925262532 Waiting for prior transaction to commit
|
---TRANSACTION 60395546739, ACTIVE (PREPARED) 60 sec
|
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 2
|
MySQL thread id 16, OS thread handle 139722467899136, query id 9925262526 Waiting for prior transaction to commit
|
---TRANSACTION 421197791175000, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
---TRANSACTION 421197791170744, not started
|
0 lock struct(s), heap size 1128, 0 row lock(s)
|
Also, there is a warning "Thread 139722467284736 has waited at btr0cur.cc line 1492 for 60.00 seconds the semaphore", possible a clue.
We tried to update the server to the version 10.5.8, but this version contains the same issue.
When we downgraded version to 10.4.15 when problem solved.
I hope you will find the difference and will port it to 10.5.X version.
{"report":{"fcp":933.5999999046326,"ttfb":203,"pageVisibility":"visible","entityId":96956,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"80ad3691-8bb5-410f-8d74-625ffe51bdf5","navigationType":0,"readyForUser":1018.5999999046326,"redirectCount":0,"resourceLoadedEnd":679.2999999523163,"resourceLoadedStart":208.19999980926514,"resourceTiming":[{"duration":201.10000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":208.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":208.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":409.2999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":201.20000004768372,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":208.39999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":208.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":409.59999990463257,"responseStart":0,"secureConnectionStart":0},{"duration":210.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":208.59999990463257,"connectEnd":208.59999990463257,"connectStart":208.59999990463257,"domainLookupEnd":208.59999990463257,"domainLookupStart":208.59999990463257,"fetchStart":208.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":208.59999990463257,"responseEnd":419.19999980926514,"responseStart":419.19999980926514,"secureConnectionStart":208.59999990463257},{"duration":324.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":208.79999995231628,"connectEnd":208.79999995231628,"connectStart":208.79999995231628,"domainLookupEnd":208.79999995231628,"domainLookupStart":208.79999995231628,"fetchStart":208.79999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":208.79999995231628,"responseEnd":533.0999999046326,"responseStart":533.0999999046326,"secureConnectionStart":208.79999995231628},{"duration":328.39999985694885,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":209,"connectEnd":209,"connectStart":209,"domainLookupEnd":209,"domainLookupStart":209,"fetchStart":209,"redirectEnd":0,"redirectStart":0,"requestStart":209,"responseEnd":537.3999998569489,"responseStart":537.3999998569489,"secureConnectionStart":209},{"duration":328.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":209.09999990463257,"connectEnd":209.09999990463257,"connectStart":209.09999990463257,"domainLookupEnd":209.09999990463257,"domainLookupStart":209.09999990463257,"fetchStart":209.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":209.09999990463257,"responseEnd":537.7999999523163,"responseStart":537.7999999523163,"secureConnectionStart":209.09999990463257},{"duration":329.10000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":209.39999985694885,"connectEnd":209.39999985694885,"connectStart":209.39999985694885,"domainLookupEnd":209.39999985694885,"domainLookupStart":209.39999985694885,"fetchStart":209.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":209.39999985694885,"responseEnd":538.5,"responseStart":538.5,"secureConnectionStart":209.39999985694885},{"duration":354,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":209.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":209.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":563.5,"responseStart":0,"secureConnectionStart":0},{"duration":329.10000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":209.69999980926514,"connectEnd":209.69999980926514,"connectStart":209.69999980926514,"domainLookupEnd":209.69999980926514,"domainLookupStart":209.69999980926514,"fetchStart":209.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":209.69999980926514,"responseEnd":538.7999999523163,"responseStart":538.7999999523163,"secureConnectionStart":209.69999980926514},{"duration":353.7000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":209.89999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":209.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":563.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":329.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":210,"connectEnd":210,"connectStart":210,"domainLookupEnd":210,"domainLookupStart":210,"fetchStart":210,"redirectEnd":0,"redirectStart":0,"requestStart":210,"responseEnd":539.5,"responseStart":539.5,"secureConnectionStart":210},{"duration":463.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":215.29999995231628,"connectEnd":215.29999995231628,"connectStart":215.29999995231628,"domainLookupEnd":215.29999995231628,"domainLookupStart":215.29999995231628,"fetchStart":215.29999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":215.29999995231628,"responseEnd":678.8999998569489,"responseStart":678.8999998569489,"secureConnectionStart":215.29999995231628},{"duration":463.90000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":215.39999985694885,"connectEnd":215.39999985694885,"connectStart":215.39999985694885,"domainLookupEnd":215.39999985694885,"domainLookupStart":215.39999985694885,"fetchStart":215.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":215.39999985694885,"responseEnd":679.2999999523163,"responseStart":679.2999999523163,"secureConnectionStart":215.39999985694885},{"duration":61,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":575.5,"connectEnd":575.5,"connectStart":575.5,"domainLookupEnd":575.5,"domainLookupStart":575.5,"fetchStart":575.5,"redirectEnd":0,"redirectStart":0,"requestStart":575.5,"responseEnd":636.5,"responseStart":636.5,"secureConnectionStart":575.5},{"duration":276.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":850,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":850,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1126.5999999046326,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":42,"responseStart":203,"responseEnd":209,"domLoading":206,"domInteractive":1077,"domContentLoadedEventStart":1077,"domContentLoadedEventEnd":1126,"domComplete":1869,"loadEventStart":1869,"loadEventEnd":1871,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1058.6999998092651},{"name":"bigPipe.sidebar-id.end","time":1059.5},{"name":"bigPipe.activity-panel-pipe-id.start","time":1059.6999998092651},{"name":"bigPipe.activity-panel-pipe-id.end","time":1062.2999999523163},{"name":"activityTabFullyLoaded","time":1138.2999999523163}],"measures":[],"correlationId":"c6ee5d6ffa1f33","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":93,"dbReadsTimeInMs":11,"dbConnsTimeInMs":19,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}