[MDEV-13983] Mariadb becomes unresponsive Created: 2017-10-03  Updated: 2023-10-25  Resolved: 2020-01-30

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.1.26, 10.0.32, 10.2.10, 10.3.1, 10.2
Fix Version/s: 10.3.3, 10.4.0, 10.2.25, 10.1.41

Type: Bug Priority: Major
Reporter: Sander Pilon Assignee: Marko Mäkelä
Resolution: Fixed Votes: 6
Labels: deadlock, hang
Environment:

10.2.9-MariaDB-10.2.9+maria~trusty-log

Attachments: Text File crash.log     File mysql-error-running.log.1    
Issue Links:
Blocks
is blocked by MDEV-15135 'show global status' can cause lock c... Closed
Duplicate
is duplicated by MDEV-17371 InnoDB: Warning: a long semaphore wait Closed
Problem/Incident
is caused by MDEV-11896 thd_get_error_context_description rac... Closed
Relates
relates to MDEV-13196 Remove handler::store_lock() Open
relates to MDEV-13754 TRANSACTION sometimes stay in "KILLL... Closed
relates to MDEV-13935 INSERT INTO stuck at state "Unlocking... Closed
relates to MDEV-15135 'show global status' can cause lock c... Closed
relates to MDEV-22529 thd_query_safe() isn’t, causing InnoD... Closed
relates to MDEV-32576 InnoDB deadlock output does not alway... Open
relates to MDEV-16467 MariaDB crashes because of "long sema... Closed
relates to MDEV-20547 mysqld crashes: Semaphore wait has la... Closed
relates to MDEV-21418 [Warning] InnoDB: A long semaphore wait Closed

 Description   

We recently upgraded from 10.1 to 10.2, but unfortunately MariaDB 10.2 becomes unresponsive multiple times every day. This is the log we see:

10.2.9-MariaDB-10.2.9+maria~trusty-log
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200436983552 has waited at read0read.cc line 579 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140202383660800 has waited at lock0lock.cc line 7784 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fceccddc0, Mutex LOCK_SYS created lock0lock.cc:495, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140203085694720 has waited at read0read.cc line 687 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200439097088 has waited at read0read.cc line 579 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200431699712 has waited at lock0lock.cc line 5075 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fceccddc0, Mutex LOCK_SYS created lock0lock.cc:495, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200429586176 has waited at read0read.cc line 579 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200443324160 has waited at trx0trx.cc line 664 for 303.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140202384189184 has waited at read0read.cc line 579 for 301.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200442267392 has waited at read0read.cc line 579 for 299.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140201363072768 has waited at read0read.cc line 579 for 299.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140200430114560 has waited at read0read.cc line 579 for 299.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140201362544384 has waited at trx0trx.cc line 3091 for 286.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140203127658240 has waited at lock0wait.cc line 484 for 282.00 seconds the semaphore:
 
Mutex at 0x7f8fceccddc0, Mutex LOCK_SYS created lock0lock.cc:495, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140203034810112 has waited at trx0trx.cc line 1275 for 242.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
 
--Thread 140203110872832 has waited at read0read.cc line 714 for 60.00 seconds the semaphore:
 
Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
 
 
 
 
 
....



 Comments   
Comment by Elena Stepanova [ 2017-10-08 ]

CrewOne, there should be much more output in the error log when it starts happening, could you please provide it all?

Does it eventually crash saying that there has been too long semaphore wait, or does it resolve on its own and proceed?

Please attach your cnf files, and next time it happens, grab from the server whatever you can (whatever the server is willing to return) – SHOW FULL PROCESSLIST, InnoDB status, etc.

Comment by Sander Pilon [ 2017-10-10 ]

I'm sorry. This has caused us much, much work and we decided to restore a backup to the latest 10.1 version and ignore the 10.2 branch for now.
It has proven too unstable to reliably work with. So I'm afraid I'm not able to grab any additional logfiles for you from our production server.

What I can tell you is that the server used to have paralell replication enabled with optimistic mode, but that caused problems as well. (As well on 10.1) Threads seem to deadlock with eachother and replication slows down to about 3 QPS (simple inserts). When using a single thread, everything speeds up again to 100+ QPS. But I'm not sure this is a bug, when reading the documentation this might also be by design.

For the rest the configuration has not changed since my last issues. MDEV-10419 MDEV-9674

Comment by MG [ 2018-01-02 ]

I ran in to a very similar problem on 5.7 and thought I would comment here. The problem went away if binary logging was disabled or innodb_thread_concurrency was changed from 0 to 32. The machine had 40 cores. Changing to sync_binlog=0 or innodb_purge_threads=1 settings had no impact.

Some status:

OS WAIT ARRAY INFO: reservation count 17759912
 
--Thread
 
140148197480192 has waited at row0sel.cc line 3511 for 48.00 seconds
 
the semaphore:
 
S-lock on RW-latch at 0x7f4e155518e0 created in file buf0buf.cc line 1460
 
a writer (thread id 140204002961152) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file row0sel.cc line 3511
 
Last time write locked in file /export/home/pb2/build/sb_0-24964902-1505320269.78/rpm/BUILD/mysql-5.7.20/mysql-5.7.20/storage/innobase/include/mtr0mtr.ic line 153
 
--Thread
 
140144180852480 has waited at row0sel.cc line 3511 for 48.00 seconds the semaphore:
 
S-lock on RW-latch at 0x7f4e155518e0 created in file buf0buf.cc line 1460
 
a writer (thread id 140204002961152) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file row0sel.cc line 3511
 
Last time write locked in file /export/home/pb2/build/sb_0-24964902-1505320269.78/rpm/BUILD/mysql-5.7.20/mysql-5.7.20/storage/innobase/include/mtr0mtr.ic line 153
 
--Thread
 
140076126754560 has waited at trx0trx.cc line 540 for 31.00 seconds the semaphore:
 
Mutex at 0x7f0857625a18, Mutex TRX_SYS created trx0sys.cc:544, lock var 1
 
 
 
 
 
 
 
--Thread
 
140256075245312 has waited at row0sel.cc line 5117 for 48.00 seconds the semaphore:
 
S-lock on RW-latch at 0x7f4e155518e0 created in file buf0buf.cc line 1460
 
a writer (thread id 140204002961152) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file row0sel.cc line 3511
 
Last time write locked in file /export/home/pb2/build/sb_0-24964902-1505320269.78/rpm/BUILD/mysql-5.7.20/mysql-5.7.20/storage/innobase/include/mtr0mtr.ic line 153
 
--Thread 140234541688576 has waited at row0ins.cc line 2997 for 50.00 seconds the semaphore:
 
X-lock on RW-latch at 0x7f2cb01f5e08 created in file buf0buf.cc line 1460 
a writer (thread id 140184718001920) has reserved it in mode  wait exclusive
 
number of readers 2, waiters flag 1, lock_word: fffffffffffffffe
 
Last time read locked in file row0sel.cc line 5117
 
Last time write locked in file /export/home/pb2/build/sb_0-24964902-1505320269.78/rpm/BUILD/mysql-5.7.20/mysql-5.7.20/storage/innobase/row/row0ins.cc line 2997
 
--Thread
 
140109085779712 has waited at trx0trx.cc line 540 for 31.00 seconds the semaphore:
 
Mutex at 0x7f0857625a18, Mutex TRX_SYS created trx0sys.cc:544, lock var 1
 
 
 
--Thread
 
140210269251328 has waited at row0sel.cc line 3751 for 49.00 seconds the semaphore:
 
S-lock on RW-latch at 0x7ee395221848 created in file buf0buf.cc line 1460
 
a writer (thread id 140021458196224) has reserved it in mode  wait exclusive
 
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
 
Last time read locked in file row0sel.cc line 3751
 
Last time write locked in file /export/home/pb2/build/sb_0-24964902-1505320269.78/rpm/BUILD/mysql-5.7.20/mysql-5.7.20/storage/innobase/row/row0ins.cc line 2997
 
--Thread
 
140132930746112 has waited at read0read.cc line 579 for 48.00 seconds
 
the semaphore:
 
Mutex at 0x7f0857625a18, Mutex TRX_SYS created trx0sys.cc:544, lock var 0
 
 
 
 
 
--Thread
 
140089830078208 has waited at trx0trx.cc line 540 for 35.00 seconds
 
the semaphore:
 
Mutex at 0x7f0857625a18, Mutex TRX_SYS created trx0sys.cc:544, lock var 0
 
 
 
 
 
--Thread
 
140092535404288 has waited at trx0trx.cc line 540 for 37.00 seconds
 
the semaphore:
 
Mutex at 0x7f0857625a18, Mutex TRX_SYS created trx0sys.cc:544, lock var 1
 
 
 
 
 
--Thread
 
140072104949504 has waited at trx0trx.cc line 540 for 31.00 seconds
 
the semaphore:
 
Mutex at 0x7f0857625a18, Mutex TRX_SYS created trx0sys.cc:544, lock var 1

Comment by Jean-François Gagné [ 2018-01-20 ]

I also hit what looks like a very similar issue when running Optimistic Parallel Replication with 128 or more threads. Lowering innodb_thread_concurrency to 32 does not solve the problem in my case.

Comment by Sergey Vojtovich [ 2018-03-09 ]

In 10.3 trx_sys was refactored and trx_sys_t mutex is now acquired a lot less frequently and for much shorter duration. Still there's one trx_sys_t mutex acquisition left, which is easy to eliminate.

10.2 should be debugged - there must be some bug here.

Comment by Marko Mäkelä [ 2018-12-31 ]

Could this possibly share a root cause with MDEV-13935, which was fixed in 10.2.14?

Comment by Daniel Carrasco [ 2019-02-04 ]

Hello,

I don't know if was fixed on 10.2.14, but I've got a similar problem on:
mysql Ver 15.1 Distrib 10.2.21-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

The database was frozen with no reason and giving the same errors on log:

2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680337733376 has waited at read0read.cc line 584 for 488.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680927573760 has waited at read0read.cc line 579 for 488.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680337528576 has waited at read0read.cc line 579 for 488.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139703951984384 has waited at read0read.cc line 579 for 488.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680642549504 has waited at read0read.cc line 584 for 487.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680641730304 has waited at read0read.cc line 579 for 487.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680927164160 has waited at read0read.cc line 579 for 487.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680346130176 has waited at read0read.cc line 579 for 487.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680337938176 has waited at read0read.cc line 579 for 487.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680341829376 has waited at trx0trx.cc line 2946 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680339986176 has waited at trx0trx.cc line 1194 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680342648576 has waited at read0read.cc line 579 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680641320704 has waited at trx0trx.cc line 458 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680340190976 has waited at trx0trx.cc line 458 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680641115904 has waited at trx0trx.cc line 1194 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139703945516800 has waited at trx0trx.cc line 458 for 486.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680338962176 has waited at read0read.cc line 579 for 485.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680342853376 has waited at trx0trx.cc line 458 for 485.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680345310976 has waited at trx0trx.cc line 458 for 485.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680340600576 has waited at trx0trx.cc line 458 for 485.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680336709376 has waited at trx0trx.cc line 458 for 484.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680336299776 has waited at trx0trx.cc line 458 for 484.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680335480576 has waited at trx0trx.cc line 458 for 484.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680926754560 has waited at trx0rec.cc line 2259 for 483.00 seconds the semaphore:
 
S-lock on RW-latch at 0x5601513e30a0 created in file trx0purge.cc line 206
 
a writer (thread id 139680725071616) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file trx0rec.cc line 2259
 
Last time write locked in file trx0purge.cc line 1690
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680971949824 has waited at lock0wait.cc line 501 for 482.00 seconds the semaphore:
 
Mutex at 0x7f0f5c4fc050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
 
 
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680338142976 has waited at trx0rec.cc line 2259 for 420.00 seconds the semaphore:
 
S-lock on RW-latch at 0x5601513e30a0 created in file trx0purge.cc line 206
 
a writer (thread id 139680725071616) has reserved it in mode  exclusive
 
number of readers 0, waiters flag 1, lock_word: 0
 
Last time read locked in file trx0rec.cc line 2259
 
Last time write locked in file trx0purge.cc line 1690
 
2019-02-04 15:14:03 139680963557120 [Note] InnoDB: A semaphore wait:
 
--Thread 139680955164416 has waited at read0read.cc line 714 for 244.00 seconds the semaphore:
 
Mutex at 0x5601513b6630, Mutex TRX_SYS created trx0sys.cc:552, lock var 2

I can't connect, so I cannot check if a process is locking the DB or something similar. The only way to recover the DB is restarting the daemon.

Maybe is related with this bug?.

Greetings!.

Comment by Andrei Elkin [ 2019-02-04 ]

Danixu86 Could you please provide show processlist for us to see what can be the execution blocker?
Notice also MDEV-15135 fixes a case that could lead to this case description. I am not sure that it's the only one though.

Comment by Daniel Carrasco [ 2019-02-04 ]

@Elkin

Hello,
Thanks for your response, but just now the DataBase is working again because I've restarted the service. When was failing I was unable to connect so I was unable to get a processlist. Is there any other way to get it?

Greetings

Comment by Andrei Elkin [ 2019-02-05 ]

Danixu86. Sorry, no other way but by querying.

Comment by Marko Mäkelä [ 2019-02-05 ]

Danixu86, your observation looks like an InnoDB bug (possibly sharing a root cause with MDEV-11624, which was reported for 10.1). During this hang, can you please obtain stack traces from all running threads?

gdb -ex 'set height 0' -ex 'thread apply all backtrace' -ex 'quit' -p $(pgrep -x mysqld)

Comment by Daniel Carrasco [ 2019-02-05 ]

Hello,

I'll try next time. I hope not too soon, because is a big production database and a hang is a big problem.

Greetings!

Comment by Brandon Ginsberg [ 2019-03-21 ]

Hi Guys,

We encountered something similar on a production AWS RDS instance. Here's a portion of the error log. It seemed to be hanging before I manually initiated a failover:

InnoDB: Warning: semaphore wait:
--Thread 47876001244928 has waited at trx0trx.cc line 339 for 920.00 seconds the semaphore:
Mutex at 0x2b8435743068 '&trx_sys->mutex', lock var 1
Last time reserved by thread 47876026271488 in file not yet reserved line 0, waiters flag 1

This occurred hundreds of times in the log.
Running 10.1.34.

Comment by Steve [ 2019-04-02 ]

We have had this (or very similar) happen twice in the last two weeks on 10.2.17. Using our logs, we have been looking for what might have occurred 241 seconds before the first log of the 'long semaphore wait'. In both cases, we found that we have 'timed out' a long running query by killing the queries connection. We have a script that does this and has been running this way for many years on MariaDB 5.5. We have only started seeing these crashes since upgrading to 10.2. This may be coincidence, but I mention it in case it helps identify this issue. A full log of he stall and crash is attached: crash.log

From mysql's error log, starts off

My title

 
2019-04-01 21:52:15 139798828275456 [Warning] InnoDB: A long semaphore wait:
 
--Thread 139798786311936 has waited at read0read.cc line 687 for 241.00 seconds the semaphore:
 
Mutex at 0x7f5dac87b400, Mutex TRX_SYS created trx0sys.cc:555, lock var 2
 
 
 
2019-04-01 21:52:15 139798828275456 [Warning] InnoDB: A long semaphore wait:
 
--Thread 140029259699968 has waited at lock0lock.cc line 6970 for 241.00 seconds the semaphore:
 
Mutex at 0x7f2715206040, Mutex LOCK_SYS created lock0lock.cc:489, lock var 2
 
 
 
2019-04-01 21:52:15 139798828275456 [Warning] InnoDB: A long semaphore wait:
 
--Thread 140023312725760 has waited at trx0trx.cc line 2976 for 241.00 seconds the semaphore:
 
Mutex at 0x7f5dac87b400, Mutex TRX_SYS created trx0sys.cc:555, lock var 2
 
 
 
2019-04-01 21:52:15 139798828275456 [Warning] InnoDB: A long semaphore wait:
 
--Thread 140014680241920 has waited at trx0sys.ic line 365 for 241.00 seconds the semaphore:
 
Mutex at 0x7f5dac87b400, Mutex TRX_SYS created trx0sys.cc:555, lock var 2
 
 
 
2019-04-01 21:52:15 139798828275456 [Warning] InnoDB: A long semaphore wait:
 
--Thread 139798803097344 has waited at trx0trx.cc line 1221 for 241.00 seconds the semaphore:
 
Mutex at 0x7f5dac87b400, Mutex TRX_SYS created trx0sys.cc:555, lock var 2

and repeats for 12 minutes before the database crashes. During this time, the database is unresponsive. In the logs, an innodb engine status is dumped.

My title

 
019-04-01 22:04:07 139798828275456 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
 
190401 22:04:07 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.17-MariaDB-10.2.17+maria~xenial-log
 
key_buffer_size=1073741824
 
read_buffer_size=131072
 
max_used_connections=521
 
max_threads=2002
 
thread_count=557
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3397620 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f5dae3cd7e5]
 
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f5dae46f15c]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x117160)[0x7f5dae46d160]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x1190a7)[0x7f5dae46f0a7]
 
/usr/sbin/mysqld(my_addr_resolve+0xde)[0x55efe62cc9ee]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1e2)[0x55efe62b3992]
 
/usr/sbin/mysqld(handle_fatal_signal+0x345)[0x55efe5d4e4e5]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f5daedbc390]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f5dae38b428]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f5dae38d02a]
 
/usr/sbin/mysqld(+0xa09f68)[0x55efe60faf68]
 
/usr/sbin/mysqld(+0x9b274c)[0x55efe60a374c]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f5daedb26ba]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5dae45d41d]

Comment by Marko Mäkelä [ 2019-08-16 ]

I would like to assign this back to replication, based on an earlier comment:

The problem went away if binary logging was disabled

Comment by Brandon Ginsberg [ 2019-09-19 ]

We had another occurrence of this last night. Log reports it occurred around 23:55 and a thread had been waiting for roughly 900s before the semaphore wait limit crashed the server. Not sure how it reached 900s if the limit is 600s?

Version: 10.2.21 on AWS RDS

Log is attached for reference:
mysql-error-running.log.1

Comment by Marko Mäkelä [ 2019-09-19 ]

The 600-second limit is some kind of a ‘soft limit’ that avoids false positive alerts from the InnoDB watchdog. Before I fixed it years ago in MySQL, you could get bogus watchdog kills on a heavily loaded system, possibly because the same mutex happened to be occupied (by varying threads) every time a sample was taken. The hard limit has always been 900 seconds (15 minutes) as long as I remember.

Comment by Marko Mäkelä [ 2020-01-30 ]

It looks like this bug was introduced in an attempt to fix MDEV-11896 and fixed for 10.3.3 in a cleanup that was in MDEV-16467 partially backported to 10.1.41 and 10.2.25.

The history of this goes all the way back to 2004, when the LOCK_thread_count acquisition was introduced in InnoDB. Back then, the logic was clear and correct: Mutexes outside InnoDB are conceptually above any InnoDB mutex or rw-lock in the latching order, and thus they should not be acquired while holding any InnoDB mutex. For displaying the details of other sessions that participate in a deadlock between InnoDB transactions, InnoDB would first acquire LOCK_thread_count upfront, and only after that acquire its own mutexes (originally kernel_mutex, later trx_sys_t::mutex, lock_sys_t::mutex, trx_t::mutex). At some point, the LOCK_thread_count acquisition was removed from the ‘upper level’ along with the function innobase_mysql_prepare_print_arbitrary_thd(). Finally, the attempted MDEV-11896 fix broke the assumptions by introducing the offending mutex acquisition at the low level.

We did some analysis of stack trace dumps of a support customer using MariaDB Server 10.2.21. The deadlocks that they were experiencing involve 3 threads:

  • Thread A (user transaction deadlock reporter in InnoDB) is holding lock_sys_t::mutex and waiting for LOCK_thread_count.
  • Thread B (handle_slave_background() but it could also be something else handling a KILL statement) acquired THD::LOCK_ha_data on the victim’s handle before entering THD::awake(), and is waiting for lock_sys_t::mutex in lock_trx_handle_wait(), invoked via innobase_hton->kill_query = innobase_kill_query.
  • Thread C (SELECT…FROM INFORMATION_SCHEMA.PROCESSLIST or SHOW PROCESSLIST) acquired LOCK_thread_count in fill_schema_processlist() or mysqld_list_processes and is waiting for THD::lock_ha_data later in the same function, on the KILL victim’s handle.

This deadlock was fixed by removing the offending wait in Thread A, in the code that InnoDB calls while holding its own mutexes.

Note: In MySQL 5.7 and MariaDB 10.2.2, the InnoDB deadlock checker was refactored. Before that, the mutex functionality was basically the same: lock_deadlock_trx_print() is holding lock_sys->mutex and trx_sys->mutex before invoking innobase_mysql_print_thd() in trx_print_low(). Starting with 10.2.2, the diagnostic output is initiated by DeadlockChecker::notify().

Comment by Sander Pilon [ 2020-01-30 ]

Wow. Thanks for fixing this. We upgraded to Percona early 2019 because of this, but good to know it's fixed.

Comment by Marko Mäkelä [ 2020-01-31 ]

I repeated this hang locally. My test ran for a couple of seconds from the server startup until the deadlock occurred. After reverting the attempted fix of MDEV-11896, the server would not hang. I interrupted the test after 3 minutes.

I will not disclose my scripts (involving 4 clients, 2 of them causing user transaction deadlocks when accessing an InnoDB table), because it might make the life too easy for script kiddies.

Here is the server error log output:

mariadb-10.2.21

 
2020-01-31  8:28:22 140120539419456 [Note] ../sql/mysqld: ready for connections.
 
Version: '10.2.21-MariaDB'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
 
2020-01-31  8:32:27 140116577302272 [Warning] InnoDB: A long semaphore wait:
 
--Thread 140120359331584 has waited at lock0lock.cc line 6956 for 241.00 seconds the semaphore:
 
Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
 
 
 
2020-01-31  8:32:27 140116577302272 [Note] InnoDB: A semaphore wait:
 
--Thread 140120359331584 has waited at lock0lock.cc line 6956 for 241.00 seconds the semaphore:
 
Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
 
 
 
2020-01-31  8:32:27 140116577302272 [Note] InnoDB: A semaphore wait:
 
--Thread 140116746655488 has waited at lock0wait.cc line 501 for 190.00 seconds the semaphore:
 
Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
 
 
 
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
 
InnoDB: Pending reads 0, writes 0

Here are the stack traces of the hung server process:

mariadb-10.2.21

 
Thread 34 (Thread 0x7f705039b700 (LWP 26484)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6f10006644) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6f100065f0, cond=0x7f6f10006618) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x7f6f10006618, mutex=0x7f6f100065f0) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x7f6f100065e0) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x7f6f100065e0, reset_sig_count=406) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba12bcbfa in lock_wait_suspend_thread (thr=0x7f6f2402c458) at /mariadb/server/storage/innobase/lock/lock0wait.cc:355
 
#6  0x0000557ba131cf74 in row_mysql_handle_errors (new_err=0x7f7050397cdc, trx=0x7f7051ebc118, thr=0x7f6f2402c458, savept=0x0) at /mariadb/server/storage/innobase/row/row0mysql.cc:741
 
#7  0x0000557ba1336dab in row_search_mvcc (buf=<optimized out>, mode=PAGE_CUR_GE, prebuilt=0x7f6f2402bd90, match_mode=<optimized out>, direction=<optimized out>) at /mariadb/server/storage/innobase/row/row0sel.cc:5693
 
#8  0x0000557ba126c3ed in ha_innobase::index_read (this=0x7f6f2402b620, buf=0x7f6f2402b238 "\377", key_ptr=0x7f6f10011a58 "\001", key_len=4, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:9431
 
#9  0x0000557ba10eb70a in handler::index_read_idx_map (this=0x7f6f2402b620, buf=0x7f6f2402b238 "\377", index=0, key=0x7f6f10011a58 "\001", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:5487
 
#10 0x0000557ba10e5f99 in handler::ha_index_read_idx_map (this=0x7f6f2402b620, buf=0x80 <error: Cannot access memory at address 0x80>, index=<optimized out>, key=0x7f6f10011a58 "\001", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:2657
 
#11 0x0000557ba0fa8606 in join_read_const (tab=0x7f6f10010928) at /mariadb/server/sql/sql_select.cc:19336
 
#12 0x0000557ba0fa81a2 in join_read_const_table (thd=0x7f6f10000c48, tab=0x7f6f10010928, pos=0x7f6f10010ee0) at /mariadb/server/sql/sql_select.cc:19216
 
#13 0x0000557ba0f7dd05 in make_join_statistics (join=0x7f6f1000fea8, tables_list=..., keyuse_array=0x7f6f10010198) at /mariadb/server/sql/sql_select.cc:4310
 
#14 JOIN::optimize_inner (this=<optimized out>) at /mariadb/server/sql/sql_select.cc:1582
 
#15 0x0000557ba0f7bbda in JOIN::optimize (this=0x7f6f1000fea8) at /mariadb/server/sql/sql_select.cc:1115
 
#16 0x0000557ba0f7980e in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /mariadb/server/sql/sql_select.cc:3804
 
#17 0x0000557ba0f7964f in handle_select (thd=0x7f6f10000c48, lex=0x7f6f100045a0, result=0x7f6f1000fe88, setup_tables_done_option=<optimized out>) at /mariadb/server/sql/sql_select.cc:364
 
#18 0x0000557ba0f5b38d in execute_sqlcom_select (thd=0x7f6f10000c48, all_tables=<optimized out>) at /mariadb/server/sql/sql_parse.cc:6481
 
#19 0x0000557ba0f53b5d in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:3487
 
#20 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f10000c48, rawbuf=0x7f6f1000f2a0 "select * from t1 where id=1 for update", length=<optimized out>, parser_state=0x7f705039a350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
 
#21 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f10000c48, packet=0x7f6f10006f89 "select * from t1 where id=1 for update", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
 
#22 0x0000557ba0f50220 in do_command (thd=0x7f6f10000c48) at /mariadb/server/sql/sql_parse.cc:1378
 
#23 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
 
#24 0x0000557ba1023973 in handle_one_connection (arg=0x557ba85fa0d8) at /mariadb/server/sql/sql_connect.cc:1241
 
#25 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#26 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 33 (Thread 0x7f70503e6700 (LWP 26472)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82a6f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82a6f10, cond=0x557ba82a6f38) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba82a6f38, mutex=0x557ba82a6f10) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82a6f00) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba82a6f00, reset_sig_count=3) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f70503e4118: 0x7f70522bb010) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469
 
#6  0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x7f705003d050, max_spins=30, max_delay=6, filename=0x557ba169c700 "/mariadb/server/storage/innobase/lock/lock0lock.cc", line=6956) at /mariadb/server/storage/innobase/include/ib0mutex.h:516
 
#7  0x0000557ba12b9808 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x7f705003d050, n_spins=30, n_delay=6, name=<optimized out>, line=6956) at /mariadb/server/storage/innobase/include/ib0mutex.h:637
 
#8  lock_trx_handle_wait (trx=0x7f7051ebb098) at /mariadb/server/storage/innobase/lock/lock0lock.cc:6956
 
#9  0x0000557ba10e32b4 in kill_handlerton (thd=0x7f6f24000c48, plugin=<optimized out>, level=0x7f70503e42f4) at /mariadb/server/sql/handler.cc:809
 
#10 0x0000557ba0f6431f in plugin_foreach_with_mask (thd=0x7f6f24000c48, func=0x557ba10e3280 <kill_handlerton(THD*, st_plugin_int*, void*)>, type=<optimized out>, state_mask=8, arg=0x7f70503e42f4) at /mariadb/server/sql/sql_plugin.cc:2396
 
#11 0x0000557ba10e3270 in ha_kill_query (thd=0x557ba82a6f60, level=<optimized out>) at /mariadb/server/sql/handler.cc:816
 
#12 0x0000557ba0f204ad in THD::awake (this=0x7f6f24000c48, state_to_set=KILL_QUERY_HARD) at /mariadb/server/sql/sql_class.cc:1715
 
#13 0x0000557ba0f5ebf2 in kill_one_thread (thd=0x7f6f1c000c48, id=<optimized out>, kill_signal=KILL_QUERY_HARD, type=KILL_TYPE_ID) at /mariadb/server/sql/sql_parse.cc:8903
 
#14 0x0000557ba0f584e8 in sql_kill (thd=0x7f6f1c000c48, id=8, state=NOT_KILLED, type=(KILL_TYPE_USER | unknown: 1538883012)) at /mariadb/server/sql/sql_parse.cc:9009
 
#15 mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:5475
 
#16 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f1c000c48, rawbuf=0x7f6f1c00f2a0 "kill query 8", length=<optimized out>, parser_state=0x7f70503e5350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
 
#17 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f1c000c48, packet=0x7f6f1c006f89 "kill query 8", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
 
#18 0x0000557ba0f50220 in do_command (thd=0x7f6f1c000c48) at /mariadb/server/sql/sql_parse.cc:1378
 
#19 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
 
#20 0x0000557ba1023973 in handle_one_connection (arg=0x557ba85fa698) at /mariadb/server/sql/sql_connect.cc:1241
 
#21 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#22 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 32 (Thread 0x7f7050431700 (LWP 26471)):
 
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
 
#1  0x00007f705bb948d9 in __GI___pthread_mutex_lock (mutex=0x7f6f240024d8) at ../nptl/pthread_mutex_lock.c:135
 
#2  0x0000557ba0fb3e61 in inline_mysql_mutex_lock (that=0x7f6f240024d8, src_file=<optimized out>, src_line=2623) at /mariadb/server/include/mysql/psi/mysql_thread.h:683
 
#3  mysqld_list_processes (thd=0x7f6f18000c48, user=0x0, verbose=<optimized out>) at /mariadb/server/sql/sql_show.cc:2623
 
#4  0x0000557ba0f57776 in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:4776
 
#5  0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f18000c48, rawbuf=0x7f6f1800f2a0 "show processlist", length=<optimized out>, parser_state=0x7f7050430350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
 
#6  0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f18000c48, packet=0x7f6f18006f89 "", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
 
#7  0x0000557ba0f50220 in do_command (thd=0x7f6f18000c48) at /mariadb/server/sql/sql_parse.cc:1378
 
#8  0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
 
#9  0x0000557ba1023973 in handle_one_connection (arg=0x557ba82f2a98) at /mariadb/server/sql/sql_connect.cc:1241
 
#10 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#11 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 31 (Thread 0x7f705047c700 (LWP 26466)):
 
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
 
#1  0x00007f705bb948d9 in __GI___pthread_mutex_lock (mutex=0x557ba1c25530 <LOCK_thread_count>) at ../nptl/pthread_mutex_lock.c:135
 
#2  0x0000557ba0fc4fbb in inline_mysql_mutex_lock (that=0xfffffffffffffe00, src_file=<optimized out>, src_line=9923) at /mariadb/server/include/mysql/psi/mysql_thread.h:683
 
#3  thd_get_error_context_description (thd=0x7f6f10000c48, buffer=0x7f7050477fe0 "H\264*\250{U", length=1024, max_query_len=3000) at /mariadb/server/sql/sql_show.cc:9923
 
#4  0x0000557ba1263a71 in innobase_mysql_print_thd (f=0x557ba82a7230, thd=0x557ba1c25530 <LOCK_thread_count>, max_query_len=1538895916) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:2153
 
#5  0x0000557ba12b9f21 in DeadlockChecker::print (trx=0x7f7051ebc118, max_query_len=3) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7224
 
#6  0x0000557ba12ba28d in DeadlockChecker::notify (this=0x7f70504784e8, lock=0x7f7051ebb140) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7371
 
#7  0x0000557ba12ba8d6 in DeadlockChecker::search (this=0x7f70504784e8) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7487
 
#8  0x0000557ba12af6ac in DeadlockChecker::check_and_resolve (lock=0x7f7051ebb298, trx=0x7f7051ebb098) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7622
 
#9  0x0000557ba12af0a1 in lock_rec_enqueue_waiting (c_lock=<optimized out>, type_mode=<optimized out>, block=<optimized out>, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>, prdt=0x0) at /mariadb/server/storage/innobase/lock/lock0lock.cc:1796
 
#10 0x0000557ba12b7a21 in lock_rec_lock_slow (impl=<optimized out>, mode=<optimized out>, block=0x7f7043cfe140, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>) at /mariadb/server/storage/innobase/lock/lock0lock.cc:2113
 
#11 lock_rec_lock (impl=<optimized out>, mode=<optimized out>, block=0x7f7043cfe140, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>) at /mariadb/server/storage/innobase/lock/lock0lock.cc:2178
 
#12 0x0000557ba12b8150 in lock_clust_rec_read_check_and_lock (flags=<optimized out>, block=0x7f7043cfe140, rec=<optimized out>, index=0x7f6f240221b0, offsets=<optimized out>, mode=<optimized out>, gap_mode=<optimized out>, thr=0x7f6f24019898) at /mariadb/server/storage/innobase/lock/lock0lock.cc:6393
 
#13 0x0000557ba133a1fa in sel_set_rec_lock (pcur=<optimized out>, rec=0x7f70444d0093 "\200", index=0x7f6f240221b0, offsets=<optimized out>, mode=<optimized out>, type=<optimized out>, thr=0x7f6f24019898, mtr=0x7f70504790f8) at /mariadb/server/storage/innobase/row/row0sel.cc:1261
 
#14 0x0000557ba1337dde in row_search_mvcc (buf=<optimized out>, mode=PAGE_CUR_GE, prebuilt=0x7f6f240191d0, match_mode=<optimized out>, direction=<optimized out>) at /mariadb/server/storage/innobase/row/row0sel.cc:5090
 
#15 0x0000557ba126c3ed in ha_innobase::index_read (this=0x7f6f24018a60, buf=0x7f6f24018678 "\377", key_ptr=0x7f6f24011a58 "\002", key_len=4, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:9431
 
#16 0x0000557ba10eb70a in handler::index_read_idx_map (this=0x7f6f24018a60, buf=0x7f6f24018678 "\377", index=0, key=0x7f6f24011a58 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:5487
 
#17 0x0000557ba10e5f99 in handler::ha_index_read_idx_map (this=0x7f6f24018a60, buf=0x80 <error: Cannot access memory at address 0x80>, index=<optimized out>, key=0x7f6f24011a58 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:2657
 
#18 0x0000557ba0fa8606 in join_read_const (tab=0x7f6f24010928) at /mariadb/server/sql/sql_select.cc:19336
 
#19 0x0000557ba0fa81a2 in join_read_const_table (thd=0x7f6f24000c48, tab=0x7f6f24010928, pos=0x7f6f24010ee0) at /mariadb/server/sql/sql_select.cc:19216
 
#20 0x0000557ba0f7dd05 in make_join_statistics (join=0x7f6f2400fea8, tables_list=..., keyuse_array=0x7f6f24010198) at /mariadb/server/sql/sql_select.cc:4310
 
#21 JOIN::optimize_inner (this=<optimized out>) at /mariadb/server/sql/sql_select.cc:1582
 
#22 0x0000557ba0f7bbda in JOIN::optimize (this=0x7f6f2400fea8) at /mariadb/server/sql/sql_select.cc:1115
 
#23 0x0000557ba0f7980e in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /mariadb/server/sql/sql_select.cc:3804
 
#24 0x0000557ba0f7964f in handle_select (thd=0x7f6f24000c48, lex=0x7f6f240045a0, result=0x7f6f2400fe88, setup_tables_done_option=<optimized out>) at /mariadb/server/sql/sql_select.cc:364
 
#25 0x0000557ba0f5b38d in execute_sqlcom_select (thd=0x7f6f24000c48, all_tables=<optimized out>) at /mariadb/server/sql/sql_parse.cc:6481
 
#26 0x0000557ba0f53b5d in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:3487
 
#27 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f24000c48, rawbuf=0x7f6f2400f2a0 "select * from t1 where id=2 for update", length=<optimized out>, parser_state=0x7f705047b350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
 
#28 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f24000c48, packet=0x7f6f24006f89 "select * from t1 where id=2 for update", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
 
#29 0x0000557ba0f50220 in do_command (thd=0x7f6f24000c48) at /mariadb/server/sql/sql_parse.cc:1378
 
#30 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
 
#31 0x0000557ba1023973 in handle_one_connection (arg=0x557ba8608058) at /mariadb/server/sql/sql_connect.cc:1241
 
#32 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#33 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 30 (Thread 0x7f70504c7700 (LWP 26463)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba1c2306c <COND_slave_background+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba1c25830 <LOCK_slave_background>, cond=0x557ba1c23040 <COND_slave_background>) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba1c23040 <COND_slave_background>, mutex=0x557ba1c25830 <LOCK_slave_background>) at pthread_cond_wait.c:655
 
#3  0x0000557ba0edca0b in inline_mysql_cond_wait (that=0x557ba1c23040 <COND_slave_background>, mutex=<optimized out>, src_file=<optimized out>, src_line=334) at /mariadb/server/include/mysql/psi/mysql_thread.h:1149
 
#4  handle_slave_background (arg=<optimized out>) at /mariadb/server/sql/slave.cc:334
 
#5  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#6  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 29 (Thread 0x7f7050512700 (LWP 26462)):
 
#0  0x00007f705b12ddd2 in __GI___sigtimedwait (set=set@entry=0x7f7050511dc8, info=info@entry=0x7f7050511cb0, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
 
#1  0x00007f705bb9c07c in __sigwait (set=0x7f7050511dc8, sig=0x7f7050511d60) at ../sysdeps/unix/sysv/linux/sigwait.c:28
 
#2  0x0000557ba0ec317b in signal_hand (arg=<optimized out>) at /mariadb/server/sql/mysqld.cc:3513
 
#3  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#4  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 28 (Thread 0x7f6f4daaf700 (LWP 26461)):
 
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f4daaed98, expected=0, futex_word=0x557ba249f178 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
 
#1  __pthread_cond_wait_common (abstime=0x7f6f4daaed98, mutex=0x557ba249f118 <LOCK_checkpoint>, cond=0x557ba249f150 <COND_checkpoint>) at pthread_cond_wait.c:539
 
#2  __pthread_cond_timedwait (cond=0x557ba249f150 <COND_checkpoint>, mutex=0x557ba249f118 <LOCK_checkpoint>, abstime=0x7f6f4daaed98) at pthread_cond_wait.c:667
 
#3  0x0000557ba14bbb5f in inline_mysql_cond_timedwait (that=0x557ba249f150 <COND_checkpoint>, mutex=0x557ba249f118 <LOCK_checkpoint>, abstime=0x7f6f4daaed98, src_file=<optimized out>, src_line=116) at /mariadb/server/include/mysql/psi/mysql_thread.h:1186
 
#4  my_service_thread_sleep (control=0x557ba1ba82b0 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/server/storage/maria/ma_servicethread.c:115
 
#5  0x0000557ba14b4c4c in ma_checkpoint_background (arg=<optimized out>) at /mariadb/server/storage/maria/ma_checkpoint.c:709
 
#6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 27 (Thread 0x7f6f4e7fc700 (LWP 26460)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6f4e7fbe00) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6f4e7fbda8, cond=0x7f6f4e7fbdd8) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x7f6f4e7fbdd8, mutex=0x7f6f4e7fbda8) at pthread_cond_wait.c:655
 
#3  0x0000557ba126327b in inline_mysql_cond_wait (that=0x7f6f4e7fbdd8, mutex=0x7f6f4e7fbda8, src_file=<optimized out>, src_line=332) at /mariadb/server/include/mysql/psi/mysql_thread.h:1149
 
#4  thd_destructor_proxy () at /mariadb/server/storage/innobase/handler/ha_innodb.cc:332
 
#5  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#6  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 26 (Thread 0x7f6f4effd700 (LWP 26459)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327b30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327ae0, cond=0x557ba4327b08) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba4327b08, mutex=0x557ba4327ae0) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327ad0) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba4327ad0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba13c9ee2 in buf_resize_thread () at /mariadb/server/storage/innobase/buf/buf0buf.cc:3102
 
#6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 25 (Thread 0x7f6f4f7fe700 (LWP 26458)):
 
#0  0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f4f7fd7a8, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
 
#1  0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225
 
#2  0x0000557ba13c675a in btr_defragment_thread () at /mariadb/server/storage/innobase/btr/btr0defragment.cc:709
 
#3  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#4  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 24 (Thread 0x7f6f4ffff700 (LWP 26457)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327a50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327a00, cond=0x557ba4327a28) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba4327a28, mutex=0x557ba4327a00) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43279f0) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba43279f0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba13deb60 in buf_dump_thread () at /mariadb/server/storage/innobase/buf/buf0dump.cc:807
 
#6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 23 (Thread 0x7f6f6c8e6700 (LWP 26456)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327900) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba43278b0, cond=0x557ba43278d8) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba43278d8, mutex=0x557ba43278b0) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43278a0) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba43278a0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba6968 <srv_sys+360>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
 
#6  srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580
 
#7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 22 (Thread 0x7f6f6d0e7700 (LWP 26455)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327890) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327840, cond=0x557ba4327868) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba4327868, mutex=0x557ba4327840) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327830) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba4327830, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba6930 <srv_sys+304>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
 
#6  srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580
 
#7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 21 (Thread 0x7f6f6d8e8700 (LWP 26454)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327820) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba43277d0, cond=0x557ba43277f8) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba43277f8, mutex=0x557ba43277d0) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43277c0) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba43277c0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba68f8 <srv_sys+248>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
 
#6  srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580
 
#7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 20 (Thread 0x7f6f76690700 (LWP 26453)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba43277b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327760, cond=0x557ba4327788) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba4327788, mutex=0x557ba4327760) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327750) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba4327750, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba135b821 in srv_resume_thread (slot=<optimized out>, sig_count=<optimized out>, wait=<optimized out>, timeout_usec=10000) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
 
#6  srv_purge_coordinator_suspend (slot=<optimized out>, rseg_history_len=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2730
 
#7  srv_purge_coordinator_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2819
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 19 (Thread 0x7f6f76e91700 (LWP 26452)):
 
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f76e90bc0, expected=0, futex_word=0x557ba8306b50) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
 
#1  __pthread_cond_wait_common (abstime=0x7f6f76e90bc0, mutex=0x557ba8306b00, cond=0x557ba8306b28) at pthread_cond_wait.c:539
 
#2  __pthread_cond_timedwait (cond=0x557ba8306b28, mutex=0x557ba8306b00, abstime=0x7f6f76e90bc0) at pthread_cond_wait.c:667
 
#3  0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f76e90bc0) at /mariadb/server/storage/innobase/os/os0event.cc:279
 
#4  0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba8306af0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:406
 
#5  0x0000557ba13995bf in ib_wqueue_timedwait (wq=0x557ba42f6f20, wait_in_usecs=5000000) at /mariadb/server/storage/innobase/ut/ut0wqueue.cc:161
 
#6  0x0000557ba14686c4 in fts_optimize_thread (arg=0x557ba42f6f20) at /mariadb/server/storage/innobase/fts/fts0opt.cc:2897
 
#7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 18 (Thread 0x7f6f77692700 (LWP 26451)):
 
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f77691dc0, expected=0, futex_word=0x557ba82aa6f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
 
#1  __pthread_cond_wait_common (abstime=0x7f6f77691dc0, mutex=0x557ba82aa6a0, cond=0x557ba82aa6c8) at pthread_cond_wait.c:539
 
#2  __pthread_cond_timedwait (cond=0x557ba82aa6c8, mutex=0x557ba82aa6a0, abstime=0x7f6f77691dc0) at pthread_cond_wait.c:667
 
#3  0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f77691dc0) at /mariadb/server/storage/innobase/os/os0event.cc:279
 
#4  0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba82aa690, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:406
 
#5  0x0000557ba1424089 in dict_stats_thread () at /mariadb/server/storage/innobase/dict/dict0stats_bg.cc:458
 
#6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 17 (Thread 0x7f6f77e93700 (LWP 26450)):
 
#0  0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f77e92c68, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
 
#1  0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225
 
#2  0x0000557ba1359e4f in srv_master_sleep () at /mariadb/server/storage/innobase/srv/srv0srv.cc:2385
 
#3  srv_master_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2426
 
#4  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#5  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 16 (Thread 0x7f6f78694700 (LWP 26449)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82ab4f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82ab4a0, cond=0x557ba82ab4c8) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba82ab4c8, mutex=0x557ba82ab4a0) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82ab490) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba82ab490, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f6f78693ca8: 0x7f70522bb090) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469
 
#6  0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x557ba82aaa80, max_spins=30, max_delay=6, filename=0x557ba16a2e36 "/mariadb/server/storage/innobase/read/read0read.cc", line=714) at /mariadb/server/storage/innobase/include/ib0mutex.h:516
 
#7  0x0000557ba12f79f0 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x557ba82aaa80, n_spins=30, n_delay=6, name=<optimized out>, line=714) at /mariadb/server/storage/innobase/include/ib0mutex.h:637
 
#8  MVCC::size (this=0x557ba42e8940) at /mariadb/server/storage/innobase/read/read0read.cc:714
 
#9  0x0000557ba1356f86 in srv_printf_innodb_monitor (file=0x7f705b2ad680 <_IO_2_1_stderr_>, nowait=<optimized out>, trx_start_pos=<optimized out>, trx_end=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:1393
 
#10 0x0000557ba135901e in srv_monitor_thread () at /mariadb/server/storage/innobase/srv/srv0srv.cc:1752
 
#11 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#12 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 15 (Thread 0x7f6f6ed13700 (LWP 26448)):
 
#0  0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f6ed12a28, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
 
#1  0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225
 
#2  0x0000557ba1365d04 in sync_array_print_long_waits (waiter=<optimized out>, sema=<optimized out>) at /mariadb/server/storage/innobase/sync/sync0arr.cc:1104
 
#3  0x0000557ba13598c5 in srv_error_monitor_thread () at /mariadb/server/storage/innobase/srv/srv0srv.cc:1861
 
#4  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#5  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 14 (Thread 0x7f6f78e95700 (LWP 26447)):
 
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82a6f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82a6f10, cond=0x557ba82a6f38) at pthread_cond_wait.c:502
 
#2  __pthread_cond_wait (cond=0x557ba82a6f38, mutex=0x557ba82a6f10) at pthread_cond_wait.c:655
 
#3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82a6f00) at /mariadb/server/storage/innobase/os/os0event.cc:162
 
#4  os_event::wait_low (this=0x557ba82a6f00, reset_sig_count=3) at /mariadb/server/storage/innobase/os/os0event.cc:329
 
#5  0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f6f78e94da8: 0x7f70522bb050) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469
 
#6  0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x7f705003d050, max_spins=30, max_delay=6, filename=0x557ba169d35a "/mariadb/server/storage/innobase/lock/lock0wait.cc", line=501) at /mariadb/server/storage/innobase/include/ib0mutex.h:516
 
#7  0x0000557ba12bd649 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x7f705003d050, n_spins=30, n_delay=6, name=<optimized out>, line=501) at /mariadb/server/storage/innobase/include/ib0mutex.h:637
 
#8  lock_wait_check_and_cancel (slot=0x7f705003d138) at /mariadb/server/storage/innobase/lock/lock0wait.cc:501
 
#9  lock_wait_timeout_thread () at /mariadb/server/storage/innobase/lock/lock0wait.cc:569
 
#10 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#11 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 13 (Thread 0x7f6f6f719700 (LWP 26444)):
 
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f6f718b70, expected=0, futex_word=0x557ba4327ac4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
 
#1  __pthread_cond_wait_common (abstime=0x7f6f6f718b70, mutex=0x557ba4327a70, cond=0x557ba4327a98) at pthread_cond_wait.c:539
 
#2  __pthread_cond_timedwait (cond=0x557ba4327a98, mutex=0x557ba4327a70, abstime=0x7f6f6f718b70) at pthread_cond_wait.c:667
 
#3  0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f6f718b70) at /mariadb/server/storage/innobase/os/os0event.cc:279
 
#4  0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba4327a60, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/server/storage/innobase/os/os0event.cc:406
 
#5  0x0000557ba13e4047 in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/server/storage/innobase/buf/buf0flu.cc:2702
 
#6  buf_flush_page_cleaner_coordinator () at /mariadb/server/storage/innobase/buf/buf0flu.cc:3168
 
#7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 12 (Thread 0x7f6f6ff1a700 (LWP 26443)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f6ff19b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f6ff19b60, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=9) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 11 (Thread 0x7f6f7071b700 (LWP 26442)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7071ab60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7071ab60, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=8) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 10 (Thread 0x7f6f70f1c700 (LWP 26441)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f70f1bb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f70f1bb60, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=7) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 9 (Thread 0x7f6f7171d700 (LWP 26440)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7171cb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7171cb60, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=6) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 8 (Thread 0x7f6f71f1e700 (LWP 26439)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f71f1db60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f71f1db60, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=5) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 7 (Thread 0x7f6f7271f700 (LWP 26438)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7271eb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7271eb60, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=4) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 6 (Thread 0x7f6f72f20700 (LWP 26437)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f72f1fb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f72f1fb60, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=3) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 5 (Thread 0x7f6f73721700 (LWP 26436)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f73720b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f73720b60, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=2) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 4 (Thread 0x7f6f73f22700 (LWP 26435)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f73f21b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f73f21b60, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=1) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 3 (Thread 0x7f6f74723700 (LWP 26434)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
 
#2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f74722b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
 
#3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f74722b60, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
 
#4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
 
#5  os_aio_handler (segment=<optimized out>, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
 
#6  0x0000557ba1437d2a in fil_aio_wait (segment=0) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
 
#7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
 
#8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 2 (Thread 0x7f705adff700 (LWP 26433)):
 
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f705adfee00, expected=0, futex_word=0x557ba24ae818 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
 
#1  __pthread_cond_wait_common (abstime=0x7f705adfee00, mutex=0x557ba24ae7c0 <LOCK_timer>, cond=0x557ba24ae7f0 <COND_timer>) at pthread_cond_wait.c:539
 
#2  __pthread_cond_timedwait (cond=0x557ba24ae7f0 <COND_timer>, mutex=0x557ba24ae7c0 <LOCK_timer>, abstime=0x7f705adfee00) at pthread_cond_wait.c:667
 
#3  0x0000557ba15a5ed6 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=0x7f705adfee00, src_file=<optimized out>, src_line=292) at /mariadb/server/include/mysql/psi/mysql_thread.h:1186
 
#4  timer_handler (arg=<optimized out>) at /mariadb/server/mysys/thr_timer.c:292
 
#5  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
 
#6  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 1 (Thread 0x7f705afa5340 (LWP 26432)):
 
#0  0x00007f705b1e2d0f in __GI___poll (fds=0x7ffc43d161a0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
 
#1  0x0000557ba0ec8e39 in handle_connections_sockets () at /mariadb/server/sql/mysqld.cc:6619
 
#2  0x0000557ba0ec4797 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/server/sql/mysqld.cc:6085
 
#3  0x00007f705b119bbb in __libc_start_main (main=0x557ba0ebf760 <main(int, char**)>, argc=5, argv=0x7ffc43d168a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc43d16898) at ../csu/libc-start.c:308
 
#4  0x0000557ba0ebf69a in _start ()

The threads that participate in the deadlock are as follows, using the identifiers introduced in my previous comment:

  • Thread A = Thread 31
  • Thread B = Thread 33
  • Thread C = Thread 32
Generated at Thu Feb 08 08:09:55 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.