[MDEV-24606] InnoDB: Semaphore wait has lasted > 600 second Created: 2021-01-16  Updated: 2021-12-02  Resolved: 2021-04-12

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 10.4.17
Fix Version/s: 10.2.37, 10.3.28, 10.4.18, 10.5.9

Type: Bug Priority: Major
Reporter: Marcin Gryszkalis Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 1
Labels: need_feedback, semaphore
Environment:

FreeBSD 11

Attachments: Text File semaphore-bt.txt     File semaphore-error.log    
Issue Links:
Relates
relates to MDEV-24378 Crashes on Semaphore wait > 600 seconds Closed
relates to MDEV-24504 [FATAL] InnoDB: Semaphore wait has la... Closed
relates to MDEV-24275 InnoDB persistent stats analyze force... Closed
relates to MDEV-25111 Long semaphore wait (> 800 secs), ser... Closed

 Description   

After upgrading 10.4.15 to 10.4.17 I experience forced crashes every few days.
This is zabbix database (like MDEV-24504 ).
Last crash is from debug build.
I'll attach full log and backtraces

 Comments   
Comment by Marcin Gryszkalis [ 2021-01-16 ]

Maybe zabbix is just good for load testing, but more people are reporting: https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/416036-mysql-error-fatal-innodb-semaphore-wait-has-lasted-600-seconds

Comment by Marcin Gryszkalis [ 2021-01-22 ]

I have 3 more coredumps from last few days (the same version, same installation, same config) - in case you need something to compare.

Comment by Marko Mäkelä [ 2021-02-26 ]

marcin.gryszkalis, thank you. The semaphore-bt.txt suggests that this could be a duplicate of MDEV-24275. Can you please test the latest release (10.4.18)?

Comment by Marcin Gryszkalis [ 2021-03-03 ]

Looks promising - no crashes with 10.4.18 (4 days or so), thanks

Comment by Marko Mäkelä [ 2021-03-05 ]

marcin.gryszkalis, that sounds good. Please let us know if the hangs stay away, so that we may close this ticket.

Comment by Vasilisc [ 2021-12-02 ]

Dec 1 20:10:23 sp mariadbd[84090]: --Thread 140072735258368 has waited at row0ins.cc line 2974 for 618.00 seconds the semaphore:
Dec 1 20:10:23 sp mariadbd[84090]: X-lock on RW-latch at 0x7f666401ac78 created in file buf0buf.cc line 1227
Dec 1 20:10:23 sp mariadbd[84090]: a writer (thread id 0) has reserved it in mode SX
Dec 1 20:10:23 sp mariadbd[84090]: number of readers 0, waiters flag 1, lock_word: 10000000
Dec 1 20:10:23 sp mariadbd[84090]: Last time write locked in file buf0flu.cc line 837
Dec 1 20:10:23 sp mariadbd[84090]: InnoDB: Pending reads 0, writes 0
Dec 1 20:10:23 sp mariadbd[84090]: 2021-12-01 20:10:23 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

# dpkg -l|grep maria

ii libdbd-mariadb-perl 1.21-3 amd64 Perl5 database interface to the MariaDB/MySQL databases
ii libmariadb3:amd64 1:10.5.12-0+deb11u1 amd64 MariaDB database client library
ii mariadb-client-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database client binaries
ii mariadb-client-core-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database core client binaries
ii mariadb-common 1:10.5.12-0+deb11u1 all MariaDB common configuration files
ii mariadb-server-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database server binaries
ii mariadb-server-core-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database core server files

Generated at Thu Feb 08 09:31:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.