[MDEV-6487] InnoDB: Warning: a long semaphore wait, leading to crash after 600s Created: 2014-07-25  Updated: 2014-09-09  Due: 2014-09-05  Resolved: 2014-09-09

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 5.5.33a
Fix Version/s: 5.5.40

Type: Bug Priority: Major
Reporter: 21Net Assignee: Jan Lindström (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: crash, semaphore
Environment:

Ubuntu 12.04 32bit
Virtual VMWare guest on ESX cluster

Attachments: HTML File syslog    

 Description   

Our database server has been running fine for a long time, but this night it crashed.

Leading to the crash we saw a JBD2: bad log_start_commit of the filesystem in syslog, followed by:

Jul 25 07:57:12 vmdb01 mysqld: InnoDB: Warning: a long semaphore wait:
 
Jul 25 07:57:12 vmdb01 mysqld: --Thread 140114759440128 has waited at row0ins.c line 2039 for 241.00 seconds the semaphore:
 
Jul 25 07:57:12 vmdb01 mysqld: X-lock (wait_ex) on RW-latch at 0x7f701da49460 '&block->lock'
 
Jul 25 07:57:12 vmdb01 mysqld: a writer (thread id 140114759440128) has reserved it in mode  wait exclusive
 
Jul 25 07:57:12 vmdb01 mysqld: number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
 
Jul 25 07:57:12 vmdb01 mysqld: Last time read locked in file buf0flu.c line 1340
 
Jul 25 07:57:12 vmdb01 mysqld: Last time write locked in file /home/buildbot/buildbot/build/mariadb-5.5.33/storage/xtradb/row/row0ins.c line 2039
 
...
 
Jul 25 08:09:05 vmdb01 mysqld: InnoDB: ###### Diagnostic info printed to the standard error stream
 
Jul 25 08:09:05 vmdb01 mysqld: InnoDB: Error: semaphore wait has lasted > 600 seconds
 
Jul 25 08:09:05 vmdb01 mysqld: InnoDB: We intentionally crash the server, because it appears to be hung.
 
Jul 25 08:09:05 vmdb01 mysqld: 140725  8:09:05  InnoDB: Assertion failure in thread 140114818750208 in file srv0srv.c line 2945

All info obtained from syslog is attached to this issue report. It might be a filesystem issue and we keep investigating, but we would like to exclude that the error originated from a bug or config fault in MariaDB.

 Comments   
Comment by Elena Stepanova [ 2014-07-28 ]

As I understand, jplindst is currently working on improving diagnostics in case of long semaphore waits, maybe it will help to find the reason of the problem if it re-occurs.

Comment by Jan Lindström (Inactive) [ 2014-07-31 ]

Can you try again with 5.5.39 ? It should contain improvement to diagnostics and help us to resolve the issue. Your syslog contains also other assertions, is your HD in working order ?

Comment by 21Net [ 2014-07-31 ]

Thank you for your reply, we will schedule a MariaDB upgrade and see... Until now this was a one time issue and of course it would be best if it remained that way

Generated at Thu Feb 08 07:12:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.