[MDEV-13373] Shutdown never completes : Waiting for master thread to exit Created: 2017-07-22  Updated: 2017-09-04  Resolved: 2017-09-04

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.1.24
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Tom Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: need_feedback
Environment:

Fedora 25 64bit

Attachments: File cnf.bz2     Text File log.txt    

 Description   

When MariaDB is shutdown (via systemd), it never finishes.

The mysqld process jumps to 100% CPU, and the log file repeats 

2017-07-22 15:06:58 140459974318848 [Note] InnoDB: Waiting for master thread to exit

at regular intervals.

Killing (with kill -KILL) the mysqld process is the only way to proceed. I've left it for hours with no change.
This is a tiny DB install : 

# du -ksh /var/lib/mysql
 
273M    /var/lib/mysql



 Comments   
Comment by Tom [ 2017-07-22 ]

Also occurs on FC24

Comment by Alice Sherepa [ 2017-07-25 ]

please provide your .cnf file(s), error.log and collect gdb stack traces

Comment by Tom [ 2017-07-25 ]

Cnf's attached, along with fuller log entries

strace -p 18308
strace: Process 18308 attached
futex(0x5613c9fa26c4, FUTEX_WAIT_PRIVATE, 7, NULL

(gdb) bt
#0 0x00007fcb01f3d460 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/usr/lib64/libpthread.so.0
#1 0x00005613c90c3463 in inline_mysql_cond_wait (that=0x5613c9fa26c0 <COND_thread_count>, mutex=0x5613c9fa2d80 <LOCK_thread_count>,
src_file=0x5613c97e5288 "/builddir/build/BUILD/mariadb-10.1.24/sql/mysqld.cc", src_line=5946)
at /usr/src/debug/mariadb-10.1.24/include/mysql/psi/mysql_thread.h:1165
#2 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/mariadb-10.1.24/sql/mysqld.cc:5946
#3 0x00007fcb0011b401 in __libc_start_main () from target:/usr/lib64/libc.so.6
#4 0x00005613c90b677a in _start ()

Comment by Tom [ 2017-08-01 ]

This has fixed itself as of

# rpm -qi mariadb-server
Name : mariadb-server
Epoch : 3
Version : 10.1.25
Release : 1.fc25
Architecture: x86_64
Install Date: Fri 28 Jul 2017 02:14:27 BST

Comment by Elena Stepanova [ 2017-08-04 ]

Tom,

Would you be able to setup the server to produce a coredump and next time when it hangs, kill it with SIGABRT rather than SIGKILL?
See details here:
https://mariadb.com/kb/en/mariadb/how-to-produce-a-full-stack-trace-for-mysqld/

Further, would you be willing to substitute mysqld with a debug binary for one cycle and get a debug stack trace and the coredump with it?

Comment by Tom [ 2017-08-07 ]

Well, it doesn't reproduce in later versions of MariaDB Fedora ship (though it was consistently broken), so I think we can just close this.

Comment by Elena Stepanova [ 2017-09-04 ]

According to the comment above, closing for now; but please comment if the problem re-appears.

Generated at Thu Feb 08 08:05:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.