[MDEV-16586] Dead-lock in MYSQL_BIN_LOG::reset_logs(..) ? Created: 2018-06-26 Updated: 2020-01-06 Resolved: 2020-01-06 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Replication, Server |
| Affects Version/s: | 10.3.7, 10.3.8, 10.3.10, 10.3.18 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Critical |
| Reporter: | David Kedves | Assignee: | Sachin Setiya (Inactive) |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | binlog_checkpoint, need_feedback | ||
| Environment: |
Ubuntu 14.04 (I used an LXC container), CentOS 6.10 Cloud Image (qemu) |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Description |
|
I've installed MariaDB 10.3.7 and initialized with my own configs (from previous setups), set a root user.. etc, then I have executed /usr/bin/mysql -NAB -u'root' -p'!password$$' -e "RESET MASTER" And this command seems to be stuck since hours (actually it is almost half a day since now...). I have attached my mysqld config, logs and the backtrace of mysqld too, it seems to be waiting for a mutex ... possibly dead-lock internally? Actually there are two issues here: Unfortunetely mariadb doesn't provide -dbg packages (or i just don't see them) so i couldn't provide better backtrace atm... |
| Comments |
| Comment by David Kedves [ 2018-06-26 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
NOTE: I've reproduced this so-far only once... will retry if it is reproducible (also i'll check if affects other distros as well..) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Kedves [ 2018-06-26 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Okay, it seems the issue is only reproducible on Ubuntu 14.04, at least this did not happen in other OS-es to me (CentOS 7, Ubuntu16.04, Debian 8 and 9) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Kedves [ 2018-07-24 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hey guys, this is still reproducible also with 10.3.8, any news there? {{ Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show full processlist
---
--- MariaDB [(none)]> Bye {{ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Kedves [ 2018-10-16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Update: this still seems to happen with MariaDB 10.3.10 as well mariadb-server-10.3 1:10.3.10+maria~trusty Bactrace still similar: Thread 5 (Thread 0x7f500c1eb700 (LWP 5251)): And MySQL CLI and mysqld seems to stuck :-S | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Sujatha Sivakumar (Inactive) [ 2019-05-31 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello David, Thanks for reporting this issue. It would be great if you can provide a backtrace which includes line numbers and filenames, for all the thread. Can please try installing the debug info packages and provide backtrace during the hang scenario. https://mariadb.com/kb/en/library/yum/#installing-debug-info-packages-with-yum | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Kedves [ 2019-10-02 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi, bt.txt It just happened to me again, this time on CentOS6 (on a qemu cloud image). It didn't "freeze" now on reset logs but on a create user command after that, however in processlist i still see the "reset master" in initializing state.. and i guess this makes the "create user" to be deadlocked somehow when mysqld tries to write into the binlogs... (just guessing..) anyhow, this time i could capture a backtrace with line numbers and everything, please see the new set of attachment to this comment. Thanks, David | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Sachin Setiya (Inactive) [ 2019-10-06 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi kedazo Thanks for providing backtrace , Can you provide mysqld.log file also ?, In your earlier mysqld.log file
So is this server master ?, or there is no replication involved at all ? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Kedves [ 2019-10-06 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sorry I have destroyed this VM already... well it is a single instance, so that is right, there is no replication involved at all. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Sachin Setiya (Inactive) [ 2019-10-06 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi kedazo, Thanks , | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andrei Elkin [ 2019-12-02 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The latest bt attachment https://jira.mariadb.org/secure/attachment/45808/backtrace.txt (thanks, kedazo) contains symbols and suggests Thread 4 "RESET MASTER" is waiting for the end of binlog checkpoint request while holing a mutex lock which Thread 2 is waiting for. It's no a deadlock between the two. And it's unclear about the reason of binlog checkpoint response delay or loss. There's no other thread that keeps a running transaction that might be blamed for the delay. Overall, I think we need some more specific instructions to how to reproduce. |