[MDEV-20821] mariadb parallel slave server shutdown hang - v10.4.8 Created: 2019-10-14  Updated: 2021-05-14  Resolved: 2020-01-21

Status: Closed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: 10.4
Fix Version/s: 10.4.12, 10.5.1

Type: Bug Priority: Critical
Reporter: Kwangbock Lee Assignee: Andrei Elkin
Resolution: Fixed Votes: 2
Labels: None

Attachments: File mariadb.STATUS.log     File mariadb.VARIABLES.log     File mariadb.service.log     Text File show global variables and status.txt    

 Description   

During on-site engagement, I installed master-slave configuration with MaxScale v2.3 on top of master node.

OS: CentOS 7.4.1708
MariaDB: v10.4.8
MaxScale: v2.3.11

Customer is using GTID to utilize auto failover of MaxScale and HA test has been done successfully.

But when I tried to shutdown mariadb server it failed with timeout.
I tried
$ systemctl stop mysql
and
$ mysqladmin shutdown and the result is the same.

The last error log is follows..
...
[Note] Slave I/O thread: connected to master 'xxx@xxxx:3306', replication starts at GTID position '0-1000-382'
----------------------------------------------

So If I stop the slave with "mysql> stop slave;" and then tried to stop the mariadb server, it stops immediately.
It looks like shutdown never stops "Slave I/O thread"... and got timeout.

Is it a known issue of v10.4.8 or any OS setting dependancies ?

FYI. innodb-thread-concurrency is set to 0.

Thanks



 Comments   
Comment by Kwangbock Lee [ 2019-10-16 ]

FYI. Customer is utilizing their infrastructures with OpenStack on top of their on-premises.

Comment by Andrei Elkin [ 2019-10-16 ]

kb, thank you for the report! I am sure there were more data in the error log around and after the attempted shutdown. It would be very helpful to share with us. Could you please?

Secondly, if the issue is reproducible, could you try out SHOW PROCESSLIST when the server is hanging.

And finally, we also need to know SHOW GLOBAL for both VARIABLES and STATUS of your slave server.

Comment by Andrei Elkin [ 2019-12-26 ]

alice: could you please verify the case? Support is also interested though there's not ticket associated. The report may related to MDEV-18450.

Comment by Volker Klasen [ 2020-01-02 ]

Hi,

we've got the same issue as the original reporter. I attached the output of SHOW GLOBAL VARIABLES and STATUS. However, I can't issue SHOW PROCESSLIST as my connection gets killed when issuing the shutdown command.

Here is the tail of mysql-error.log:

2020-01-02 10:53:45 0 [Note] /opt/mysql/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
 
2020-01-02 10:53:45 0 [Note] Event Scheduler: Purging the queue. 0 events
 
2020-01-02 10:53:45 8 [Note] Error reading relay log event: slave SQL thread was killed
 
2020-01-02 10:53:45 8 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin.000045' at position 3382; GTID position '0-2-174394'
 
2020-01-02 10:53:45 0 [Note] InnoDB: FTS optimize thread exiting.
 
2020-01-02 10:53:45 7 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000045', position 3382; GTID position 0-2-174394

Cheers,
Volker

P.S. Setting innodb-thread-concurrency to 0 makes no difference (cf. MDEV-20288)

Comment by Devin Henderson [ 2020-01-21 ]

Happening to me on a fresh MariaDB 10.4.11 install from the mariadb.org CentOS 8 repo on CentOS 8.1.1911 (fully updated). `innodb-thread-concurrency=0` makes no difference for me either. I'm using `slave_parallel_threads=2` but commenting that out doesn't help. I'm also using `thread_handling=pool-of-threads` but I can't recall if I tried commenting that out, I will report back on that.

What I have noticed is that a maraidb.service stop or restart does work if I first do `STOP SLAVE` otherwise it hangs. See attached my journalctl for mariadb.service as well as GLOBAL VARIABLES and STATUS.

mariadb.service.log
mariadb.VARIABLES.log
mariadb.STATUS.log

Comment by Devin Henderson [ 2020-02-04 ]

Can verify, this is no longer happening to me on 10.4.12. Thanks!!!

Generated at Thu Feb 08 09:02:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.