[MDEV-6669] Restarting mariadb-server 10 service on ubuntu 14.04 fails Created: 2014-08-30  Updated: 2014-10-09  Due: 2014-10-16  Resolved: 2014-10-09

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 10.0.13-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Bruce Kirkpatrick Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

ubuntu 14.04

Attachments: Text File my.cnf.txt     Text File mysql-confd-file.txt    

 Description   

I installed mariadb via the ubuntu 14.04 repo and documentation provided on mariadb.org.

I found the /etc/init.d/mysql script wasn't waiting for mysqld to fully shutdown after running mysqladmin shutdown in the init script. There is logic that looks for an error condition and then runs a sequence to verify it has finished shutting down.

For some reason, it is not able to return an error condition for mysqladmin shutdown on this platform, so the stop() function completes too early and then start() fails and this leaves the mysql process not running. If you run service mysql restart again, it works, because enough time has passed for mysqladmin shutdown to complete.

I found I could fix this by forcing the following shell script code to run immediately after this line:

    shutdown_out=`$MYADMIN shutdown 2>&1`; r=$?

but before this line:

     if [ "$r" -ne 0 ]; then

new code:

for i in `seq 1 600`; do
 
sleep 1
 
if mysqld_status check_dead nowarn; then server_down=1; break; fi
 
done

Please verify/fix for the benefit of others.

I'm also using the innodb pool dump feature, which makes it take longer to shutdown possibly.

 Comments   
Comment by Sergei Golubchik [ 2014-09-04 ]

Unfortunately, I was not able to repeat this issue. Nor can I see (by looking at the source code) how could mysqladmin possibly return wrong error code. Even with long shutdown times mysqladmin will wait.

Comment by Sergei Golubchik [ 2014-09-04 ]

Could you provide some additional information that would help me to repeat this issue?

Comment by Bruce Kirkpatrick [ 2014-09-04 ]

I attached the my.cnf options I'm using. If this doesn't reproduce the service mysql restart fail problem, let me know. I had this happen on 3 different installs of ubuntu 14/mysql using the same steps - i changed the system in several ways, so it could be something else. I do have my complete installation steps for setting up ubuntu published online, but that would be too much work to ask you to go through.

Comment by Elena Stepanova [ 2014-10-04 ]

Hi Bruce,

Are you actually using 10.0.13-galera, as the "Affected Version/s" field says?
Could you please provide the output of dpkg -l | grep -iE 'mysql|maria'?
If it's so easily reproducible for you, can you try to run the same mysqladmin shutdown manually and see what it returns?

Comment by Bruce Kirkpatrick [ 2014-10-04 ]

I use the default /etc/mysql/my.cnf and also include this file in conf.d directory. It seems I already included the configuration file in original post...

Comment by Bruce Kirkpatrick [ 2014-10-04 ]

Hi Elena,

mysqladmin shutdown has no output, because it does complete successfully. The problem with "service mysql restart" is that it doesn't wait long enough for that shutdown to complete, so then it can't start the server again. It is like mysqladmin shutdown is an asynchronous command, but the init script assumes it is synchronous. When I set the /etc/init.d/mysql back to how it was, this issue is still able to be reproduced.

I attached my mysql configuration settings.

Version output below:

root@jetendo:~# dpkg -l | grep -iE 'mysql|maria'
ii libdbd-mysql-perl 4.025-1 amd64 Perl5 database interface to the MySQL database
ii libmariadbclient18 10.0.13+maria-1~trusty amd64 MariaDB database client library
ii libmysqlclient18 10.0.13+maria-1~trusty amd64 Virtual package to satisfy external depends
ii mariadb-client-10.0 10.0.13+maria-1~trusty amd64 MariaDB database client binaries
ii mariadb-client-core-10.0 10.0.13+maria-1~trusty amd64 MariaDB database core client binaries
ii mariadb-common 10.0.13+maria-1~trusty all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
ii mariadb-server 10.0.13+maria-1~trusty all MariaDB database server (metapackage depending on the latest version)
ii mariadb-server-10.0 10.0.13+maria-1~trusty amd64 MariaDB database server binaries
ii mariadb-server-core-10.0 10.0.13+maria-1~trusty amd64 MariaDB database core server files
ii mysql-common 10.0.13+maria-1~trusty all MariaDB database common files (e.g. /etc/mysql/my.cnf)
ii php5-mysql 5.5.16+dfsg-1+deb.sury.org~trusty+1 amd64 MySQL module for php5

Comment by Elena Stepanova [ 2014-10-08 ]

Hi Bruce,

Thanks for the data.

mysqladmin shutdown has no output, because it does complete successfully. The problem with "service mysql restart" is that it doesn't wait long enough for that shutdown to complete

When you run it manually, you can run it with --verbose option ( mysqadmin --defaults-file=/etc/mysql/debian.cnf --verbose shutdown )
If mysqladmin is using the unix socket, as it's supposed to, in verbose mode it should report waiting for pid file to disappear (and should actually wait).
If it doesn't log the message, it might mean that somehow it is forced to connect via a TCP port, in which case, indeed, it doesn't wait for anything.
If you run it in verbose mode and it's still silent, it makes sense to check your /etc/mysql/debian.cnf – maybe it was tampered with, and the client connection parameters are modified to use TCP instead of a socket?
If it reports waiting for the pid file in the verbose mode, it would be interesting to try to see if it exits before the pid file actually disappears.

Comment by Bruce Kirkpatrick [ 2014-10-08 ]

I intentionally changed debian.cnf to use TCP because it wasn't working right with the default socket method at some point. It appears you've isolated the problem. I could stop using TCP or continue to use the workaround I came up with.

Comment by Elena Stepanova [ 2014-10-09 ]

The difference between socket (local) connections and TCP connections in mysqladmin is intentional – obviously, it only makes sense to wait on the pid file if the connection is definitely local.

I suppose hypothetically it could be made to distinguish between local TCP connections and remote TCP connections, but I'm not sure it's worth the trouble. So, for now I'm closing the issue as 'Not a bug'. Please comment to re-open if you disagree.

Generated at Thu Feb 08 07:13:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.