[MDEV-11046] systemd loses control of running daemon after RPM upgrade - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Duplicate
Affects Version/s: 10.1.18
Fix Version/s: 10.4.4
Component/s: Packaging
Labels:
- packaging
Environment:
CentOS 7

Description

I receive MariaDB updates from the mariadb-enterprise-main repo. Yesterday I updated from 10.1.17 to 10.1.18 using yum but after the upgrade, system no longer has control of the running daemon.

The workaround I've found is to stop the daemon with the legacy init script (which shouldn't exist on EL7, see ~~MDEV-10797~~) and then restart it with the systemd unit. After that, systemd has control again. This is a dangerous workaround as it requires taking down a live server.

I have also reported ~~MDEV-11045~~ about the systemctl daemon-reload issue, which may or may not be related.

[jg4461@db2-dev ~]$ sudo cat /var/log/yum.log | grep Maria

Oct 12 06:20:32 Updated: MariaDB-common-10.1.18-1.el7.centos.x86_64

Oct 12 06:20:39 Updated: MariaDB-client-10.1.18-1.el7.centos.x86_64

Oct 12 06:20:58 Updated: MariaDB-server-10.1.18-1.el7.centos.x86_64

Oct 12 06:20:58 Updated: MariaDB-shared-10.1.18-1.el7.centos.x86_64

[jg4461@db2-dev ~]$ sudo systemctl status mariadb

● mariadb.service - MariaDB database server

   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           └─migrated-from-my.cnf-settings.conf, timeout.conf

   Active: failed (Result: exit-code) since Thu 2016-10-13 09:16:21 UTC; 31min ago

  Process: 7403 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 5247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 5148 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=1/FAILURE)

  Process: 5146 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 5247 (code=exited, status=0/SUCCESS)

Oct 13 09:14:10 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Starting MariaDB database server...

Oct 13 09:14:10 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:10 140368927991936 [Warning] option 'wsrep_max_ws_size': unsigned value 2147483648 adjusted to 2147483647

Oct 13 09:14:10 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:10 140368927991936 [Note] /usr/sbin/mysqld (mysqld 10.1.18-MariaDB-enterprise) starting as process 5612 ...

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk sh[5148]: WSREP: Failed to start mysqld for wsrep recovery: '2016-10-13  9:14:10 140368927991936 [ERROR] mysqld: Can't lock aria control file '/v...r 30 seconds

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:41 140368927991936 [ERROR] mysqld: Got error 'Could not get an exclusive lock; file is probably in use by another proc...log_control'

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:41 140368927991936 [ERROR] Plugin 'Aria' init function returned error.

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: mariadb.service: control process exited, code=exited status=1

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Failed to start MariaDB database server.

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Unit mariadb.service entered failed state.

Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: mariadb.service failed.

Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Hint: Some lines were ellipsized, use -l to show in full.

[jg4461@db2-dev ~]$ ps -ef | grep mysql

root      4480     1  0 Oct12 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql/data --pid-file=/var/lib/mysql/data/db2-dev.nomadic-core.bris.ac.uk.pid

mysql     5082  4480  0 Oct12 ?        00:02:52 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql/data --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=on --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/lib/mysql/data/db2-dev.nomadic-core.bris.ac.uk.pid --socket=/var/lib/mysql/mysql.sock --port=3306 --wsrep_start_position=01277571-0c67-11e6-961c-32aa2e838dbc:304

jg4461    9415  8089  0 09:48 pts/0    00:00:00 grep --color=auto mysql

[jg4461@db2-dev ~]$ sudo /etc/init.d/mysql stop

Shutting down MySQL....                                    [  OK  ]

[jg4461@db2-dev ~]$ sudo systemctl start mariadb

Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[jg4461@db2-dev ~]$ sudo systemctl status mariadb

● mariadb.service - MariaDB database server

   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           └─migrated-from-my.cnf-settings.conf, timeout.conf

   Active: active (running) since Thu 2016-10-13 09:48:40 UTC; 12s ago

  Process: 10054 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 9470 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)

  Process: 9467 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 9977 (mysqld)

   Status: "Taking your SQL requests now..."

   CGroup: /system.slice/mariadb.service

           └─9977 /usr/sbin/mysqld --wsrep_start_position=01277571-0c67-11e6-961c-32aa2e838dbc:304

Oct 13 09:48:36 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Starting MariaDB database server...

Oct 13 09:48:37 db2-dev.nomadic-core.bris.ac.uk sh[9470]: 2016-10-13  9:48:37 140270106884224 [Warning] option 'wsrep_max_ws_size': unsigned value 2147483648 adjusted to 2147483647

Oct 13 09:48:37 db2-dev.nomadic-core.bris.ac.uk sh[9470]: 2016-10-13  9:48:37 140270106884224 [Note] /usr/sbin/mysqld (mysqld 10.1.18-MariaDB-enterprise) starting as process 9933 ...

Oct 13 09:48:39 db2-dev.nomadic-core.bris.ac.uk sh[9470]: WSREP: Recovered position 01277571-0c67-11e6-961c-32aa2e838dbc:304

Oct 13 09:48:39 db2-dev.nomadic-core.bris.ac.uk mysqld[9977]: 2016-10-13  9:48:39 140308169877632 [Warning] option 'wsrep_max_ws_size': unsigned value 2147483648 adjusted to 2147483647

Oct 13 09:48:39 db2-dev.nomadic-core.bris.ac.uk mysqld[9977]: 2016-10-13  9:48:39 140308169877632 [Note] /usr/sbin/mysqld (mysqld 10.1.18-MariaDB-enterprise) starting as process 9977 ...

Oct 13 09:48:40 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Started MariaDB database server.

Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Attachments

Issue Links

is duplicated by

MDEV-10797 RPM includes init script and a systemd unit

Closed

relates to

MDEV-11045 RPM should execute systemctl daemon-reload after update

Closed

MDEV-19706 RPM no longer installs init script on systemd systems, but preun script still tries to erase it

Closed

systemd loses control of running daemon after RPM upgrade

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration