[MDEV-8956] systemd loses MariaDB service after disable / enable Created: 2015-10-16  Updated: 2022-11-05  Resolved: 2022-11-05

Status: Closed
Project: MariaDB Server
Component/s: Packaging, Platform Debian, Platform RedHat
Affects Version/s: 10.1
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Elena Stepanova Assignee: Faustin Lammler
Resolution: Fixed Votes: 0
Labels: foundation, systemd, upstream


 Description   

That's the notorious problem with "inactive (dead)", I'm filing it to track upstream bugs further, and also hopefully to have a workaround by the next release.

It has been observed on Ubuntu Vivid, Debian Sid, Fedora 22.

Quoting my email from earlier:

sudo systemctl stop mariadb
 
 
 
# needs to be fast, do not sleep more than 1 sec
 
 
 
sudo systemctl start mariadb
 
 
 
# delay does not matter
 
 
 
sudo systemctl disable mariadb
 
 
 
# delay does not matter
 
 
 
sudo systemctl enable mariadb
 
 
 
# delay does not matter
 
 
 
sudo systemctl status mariadb
 
 
 
############################
 
 
 
Here the service starts showing the infamous
 
 
 
############################
 
 
 
● mariadb.service - MariaDB database server
 
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
 
  Drop-In: /etc/systemd/system/mariadb.service.d
 
           └─migrated-from-my.cnf-settings.conf
 
   Active: inactive (dead) since Wed 2015-10-14 15:04:58 UTC; 58s ago
 
 Main PID: 880 (mysqld)
 
   Status: "Taking your SQL requests now..."
 
   CGroup: /system.slice/mariadb.service
 
           └─880 /usr/sbin/mysqld
 
 
 
############################
 
 
 
If I then shut down the MariaDB server, e.g. via the shutdown command (or just SIGTERM the process), I get
 
 
 
############################
 
 
 
Broadcast message from systemd-journald@ubuntu-vivid-amd64 (Wed 2015-10-14 15:14:18 UTC):
 
 
 
systemd[1]: Caught <ABRT>, dumped core as pid 960.
 
 
 
 
 
Broadcast message from systemd-journald@ubuntu-vivid-amd64 (Wed 2015-10-14 15:14:18 UTC):
 
 
 
systemd[1]: Freezing execution.
 
 
 
############################
 
 
 
systemd:
 
 
 
ii  libpam-systemd:amd64                219-7ubuntu3 amd64        system and service manager - PAM module
 
ii  libsystemd0:amd64                   219-7ubuntu3 amd64        systemd utility library
 
ii  systemd                             219-7ubuntu3 amd64        system and service manager
 
ii  systemd-sysv                        219-7ubuntu3 amd64        system and service manager - SysV links

Unfortunately, it happens not only when a user stops/starts the server quickly, but also apparently when installation process does the same. My experiments above were performed when MariaDB was already installed, and the machine was rebooted; but it also happens right after installation, and then it's not necessary to stop/start server manually, the sequence is

  • install server
  • disable service
  • enable service
  • run status
    => observe the problem.

It means that a user has no control over the situation, which makes it more important.



 Comments   
Comment by Elena Stepanova [ 2015-11-02 ]

svoj,

While there is still time, could you please try to add a few second delays in installation/upgrade scripts, wherever the server is restarted?
We are getting many failures of this kind in buildbot, so I'm worried about real-life installation/upgrade.
Please push it into bb-10.1-systemd (note "10.1"), I have temporarily added it to the list of main trees so it builds everywhere.

Comment by Elena Stepanova [ 2015-11-03 ]

It turns out that the server restart (quick or not) is irrelevant. It was not obvious because the problem happens sporadically, so after adding a delay the test was passing by pure chance.
Further experiments have shown that it is enough to do 

# After reboot, mariadb service is already running
 
sudo systemctl disable mariadb
 
# optional delay
 
sudo systemctl enable mariadb
 
# optional delay
 
sudo systemctl status mariadb

If it only happens after disable/enable, it makes the problem much less important, since it is not a common scenario.
I've commented disable/enable in the buildbot test, two builds have passed without the error upon installation. If it continues to be so, we can just wait for the systemd fix, no need to do anything else.

Comment by Elena Stepanova [ 2015-11-03 ]

danblack, I added a comment to https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1506206, could you please repost it to https://bugzilla.redhat.com/show_bug.cgi?id=1271832 ? I don't have an account there.

Comment by Daniel Black [ 2015-11-03 ]

could you please repost it to https://bugzilla.redhat.com/show_bug.cgi?id=1271832

done

Comment by Daniel Black [ 2018-04-01 ]

I attempted reproduce this on systemd 237 with a timed out service that still had a process active. Performing the enable/disable. Showing the status and then killing of the process. Seems to not result in systemd aborts and a fairly sane behaviour.

# cat /etc/systemd/system/sleep.service
 
 
 
[Install]
 
WantedBy=multi-user.target
 
 
 
[Service]
 
ExecStart=/bin/nohup /bin/sleep 600
 
KillSignal=HUP
 
SendSIGKILL=no
 
[root@6adbfc4b13bc /]# systemctl start sleep 
[root@6adbfc4b13bc /]# systemctl stop sleep
 
 
 
[root@6adbfc4b13bc /]# journalctl -f -u sleep.service 
Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service.
 
Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service...
 
Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
 
^C
 
[root@6adbfc4b13bc /]# systemctl  disable sleep 
Removed /etc/systemd/system/multi-user.target.wants/sleep.service.
 
[root@6adbfc4b13bc /]# systemctl  enable sleep 
Created symlink /etc/systemd/system/multi-user.target.wants/sleep.service → /etc/systemd/system/sleep.service.
 
[root@6adbfc4b13bc /]# systemctl  status sleep    
● sleep.service
 
   Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di
 
sabled)
 
   Active: deactivating (final-sigterm) (Result: timeout) since Sun 2018-04-01 0
 
2:16:31 UTC; 1min 41s ago
 
 Main PID: 394 (sleep)
 
    Tasks: 1 (limit: 4915)
 
   CGroup: /system.slice/docker.service/system.slice/sleep.service
 
           └─394 /usr/bin/coreutils --coreutils-prog-shebang=sleep /bin/sleep 600
 
 
 
Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service.
 
Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service...
 
Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: 
State 'stop-sigterm' timed out. Skipping SIGKILL.
 
[root@6adbfc4b13bc /]# kill 394
 
[root@6adbfc4b13bc /]# systemctl  status sleep
 
● sleep.service
 
   Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di
 
sabled)
 
   Active: failed (Result: timeout) since Sun 2018-04-01 02:18:20 UTC
 
; 2s ago
 
  Process: 394 ExecStart=/bin/nohup /bin/sleep 600 (code=killed, signal=TERM)
 
 Main PID: 394 (code=killed, signal=TERM)
 
 
 
Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service.
 
Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service...
 
Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service:
 
State 'stop-sigterm' timed out. Skipping SIGKILL.
 
Apr 01 02:18:20 6adbfc4b13bc systemd[1]: sleep.service:
 
Failed with result 'timeout'.
 
Apr 01 02:18:20 6adbfc4b13bc systemd[1]: Stopped sleep.service.
 
[root@6adbfc4b13bc /]# systemctl  start sleep
 
[root@6adbfc4b13bc /]# systemctl  status sleep
 
● sleep.service
 
   Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di
 
sabled)
 
   Active: active (running) since Sun 2018-04-01 02:19:05 UTC; 1s ago
 
 Main PID: 429 (sleep)
 
    Tasks: 1 (limit: 4915)
 
   CGroup: /system.slice/docker.service/system.slice/sleep.service
 
           └─429 /usr/bin/coreutils --coreutils-prog-shebang=sleep /bin/sleep 600
 
 
 
Apr 01 02:19:05 6adbfc4b13bc systemd[1]: Started sleep.service.

Comment by Faustin Lammler [ 2018-05-09 ]

Hi elenst,
what is the status of this bug, I can not reproduce the problem.

# while true; do systemctl disable mariadb; sleep 1; systemctl enable mariadb; sleep 5; done

Gives no error after various reboot.

$ cat /etc/debian_version 
buster/sid
 
$ dpkg -l | egrep 'systemd|mariadb'
 
ii  libmariadbclient18:amd64      1:10.1.29-6                    amd64        MariaDB database client library
 
ii  libpam-systemd:amd64          238-4                          amd64        system and service manager - PAM module
 
ii  libsystemd0:amd64             238-4                          amd64        systemd utility library
 
ii  mariadb-client-10.1           1:10.1.29-6                    amd64        MariaDB database client binaries
 
ii  mariadb-client-core-10.1      1:10.1.29-6                    amd64        MariaDB database core client binaries
 
ii  mariadb-common                1:10.1.29-6                    all          MariaDB common metapackage
 
ii  mariadb-server                1:10.1.29-6                    all          MariaDB database server (metapackage depending on the latest version)
 
ii  mariadb-server-10.1           1:10.1.29-6                    amd64        MariaDB database server binaries
 
ii  mariadb-server-core-10.1      1:10.1.29-6                    amd64        MariaDB database core server files
 
ii  systemd                       238-4                          amd64        system and service manager
 
ii  systemd-sysv                  238-4                          amd64        system and service manager - SysV links

Comment by Faustin Lammler [ 2022-11-04 ]

Hi elenst do you think we should close this?

Comment by Elena Stepanova [ 2022-11-04 ]

I guess so. Both upstream bugs are closed as won't fix, so there is nothing more to track.

Comment by Faustin Lammler [ 2022-11-04 ]

thanks!!

Generated at Thu Feb 08 07:31:03 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.