Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8956

systemd loses MariaDB service after disable / enable

Details

    Description

      That's the notorious problem with "inactive (dead)", I'm filing it to track upstream bugs further, and also hopefully to have a workaround by the next release.

      It has been observed on Ubuntu Vivid, Debian Sid, Fedora 22.

      Quoting my email from earlier:

      sudo systemctl stop mariadb
       
      # needs to be fast, do not sleep more than 1 sec
       
      sudo systemctl start mariadb
       
      # delay does not matter
       
      sudo systemctl disable mariadb
       
      # delay does not matter
       
      sudo systemctl enable mariadb
       
      # delay does not matter
       
      sudo systemctl status mariadb
       
      ############################
       
      Here the service starts showing the infamous
       
      ############################
       
      ● mariadb.service - MariaDB database server
         Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
        Drop-In: /etc/systemd/system/mariadb.service.d
                 └─migrated-from-my.cnf-settings.conf
         Active: inactive (dead) since Wed 2015-10-14 15:04:58 UTC; 58s ago
       Main PID: 880 (mysqld)
         Status: "Taking your SQL requests now..."
         CGroup: /system.slice/mariadb.service
                 └─880 /usr/sbin/mysqld
       
      ############################
       
      If I then shut down the MariaDB server, e.g. via the shutdown command (or just SIGTERM the process), I get
       
      ############################
       
      Broadcast message from systemd-journald@ubuntu-vivid-amd64 (Wed 2015-10-14 15:14:18 UTC):
       
      systemd[1]: Caught <ABRT>, dumped core as pid 960.
       
       
      Broadcast message from systemd-journald@ubuntu-vivid-amd64 (Wed 2015-10-14 15:14:18 UTC):
       
      systemd[1]: Freezing execution.
       
      ############################
       
      systemd:
       
      ii  libpam-systemd:amd64                219-7ubuntu3 amd64        system and service manager - PAM module
      ii  libsystemd0:amd64                   219-7ubuntu3 amd64        systemd utility library
      ii  systemd                             219-7ubuntu3 amd64        system and service manager
      ii  systemd-sysv                        219-7ubuntu3 amd64        system and service manager - SysV links 

      Unfortunately, it happens not only when a user stops/starts the server quickly, but also apparently when installation process does the same. My experiments above were performed when MariaDB was already installed, and the machine was rebooted; but it also happens right after installation, and then it's not necessary to stop/start server manually, the sequence is

      • install server
      • disable service
      • enable service
      • run status
        => observe the problem.

      It means that a user has no control over the situation, which makes it more important.

      Attachments

        Activity

          danblack Daniel Black added a comment -

          I attempted reproduce this on systemd 237 with a timed out service that still had a process active. Performing the enable/disable. Showing the status and then killing of the process. Seems to not result in systemd aborts and a fairly sane behaviour.

          # cat /etc/systemd/system/sleep.service
           
          [Install]
          WantedBy=multi-user.target
           
          [Service]
          ExecStart=/bin/nohup /bin/sleep 600
          KillSignal=HUP
          SendSIGKILL=no
          [root@6adbfc4b13bc /]# systemctl start sleep 
          [root@6adbfc4b13bc /]# systemctl stop sleep
           
          [root@6adbfc4b13bc /]# journalctl -f -u sleep.service 
          Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service.
          Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service...
          Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
          ^C
          [root@6adbfc4b13bc /]# systemctl  disable sleep 
          Removed /etc/systemd/system/multi-user.target.wants/sleep.service.
          [root@6adbfc4b13bc /]# systemctl  enable sleep 
          Created symlink /etc/systemd/system/multi-user.target.wants/sleep.service → /etc/systemd/system/sleep.service.
          [root@6adbfc4b13bc /]# systemctl  status sleep    
          ● sleep.service
             Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di
          sabled)
             Active: deactivating (final-sigterm) (Result: timeout) since Sun 2018-04-01 0
          2:16:31 UTC; 1min 41s ago
           Main PID: 394 (sleep)
              Tasks: 1 (limit: 4915)
             CGroup: /system.slice/docker.service/system.slice/sleep.service
                     └─394 /usr/bin/coreutils --coreutils-prog-shebang=sleep /bin/sleep 600
           
          Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service.
          Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service...
          Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: 
          State 'stop-sigterm' timed out. Skipping SIGKILL.
          [root@6adbfc4b13bc /]# kill 394
          [root@6adbfc4b13bc /]# systemctl  status sleep
          ● sleep.service
             Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di
          sabled)
             Active: failed (Result: timeout) since Sun 2018-04-01 02:18:20 UTC
          ; 2s ago
            Process: 394 ExecStart=/bin/nohup /bin/sleep 600 (code=killed, signal=TERM)
           Main PID: 394 (code=killed, signal=TERM)
           
          Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service.
          Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service...
          Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service:
          State 'stop-sigterm' timed out. Skipping SIGKILL.
          Apr 01 02:18:20 6adbfc4b13bc systemd[1]: sleep.service:
          Failed with result 'timeout'.
          Apr 01 02:18:20 6adbfc4b13bc systemd[1]: Stopped sleep.service.
          [root@6adbfc4b13bc /]# systemctl  start sleep
          [root@6adbfc4b13bc /]# systemctl  status sleep
          ● sleep.service
             Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di
          sabled)
             Active: active (running) since Sun 2018-04-01 02:19:05 UTC; 1s ago
           Main PID: 429 (sleep)
              Tasks: 1 (limit: 4915)
             CGroup: /system.slice/docker.service/system.slice/sleep.service
                     └─429 /usr/bin/coreutils --coreutils-prog-shebang=sleep /bin/sleep 600
           
          Apr 01 02:19:05 6adbfc4b13bc systemd[1]: Started sleep.service.
          

          danblack Daniel Black added a comment - I attempted reproduce this on systemd 237 with a timed out service that still had a process active. Performing the enable/disable. Showing the status and then killing of the process. Seems to not result in systemd aborts and a fairly sane behaviour. # cat /etc/systemd/system/sleep.service   [Install] WantedBy=multi-user.target   [Service] ExecStart=/bin/nohup /bin/sleep 600 KillSignal=HUP SendSIGKILL=no [root@6adbfc4b13bc /]# systemctl start sleep [root@6adbfc4b13bc /]# systemctl stop sleep   [root@6adbfc4b13bc /]# journalctl -f -u sleep.service Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service. Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service... Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: State 'stop-sigterm' timed out. Skipping SIGKILL. ^C [root@6adbfc4b13bc /]# systemctl disable sleep Removed /etc/systemd/system/multi-user.target.wants/sleep.service. [root@6adbfc4b13bc /]# systemctl enable sleep Created symlink /etc/systemd/system/multi-user.target.wants/sleep.service → /etc/systemd/system/sleep.service. [root@6adbfc4b13bc /]# systemctl status sleep ● sleep.service Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di sabled) Active: deactivating (final-sigterm) (Result: timeout) since Sun 2018-04-01 0 2:16:31 UTC; 1min 41s ago Main PID: 394 (sleep) Tasks: 1 (limit: 4915) CGroup: /system.slice/docker.service/system.slice/sleep.service └─394 /usr/bin/coreutils --coreutils-prog-shebang=sleep /bin/sleep 600   Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service. Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service... Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: State 'stop-sigterm' timed out. Skipping SIGKILL. [root@6adbfc4b13bc /]# kill 394 [root@6adbfc4b13bc /]# systemctl status sleep ● sleep.service Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di sabled) Active: failed (Result: timeout) since Sun 2018-04-01 02:18:20 UTC ; 2s ago Process: 394 ExecStart=/bin/nohup /bin/sleep 600 (code=killed, signal=TERM) Main PID: 394 (code=killed, signal=TERM)   Apr 01 02:16:11 6adbfc4b13bc systemd[1]: Started sleep.service. Apr 01 02:16:31 6adbfc4b13bc systemd[1]: Stopping sleep.service... Apr 01 02:18:01 6adbfc4b13bc systemd[1]: sleep.service: State 'stop-sigterm' timed out. Skipping SIGKILL. Apr 01 02:18:20 6adbfc4b13bc systemd[1]: sleep.service: Failed with result 'timeout'. Apr 01 02:18:20 6adbfc4b13bc systemd[1]: Stopped sleep.service. [root@6adbfc4b13bc /]# systemctl start sleep [root@6adbfc4b13bc /]# systemctl status sleep ● sleep.service Loaded: loaded (/etc/systemd/system/sleep.service; enabled; vendor preset: di sabled) Active: active (running) since Sun 2018-04-01 02:19:05 UTC; 1s ago Main PID: 429 (sleep) Tasks: 1 (limit: 4915) CGroup: /system.slice/docker.service/system.slice/sleep.service └─429 /usr/bin/coreutils --coreutils-prog-shebang=sleep /bin/sleep 600   Apr 01 02:19:05 6adbfc4b13bc systemd[1]: Started sleep.service.
          faust Faustin Lammler added a comment - - edited

          Hi elenst,
          what is the status of this bug, I can not reproduce the problem.

          # while true; do systemctl disable mariadb; sleep 1; systemctl enable mariadb; sleep 5; done
          

          Gives no error after various reboot.

          $ cat /etc/debian_version 
          buster/sid
          $ dpkg -l | egrep 'systemd|mariadb'
          ii  libmariadbclient18:amd64      1:10.1.29-6                    amd64        MariaDB database client library
          ii  libpam-systemd:amd64          238-4                          amd64        system and service manager - PAM module
          ii  libsystemd0:amd64             238-4                          amd64        systemd utility library
          ii  mariadb-client-10.1           1:10.1.29-6                    amd64        MariaDB database client binaries
          ii  mariadb-client-core-10.1      1:10.1.29-6                    amd64        MariaDB database core client binaries
          ii  mariadb-common                1:10.1.29-6                    all          MariaDB common metapackage
          ii  mariadb-server                1:10.1.29-6                    all          MariaDB database server (metapackage depending on the latest version)
          ii  mariadb-server-10.1           1:10.1.29-6                    amd64        MariaDB database server binaries
          ii  mariadb-server-core-10.1      1:10.1.29-6                    amd64        MariaDB database core server files
          ii  systemd                       238-4                          amd64        system and service manager
          ii  systemd-sysv                  238-4                          amd64        system and service manager - SysV links
          

          faust Faustin Lammler added a comment - - edited Hi elenst , what is the status of this bug, I can not reproduce the problem. # while true; do systemctl disable mariadb; sleep 1; systemctl enable mariadb; sleep 5; done Gives no error after various reboot. $ cat /etc/debian_version buster/sid $ dpkg -l | egrep 'systemd|mariadb' ii libmariadbclient18:amd64 1:10.1.29-6 amd64 MariaDB database client library ii libpam-systemd:amd64 238-4 amd64 system and service manager - PAM module ii libsystemd0:amd64 238-4 amd64 systemd utility library ii mariadb-client-10.1 1:10.1.29-6 amd64 MariaDB database client binaries ii mariadb-client-core-10.1 1:10.1.29-6 amd64 MariaDB database core client binaries ii mariadb-common 1:10.1.29-6 all MariaDB common metapackage ii mariadb-server 1:10.1.29-6 all MariaDB database server (metapackage depending on the latest version) ii mariadb-server-10.1 1:10.1.29-6 amd64 MariaDB database server binaries ii mariadb-server-core-10.1 1:10.1.29-6 amd64 MariaDB database core server files ii systemd 238-4 amd64 system and service manager ii systemd-sysv 238-4 amd64 system and service manager - SysV links

          Hi elenst do you think we should close this?

          faust Faustin Lammler added a comment - Hi elenst do you think we should close this?

          I guess so. Both upstream bugs are closed as won't fix, so there is nothing more to track.

          elenst Elena Stepanova added a comment - I guess so. Both upstream bugs are closed as won't fix, so there is nothing more to track.

          thanks!!

          faust Faustin Lammler added a comment - thanks!!

          People

            faust Faustin Lammler
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.