Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11046

systemd loses control of running daemon after RPM upgrade

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.1.18
    • 10.4.4
    • Packaging
    • CentOS 7

    Description

      I receive MariaDB updates from the mariadb-enterprise-main repo. Yesterday I updated from 10.1.17 to 10.1.18 using yum but after the upgrade, system no longer has control of the running daemon.

      The workaround I've found is to stop the daemon with the legacy init script (which shouldn't exist on EL7, see MDEV-10797) and then restart it with the systemd unit. After that, systemd has control again. This is a dangerous workaround as it requires taking down a live server.

      I have also reported MDEV-11045 about the systemctl daemon-reload issue, which may or may not be related.

      [jg4461@db2-dev ~]$ sudo cat /var/log/yum.log | grep Maria
      		 
      Oct 12 06:20:32 Updated: MariaDB-common-10.1.18-1.el7.centos.x86_64
      		 
      Oct 12 06:20:39 Updated: MariaDB-client-10.1.18-1.el7.centos.x86_64
      		 
      Oct 12 06:20:58 Updated: MariaDB-server-10.1.18-1.el7.centos.x86_64
      		 
      Oct 12 06:20:58 Updated: MariaDB-shared-10.1.18-1.el7.centos.x86_64
      		 
      [jg4461@db2-dev ~]$ sudo systemctl status mariadb
      		 
      ● mariadb.service - MariaDB database server
      		 
         Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      		 
        Drop-In: /etc/systemd/system/mariadb.service.d
      		 
                 └─migrated-from-my.cnf-settings.conf, timeout.conf
      		 
         Active: failed (Result: exit-code) since Thu 2016-10-13 09:16:21 UTC; 31min ago
      		 
        Process: 7403 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      		 
        Process: 5247 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      		 
        Process: 5148 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=1/FAILURE)
      		 
        Process: 5146 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      		 
       Main PID: 5247 (code=exited, status=0/SUCCESS)
      		 
       
      		 
      Oct 13 09:14:10 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Starting MariaDB database server...
      		 
      Oct 13 09:14:10 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:10 140368927991936 [Warning] option 'wsrep_max_ws_size': unsigned value 2147483648 adjusted to 2147483647
      		 
      Oct 13 09:14:10 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:10 140368927991936 [Note] /usr/sbin/mysqld (mysqld 10.1.18-MariaDB-enterprise) starting as process 5612 ...
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk sh[5148]: WSREP: Failed to start mysqld for wsrep recovery: '2016-10-13  9:14:10 140368927991936 [ERROR] mysqld: Can't lock aria control file '/v...r 30 seconds
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:41 140368927991936 [ERROR] mysqld: Got error 'Could not get an exclusive lock; file is probably in use by another proc...log_control'
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk sh[5148]: 2016-10-13  9:14:41 140368927991936 [ERROR] Plugin 'Aria' init function returned error.
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: mariadb.service: control process exited, code=exited status=1
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Failed to start MariaDB database server.
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Unit mariadb.service entered failed state.
      		 
      Oct 13 09:16:21 db2-dev.nomadic-core.bris.ac.uk systemd[1]: mariadb.service failed.
      		 
      Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.
      		 
      Hint: Some lines were ellipsized, use -l to show in full.
      		 
      [jg4461@db2-dev ~]$ ps -ef | grep mysql
      		 
      root      4480     1  0 Oct12 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql/data --pid-file=/var/lib/mysql/data/db2-dev.nomadic-core.bris.ac.uk.pid
      		 
      mysql     5082  4480  0 Oct12 ?        00:02:52 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql/data --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=on --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/lib/mysql/data/db2-dev.nomadic-core.bris.ac.uk.pid --socket=/var/lib/mysql/mysql.sock --port=3306 --wsrep_start_position=01277571-0c67-11e6-961c-32aa2e838dbc:304
      		 
      jg4461    9415  8089  0 09:48 pts/0    00:00:00 grep --color=auto mysql
      		 
      [jg4461@db2-dev ~]$ sudo /etc/init.d/mysql stop
      		 
      Shutting down MySQL....                                    [  OK  ]
      		 
      [jg4461@db2-dev ~]$ sudo systemctl start mariadb
      		 
      Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.
      		 
      [jg4461@db2-dev ~]$ sudo systemctl status mariadb
      		 
      ● mariadb.service - MariaDB database server
      		 
         Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      		 
        Drop-In: /etc/systemd/system/mariadb.service.d
      		 
                 └─migrated-from-my.cnf-settings.conf, timeout.conf
      		 
         Active: active (running) since Thu 2016-10-13 09:48:40 UTC; 12s ago
      		 
        Process: 10054 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      		 
        Process: 9470 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
      		 
        Process: 9467 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      		 
       Main PID: 9977 (mysqld)
      		 
         Status: "Taking your SQL requests now..."
      		 
         CGroup: /system.slice/mariadb.service
      		 
                 └─9977 /usr/sbin/mysqld --wsrep_start_position=01277571-0c67-11e6-961c-32aa2e838dbc:304
      		 
       
      		 
      Oct 13 09:48:36 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Starting MariaDB database server...
      		 
      Oct 13 09:48:37 db2-dev.nomadic-core.bris.ac.uk sh[9470]: 2016-10-13  9:48:37 140270106884224 [Warning] option 'wsrep_max_ws_size': unsigned value 2147483648 adjusted to 2147483647
      		 
      Oct 13 09:48:37 db2-dev.nomadic-core.bris.ac.uk sh[9470]: 2016-10-13  9:48:37 140270106884224 [Note] /usr/sbin/mysqld (mysqld 10.1.18-MariaDB-enterprise) starting as process 9933 ...
      		 
      Oct 13 09:48:39 db2-dev.nomadic-core.bris.ac.uk sh[9470]: WSREP: Recovered position 01277571-0c67-11e6-961c-32aa2e838dbc:304
      		 
      Oct 13 09:48:39 db2-dev.nomadic-core.bris.ac.uk mysqld[9977]: 2016-10-13  9:48:39 140308169877632 [Warning] option 'wsrep_max_ws_size': unsigned value 2147483648 adjusted to 2147483647
      		 
      Oct 13 09:48:39 db2-dev.nomadic-core.bris.ac.uk mysqld[9977]: 2016-10-13  9:48:39 140308169877632 [Note] /usr/sbin/mysqld (mysqld 10.1.18-MariaDB-enterprise) starting as process 9977 ...
      		 
      Oct 13 09:48:40 db2-dev.nomadic-core.bris.ac.uk systemd[1]: Started MariaDB database server.
      		 
      Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-10797 RPM includes init script and a systemd unit

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11045 RPM should execute systemctl daemon-reload after update

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19706 RPM no longer installs init script on systemd systems, but preun script still tries to erase it

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              serg Sergei Golubchik
              jgazeley Jonathan Gazeley
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.