Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11789

MariaDB fails to restart after 10.0.29-1.el7 update

Details

    • 10.0.30

    Description

      A few minutes ago, MariaDB updated on my CentOS 7.3 server as per the below from /var/log/yum.log.

      Jan 14 10:10:04 Updated: MariaDB-common-10.0.29-1.el7.centos.x86_64
      Jan 14 10:10:05 Updated: MariaDB-client-10.0.29-1.el7.centos.x86_64
      Jan 14 10:10:14 Updated: MariaDB-server-10.0.29-1.el7.centos.x86_64
      Jan 14 10:10:14 Updated: MariaDB-shared-10.0.29-1.el7.centos.x86_64
      

      The error when manually performing a restart is shown below:

      Jan 14 10:21:36 server systemd[1]: Starting LSB: start and stop MySQL...
      Jan 14 10:21:36 server mysql[14399]: Starting MySQL.170114 10:21:36 mysqld_safe Logging to '/var/lib/mysql/server.err'.
      Jan 14 10:21:36 server mysql[14399]: 170114 10:21:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
      Jan 14 10:21:36 server mysql[14399]: /usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)
      Jan 14 10:21:37 server mysql[14399]: ERROR!
      Jan 14 10:21:37 server systemd[1]: mysql.service: control process exited, code=exited status=1
      Jan 14 10:21:37 server systemd[1]: Failed to start LSB: start and stop MySQL.
      Jan 14 10:21:37 server systemd[1]: Unit mysql.service entered failed state.
      Jan 14 10:21:37 server systemd[1]: mysql.service failed.
      

      After further investigation, I found the two SELinux errors from running 'sealert -a /var/log/audit/audit.log'.

      SELinux is preventing /usr/bin/mysqld_safe_helper from using the setgid capability.
      SELinux is preventing /usr/bin/mysqld_safe_helper from using the setuid capability.
      

      For now I have fixed this by creating a local policy and restarting MariaDB, however it appears that by default SELinux prevents /usr/bin/mysqld_safe_helper making use of setuid and setgid which causes it to fail to start back up after the upgrade.

      I'm not sure if this is a MariaDB specific issue, or MySQL, or maybe even the SELinux policy has changed since the last MariaDB update.

      Attachments

        Issue Links

          Activity

            Also wanted to note that I tested a fresh install of 10.0.29 in a new CentOS 7.3 virtual machine, upon trying to start the service with SELinux in enforcing mode I receive the same error, so it appears to be reproducible.

            Jarrod Farncomb Jarrod Farncomb added a comment - Also wanted to note that I tested a fresh install of 10.0.29 in a new CentOS 7.3 virtual machine, upon trying to start the service with SELinux in enforcing mode I receive the same error, so it appears to be reproducible.
            hyral Hyral Sacai added a comment -

            Just wanted to chime in here and say that I've also ran into the exact same problem on my CentOS 7.3 server after performing 'yum update' and was able to resolve the issue by running the suggested commands the error logs brought up (with confidence, thanks to Jarrod's article on the issue)

            hyral Hyral Sacai added a comment - Just wanted to chime in here and say that I've also ran into the exact same problem on my CentOS 7.3 server after performing 'yum update' and was able to resolve the issue by running the suggested commands the error logs brought up (with confidence, thanks to Jarrod's article on the issue )

            serg at a minimum, the work you did in MDEV-11676 needs to be extended to RHEL 6 and RHEL/CentOS 7. The policy should also probably be installed on Fedora and other OSs where SELinux is used. Perhaps instead of checking for the OS name, it would be enough to check whether the sestatus tool is available (`command -v sestatus >/dev/null 2>&1`)?

            kolbe Kolbe Kegel (Inactive) added a comment - serg at a minimum, the work you did in MDEV-11676 needs to be extended to RHEL 6 and RHEL/CentOS 7. The policy should also probably be installed on Fedora and other OSs where SELinux is used. Perhaps instead of checking for the OS name, it would be enough to check whether the sestatus tool is available (`command -v sestatus >/dev/null 2>&1`)?
            miken32 Michael Newton added a comment - - edited

            For the record, this is also affecting 10.1. I'm running Scientific Linux 6.8, and was impacted by this issue after updating from 10.1.14 to 10.1.21. Based on the logs, it appears in my case to only be denying setgid to the process, not setuid:

            type=AVC msg=audit(1485280894.516:511332): avc:  denied  { setgid } for  pid=32564 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
            type=SYSCALL msg=audit(1485280894.516:511332): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7f1f69e853c0 a2=10000 a3=1 items=0 ppid=32473 pid=32564 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
            type=AVC msg=audit(1485280894.516:511333): avc:  denied  { setgid } for  pid=32564 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
            type=SYSCALL msg=audit(1485280894.516:511333): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7f1f69e853c0 a2=40 a3=1 items=0 ppid=32473 pid=32564 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
            type=AVC msg=audit(1485280894.546:511334): avc:  denied  { setgid } for  pid=32575 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
            type=SYSCALL msg=audit(1485280894.546:511334): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7fd400c743c0 a2=10000 a3=1 items=0 ppid=32473 pid=32575 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
            type=AVC msg=audit(1485280894.550:511335): avc:  denied  { setgid } for  pid=32575 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
            type=SYSCALL msg=audit(1485280894.550:511335): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7fd400c743c0 a2=40 a3=1 items=0 ppid=32473 pid=32575 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
            type=AVC msg=audit(1485280894.699:511336): avc:  denied  { setgid } for  pid=32579 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
            type=SYSCALL msg=audit(1485280894.699:511336): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7f00af9d83c0 a2=10000 a3=1 items=0 ppid=32473 pid=32579 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
            type=AVC msg=audit(1485280894.700:511337): avc:  denied  { setgid } for  pid=32579 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
            type=SYSCALL msg=audit(1485280894.700:511337): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7f00af9d83c0 a2=40 a3=1 items=0 ppid=32473 pid=32579 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
            

            Edit: after allowing setgid, then it starts complaining about not being able to do setuid, as in the original report.

            miken32 Michael Newton added a comment - - edited For the record, this is also affecting 10.1. I'm running Scientific Linux 6.8, and was impacted by this issue after updating from 10.1.14 to 10.1.21. Based on the logs, it appears in my case to only be denying setgid to the process, not setuid: type=AVC msg=audit(1485280894.516:511332): avc: denied { setgid } for pid=32564 comm="mysqld_safe_hel" capability=6 scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability type=SYSCALL msg=audit(1485280894.516:511332): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7f1f69e853c0 a2=10000 a3=1 items=0 ppid=32473 pid=32564 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) type=AVC msg=audit(1485280894.516:511333): avc: denied { setgid } for pid=32564 comm="mysqld_safe_hel" capability=6 scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability type=SYSCALL msg=audit(1485280894.516:511333): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7f1f69e853c0 a2=40 a3=1 items=0 ppid=32473 pid=32564 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) type=AVC msg=audit(1485280894.546:511334): avc: denied { setgid } for pid=32575 comm="mysqld_safe_hel" capability=6 scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability type=SYSCALL msg=audit(1485280894.546:511334): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7fd400c743c0 a2=10000 a3=1 items=0 ppid=32473 pid=32575 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) type=AVC msg=audit(1485280894.550:511335): avc: denied { setgid } for pid=32575 comm="mysqld_safe_hel" capability=6 scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability type=SYSCALL msg=audit(1485280894.550:511335): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7fd400c743c0 a2=40 a3=1 items=0 ppid=32473 pid=32575 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) type=AVC msg=audit(1485280894.699:511336): avc: denied { setgid } for pid=32579 comm="mysqld_safe_hel" capability=6 scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability type=SYSCALL msg=audit(1485280894.699:511336): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7f00af9d83c0 a2=10000 a3=1 items=0 ppid=32473 pid=32579 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) type=AVC msg=audit(1485280894.700:511337): avc: denied { setgid } for pid=32579 comm="mysqld_safe_hel" capability=6 scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability type=SYSCALL msg=audit(1485280894.700:511337): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7f00af9d83c0 a2=40 a3=1 items=0 ppid=32473 pid=32579 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) Edit: after allowing setgid, then it starts complaining about not being able to do setuid, as in the original report.

            kolbe, I took a conservative approach — while we compile and install the file with the policy everywhere, we only enable it on distributions that are known to have the issue. I believed that most distributions didn't restrict setuid/setgid in their policies and it was only CentOS/RHEL that did.

            May be a conservative approach should've been the opposite, assuming that all distributions restrict setuid/setgid and enable our policy everywhere... I'll do that.

            serg Sergei Golubchik added a comment - kolbe , I took a conservative approach — while we compile and install the file with the policy everywhere, we only enable it on distributions that are known to have the issue. I believed that most distributions didn't restrict setuid/setgid in their policies and it was only CentOS/RHEL that did. May be a conservative approach should've been the opposite, assuming that all distributions restrict setuid/setgid and enable our policy everywhere... I'll do that.

            I have tested MariaDB 10.0.30 on CentOS 7 and this same problem still happens so it does not seem that the bug has been resolved.

            Jarrod Farncomb Jarrod Farncomb added a comment - I have tested MariaDB 10.0.30 on CentOS 7 and this same problem still happens so it does not seem that the bug has been resolved.

            Same problem — do you mean it's still "/usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)" and "SELinux is preventing /usr/bin/mysqld_safe_helper from using the setuid capability." ? Is the new policy installed when you install a 10.0.30 rpm?

            The post-install script has

            if [ -x /usr/sbin/semodule ] ; then
              /usr/sbin/semodule -i /usr/share/mysql/policy/selinux/mariadb.pp
            fi
            

            So I don't quite see how it could not have been installed. Do you have the /usr/sbin/semodule executable?

            serg Sergei Golubchik added a comment - Same problem — do you mean it's still "/usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)" and "SELinux is preventing /usr/bin/mysqld_safe_helper from using the setuid capability." ? Is the new policy installed when you install a 10.0.30 rpm? The post-install script has if [ -x /usr/sbin/semodule ] ; then /usr/sbin/semodule -i /usr/share/mysql/policy/selinux/mariadb .pp fi So I don't quite see how it could not have been installed. Do you have the /usr/sbin/semodule executable?

            Ok, ignore the question, please. I see you created MDEV-12231, let's continue there.

            serg Sergei Golubchik added a comment - Ok, ignore the question, please. I see you created MDEV-12231 , let's continue there.

            People

              serg Sergei Golubchik
              Jarrod Farncomb Jarrod Farncomb
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.