[MDEV-11789] MariaDB fails to restart after 10.0.29-1.el7 update Created: 2017-01-14  Updated: 2017-03-11  Resolved: 2017-02-27

Status: Closed
Project: MariaDB Server
Component/s: Platform RedHat
Affects Version/s: 10.0.29, 10.0.30
Fix Version/s: 5.5.55, 10.0.30, 10.1.22

Type: Bug Priority: Critical
Reporter: Jarrod Farncomb Assignee: Sergei Golubchik
Resolution: Fixed Votes: 2
Labels: None
Environment:

CentOS 7.3

Issue Links:
Relates
relates to MDEV-12231 MariaDB fails to restart after 10.0.3... Closed
Sprint: 10.0.30

 Description   

A few minutes ago, MariaDB updated on my CentOS 7.3 server as per the below from /var/log/yum.log.

Jan 14 10:10:04 Updated: MariaDB-common-10.0.29-1.el7.centos.x86_64
 
Jan 14 10:10:05 Updated: MariaDB-client-10.0.29-1.el7.centos.x86_64
 
Jan 14 10:10:14 Updated: MariaDB-server-10.0.29-1.el7.centos.x86_64
 
Jan 14 10:10:14 Updated: MariaDB-shared-10.0.29-1.el7.centos.x86_64

The error when manually performing a restart is shown below:

Jan 14 10:21:36 server systemd[1]: Starting LSB: start and stop MySQL...
 
Jan 14 10:21:36 server mysql[14399]: Starting MySQL.170114 10:21:36 mysqld_safe Logging to '/var/lib/mysql/server.err'.
 
Jan 14 10:21:36 server mysql[14399]: 170114 10:21:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
 
Jan 14 10:21:36 server mysql[14399]: /usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)
 
Jan 14 10:21:37 server mysql[14399]: ERROR!
 
Jan 14 10:21:37 server systemd[1]: mysql.service: control process exited, code=exited status=1
 
Jan 14 10:21:37 server systemd[1]: Failed to start LSB: start and stop MySQL.
 
Jan 14 10:21:37 server systemd[1]: Unit mysql.service entered failed state.
 
Jan 14 10:21:37 server systemd[1]: mysql.service failed.

After further investigation, I found the two SELinux errors from running 'sealert -a /var/log/audit/audit.log'.

SELinux is preventing /usr/bin/mysqld_safe_helper from using the setgid capability.
 
SELinux is preventing /usr/bin/mysqld_safe_helper from using the setuid capability.

For now I have fixed this by creating a local policy and restarting MariaDB, however it appears that by default SELinux prevents /usr/bin/mysqld_safe_helper making use of setuid and setgid which causes it to fail to start back up after the upgrade.

I'm not sure if this is a MariaDB specific issue, or MySQL, or maybe even the SELinux policy has changed since the last MariaDB update.

 Comments   
Comment by Jarrod Farncomb [ 2017-01-14 ]

Also wanted to note that I tested a fresh install of 10.0.29 in a new CentOS 7.3 virtual machine, upon trying to start the service with SELinux in enforcing mode I receive the same error, so it appears to be reproducible.

Comment by Hyral Sacai [ 2017-01-17 ]

Just wanted to chime in here and say that I've also ran into the exact same problem on my CentOS 7.3 server after performing 'yum update' and was able to resolve the issue by running the suggested commands the error logs brought up (with confidence, thanks to Jarrod's article on the issue)

Comment by Kolbe Kegel (Inactive) [ 2017-01-23 ]

serg at a minimum, the work you did in MDEV-11676 needs to be extended to RHEL 6 and RHEL/CentOS 7. The policy should also probably be installed on Fedora and other OSs where SELinux is used. Perhaps instead of checking for the OS name, it would be enough to check whether the sestatus tool is available (`command -v sestatus >/dev/null 2>&1`)?

Comment by Michael Newton [ 2017-01-24 ]

For the record, this is also affecting 10.1. I'm running Scientific Linux 6.8, and was impacted by this issue after updating from 10.1.14 to 10.1.21. Based on the logs, it appears in my case to only be denying setgid to the process, not setuid:

type=AVC msg=audit(1485280894.516:511332): avc:  denied  { setgid } for  pid=32564 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
 
type=SYSCALL msg=audit(1485280894.516:511332): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7f1f69e853c0 a2=10000 a3=1 items=0 ppid=32473 pid=32564 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
 
type=AVC msg=audit(1485280894.516:511333): avc:  denied  { setgid } for  pid=32564 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
 
type=SYSCALL msg=audit(1485280894.516:511333): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7f1f69e853c0 a2=40 a3=1 items=0 ppid=32473 pid=32564 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
 
type=AVC msg=audit(1485280894.546:511334): avc:  denied  { setgid } for  pid=32575 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
 
type=SYSCALL msg=audit(1485280894.546:511334): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7fd400c743c0 a2=10000 a3=1 items=0 ppid=32473 pid=32575 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
 
type=AVC msg=audit(1485280894.550:511335): avc:  denied  { setgid } for  pid=32575 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
 
type=SYSCALL msg=audit(1485280894.550:511335): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7fd400c743c0 a2=40 a3=1 items=0 ppid=32473 pid=32575 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
 
type=AVC msg=audit(1485280894.699:511336): avc:  denied  { setgid } for  pid=32579 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
 
type=SYSCALL msg=audit(1485280894.699:511336): arch=c000003e syscall=116 success=no exit=-1 a0=1 a1=7f00af9d83c0 a2=10000 a3=1 items=0 ppid=32473 pid=32579 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)
 
type=AVC msg=audit(1485280894.700:511337): avc:  denied  { setgid } for  pid=32579 comm="mysqld_safe_hel" capability=6  scontext=unconfined_u:system_r:mysqld_safe_t:s0 tcontext=unconfined_u:system_r:mysqld_safe_t:s0 tclass=capability
 
type=SYSCALL msg=audit(1485280894.700:511337): arch=c000003e syscall=106 success=no exit=-1 a0=1b a1=7f00af9d83c0 a2=40 a3=1 items=0 ppid=32473 pid=32579 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=58566 comm="mysqld_safe_hel" exe="/usr/bin/mysqld_safe_helper" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null)

Edit: after allowing setgid, then it starts complaining about not being able to do setuid, as in the original report.

Comment by Sergei Golubchik [ 2017-02-23 ]

kolbe, I took a conservative approach — while we compile and install the file with the policy everywhere, we only enable it on distributions that are known to have the issue. I believed that most distributions didn't restrict setuid/setgid in their policies and it was only CentOS/RHEL that did.

May be a conservative approach should've been the opposite, assuming that all distributions restrict setuid/setgid and enable our policy everywhere... I'll do that.

Comment by Jarrod Farncomb [ 2017-03-11 ]

I have tested MariaDB 10.0.30 on CentOS 7 and this same problem still happens so it does not seem that the bug has been resolved.

Comment by Sergei Golubchik [ 2017-03-11 ]

Same problem — do you mean it's still "/usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)" and "SELinux is preventing /usr/bin/mysqld_safe_helper from using the setuid capability." ? Is the new policy installed when you install a 10.0.30 rpm?

The post-install script has

if [ -x /usr/sbin/semodule ] ; then
 
  /usr/sbin/semodule -i /usr/share/mysql/policy/selinux/mariadb.pp
 
fi

So I don't quite see how it could not have been installed. Do you have the /usr/sbin/semodule executable?

Comment by Sergei Golubchik [ 2017-03-11 ]

Ok, ignore the question, please. I see you created MDEV-12231, let's continue there.

Generated at Thu Feb 08 07:52:40 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.