Galera Docs: SELinux makes server from RPM installation throw errors 2 (No such file or directory) in the log and crash (MDEV-4238)

[MDEV-4242] Galera in buildbot: SELinux in CentOS 5 and RHEL5 builders breaks Galera installation tests Created: 2013-03-05  Updated: 2013-09-28  Resolved: 2013-09-28

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Technical task Priority: Minor
Reporter: Elena Stepanova Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: buildbot, galera
Environment:

CentOS 5 buildbot, SELinux

Issue Links:
Relates
relates to MDEV-4344 Galera: Server crashes on setting wsr... Open

 Description   

Galera installation tests on CentOS 5 in buildbot fail with an obscure "Lost connection to MySQL server during query" error on enabling wsrep_provider:
http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos5-x86/builds/2248/steps/test/logs/stdio
http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos5-amd64/builds/1981/steps/test/logs/stdio

I did some digging and it turned out that the actual cause is strict SELinux settings which Galera cannot deal with (it's documented in Galera FAQ: http://www.codership.com/wiki/doku.php?id=faq#qnothing_works_damnit)

In my experiments, it was enough to switch the level from Enforcing which it is in the VM now, to Permissive at runtime using setenforce. Setting the level permanently and rebooting the machine also helped, of course.

If the level is set/kept high on purpose, please consider a conditional setting it to Permissive at runtime for Galera tests (if you do so, please make sure setenforce comes with the full path, it's not on the PATH). If there is no particular reason to have it Enforcing by default, maybe it's easier to reconfigure it permanently.

I was checking 64-bit build, but I suppose the reason of the failure on x86 is the same.

 Comments   
Comment by Daniel Bartholomew [ 2013-03-05 ]

Let me know if other VMs need the same thing done to them and/or if we need to do more to the CentOS VMs.

Comment by Elena Stepanova [ 2013-03-05 ]

Yes, thanks, lets wait and see how the next buildbot run goes (I don't want to trigger re-run now since the buildbot is busy as it is, and our question is not urgent).

Comment by Daniel Bartholomew [ 2013-03-21 ]

Is this issue resolved now?

Comment by Elena Stepanova [ 2013-03-21 ]

We haven't had a new Galera build in buildbot since then, so the status is still unknown.

Comment by Elena Stepanova [ 2013-03-27 ]

We finally have a new build. Strangely, the fix worked for amd64, but not for x86.
amd64 (good):http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos5-amd64/builds/2016/steps/test/logs/stdio
x86 (not good): http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos5-x86/builds/2285/steps/test/logs/stdio

If you are sure that x86 was modified, I'll try later to see what else might cause the problem, but it looks very much like the previous one

Comment by Daniel Bartholomew [ 2013-03-27 ]

I've checked both of the x86 VMs,

  • vm-centos5-i386-install.qcow2
  • vm-centos6-i386-install.qcow2

And they both have SELINUX=permissive set.

Very strange that the fix on amd64 worked and the exact same fix on x86 didn't.

Comment by Elena Stepanova [ 2013-03-27 ]

It means I'll need to repeat the same exercise of manually reproducing the problem in a cloned VM to see what's going on there.

Comment by Elena Stepanova [ 2013-03-31 ]

Hi Daniel,

It turns out that wsrep crashes when the server is run with an old version of libgcc. The VM (vm-centos5-i386-install.qcow2) has
libgcc.i386 4.1.2-44.el5

while currently available is
libgcc i386 4.1.2-54.el5

I'll file a bug report for Galera so Codership could check why it crashes, but meanwhile we might upgrade the library in the VM (or maybe not, I don't know what our VM upgrade policies are). If you think it makes sense to upgrade, please go ahead; if not, please close this issue as fixed, since the initial one with SELinux has been resolved anyway.

Comment by Daniel Bartholomew [ 2013-04-01 ]

I'm fine with upgrading the library in the VM

Comment by Elena Stepanova [ 2013-04-01 ]

If you do that, could you please backup a VM image with the old library, so I could re-test the fix if the bug is fixed?
(the bug was filed as MDEV-4344)

Comment by Elena Stepanova [ 2013-06-15 ]

With the new version of Galera, SELinux breaks tests on RHEL5 (I've seen and checked failures on x86, but I suppose it's the same on amd64, it just currently lags behind).
The failure is a bit different, it hides inside the init script, and shows up in messages as follows:

SELinux is preventing the mysqld from using potentially mislabeled files (./tmp.wBeQpv4640). For complete SELinux messages. run sealert -l 3efbde02-94d1-4e88-88ce-88a8a5cd3987

Switching to Permissive fixes it, please do so for Galera tests.

I suppose if we don't want to duplicate VM images and don't want to change it for regular tests, we can switch it off dynamically for the test, adding a conditional step
sudo bash -c "echo 0 > /selinux/enforce"
or alike.

Comment by Elena Stepanova [ 2013-09-28 ]

RHEL and CentOS builders are green now, closing as fixed.

Generated at Thu Feb 08 06:54:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.