|
claudio.nanni, so, what's the request – to stop providing the scripts? I don't know if we can do it in 10.2, it will probably break things for those who uses them.
How do they cause race conditions if systemd service is disabled?
|
|
Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages.
The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system.
- Disable systemd service
- reboot the server
- init will start mariadb
- start mariadb via systemctl (systemd)
|
|
> Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages.
As a user, I would disagree. Even if I happen to have systemd on the system, it doesn't yet mean I want to be forced to change all my routines to use it, and it only.
> The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system.
It's not really a race condition, it's a human error. Same way you could say that even without systemd a user can have the server started by init script automatically and then also start it manually, by running mysqld_safe or mysqld or what not.
In any case, I'll leave it to svoj as an author of MariaDB systemd to decide.
|
|
cvicentiu kindly offered help on systemd, so reassigning to him.
|
|
>>Claudio: Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages.
>Elena: As a user, I would disagree. Even if I happen to have systemd on the system, it doesn't yet mean I want to be forced to change all my routines to use it, and it only.
Claudio: It's not that you happen to have systemd on the system, the whole distribution is either based on SysV init or SystemD, if you upgrade to a distro based on Systemd then the norm is to use that.
>>Claudio: The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system.
>Elena: It's not really a race condition, it's a human error. Same way you could say that even without systemd a user can have the server started by init script automatically and then also start it manually, by running mysqld_safe or mysqld or what not.
Claudio: I find just wrong to have init start MariaDB on a Systemd distro, why would you want init to start MariaDB on a systemd system?
There was a 'transition' time when MariaDB systemd scripts were 'not ready' and so systemd would use the init scripts but that should belong to the past.
|
|
The problem right now is that both mechanisms are enabled by default.
So after installing we have the systemd service enabled, and mysqld (without mysqld_safe) is running, and is restarted on reboot, as expected.
The sysV init file is also enabled, but only processed after the systemd service, so it just sees that the pid file already exists and a mysqld process is running under that pid, so it does not take startup actions.
Now when you disable the systemd service with "systemctl disable mariadb.service" on the next restart the sysV init file will now not see a running mysqld process, and will start one with the help of mysqld_safe.
So the problem is not so much that both files are installed, but that both are enabled.
$ systemctl list-units mariadb.service
|
UNIT LOAD ACTIVE SUB DESCRIPTION
|
mariadb.service loaded active running MariaDB 10.2.14 database server
|
|
$ chkconfig --list mysql
|
mysql 0:off 1:off 2:on 3:on 4:on 5:on 6:off
|
IMHO the sysV init file should be installed, but the service completely disabled by default, when installing on distribution release that uses SystemD by default. So these lines in the init file would need to be changed to empty
# Default-Start: 2 3 4 5
|
# Default-Stop: 0 1 6
|
and users who want to use the sysV script instead of the systemd service would have to enable it explicitly with chkconfig (RedHat/CentOS) or update-rc.d (Debian/Ubuntu)
|
|
Hi hholzgra,
so if I understand, this is what you propose.
Keep providing both version (systemd/sysV init) with the following condition (and version):
1/ IF systemd is installed AND used by default, THEN provide a disabled version of the init script ;
2/ IF systemd is not installed OR is installed but not used by default, THEN provide an enabled version of the init script ;
The second condition seems pretty difficult test IMO but I have not yet looked into it, maybe not.
But this situation where systemd is not used by default should be tested because we could have some unwanted issues.
And of course we should document how to enable sysv init script for users that are willing to use them.
And finally, I totally agree with elenst, this is not our decision to force users to switch to systemd and as long as sysv init script can be used we have to keep providing sysv init script.
So this is what I think but I am leaving this to cvicentiu and to discussion.
|
|
Ok, this affects all versions and can easily be reproduced, it also affects the official Debian package (10.1.26-MariaDB-0+deb9u1).
Install any version from the official mariadb repo (https://downloads.mariadb.org/mariadb/repositories/) or official Debian stable version;
$ sudo systemctl stop mariadb
|
$ sudo systemctl disable mariadb
|
$ sudo reboot
|
$ sudo systemctl status mariadb
|
● mariadb.service - MariaDB 10.1.36 database server
|
Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
|
Drop-In: /etc/systemd/system/mariadb.service.d
|
└─migrated-from-my.cnf-settings.conf
|
Active: inactive (dead)
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
$ sudo /etc/init.d/mysql status
|
● mysql.service - LSB: Start and stop the mysql database server daemon
|
Loaded: loaded (/etc/init.d/mysql; generated; vendor preset: enabled)
|
Active: active (running) since Tue 2018-10-30 16:52:57 -03; 3min 46s ago
|
Docs: man:systemd-sysv-generator(8)
|
Process: 419 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
|
Tasks: 30 (limit: 4915)
|
CGroup: /system.slice/mysql.service
|
├─463 /bin/bash /usr/bin/mysqld_safe
|
├─672 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/v…06
|
└─673 logger -t mysqld -p daemon error
|
$ sudo mysql
|
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 45
|
Server version: 10.1.36-MariaDB-1~stretch mariadb.org binary distribution
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
|
So this is indeed critical as users will not understand why mariadb server is reactivated "magically" after reboot.
|
|
Ok the problem comes from the naming (mariadb/mysql).
Normally when disabling a service, the following script is used `/lib/systemd/systemd-sysv-install` in order to synchronize systemd disabled/enabled services with SysV service script.
1/ So with mysql it works because there is a `/etc/init.d/mysql` script:
$ sudo systemctl disable mysql
|
Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.
|
Executing: /lib/systemd/systemd-sysv-install disable mysql
|
Removed /etc/systemd/system/mysql.service.
|
Removed /etc/systemd/system/multi-user.target.wants/mariadb.service.
|
Removed /etc/systemd/system/mysqld.service.
|
2/ But with mariadb, it is not working since there is no `/etc/init.d/mariadb` script (not calling /lib/systemd/systemd-sysv-install):
$ sudo systemctl disable mariadb
|
Removed /etc/systemd/system/mysql.service.
|
Removed /etc/systemd/system/multi-user.target.wants/mariadb.service.
|
Removed /etc/systemd/system/mysqld.service.
|
For the moment, I can not propose a clean solution to this.
|
|
Ok, I posted on systemd-devel ML and I get good information from Michael:
https://lists.freedesktop.org/archives/systemd-devel/2018-November/041658.html
He updated the Debian wiki:
https://wiki.debian.org/Teams/pkg-systemd/Packaging#systemd_unit_files_naming_and_installation
I will try this on upstream 10.1 release.
|
|
This seems to be working fine.
Please find attached a bash script that I developed quickly to be able to reproduce this and test this quickly:
# remove previously installed services
|
rm -vf /etc/systemd/system/mysql*.service
|
rm -vf /lib/systemd/system/mysql*.service
|
|
# remove aliases entry from services
|
sed -i -r 's/Alias=(mysql|mysqld).service//g' /lib/systemd/system/mariadb.service
|
sed -i -r 's/Alias=(mysql|mysqld).service//g' /lib/systemd/system/mariadb@.service
|
|
# create symlinks
|
cd /lib/systemd/system
|
ln -s mariadb.service mysql.service
|
ln -s mariadb.service mysqld.service
|
|
# update systemd services
|
systemctl daemon-reload
|
config_systemd.sh
|
|
Please update the bug report title to match the problem and solution in the conclusion.
This seems to be like rather risky business so I suggest that you apply this on 10.4 (current development version) first and only later backport to production versions when you are 100 % sure this is the correct thing to do.
|
|
I have renamed the title and created a branch on my fork with the fix (48358cca9f6) https://github.com/fauust/mariadb/tree/fix-MDEV-15526-10.4.
I will test this in the next days...
|
|
What functionality does providing symlinks provide explicitly? Looks to be just adding to a mess to me. It would seem to break starting mariadb.service and mysqld.service at the same time (as might happen if both are listed as service dependences), as nothing will really be aware of the relationship between them. Both would start, one would fail, hopefully without impact (likely duplicate process detection is quite mature).
If a user on a systemd distro wants a sysv init script they can disable the mariadb@service take it out of /usr/share somewhere and install it themselves. Installing both just causes problems as MDEV-10797 highlights. Its not sane to try to support both at the same time, the ugly hacks that attempt to provide dependency relationships between them are flaky.
There is no such thing as a disabled /etc/init.d/mysql|mariadb service. The existence this file on a systemd system has some active status, changing `Default-Start/Stop: ` lines isn't sufficient as systemd and some init systems doesn't read them. Install it in /usr/share/... for the users that want to create a mess of their system, but please, keep it clean for everyone else.
The assumption that there are users consciously aware and appreciative of the broken packaging provided seems odd. Sure fix 10.4 first, but don't forget the backport.
|
|
We do something similar for Arch Linux:
0001-arch-specific.patch at line 100
PKGBUILD at line 194
|
|
Thanks a lot eworm for you input, indeed this looks pretty similar.
Are you at Arch Linux deploying sysv init script? I saw this but I would appreciate an explanation of your policy regarding sysv init support if systemd is the default init system.
There seems to be a consensus regarding not using Aliases, now we should confirm that using symlink is the right thing to do.
danblack, these symlinks have no other utility than permitting the usage of the 2 names (mariadb or mysql) with systemctl commands (even if it also seems to resolve the disable issue that hholzgra reported and that seems to be caused by sysv init script presence).
This for instance, should behave exactly equal :
$ sudo systemctl start|stop|reload|disable|enable mariadb
|
$ sudo systemctl start|stop|reload|disable|enable mysql
|
|
|
We do not ship any sysvinit scripts with Arch Linux.
Using symlinks is the right thing to do. In fact the Alias= makes systemd create symlinks in /etc/systemd/system/... But these are available after enabling the service only.
|
|
Tested symlinks, seems to be exhibiting the right behaviour, for now.
$ pwd
|
/etc/systemd/system
|
$ cat >> mariadbdependentservice.service
|
[Unit]
|
Requires=mariadb.service
|
|
[Service]
|
ExecStart=/bin/sleep 3600
|
$ cat >> mysqldependentservice.service
|
[Unit]
|
Requires=mysql.service
|
[Service]
|
ExecStart=/bin/sleep 3600
|
$ ln -s mariadb.service mysql.service
|
$ systemctl daemon-reload
|
$ systemctl start mariadbdependentservice.service
|
$ systemctl status mariadbdependentservice.service
|
● mariadbdependentservice.service
|
Loaded: loaded (/etc/systemd/system/mariadbdependentservice.service; static; vendor preset: disabled)
|
Active: active (running) since Thu 2019-02-07 08:31:52 AEDT; 8s ago
|
Main PID: 2269 (sleep)
|
Tasks: 1 (limit: 4915)
|
Memory: 160.0K
|
CPU: 2ms
|
CGroup: /system.slice/mariadbdependentservice.service
|
└─2269 /bin/sleep 3600
|
|
Feb 07 08:31:52 volution systemd[1]: Started mariadbdependentservice.service.
|
$ systemctl status mariadb.service
|
● mariadb.service - MariaDB @VERSION@ database server
|
Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)
|
Drop-In: /usr/lib/systemd/system/mariadb.service.d
|
└─tokudb.conf
|
Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 16s ago
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Main PID: 2271 (mysqld)
|
Status: "Taking your SQL requests now..."
|
Tasks: 68 (limit: 4915)
|
Memory: 246.0M
|
CPU: 806ms
|
CGroup: /system.slice/mariadb.service
|
└─2271 /usr/libexec/mysqld
|
|
Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server...
|
Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07 8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ...
|
Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server.
|
$ systemctl start mysqldependentservice.service
|
$ systemctl status mysqldependentservice.service
|
● mysqldependentservice.service
|
Loaded: loaded (/etc/systemd/system/mysqldependentservice.service; static; vendor preset: disabled)
|
Active: active (running) since Thu 2019-02-07 08:32:38 AEDT; 7s ago
|
Main PID: 2375 (sleep)
|
Tasks: 1 (limit: 4915)
|
Memory: 248.0K
|
CPU: 2ms
|
CGroup: /system.slice/mysqldependentservice.service
|
└─2375 /bin/sleep 3600
|
|
Feb 07 08:32:38 volution systemd[1]: Started mysqldependentservice.service.
|
$ systemctl status mysql.service
|
● mariadb.service - MariaDB @VERSION@ database server
|
Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)
|
Drop-In: /usr/lib/systemd/system/mariadb.service.d
|
└─tokudb.conf
|
Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 1min 6s ago
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Main PID: 2271 (mysqld)
|
Status: "Taking your SQL requests now..."
|
Tasks: 68 (limit: 4915)
|
Memory: 246.1M
|
CPU: 857ms
|
CGroup: /system.slice/mariadb.service
|
└─2271 /usr/libexec/mysqld
|
|
Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server...
|
Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07 8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ...
|
Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server.
|
$ systemctl stop mysqldependentservice.service
|
$ systemctl status mysqldependentservice.service
|
● mysqldependentservice.service
|
Loaded: loaded (/etc/systemd/system/mysqldependentservice.service; static; vendor preset: disabled)
|
Active: inactive (dead)
|
|
Feb 07 08:32:38 volution systemd[1]: Started mysqldependentservice.service.
|
Feb 07 08:34:36 volution systemd[1]: Stopping mysqldependentservice.service...
|
Feb 07 08:34:36 volution systemd[1]: Stopped mysqldependentservice.service.
|
Feb 07 08:34:36 volution systemd[1]: mysqldependentservice.service: Consumed 2ms CPU time
|
$ systemctl status mysql.service
|
● mariadb.service - MariaDB @VERSION@ database server
|
Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)
|
Drop-In: /usr/lib/systemd/system/mariadb.service.d
|
└─tokudb.conf
|
Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 22min ago
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Main PID: 2271 (mysqld)
|
Status: "Taking your SQL requests now..."
|
Tasks: 68 (limit: 4915)
|
Memory: 246.2M
|
CPU: 2.108s
|
CGroup: /system.slice/mariadb.service
|
└─2271 /usr/libexec/mysqld
|
|
Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server...
|
Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07 8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ...
|
Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server.
|
$ systemctl stop mysql.service
|
$ systemctl status mariadb.service
|
● mariadb.service - MariaDB @VERSION@ database server
|
Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)
|
Drop-In: /usr/lib/systemd/system/mariadb.service.d
|
└─tokudb.conf
|
Active: inactive (dead)
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
|
|
Thanks danblack, can you also check the disable|enable command?
sudo systemctl disable mariadb
|
sudo reboot
|
sudo systemctl status mariadb # should be disabled
|
And the same with mysql (but mariadb is more significant since there is no /etc/init.d/mariadb init script).
I will propose a PR for theses modifications from my branch https://github.com/fauust/mariadb/tree/fix-MDEV-15526-10.4.
|
|
PR submitted https://github.com/MariaDB/server/pull/1172
|
|
Another user stuffed around by the existence of /etc/init.d/mysql
https://stackoverflow.com/questions/54662100/mariadb-server-is-down-but-i-can-still-login-with-mysql-client/54662193?noredirect=1#comment96119410_54662193
Please stop installing it on systemd systems.
|
|
danblack this problem is resolved by removing aliases and creating symlinks (I have just tested it again on last 10.4 official version):
https://github.com/MariaDB/server/pull/1172
As discussed, the move of not providing "old" sysv init files is not easy to do IMHO because I know a lot of sysadmin (on Debian at least) that are still using sysv init scripts and I still agree with elenst.
|
|
I think that this report renaming from " Latest packages on Systemd still install SysV init service file /etc/init.d/mysql" to "Systemd unit files naming and installation (alias in unit are not recommended, create symlink)" does not help people with this problem in finding this MDEV. Most people report(and so research) what they observe, not the underlying technical reasons for some observed behaviour.
|
|
Good point claudio.nanni, I will find a better title (but naming problem is also important IMO in that title).
|
|
I appreciate that there are people that want to use sysv init scripts. I'm all for a sysv init script continuing to exist for people that are using those. The rest of the users either don't know/care which init system is there, or want to use the existing system (like distros are mean to do in providing packages). In these cases they want packages that can be updated (without breaking in the middle of installation), and to be able to disable mariadb as a service. Currently this isn't the case.
serg even suggested to install it in a @sharedir@.
To help get to some resolution, would it help if we break this up into a couple of usage scenarios for systemd enabled systems:
| Case prior to installation |
Recommended packaging action |
Pros |
Cons |
| Greenfields - no prior install |
install sysv init script to @sharedir@ |
Eliminates all incompatibilities with sysv scripts |
A sysv user will need to realise that its systemd OS installed, or copy the sysv script to /etc/init.d |
| Updating from a previous MariaDB packaged version (10.1>=) |
Test if they are using sysv, if so install to /etc/init.d, otherwise to @sharedir@ and remove /etc/init.d/mysql |
Tries to be compatible with what users want and keeps sysv init scripts out of the way if systemd was used |
|
| Updating from a previous MariaDB packaged version <10.1 |
Not supported? Install on the assumption that a systemd is desired |
Simplicity and conforming to the expectations of the distro |
They may want init.d, in which case its installed to @shardir@ |
| Updates from distro MariaDB packages |
TODO |
|
|
| Updates from distro MySQL packages |
TODO |
|
|
| Updates from Oracle MySQL packages |
TODO |
|
|
Yes this is going to get complicated. Yes some automated testing probably should be done to (or at least documented process). This is the 12th most watched open bug, like MDEV-10797, because users do care. I realise packaging isn't easy. While a lot of care should be taken to maintain compatibility, it always comes with a maintenance burden. A new 10.4 release is a good opportunity to try to get this resolved for all users while its still in beta. There may be a couple of breaks for systems with maintaince debt for which we can develop a richer set of test case, but hopefully most users will get a smoother path.
MDEV-10797 has lingered without fixing for the same causes as this for over two major releases because there wasn't the willingness to fix. There seems to be the mentality that somehow outstanding bugs are more tolerable than new bugs, even if fixing helps more people than than what the new bug affects. This is a poor risk management decision. I looking forward to the resolution of both these packaging bugs.
|
|
faust Stop ignoring this problem and considering it too hard. Please engage in a discussion of what could be an acceptable solution for elenst, your sysv users and all the users who use systemd, want upgrades to work and want mariadb to be packaged professionally.
|
|
danblack, I provided my opinion as a user only. I'm not going to argue against any decision from a position of MariaDB engineering member, neither do I have the authority to forbid the change even if I wanted to. However, to my understanding of the policies, if the script is going to be removed, it must be done before GA (or even before RC, I'm not quite sure which).
|
|
Although not clearly stated in this issue, this was the Debian packaging side and as such is unresolved.
debian/rules -> override_dh_installinit-arch also needs fixing as it installs init.d script and systemd script of different names, causing problems exactly like the RPM packaging and is inconsistent with Debians packaging standards (written because of the problems described here):
https://wiki.debian.org/Teams/pkg-systemd/Packaging?action=show&redirect=systemd%2FPackaging#systemd_unit_files_naming_and_installation
|
|
For refrence, a comparison of current Debian mariadb-10.3 rules snippet to the left, and upstream mariadb-10.4 rules file to the right.
I am not an expert on systemd and since Docker does not support systemd if don't have any easily available testing setup to check which one works better, but knowing the alternative implementation might be useful.
|
|
Taking an interest in the dominate system manager, on the dominate OS that MariaDB supports, in which packaging pays a critical role, would be useful with bugs like this (which attract watchers while nothing is resolved).
I understand the buildbot is docker based which has a complicated relationship with systemd however there is https://hub.docker.com/r/centos/systemd and https://hub.docker.com/r/jrei/systemd-debian (which might not be official, but has the same principle as the Centos Dockerfile) and these could start to form the basis of docker systemd tests, and at the same time, giving you the systemd skills needed to maintain Debian packages.
The notes about having a init script having the same name as the service file where there on the above wiki page in 2013 and is not recommended, which mariadb and debian does. I suggest /etc/init.d/mysql -> /etc/init.d/mariadb as the first move perhaps?
Summarising a vagrant conversation with colleague that could be used for full VM tests (and galera):
- Use vagrant for the minimal amount possible (avoid complex features of vagrant)
- Use scripts to wait for full networking as vagrants doesn't always do this right
- Use ansible to install packages and trigger tests once vagrant boxes
Vagrant implementation as a result of this was these files: https://git.samba.org/?p=autocluster.git;a=tree;f=vagrant;hb=HEAD, tests aren't in this repository.
|
|
https://github.com/MariaDB/server/pull/593#issuecomment-479338028
|
|
Thank you danblack for your suggestion on how we could implement our systemd tests.
teodor do you think this is doable on buildbot? I have looked at https://hub.docker.com/r/jrei/systemd-debian/dockerfile and I if I am understanding correctly, BB worker/runner should be a VM or a host and should share `/sys/fs/cgroup` with the container.
Then I would be happy to implement the first basic tests:
- reload|restart|start|stop|enable|disable
- multi-instance mode
- ...
And maybe Daniel will give us some good test suggestions.
|
|
faust all assumptions correct. Regarding /sys/fs/cgroup, I think, not sure by any means, that each container should have a different host subdirectory off this mounted though to give each container a different path in the cgroup hierarchy so /sys/fs/cgroup/testX -> /sys/fs/cgroup. While container based systemd tests will be quite effective for some basic things, eventually some vm/vagrant infrastructure will be needed to effectively test selinux (there is no separation from container host in a selinux context), and would make galera based packaging tests more realistic. Happy to provide other suggestions however lets put that in a new issue.
|
|
faust I'm not too knowledgeable of systemd internals and workings. One question though from Buildbot's perspective, does mounting /sys/fs/cgroup from host to guests breach the security and isolation contracts that Docker provides? Remember that these builders will run on Github pull requests too and we don't want a mere opening of a PR to be able to compromise our worker machines. In a sense, Docker is Systemd for containers, as it uses the same kernel tools (cgroups, namespaces) to limit, isolate, control and cleanup the user namespace (apart from zombie processes).
|
|
In a Debian upgrade bug report I now used the systemd-debian images to test and the steps are outlined in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926231#58 in case somebody wants to replicate that.
|
|
teodor this is a very good question and consideration.
I have just found yesterday this documentation that I did not had time to read with calm.
It maybe interesting for you too :
https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/
https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/
And thanks otto I will look at your link too.
|
|
axel, there's a long discussion here, but the only important part is this comment.
I've merged https://github.com/MariaDB/server/commit/74c9872a9aae825ca6ab73feff048ff2ccb4beb8 but danblack noted that it only fixes RPM, so Debian still needs to be fixed. This is why this issue is not closed yet, it's waiting for a Debian-specific fix.
|
|
serg thanks for the fix. does debian/rules -> override_dh_installinit-arch: need to remove dh_installinit --name=mysql – defaults 19 21? It looks odd since the mysql.init is now in /usr/share.
teodor I see you haven't made much progress with systemd docker configs or scripts in the last few months, I did mention that I'd help in a new issue. https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/ is quite new as a blog on non-priviledged docker container without cgroup hacks. Hope it works for you.
|
|
danblack, yes, I suppose it should go, thanks
|
|
Hi!
I believe there is still some problematic situation with this (at least on Debian):
- versions < 10.4 (Debian Buster provides 10.3 branch);
- upgrading from previous versions.
This https://github.com/MariaDB/server/pull/1172 resolves it by removing dynamically created symlinks (via aliases) and by creating those symlinks on postinst (so symlinks are never removed and synchronization between sysvinit and systemd is working).
Then, I think that we should remove symlink postinst creation in 10.5 branch next release (and communicate that systemctl commands does not support mysql naming anymore). That would be the cleanest situation for the future IMO.
If you think that this concerns only Debian, I will fix this downstream only.
|
|
After a lot of research into this:
Root problem:
There is no /etc/init.d/mariadb so disabling mariadb in systemd with `systemctl disable mariadb` does not trigger /lib/systemd/systemd-sysv-install for service name 'mysql' and thus disabling the sysvinit service mysql service stays enabled:
Synchronizing state of mariadb.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mariadb
Same problem applies for running `systemctl disable mysqld`.
The systemd itself supports running `systemd-sysv-install` only if the sysvinit=systemd name. Neither service alias= or service symlinks are supported by this feature.
The problem only applies for systems with systemd. On other systems the command `systemctl disable mariadb` would fail because there is no sysvinit file with that name (note that the command `systemctl` may work on systems that lack systemd but have a systemctl-shim installed.
Correct solution here is to:
- don't install any sysvinit files on systems that have systemd (in effect now, but implemented wrongly)
- keep installing sysvinit files on systems that lack systemd, otherwise they would not work at all (needs to be fixed)
- systemctl aliases are good, they increase compatibility with mysql - skipping them to "fix" this bug is the wrong tradeoff
Code changes:
|
|
A similar example is package openssh-server in Debian. It has only /etc/init.d/ssh but both /lib/systemd/system/ssh.service with alias=sshd. There is also a symlink `/etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service`.
There is a related upstream bug report in https://github.com/systemd/systemd/issues/7874 but this exact problem "systemctl disable does not disable sysvinit with alias names" apparently has never been filed upstream, so I filed it now: https://github.com/systemd/systemd/issues/15394
systemd code where it tires to detect if there is a sysvinit file with identical name: https://github.com/systemd/systemd/blob/4c39820562e562b98c2fd269ad3a4f84949d6059/src/systemctl/systemctl.c#L6680
|
|
otto thanks for looking deeply into this!
I agree with most your solution proposal.
But I believe this one is not totally correct:
> systemctl aliases are good, they increase compatibility with mysql - skipping them to "fix" this bug is the wrong tradeoff
Aliases is not the best solution to increase compatibility with mysql. Instead, "shipping a (static) symlink in the package is the second best thing you can do to align the sysv init script name and the service name, The best you can do (obviously) is to use the same name for the sysv init script and the service file.", I am quoting M. Biebl, see: https://github.com/MariaDB/server/pull/1172#discussion_r305863850
|
|
They are called aliases also when they are implemented via symlinks.
I have been testing with a recent Debian Sid environment and I am actually at the moment unable to reproduce the original bug. Seems like the `/lib/lsb/init-functions` which ends up calling `/lib/lsb/init-functions.d/40-systemd` handles this correctly already.
Sources at https://sources.debian.org/src/systemd/245.4-4/debian/extra/init-functions.d/40-systemd/
|
|
faust Just to be sure, can you please test that latest 10.5 surely does not suffer from this issue anymore? And if there is some corner case where it does suffer, then document how to reproduce that.
|
|
Hi otto!
I have tested the latest upstream version (10.5.4) against buster and sid.
I am not able to reproduce the issue, and synchronization (between sysvinit and systemd) seems very sane to me, also disable works (and persists after reboot) in both cases (mariadb or mysql).
The unique "not so sane" but not problematic scenario is the following (change on /etc are tracked with etckeeper):
$ cd /etc
|
$ sudo apt install etckeeper
|
$ sudo systemctl disable mariadb
|
$ sudo git status
|
On branch master
|
Changes not staged for commit:
|
(use "git add/rm <file>..." to update what will be committed)
|
(use "git restore <file>..." to discard changes in working directory)
|
deleted: rc2.d/S01mariadb
|
deleted: rc3.d/S01mariadb
|
deleted: rc4.d/S01mariadb
|
deleted: rc5.d/S01mariadb
|
deleted: systemd/system/multi-user.target.wants/mariadb.service
|
|
Untracked files:
|
(use "git add <file>..." to include in what will be committed)
|
rc2.d/K01mariadb
|
rc3.d/K01mariadb
|
rc4.d/K01mariadb
|
rc5.d/K01mariadb
|
$ sudo systemctl enable mysql
|
On branch master
|
Changes not staged for commit:
|
(use "git add/rm <file>..." to update what will be committed)
|
(use "git restore <file>..." to discard changes in working directory)
|
deleted: rc2.d/S01mariadb
|
deleted: rc3.d/S01mariadb
|
deleted: rc4.d/S01mariadb
|
deleted: rc5.d/S01mariadb
|
|
Untracked files:
|
(use "git add <file>..." to include in what will be committed)
|
rc2.d/K01mariadb
|
rc3.d/K01mariadb
|
rc4.d/K01mariadb
|
rc5.d/K01mariadb
|
$ sudo reboot
|
$ sudo systemctl status mariadb
|
● mariadb.service - MariaDB 10.5.4 database server
|
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
|
Drop-In: /etc/systemd/system/mariadb.service.d
|
└─migrated-from-my.cnf-settings.conf
|
Active: active (running) since Thu 2020-05-21 11:34:42 CEST; 25min ago
|
...
|
Systemd is starting mariadb even if sysv runlevel files were not synchronized by systemctl enable mysql.
But this is a already known behavior (https://github.com/systemd/systemd/issues/15394).
|
|
Thanks for verifying we are all good now.
Since in systemd the 'mysql' is a alias for 'mariadb', it is correct that 'sudo systemctl enable mysql' will enable MariaDB and MariaDB will be running after the next reboot. It does not matter what state the rcXd./XXmariadb files are in, since if the system has systemd (and systemctl), it will do what systemd says ignoring rcXd./XXmariadb status.
|
Debian vs upstream comparison.png
config_systemd.sh
