[MDEV-15526] SysV init service deployed file '/etc/init.d/mysql' prevents systemctl disable command to work correctly (mariadb|mysql naming support) (debian/ubuntu) - Jira

Elena Stepanova added a comment - 2018-03-12 16:51

claudio.nanni, so, what's the request – to stop providing the scripts? I don't know if we can do it in 10.2, it will probably break things for those who uses them.
How do they cause race conditions if systemd service is disabled?

Elena Stepanova added a comment - 2018-03-12 16:51 claudio.nanni , so, what's the request – to stop providing the scripts? I don't know if we can do it in 10.2, it will probably break things for those who uses them. How do they cause race conditions if systemd service is disabled?

Claudio Nanni added a comment - 2018-03-29 08:50

Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages.
The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system.

Disable systemd service
reboot the server
init will start mariadb
start mariadb via systemctl (systemd)

Claudio Nanni added a comment - 2018-03-29 08:50 Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages. The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system. Disable systemd service reboot the server init will start mariadb start mariadb via systemctl (systemd)

Elena Stepanova added a comment - 2018-03-29 10:41

> Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages.

As a user, I would disagree. Even if I happen to have systemd on the system, it doesn't yet mean I want to be forced to change all my routines to use it, and it only.

> The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system.

It's not really a race condition, it's a human error. Same way you could say that even without systemd a user can have the server started by init script automatically and then also start it manually, by running mysqld_safe or mysqld or what not.

In any case, I'll leave it to svoj as an author of MariaDB systemd to decide.

Elena Stepanova added a comment - 2018-03-29 10:41 > Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages. As a user, I would disagree. Even if I happen to have systemd on the system, it doesn't yet mean I want to be forced to change all my routines to use it, and it only. > The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system. It's not really a race condition, it's a human error. Same way you could say that even without systemd a user can have the server started by init script automatically and then also start it manually, by running mysqld_safe or mysqld or what not. In any case, I'll leave it to svoj as an author of MariaDB systemd to decide.

Sergey Vojtovich added a comment - 2018-03-29 11:03

cvicentiu kindly offered help on systemd, so reassigning to him.

Sergey Vojtovich added a comment - 2018-03-29 11:03 cvicentiu kindly offered help on systemd, so reassigning to him.

Claudio Nanni added a comment - 2018-04-09 11:18

>>Claudio: Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages.

>Elena: As a user, I would disagree. Even if I happen to have systemd on the system, it doesn't yet mean I want to be forced to change all my routines to use it, and it only.

Claudio: It's not that you happen to have systemd on the system, the whole distribution is either based on SysV init or SystemD, if you upgrade to a distro based on Systemd then the norm is to use that.

>>Claudio: The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system.

>Elena: It's not really a race condition, it's a human error. Same way you could say that even without systemd a user can have the server started by init script automatically and then also start it manually, by running mysqld_safe or mysqld or what not.

Claudio: I find just wrong to have init start MariaDB on a Systemd distro, why would you want init to start MariaDB on a systemd system?
There was a 'transition' time when MariaDB systemd scripts were 'not ready' and so systemd would use the init scripts but that should belong to the past.

Claudio Nanni added a comment - 2018-04-09 11:18 >>Claudio: Elena, on systemd the init scripts should not be installed even less enabled, moreover on a systemd distribution the package is a dedicated one so maybe it's possible to exclude them completely from such packages. >Elena: As a user, I would disagree. Even if I happen to have systemd on the system, it doesn't yet mean I want to be forced to change all my routines to use it, and it only. Claudio: It's not that you happen to have systemd on the system, the whole distribution is either based on SysV init or SystemD, if you upgrade to a distro based on Systemd then the norm is to use that. >>Claudio: The race condition happens because even if you disable the systemd service you can still start it manually, and in any case you don't usually expect init to start a service if you are on a systemd system. >Elena: It's not really a race condition, it's a human error. Same way you could say that even without systemd a user can have the server started by init script automatically and then also start it manually, by running mysqld_safe or mysqld or what not. Claudio: I find just wrong to have init start MariaDB on a Systemd distro, why would you want init to start MariaDB on a systemd system? There was a 'transition' time when MariaDB systemd scripts were 'not ready' and so systemd would use the init scripts but that should belong to the past.

Hartmut Holzgraefe added a comment - 2018-04-09 12:31

The problem right now is that both mechanisms are enabled by default.

So after installing we have the systemd service enabled, and mysqld (without mysqld_safe) is running, and is restarted on reboot, as expected.

The sysV init file is also enabled, but only processed after the systemd service, so it just sees that the pid file already exists and a mysqld process is running under that pid, so it does not take startup actions.

Now when you disable the systemd service with "systemctl disable mariadb.service" on the next restart the sysV init file will now not see a running mysqld process, and will start one with the help of mysqld_safe.

So the problem is not so much that both files are installed, but that both are enabled.

$ systemctl list-units mariadb.service

UNIT            LOAD   ACTIVE SUB     DESCRIPTION

mariadb.service loaded active running MariaDB 10.2.14 database server

$ chkconfig --list mysql

mysql          	0:off	1:off	2:on	3:on	4:on	5:on	6:off

IMHO the sysV init file should be installed, but the service completely disabled by default, when installing on distribution release that uses SystemD by default. So these lines in the init file would need to be changed to empty

# Default-Start:  2 3 4 5

# Default-Stop: 0 1 6

and users who want to use the sysV script instead of the systemd service would have to enable it explicitly with chkconfig (RedHat/CentOS) or update-rc.d (Debian/Ubuntu)

Hartmut Holzgraefe added a comment - 2018-04-09 12:31 The problem right now is that both mechanisms are enabled by default. So after installing we have the systemd service enabled, and mysqld (without mysqld_safe) is running, and is restarted on reboot, as expected. The sysV init file is also enabled, but only processed after the systemd service, so it just sees that the pid file already exists and a mysqld process is running under that pid, so it does not take startup actions. Now when you disable the systemd service with "systemctl disable mariadb.service" on the next restart the sysV init file will now not see a running mysqld process, and will start one with the help of mysqld_safe. So the problem is not so much that both files are installed, but that both are enabled. $ systemctl list-units mariadb.service UNIT LOAD ACTIVE SUB DESCRIPTION mariadb.service loaded active running MariaDB 10.2.14 database server $ chkconfig --list mysql mysql 0:off 1:off 2:on 3:on 4:on 5:on 6:off IMHO the sysV init file should be installed, but the service completely disabled by default, when installing on distribution release that uses SystemD by default. So these lines in the init file would need to be changed to empty # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 and users who want to use the sysV script instead of the systemd service would have to enable it explicitly with chkconfig (RedHat/CentOS) or update-rc.d (Debian/Ubuntu)

Faustin Lammler added a comment - 2018-10-30 19:21 - edited

Hi hholzgra,
so if I understand, this is what you propose.

Keep providing both version (systemd/sysV init) with the following condition (and version):
1/ IF systemd is installed AND used by default, THEN provide a disabled version of the init script ;
2/ IF systemd is not installed OR is installed but not used by default, THEN provide an enabled version of the init script ;

The second condition seems pretty difficult test IMO but I have not yet looked into it, maybe not.
But this situation where systemd is not used by default should be tested because we could have some unwanted issues.

And of course we should document how to enable sysv init script for users that are willing to use them.

And finally, I totally agree with elenst, this is not our decision to force users to switch to systemd and as long as sysv init script can be used we have to keep providing sysv init script.

So this is what I think but I am leaving this to cvicentiu and to discussion.

Faustin Lammler added a comment - 2018-10-30 19:21 - edited Hi hholzgra , so if I understand, this is what you propose. Keep providing both version (systemd/sysV init) with the following condition (and version): 1/ IF systemd is installed AND used by default, THEN provide a disabled version of the init script ; 2/ IF systemd is not installed OR is installed but not used by default, THEN provide an enabled version of the init script ; The second condition seems pretty difficult test IMO but I have not yet looked into it, maybe not. But this situation where systemd is not used by default should be tested because we could have some unwanted issues. And of course we should document how to enable sysv init script for users that are willing to use them. And finally, I totally agree with elenst , this is not our decision to force users to switch to systemd and as long as sysv init script can be used we have to keep providing sysv init script. So this is what I think but I am leaving this to cvicentiu and to discussion.

Faustin Lammler added a comment - 2018-10-30 20:06

Ok, this affects all versions and can easily be reproduced, it also affects the official Debian package (10.1.26-MariaDB-0+deb9u1).

Install any version from the official mariadb repo (https://downloads.mariadb.org/mariadb/repositories/) or official Debian stable version;

$ sudo systemctl stop mariadb

$ sudo systemctl disable mariadb

$ sudo reboot

$ sudo systemctl status mariadb

● mariadb.service - MariaDB 10.1.36 database server

   Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           └─migrated-from-my.cnf-settings.conf

   Active: inactive (dead)

     Docs: man:mysqld(8)

           https://mariadb.com/kb/en/library/systemd/

$ sudo /etc/init.d/mysql status

● mysql.service - LSB: Start and stop the mysql database server daemon

   Loaded: loaded (/etc/init.d/mysql; generated; vendor preset: enabled)

   Active: active (running) since Tue 2018-10-30 16:52:57 -03; 3min 46s ago

     Docs: man:systemd-sysv-generator(8)

  Process: 419 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

    Tasks: 30 (limit: 4915)

   CGroup: /system.slice/mysql.service

           ├─463 /bin/bash /usr/bin/mysqld_safe

           ├─672 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/v…06

           └─673 logger -t mysqld -p daemon error

$ sudo mysql

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 45

Server version: 10.1.36-MariaDB-1~stretch mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

So this is indeed critical as users will not understand why mariadb server is reactivated "magically" after reboot.

Faustin Lammler added a comment - 2018-10-30 20:06 Ok, this affects all versions and can easily be reproduced, it also affects the official Debian package (10.1.26-MariaDB-0+deb9u1). Install any version from the official mariadb repo ( https://downloads.mariadb.org/mariadb/repositories/ ) or official Debian stable version; $ sudo systemctl stop mariadb $ sudo systemctl disable mariadb $ sudo reboot $ sudo systemctl status mariadb ● mariadb.service - MariaDB 10.1.36 database server Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: inactive (dead) Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ $ sudo /etc/init.d/mysql status ● mysql.service - LSB: Start and stop the mysql database server daemon Loaded: loaded (/etc/init.d/mysql; generated; vendor preset: enabled) Active: active (running) since Tue 2018-10-30 16:52:57 -03; 3min 46s ago Docs: man:systemd-sysv-generator(8) Process: 419 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS) Tasks: 30 (limit: 4915) CGroup: /system.slice/mysql.service ├─463 /bin/bash /usr/bin/mysqld_safe ├─672 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/v…06 └─673 logger -t mysqld -p daemon error $ sudo mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 45 Server version: 10.1.36-MariaDB-1~stretch mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. So this is indeed critical as users will not understand why mariadb server is reactivated "magically" after reboot.

Faustin Lammler added a comment - 2018-11-09 21:51

Ok the problem comes from the naming (mariadb/mysql).

Normally when disabling a service, the following script is used `/lib/systemd/systemd-sysv-install` in order to synchronize systemd disabled/enabled services with SysV service script.

1/ So with mysql it works because there is a `/etc/init.d/mysql` script:

$ sudo systemctl disable mysql

Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.

Executing: /lib/systemd/systemd-sysv-install disable mysql

Removed /etc/systemd/system/mysql.service.

Removed /etc/systemd/system/multi-user.target.wants/mariadb.service.

Removed /etc/systemd/system/mysqld.service.

2/ But with mariadb, it is not working since there is no `/etc/init.d/mariadb` script (not calling /lib/systemd/systemd-sysv-install):

$ sudo systemctl disable mariadb

Removed /etc/systemd/system/mysql.service.

Removed /etc/systemd/system/multi-user.target.wants/mariadb.service.

Removed /etc/systemd/system/mysqld.service.

For the moment, I can not propose a clean solution to this.

Faustin Lammler added a comment - 2018-11-09 21:51 Ok the problem comes from the naming (mariadb/mysql). Normally when disabling a service, the following script is used `/lib/systemd/systemd-sysv-install` in order to synchronize systemd disabled/enabled services with SysV service script. 1/ So with mysql it works because there is a `/etc/init.d/mysql` script: $ sudo systemctl disable mysql Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable mysql Removed /etc/systemd/system/mysql.service. Removed /etc/systemd/system/multi-user.target.wants/mariadb.service. Removed /etc/systemd/system/mysqld.service. 2/ But with mariadb, it is not working since there is no `/etc/init.d/mariadb` script (not calling /lib/systemd/systemd-sysv-install): $ sudo systemctl disable mariadb Removed /etc/systemd/system/mysql.service. Removed /etc/systemd/system/multi-user.target.wants/mariadb.service. Removed /etc/systemd/system/mysqld.service. For the moment, I can not propose a clean solution to this.

Faustin Lammler added a comment - 2018-11-10 22:56

Ok, I posted on systemd-devel ML and I get good information from Michael:
https://lists.freedesktop.org/archives/systemd-devel/2018-November/041658.html

He updated the Debian wiki:
https://wiki.debian.org/Teams/pkg-systemd/Packaging#systemd_unit_files_naming_and_installation

I will try this on upstream 10.1 release.

Faustin Lammler added a comment - 2018-11-10 22:56 Ok, I posted on systemd-devel ML and I get good information from Michael: https://lists.freedesktop.org/archives/systemd-devel/2018-November/041658.html He updated the Debian wiki: https://wiki.debian.org/Teams/pkg-systemd/Packaging#systemd_unit_files_naming_and_installation I will try this on upstream 10.1 release.

Faustin Lammler added a comment - 2018-11-11 18:53

This seems to be working fine.

Please find attached a bash script that I developed quickly to be able to reproduce this and test this quickly:

# remove previously installed services

rm -vf /etc/systemd/system/mysql*.service

rm -vf /lib/systemd/system/mysql*.service

# remove aliases entry from services

sed -i -r 's/Alias=(mysql|mysqld).service//g' /lib/systemd/system/mariadb.service

sed -i -r 's/Alias=(mysql|mysqld).service//g' /lib/systemd/system/mariadb@.service

# create symlinks

cd /lib/systemd/system

ln -s mariadb.service mysql.service

ln -s mariadb.service mysqld.service

# update systemd services

systemctl daemon-reload

config_systemd.sh

Faustin Lammler added a comment - 2018-11-11 18:53 This seems to be working fine. Please find attached a bash script that I developed quickly to be able to reproduce this and test this quickly: # remove previously installed services rm -vf /etc/systemd/system/mysql*.service rm -vf /lib/systemd/system/mysql*.service # remove aliases entry from services sed -i -r 's/Alias=(mysql|mysqld).service//g' /lib/systemd/system/mariadb.service sed -i -r 's/Alias=(mysql|mysqld).service//g' /lib/systemd/system/mariadb@.service # create symlinks cd /lib/systemd/system ln -s mariadb.service mysql.service ln -s mariadb.service mysqld.service # update systemd services systemctl daemon-reload config_systemd.sh

Otto Kekäläinen added a comment - 2019-01-11 21:05

Please update the bug report title to match the problem and solution in the conclusion.

This seems to be like rather risky business so I suggest that you apply this on 10.4 (current development version) first and only later backport to production versions when you are 100 % sure this is the correct thing to do.

Otto Kekäläinen added a comment - 2019-01-11 21:05 Please update the bug report title to match the problem and solution in the conclusion. This seems to be like rather risky business so I suggest that you apply this on 10.4 (current development version) first and only later backport to production versions when you are 100 % sure this is the correct thing to do.

Faustin Lammler added a comment - 2019-01-11 21:31

I have renamed the title and created a branch on my fork with the fix (48358cca9f6) https://github.com/fauust/mariadb/tree/fix-MDEV-15526-10.4.
I will test this in the next days...

Faustin Lammler added a comment - 2019-01-11 21:31 I have renamed the title and created a branch on my fork with the fix (48358cca9f6) https://github.com/fauust/mariadb/tree/fix-MDEV-15526-10.4 . I will test this in the next days...

Daniel Black added a comment - 2019-01-16 01:22 - edited

What functionality does providing symlinks provide explicitly? Looks to be just adding to a mess to me. It would seem to break starting mariadb.service and mysqld.service at the same time (as might happen if both are listed as service dependences), as nothing will really be aware of the relationship between them. Both would start, one would fail, hopefully without impact (likely duplicate process detection is quite mature).

If a user on a systemd distro wants a sysv init script they can disable the mariadb@service take it out of /usr/share somewhere and install it themselves. Installing both just causes problems as ~~MDEV-10797~~ highlights. Its not sane to try to support both at the same time, the ugly hacks that attempt to provide dependency relationships between them are flaky.

There is no such thing as a disabled /etc/init.d/mysql|mariadb service. The existence this file on a systemd system has some active status, changing `Default-Start/Stop: ` lines isn't sufficient as systemd and some init systems doesn't read them. Install it in /usr/share/... for the users that want to create a mess of their system, but please, keep it clean for everyone else.

The assumption that there are users consciously aware and appreciative of the broken packaging provided seems odd. Sure fix 10.4 first, but don't forget the backport.

Daniel Black added a comment - 2019-01-16 01:22 - edited What functionality does providing symlinks provide explicitly? Looks to be just adding to a mess to me. It would seem to break starting mariadb.service and mysqld.service at the same time (as might happen if both are listed as service dependences), as nothing will really be aware of the relationship between them. Both would start, one would fail, hopefully without impact (likely duplicate process detection is quite mature). If a user on a systemd distro wants a sysv init script they can disable the mariadb@service take it out of /usr/share somewhere and install it themselves. Installing both just causes problems as MDEV-10797 highlights. Its not sane to try to support both at the same time, the ugly hacks that attempt to provide dependency relationships between them are flaky. There is no such thing as a disabled /etc/init.d/mysql|mariadb service. The existence this file on a systemd system has some active status, changing `Default-Start/Stop: ` lines isn't sufficient as systemd and some init systems doesn't read them. Install it in /usr/share/... for the users that want to create a mess of their system, but please, keep it clean for everyone else. The assumption that there are users consciously aware and appreciative of the broken packaging provided seems odd. Sure fix 10.4 first, but don't forget the backport.

Christian Hesse added a comment - 2019-02-05 22:13

We do something similar for Arch Linux:
0001-arch-specific.patch at line 100
PKGBUILD at line 194

Christian Hesse added a comment - 2019-02-05 22:13 We do something similar for Arch Linux: 0001-arch-specific.patch at line 100 PKGBUILD at line 194

Faustin Lammler added a comment - 2019-02-06 10:42 - edited

Thanks a lot eworm for you input, indeed this looks pretty similar.

Are you at Arch Linux deploying sysv init script? I saw this but I would appreciate an explanation of your policy regarding sysv init support if systemd is the default init system.

There seems to be a consensus regarding not using Aliases, now we should confirm that using symlink is the right thing to do.

danblack, these symlinks have no other utility than permitting the usage of the 2 names (mariadb or mysql) with systemctl commands (even if it also seems to resolve the disable issue that hholzgra reported and that seems to be caused by sysv init script presence).

This for instance, should behave exactly equal :

$ sudo systemctl start|stop|reload|disable|enable mariadb

$ sudo systemctl start|stop|reload|disable|enable mysql

Faustin Lammler added a comment - 2019-02-06 10:42 - edited Thanks a lot eworm for you input, indeed this looks pretty similar. Are you at Arch Linux deploying sysv init script? I saw this but I would appreciate an explanation of your policy regarding sysv init support if systemd is the default init system. There seems to be a consensus regarding not using Aliases, now we should confirm that using symlink is the right thing to do. danblack , these symlinks have no other utility than permitting the usage of the 2 names (mariadb or mysql) with systemctl commands (even if it also seems to resolve the disable issue that hholzgra reported and that seems to be caused by sysv init script presence). This for instance, should behave exactly equal : $ sudo systemctl start|stop|reload|disable| enable mariadb $ sudo systemctl start|stop|reload|disable| enable mysql

Christian Hesse added a comment - 2019-02-06 11:04

We do not ship any sysvinit scripts with Arch Linux.

Using symlinks is the right thing to do. In fact the Alias= makes systemd create symlinks in /etc/systemd/system/... But these are available after enabling the service only.

Christian Hesse added a comment - 2019-02-06 11:04 We do not ship any sysvinit scripts with Arch Linux. Using symlinks is the right thing to do. In fact the Alias= makes systemd create symlinks in /etc/systemd/system/... But these are available after enabling the service only.

Daniel Black added a comment - 2019-02-06 21:58

Tested symlinks, seems to be exhibiting the right behaviour, for now.

$ pwd

/etc/systemd/system

$ cat >> mariadbdependentservice.service

[Unit]

Requires=mariadb.service

[Service]

ExecStart=/bin/sleep 3600

$ cat >> mysqldependentservice.service

[Unit]

Requires=mysql.service

[Service]

ExecStart=/bin/sleep 3600

$ ln -s mariadb.service mysql.service

$ systemctl daemon-reload

$ systemctl start mariadbdependentservice.service

$ systemctl status mariadbdependentservice.service

● mariadbdependentservice.service

   Loaded: loaded (/etc/systemd/system/mariadbdependentservice.service; static; vendor preset: disabled)

   Active: active (running) since Thu 2019-02-07 08:31:52 AEDT; 8s ago

 Main PID: 2269 (sleep)

    Tasks: 1 (limit: 4915)

   Memory: 160.0K

      CPU: 2ms

   CGroup: /system.slice/mariadbdependentservice.service

           └─2269 /bin/sleep 3600

Feb 07 08:31:52 volution systemd[1]: Started mariadbdependentservice.service.

$ systemctl status mariadb.service

● mariadb.service - MariaDB @VERSION@ database server

   Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)

  Drop-In: /usr/lib/systemd/system/mariadb.service.d

           └─tokudb.conf

   Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 16s ago

     Docs: man:mysqld(8)

           https://mariadb.com/kb/en/library/systemd/

  Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 2271 (mysqld)

   Status: "Taking your SQL requests now..."

    Tasks: 68 (limit: 4915)

   Memory: 246.0M

      CPU: 806ms

   CGroup: /system.slice/mariadb.service

           └─2271 /usr/libexec/mysqld

Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server...

Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07  8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ...

Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server.

$ systemctl start  mysqldependentservice.service

$ systemctl status  mysqldependentservice.service

● mysqldependentservice.service

   Loaded: loaded (/etc/systemd/system/mysqldependentservice.service; static; vendor preset: disabled)

   Active: active (running) since Thu 2019-02-07 08:32:38 AEDT; 7s ago

 Main PID: 2375 (sleep)

    Tasks: 1 (limit: 4915)

   Memory: 248.0K

      CPU: 2ms

   CGroup: /system.slice/mysqldependentservice.service

           └─2375 /bin/sleep 3600

Feb 07 08:32:38 volution systemd[1]: Started mysqldependentservice.service.

$ systemctl status  mysql.service

● mariadb.service - MariaDB @VERSION@ database server

   Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)

  Drop-In: /usr/lib/systemd/system/mariadb.service.d

           └─tokudb.conf

   Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 1min 6s ago

     Docs: man:mysqld(8)

           https://mariadb.com/kb/en/library/systemd/

  Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 2271 (mysqld)

   Status: "Taking your SQL requests now..."

    Tasks: 68 (limit: 4915)

   Memory: 246.1M

      CPU: 857ms

   CGroup: /system.slice/mariadb.service

           └─2271 /usr/libexec/mysqld

Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server...

Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07  8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ...

Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server.

$ systemctl stop mysqldependentservice.service

$ systemctl status mysqldependentservice.service

● mysqldependentservice.service

   Loaded: loaded (/etc/systemd/system/mysqldependentservice.service; static; vendor preset: disabled)

   Active: inactive (dead)

Feb 07 08:32:38 volution systemd[1]: Started mysqldependentservice.service.

Feb 07 08:34:36 volution systemd[1]: Stopping mysqldependentservice.service...

Feb 07 08:34:36 volution systemd[1]: Stopped mysqldependentservice.service.

Feb 07 08:34:36 volution systemd[1]: mysqldependentservice.service: Consumed 2ms CPU time

$ systemctl status  mysql.service

● mariadb.service - MariaDB @VERSION@ database server

   Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)

  Drop-In: /usr/lib/systemd/system/mariadb.service.d

           └─tokudb.conf

   Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 22min ago

     Docs: man:mysqld(8)

           https://mariadb.com/kb/en/library/systemd/

  Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

  Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 2271 (mysqld)

   Status: "Taking your SQL requests now..."

    Tasks: 68 (limit: 4915)

   Memory: 246.2M

      CPU: 2.108s

   CGroup: /system.slice/mariadb.service

           └─2271 /usr/libexec/mysqld

Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server...

Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07  8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ...

Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server.

$ systemctl stop  mysql.service

$ systemctl status  mariadb.service

● mariadb.service - MariaDB @VERSION@ database server

   Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled)

  Drop-In: /usr/lib/systemd/system/mariadb.service.d

           └─tokudb.conf

   Active: inactive (dead)

     Docs: man:mysqld(8)

           https://mariadb.com/kb/en/library/systemd/

Daniel Black added a comment - 2019-02-06 21:58 Tested symlinks, seems to be exhibiting the right behaviour, for now. $ pwd /etc/systemd/system $ cat >> mariadbdependentservice.service [Unit] Requires=mariadb.service [Service] ExecStart=/bin/sleep 3600 $ cat >> mysqldependentservice.service [Unit] Requires=mysql.service [Service] ExecStart=/bin/sleep 3600 $ ln -s mariadb.service mysql.service $ systemctl daemon-reload $ systemctl start mariadbdependentservice.service $ systemctl status mariadbdependentservice.service ● mariadbdependentservice.service Loaded: loaded (/etc/systemd/system/mariadbdependentservice.service; static; vendor preset: disabled) Active: active (running) since Thu 2019-02-07 08:31:52 AEDT; 8s ago Main PID: 2269 (sleep) Tasks: 1 (limit: 4915) Memory: 160.0K CPU: 2ms CGroup: /system.slice/mariadbdependentservice.service └─2269 /bin/sleep 3600 Feb 07 08:31:52 volution systemd[1]: Started mariadbdependentservice.service. $ systemctl status mariadb.service ● mariadb.service - MariaDB @VERSION@ database server Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/mariadb.service.d └─tokudb.conf Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 16s ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Main PID: 2271 (mysqld) Status: "Taking your SQL requests now..." Tasks: 68 (limit: 4915) Memory: 246.0M CPU: 806ms CGroup: /system.slice/mariadb.service └─2271 /usr/libexec/mysqld Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server... Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07 8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ... Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server. $ systemctl start mysqldependentservice.service $ systemctl status mysqldependentservice.service ● mysqldependentservice.service Loaded: loaded (/etc/systemd/system/mysqldependentservice.service; static; vendor preset: disabled) Active: active (running) since Thu 2019-02-07 08:32:38 AEDT; 7s ago Main PID: 2375 (sleep) Tasks: 1 (limit: 4915) Memory: 248.0K CPU: 2ms CGroup: /system.slice/mysqldependentservice.service └─2375 /bin/sleep 3600 Feb 07 08:32:38 volution systemd[1]: Started mysqldependentservice.service. $ systemctl status mysql.service ● mariadb.service - MariaDB @VERSION@ database server Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/mariadb.service.d └─tokudb.conf Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 1min 6s ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Main PID: 2271 (mysqld) Status: "Taking your SQL requests now..." Tasks: 68 (limit: 4915) Memory: 246.1M CPU: 857ms CGroup: /system.slice/mariadb.service └─2271 /usr/libexec/mysqld Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server... Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07 8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ... Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server. $ systemctl stop mysqldependentservice.service $ systemctl status mysqldependentservice.service ● mysqldependentservice.service Loaded: loaded (/etc/systemd/system/mysqldependentservice.service; static; vendor preset: disabled) Active: inactive (dead) Feb 07 08:32:38 volution systemd[1]: Started mysqldependentservice.service. Feb 07 08:34:36 volution systemd[1]: Stopping mysqldependentservice.service... Feb 07 08:34:36 volution systemd[1]: Stopped mysqldependentservice.service. Feb 07 08:34:36 volution systemd[1]: mysqldependentservice.service: Consumed 2ms CPU time $ systemctl status mysql.service ● mariadb.service - MariaDB @VERSION@ database server Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/mariadb.service.d └─tokudb.conf Active: active (running) since Thu 2019-02-07 08:31:53 AEDT; 22min ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 2360 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 2270 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Main PID: 2271 (mysqld) Status: "Taking your SQL requests now..." Tasks: 68 (limit: 4915) Memory: 246.2M CPU: 2.108s CGroup: /system.slice/mariadb.service └─2271 /usr/libexec/mysqld Feb 07 08:31:52 volution systemd[1]: Starting MariaDB @VERSION@ database server... Feb 07 08:31:52 volution mysqld[2271]: 2019-02-07 8:31:52 0 [Note] /usr/libexec/mysqld (mysqld 10.3.11-MariaDB) starting as process 2271 ... Feb 07 08:31:53 volution systemd[1]: Started MariaDB @VERSION@ database server. $ systemctl stop mysql.service $ systemctl status mariadb.service ● mariadb.service - MariaDB @VERSION@ database server Loaded: loaded (/etc/systemd/system/mariadb.service; indirect; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/mariadb.service.d └─tokudb.conf Active: inactive (dead) Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/

Faustin Lammler added a comment - 2019-02-07 08:15 - edited

Thanks danblack, can you also check the disable|enable command?

sudo systemctl disable mariadb

sudo reboot

sudo systemctl status mariadb # should be disabled

And the same with mysql (but mariadb is more significant since there is no /etc/init.d/mariadb init script).

I will propose a PR for theses modifications from my branch https://github.com/fauust/mariadb/tree/fix-MDEV-15526-10.4.

Faustin Lammler added a comment - 2019-02-07 08:15 - edited Thanks danblack , can you also check the disable|enable command? sudo systemctl disable mariadb sudo reboot sudo systemctl status mariadb # should be disabled And the same with mysql (but mariadb is more significant since there is no /etc/init.d/mariadb init script). I will propose a PR for theses modifications from my branch https://github.com/fauust/mariadb/tree/fix-MDEV-15526-10.4 .

Faustin Lammler added a comment - 2019-02-11 11:14

PR submitted https://github.com/MariaDB/server/pull/1172

Faustin Lammler added a comment - 2019-02-11 11:14 PR submitted https://github.com/MariaDB/server/pull/1172

Daniel Black added a comment - 2019-02-13 08:46

Another user stuffed around by the existence of /etc/init.d/mysql

https://stackoverflow.com/questions/54662100/mariadb-server-is-down-but-i-can-still-login-with-mysql-client/54662193?noredirect=1#comment96119410_54662193

Please stop installing it on systemd systems.

Daniel Black added a comment - 2019-02-13 08:46 Another user stuffed around by the existence of /etc/init.d/mysql https://stackoverflow.com/questions/54662100/mariadb-server-is-down-but-i-can-still-login-with-mysql-client/54662193?noredirect=1#comment96119410_54662193 Please stop installing it on systemd systems.

Faustin Lammler added a comment - 2019-02-13 10:03

danblack this problem is resolved by removing aliases and creating symlinks (I have just tested it again on last 10.4 official version):
https://github.com/MariaDB/server/pull/1172

As discussed, the move of not providing "old" sysv init files is not easy to do IMHO because I know a lot of sysadmin (on Debian at least) that are still using sysv init scripts and I still agree with elenst.

Faustin Lammler added a comment - 2019-02-13 10:03 danblack this problem is resolved by removing aliases and creating symlinks (I have just tested it again on last 10.4 official version): https://github.com/MariaDB/server/pull/1172 As discussed, the move of not providing "old" sysv init files is not easy to do IMHO because I know a lot of sysadmin (on Debian at least) that are still using sysv init scripts and I still agree with elenst .

Claudio Nanni added a comment - 2019-02-13 11:06 - edited

I think that this report renaming from " Latest packages on Systemd still install SysV init service file /etc/init.d/mysql" to "Systemd unit files naming and installation (alias in unit are not recommended, create symlink)" does not help people with this problem in finding this MDEV. Most people report(and so research) what they observe, not the underlying technical reasons for some observed behaviour.

Claudio Nanni added a comment - 2019-02-13 11:06 - edited I think that this report renaming from " Latest packages on Systemd still install SysV init service file /etc/init.d/mysql" to "Systemd unit files naming and installation (alias in unit are not recommended, create symlink)" does not help people with this problem in finding this MDEV. Most people report(and so research) what they observe, not the underlying technical reasons for some observed behaviour.

Faustin Lammler added a comment - 2019-02-13 11:32

Good point claudio.nanni, I will find a better title (but naming problem is also important IMO in that title).

Faustin Lammler added a comment - 2019-02-13 11:32 Good point claudio.nanni , I will find a better title (but naming problem is also important IMO in that title).

Daniel Black added a comment - 2019-02-13 23:39

I appreciate that there are people that want to use sysv init scripts. I'm all for a sysv init script continuing to exist for people that are using those. The rest of the users either don't know/care which init system is there, or want to use the existing system (like distros are mean to do in providing packages). In these cases they want packages that can be updated (without breaking in the middle of installation), and to be able to disable mariadb as a service. Currently this isn't the case.

serg even suggested to install it in a @sharedir@.

To help get to some resolution, would it help if we break this up into a couple of usage scenarios for systemd enabled systems:

Case prior to installation	Recommended packaging action	Pros	Cons
Greenfields - no prior install	install sysv init script to @sharedir@	Eliminates all incompatibilities with sysv scripts	A sysv user will need to realise that its systemd OS installed, or copy the sysv script to /etc/init.d
Updating from a previous MariaDB packaged version (10.1>=)	Test if they are using sysv, if so install to /etc/init.d, otherwise to @sharedir@ and remove /etc/init.d/mysql	Tries to be compatible with what users want and keeps sysv init scripts out of the way if systemd was used
Updating from a previous MariaDB packaged version <10.1	Not supported? Install on the assumption that a systemd is desired	Simplicity and conforming to the expectations of the distro	They may want init.d, in which case its installed to @shardir@
Updates from distro MariaDB packages	TODO
Updates from distro MySQL packages	TODO
Updates from Oracle MySQL packages	TODO

Yes this is going to get complicated. Yes some automated testing probably should be done to (or at least documented process). This is the 12th most watched open bug, like ~~MDEV-10797~~, because users do care. I realise packaging isn't easy. While a lot of care should be taken to maintain compatibility, it always comes with a maintenance burden. A new 10.4 release is a good opportunity to try to get this resolved for all users while its still in beta. There may be a couple of breaks for systems with maintaince debt for which we can develop a richer set of test case, but hopefully most users will get a smoother path.

~~MDEV-10797~~ has lingered without fixing for the same causes as this for over two major releases because there wasn't the willingness to fix. There seems to be the mentality that somehow outstanding bugs are more tolerable than new bugs, even if fixing helps more people than than what the new bug affects. This is a poor risk management decision. I looking forward to the resolution of both these packaging bugs.

Daniel Black added a comment - 2019-02-13 23:39 I appreciate that there are people that want to use sysv init scripts. I'm all for a sysv init script continuing to exist for people that are using those . The rest of the users either don't know/care which init system is there, or want to use the existing system (like distros are mean to do in providing packages). In these cases they want packages that can be updated (without breaking in the middle of installation), and to be able to disable mariadb as a service. Currently this isn't the case. serg even suggested to install it in a @sharedir@. To help get to some resolution, would it help if we break this up into a couple of usage scenarios for systemd enabled systems: Case prior to installation Recommended packaging action Pros Cons Greenfields - no prior install install sysv init script to @sharedir@ Eliminates all incompatibilities with sysv scripts A sysv user will need to realise that its systemd OS installed, or copy the sysv script to /etc/init.d Updating from a previous MariaDB packaged version (10.1>=) Test if they are using sysv, if so install to /etc/init.d, otherwise to @sharedir@ and remove /etc/init.d/mysql Tries to be compatible with what users want and keeps sysv init scripts out of the way if systemd was used Updating from a previous MariaDB packaged version <10.1 Not supported? Install on the assumption that a systemd is desired Simplicity and conforming to the expectations of the distro They may want init.d, in which case its installed to @shardir@ Updates from distro MariaDB packages TODO Updates from distro MySQL packages TODO Updates from Oracle MySQL packages TODO Yes this is going to get complicated. Yes some automated testing probably should be done to (or at least documented process). This is the 12th most watched open bug, like MDEV-10797 , because users do care. I realise packaging isn't easy. While a lot of care should be taken to maintain compatibility, it always comes with a maintenance burden. A new 10.4 release is a good opportunity to try to get this resolved for all users while its still in beta. There may be a couple of breaks for systems with maintaince debt for which we can develop a richer set of test case, but hopefully most users will get a smoother path. MDEV-10797 has lingered without fixing for the same causes as this for over two major releases because there wasn't the willingness to fix. There seems to be the mentality that somehow outstanding bugs are more tolerable than new bugs, even if fixing helps more people than than what the new bug affects. This is a poor risk management decision. I looking forward to the resolution of both these packaging bugs.

Daniel Black added a comment - 2019-03-13 22:42

faust Stop ignoring this problem and considering it too hard. Please engage in a discussion of what could be an acceptable solution for elenst, your sysv users and all the users who use systemd, want upgrades to work and want mariadb to be packaged professionally.

Daniel Black added a comment - 2019-03-13 22:42 faust Stop ignoring this problem and considering it too hard. Please engage in a discussion of what could be an acceptable solution for elenst , your sysv users and all the users who use systemd, want upgrades to work and want mariadb to be packaged professionally.

Elena Stepanova added a comment - 2019-03-13 23:02

danblack, I provided my opinion as a user only. I'm not going to argue against any decision from a position of MariaDB engineering member, neither do I have the authority to forbid the change even if I wanted to. However, to my understanding of the policies, if the script is going to be removed, it must be done before GA (or even before RC, I'm not quite sure which).

Elena Stepanova added a comment - 2019-03-13 23:02 danblack , I provided my opinion as a user only. I'm not going to argue against any decision from a position of MariaDB engineering member, neither do I have the authority to forbid the change even if I wanted to. However, to my understanding of the policies, if the script is going to be removed, it must be done before GA (or even before RC, I'm not quite sure which).

Daniel Black added a comment - 2019-04-03 21:28

Although not clearly stated in this issue, this was the Debian packaging side and as such is unresolved.

debian/rules -> override_dh_installinit-arch also needs fixing as it installs init.d script and systemd script of different names, causing problems exactly like the RPM packaging and is inconsistent with Debians packaging standards (written because of the problems described here):

https://wiki.debian.org/Teams/pkg-systemd/Packaging?action=show&redirect=systemd%2FPackaging#systemd_unit_files_naming_and_installation

Daniel Black added a comment - 2019-04-03 21:28 Although not clearly stated in this issue, this was the Debian packaging side and as such is unresolved. debian/rules -> override_dh_installinit-arch also needs fixing as it installs init.d script and systemd script of different names, causing problems exactly like the RPM packaging and is inconsistent with Debians packaging standards (written because of the problems described here): https://wiki.debian.org/Teams/pkg-systemd/Packaging?action=show&redirect=systemd%2FPackaging#systemd_unit_files_naming_and_installation

Otto Kekäläinen added a comment - 2019-04-04 08:14

For refrence, a comparison of current Debian mariadb-10.3 rules snippet to the left, and upstream mariadb-10.4 rules file to the right.

I am not an expert on systemd and since Docker does not support systemd if don't have any easily available testing setup to check which one works better, but knowing the alternative implementation might be useful.

Otto Kekäläinen added a comment - 2019-04-04 08:14 For refrence, a comparison of current Debian mariadb-10.3 rules snippet to the left, and upstream mariadb-10.4 rules file to the right. I am not an expert on systemd and since Docker does not support systemd if don't have any easily available testing setup to check which one works better, but knowing the alternative implementation might be useful.

Daniel Black added a comment - 2019-04-05 00:58 - edited

Taking an interest in the dominate system manager, on the dominate OS that MariaDB supports, in which packaging pays a critical role, would be useful with bugs like this (which attract watchers while nothing is resolved).

I understand the buildbot is docker based which has a complicated relationship with systemd however there is https://hub.docker.com/r/centos/systemd and https://hub.docker.com/r/jrei/systemd-debian (which might not be official, but has the same principle as the Centos Dockerfile) and these could start to form the basis of docker systemd tests, and at the same time, giving you the systemd skills needed to maintain Debian packages.

The notes about having a init script having the same name as the service file where there on the above wiki page in 2013 and is not recommended, which mariadb and debian does. I suggest /etc/init.d/mysql -> /etc/init.d/mariadb as the first move perhaps?

Summarising a vagrant conversation with colleague that could be used for full VM tests (and galera):

Use vagrant for the minimal amount possible (avoid complex features of vagrant)
Use scripts to wait for full networking as vagrants doesn't always do this right
Use ansible to install packages and trigger tests once vagrant boxes

Vagrant implementation as a result of this was these files: https://git.samba.org/?p=autocluster.git;a=tree;f=vagrant;hb=HEAD, tests aren't in this repository.

Daniel Black added a comment - 2019-04-05 00:58 - edited Taking an interest in the dominate system manager, on the dominate OS that MariaDB supports, in which packaging pays a critical role, would be useful with bugs like this (which attract watchers while nothing is resolved). I understand the buildbot is docker based which has a complicated relationship with systemd however there is https://hub.docker.com/r/centos/systemd and https://hub.docker.com/r/jrei/systemd-debian (which might not be official, but has the same principle as the Centos Dockerfile) and these could start to form the basis of docker systemd tests, and at the same time, giving you the systemd skills needed to maintain Debian packages. The notes about having a init script having the same name as the service file where there on the above wiki page in 2013 and is not recommended, which mariadb and debian does. I suggest /etc/init.d/mysql -> /etc/init.d/mariadb as the first move perhaps? Summarising a vagrant conversation with colleague that could be used for full VM tests (and galera): Use vagrant for the minimal amount possible (avoid complex features of vagrant) Use scripts to wait for full networking as vagrants doesn't always do this right Use ansible to install packages and trigger tests once vagrant boxes Vagrant implementation as a result of this was these files: https://git.samba.org/?p=autocluster.git;a=tree;f=vagrant;hb=HEAD , tests aren't in this repository.

Sergei Golubchik added a comment - 2019-04-07 14:26

https://github.com/MariaDB/server/pull/593#issuecomment-479338028

Sergei Golubchik added a comment - 2019-04-07 14:26 https://github.com/MariaDB/server/pull/593#issuecomment-479338028

Faustin Lammler added a comment - 2019-04-10 10:01

Thank you danblack for your suggestion on how we could implement our systemd tests.

teodor do you think this is doable on buildbot? I have looked at https://hub.docker.com/r/jrei/systemd-debian/dockerfile and I if I am understanding correctly, BB worker/runner should be a VM or a host and should share `/sys/fs/cgroup` with the container.

Then I would be happy to implement the first basic tests:

reload|restart|start|stop|enable|disable
multi-instance mode
...

And maybe Daniel will give us some good test suggestions.

Faustin Lammler added a comment - 2019-04-10 10:01 Thank you danblack for your suggestion on how we could implement our systemd tests. teodor do you think this is doable on buildbot? I have looked at https://hub.docker.com/r/jrei/systemd-debian/dockerfile and I if I am understanding correctly, BB worker/runner should be a VM or a host and should share `/sys/fs/cgroup` with the container. Then I would be happy to implement the first basic tests: reload|restart|start|stop|enable|disable multi-instance mode ... And maybe Daniel will give us some good test suggestions.

Daniel Black added a comment - 2019-04-10 23:20

faust all assumptions correct. Regarding /sys/fs/cgroup, I think, not sure by any means, that each container should have a different host subdirectory off this mounted though to give each container a different path in the cgroup hierarchy so /sys/fs/cgroup/testX -> /sys/fs/cgroup. While container based systemd tests will be quite effective for some basic things, eventually some vm/vagrant infrastructure will be needed to effectively test selinux (there is no separation from container host in a selinux context), and would make galera based packaging tests more realistic. Happy to provide other suggestions however lets put that in a new issue.

Daniel Black added a comment - 2019-04-10 23:20 faust all assumptions correct. Regarding /sys/fs/cgroup , I think, not sure by any means, that each container should have a different host subdirectory off this mounted though to give each container a different path in the cgroup hierarchy so /sys/fs/cgroup/testX -> /sys/fs/cgroup . While container based systemd tests will be quite effective for some basic things, eventually some vm/vagrant infrastructure will be needed to effectively test selinux (there is no separation from container host in a selinux context), and would make galera based packaging tests more realistic. Happy to provide other suggestions however lets put that in a new issue.

Teodor Mircea Ionita (Inactive) added a comment - 2019-04-16 07:20 - edited

faust I'm not too knowledgeable of systemd internals and workings. One question though from Buildbot's perspective, does mounting /sys/fs/cgroup from host to guests breach the security and isolation contracts that Docker provides? Remember that these builders will run on Github pull requests too and we don't want a mere opening of a PR to be able to compromise our worker machines. In a sense, Docker is Systemd for containers, as it uses the same kernel tools (cgroups, namespaces) to limit, isolate, control and cleanup the user namespace (apart from zombie processes).

Teodor Mircea Ionita (Inactive) added a comment - 2019-04-16 07:20 - edited faust I'm not too knowledgeable of systemd internals and workings. One question though from Buildbot's perspective, does mounting /sys/fs/cgroup from host to guests breach the security and isolation contracts that Docker provides? Remember that these builders will run on Github pull requests too and we don't want a mere opening of a PR to be able to compromise our worker machines. In a sense, Docker is Systemd for containers, as it uses the same kernel tools (cgroups, namespaces) to limit, isolate, control and cleanup the user namespace (apart from zombie processes).

Otto Kekäläinen added a comment - 2019-04-16 09:17

In a Debian upgrade bug report I now used the systemd-debian images to test and the steps are outlined in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926231#58 in case somebody wants to replicate that.

Otto Kekäläinen added a comment - 2019-04-16 09:17 In a Debian upgrade bug report I now used the systemd-debian images to test and the steps are outlined in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926231#58 in case somebody wants to replicate that.

Faustin Lammler added a comment - 2019-04-16 15:16

teodor this is a very good question and consideration.

I have just found yesterday this documentation that I did not had time to read with calm.
It maybe interesting for you too :
https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/
https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/

And thanks otto I will look at your link too.

Faustin Lammler added a comment - 2019-04-16 15:16 teodor this is a very good question and consideration. I have just found yesterday this documentation that I did not had time to read with calm. It maybe interesting for you too : https://developers.redhat.com/blog/2014/05/05/running-systemd-within-docker-container/ https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ And thanks otto I will look at your link too.

Sergei Golubchik added a comment - 2019-04-16 17:35

axel, there's a long discussion here, but the only important part is this comment.

I've merged https://github.com/MariaDB/server/commit/74c9872a9aae825ca6ab73feff048ff2ccb4beb8 but danblack noted that it only fixes RPM, so Debian still needs to be fixed. This is why this issue is not closed yet, it's waiting for a Debian-specific fix.

Sergei Golubchik added a comment - 2019-04-16 17:35 axel , there's a long discussion here, but the only important part is this comment . I've merged https://github.com/MariaDB/server/commit/74c9872a9aae825ca6ab73feff048ff2ccb4beb8 but danblack noted that it only fixes RPM, so Debian still needs to be fixed. This is why this issue is not closed yet, it's waiting for a Debian-specific fix.

Daniel Black added a comment - 2019-06-18 01:30

serg thanks for the fix. does debian/rules -> override_dh_installinit-arch: need to remove dh_installinit --name=mysql – defaults 19 21? It looks odd since the mysql.init is now in /usr/share.

teodor I see you haven't made much progress with systemd docker configs or scripts in the last few months, I did mention that I'd help in a new issue. https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/ is quite new as a blog on non-priviledged docker container without cgroup hacks. Hope it works for you.

Daniel Black added a comment - 2019-06-18 01:30 serg thanks for the fix. does debian/rules -> override_dh_installinit-arch: need to remove dh_installinit --name=mysql – defaults 19 21 ? It looks odd since the mysql.init is now in /usr/share. teodor I see you haven't made much progress with systemd docker configs or scripts in the last few months, I did mention that I'd help in a new issue . https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/ is quite new as a blog on non-priviledged docker container without cgroup hacks. Hope it works for you.

Sergei Golubchik added a comment - 2019-08-26 13:39

danblack, yes, I suppose it should go, thanks

Sergei Golubchik added a comment - 2019-08-26 13:39 danblack , yes, I suppose it should go, thanks

Faustin Lammler added a comment - 2019-09-18 14:08

Hi!
I believe there is still some problematic situation with this (at least on Debian):

versions < 10.4 (Debian Buster provides 10.3 branch);
upgrading from previous versions.

This https://github.com/MariaDB/server/pull/1172 resolves it by removing dynamically created symlinks (via aliases) and by creating those symlinks on postinst (so symlinks are never removed and synchronization between sysvinit and systemd is working).

Then, I think that we should remove symlink postinst creation in 10.5 branch next release (and communicate that systemctl commands does not support mysql naming anymore). That would be the cleanest situation for the future IMO.

If you think that this concerns only Debian, I will fix this downstream only.

Faustin Lammler added a comment - 2019-09-18 14:08 Hi! I believe there is still some problematic situation with this (at least on Debian): versions < 10.4 (Debian Buster provides 10.3 branch); upgrading from previous versions. This https://github.com/MariaDB/server/pull/1172 resolves it by removing dynamically created symlinks (via aliases) and by creating those symlinks on postinst (so symlinks are never removed and synchronization between sysvinit and systemd is working). Then, I think that we should remove symlink postinst creation in 10.5 branch next release (and communicate that systemctl commands does not support mysql naming anymore). That would be the cleanest situation for the future IMO. If you think that this concerns only Debian, I will fix this downstream only.

Otto Kekäläinen added a comment - 2020-04-10 11:31

After a lot of research into this:

Root problem:

There is no /etc/init.d/mariadb so disabling mariadb in systemd with `systemctl disable mariadb` does not trigger /lib/systemd/systemd-sysv-install for service name 'mysql' and thus disabling the sysvinit service mysql service stays enabled:
Synchronizing state of mariadb.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mariadb

Same problem applies for running `systemctl disable mysqld`.

The systemd itself supports running `systemd-sysv-install` only if the sysvinit=systemd name. Neither service alias= or service symlinks are supported by this feature.

The problem only applies for systems with systemd. On other systems the command `systemctl disable mariadb` would fail because there is no sysvinit file with that name (note that the command `systemctl` may work on systems that lack systemd but have a systemctl-shim installed.

Correct solution here is to:

don't install any sysvinit files on systems that have systemd (in effect now, but implemented wrongly)
keep installing sysvinit files on systems that lack systemd, otherwise they would not work at all (needs to be fixed)
systemctl aliases are good, they increase compatibility with mysql - skipping them to "fix" this bug is the wrong tradeoff

Code changes:

merge https://github.com/MariaDB/server/pull/1172
skip it's smaller version https://github.com/MariaDB/server/pull/1494 (=just close it)
wait until I fix https://github.com/MariaDB/server/pull/1463

Otto Kekäläinen added a comment - 2020-04-10 11:31 After a lot of research into this: Root problem: There is no /etc/init.d/mariadb so disabling mariadb in systemd with `systemctl disable mariadb` does not trigger /lib/systemd/systemd-sysv-install for service name 'mysql' and thus disabling the sysvinit service mysql service stays enabled: Synchronizing state of mariadb.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable mariadb Same problem applies for running `systemctl disable mysqld`. The systemd itself supports running `systemd-sysv-install` only if the sysvinit=systemd name. Neither service alias= or service symlinks are supported by this feature. The problem only applies for systems with systemd. On other systems the command `systemctl disable mariadb` would fail because there is no sysvinit file with that name (note that the command `systemctl` may work on systems that lack systemd but have a systemctl-shim installed. Correct solution here is to: don't install any sysvinit files on systems that have systemd (in effect now, but implemented wrongly) keep installing sysvinit files on systems that lack systemd, otherwise they would not work at all (needs to be fixed) systemctl aliases are good, they increase compatibility with mysql - skipping them to "fix" this bug is the wrong tradeoff Code changes: merge https://github.com/MariaDB/server/pull/1172 skip it's smaller version https://github.com/MariaDB/server/pull/1494 (=just close it) wait until I fix https://github.com/MariaDB/server/pull/1463

Otto Kekäläinen added a comment - 2020-04-10 12:36

A similar example is package openssh-server in Debian. It has only /etc/init.d/ssh but both /lib/systemd/system/ssh.service with alias=sshd. There is also a symlink `/etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service`.

There is a related upstream bug report in https://github.com/systemd/systemd/issues/7874 but this exact problem "systemctl disable does not disable sysvinit with alias names" apparently has never been filed upstream, so I filed it now: https://github.com/systemd/systemd/issues/15394

systemd code where it tires to detect if there is a sysvinit file with identical name: https://github.com/systemd/systemd/blob/4c39820562e562b98c2fd269ad3a4f84949d6059/src/systemctl/systemctl.c#L6680

Otto Kekäläinen added a comment - 2020-04-10 12:36 A similar example is package openssh-server in Debian. It has only /etc/init.d/ssh but both /lib/systemd/system/ssh.service with alias=sshd. There is also a symlink `/etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service`. There is a related upstream bug report in https://github.com/systemd/systemd/issues/7874 but this exact problem "systemctl disable does not disable sysvinit with alias names" apparently has never been filed upstream, so I filed it now: https://github.com/systemd/systemd/issues/15394 systemd code where it tires to detect if there is a sysvinit file with identical name: https://github.com/systemd/systemd/blob/4c39820562e562b98c2fd269ad3a4f84949d6059/src/systemctl/systemctl.c#L6680

Faustin Lammler added a comment - 2020-04-10 14:15

otto thanks for looking deeply into this!

I agree with most your solution proposal.

But I believe this one is not totally correct:
> systemctl aliases are good, they increase compatibility with mysql - skipping them to "fix" this bug is the wrong tradeoff

Aliases is not the best solution to increase compatibility with mysql. Instead, "shipping a (static) symlink in the package is the second best thing you can do to align the sysv init script name and the service name, The best you can do (obviously) is to use the same name for the sysv init script and the service file.", I am quoting M. Biebl, see: https://github.com/MariaDB/server/pull/1172#discussion_r305863850

Faustin Lammler added a comment - 2020-04-10 14:15 otto thanks for looking deeply into this! I agree with most your solution proposal. But I believe this one is not totally correct: > systemctl aliases are good, they increase compatibility with mysql - skipping them to "fix" this bug is the wrong tradeoff Aliases is not the best solution to increase compatibility with mysql. Instead, "shipping a (static) symlink in the package is the second best thing you can do to align the sysv init script name and the service name, The best you can do (obviously) is to use the same name for the sysv init script and the service file.", I am quoting M. Biebl, see: https://github.com/MariaDB/server/pull/1172#discussion_r305863850

Otto Kekäläinen added a comment - 2020-04-10 14:25 - edited

They are called aliases also when they are implemented via symlinks.

I have been testing with a recent Debian Sid environment and I am actually at the moment unable to reproduce the original bug. Seems like the `/lib/lsb/init-functions` which ends up calling `/lib/lsb/init-functions.d/40-systemd` handles this correctly already.

Sources at https://sources.debian.org/src/systemd/245.4-4/debian/extra/init-functions.d/40-systemd/

Otto Kekäläinen added a comment - 2020-04-10 14:25 - edited They are called aliases also when they are implemented via symlinks. I have been testing with a recent Debian Sid environment and I am actually at the moment unable to reproduce the original bug. Seems like the `/lib/lsb/init-functions` which ends up calling `/lib/lsb/init-functions.d/40-systemd` handles this correctly already. Sources at https://sources.debian.org/src/systemd/245.4-4/debian/extra/init-functions.d/40-systemd/

Otto Kekäläinen added a comment - 2020-05-17 09:54

faust Just to be sure, can you please test that latest 10.5 surely does not suffer from this issue anymore? And if there is some corner case where it does suffer, then document how to reproduce that.

Otto Kekäläinen added a comment - 2020-05-17 09:54 faust Just to be sure, can you please test that latest 10.5 surely does not suffer from this issue anymore? And if there is some corner case where it does suffer, then document how to reproduce that.

Faustin Lammler added a comment - 2020-05-21 10:07

Hi otto!
I have tested the latest upstream version (10.5.4) against buster and sid.
I am not able to reproduce the issue, and synchronization (between sysvinit and systemd) seems very sane to me, also disable works (and persists after reboot) in both cases (mariadb or mysql).

The unique "not so sane" but not problematic scenario is the following (change on /etc are tracked with etckeeper):

$ cd /etc

$ sudo apt install etckeeper

$ sudo systemctl disable mariadb

$ sudo git status

On branch master

Changes not staged for commit:

  (use "git add/rm <file>..." to update what will be committed)

  (use "git restore <file>..." to discard changes in working directory)

	deleted:    rc2.d/S01mariadb

	deleted:    rc3.d/S01mariadb

	deleted:    rc4.d/S01mariadb

	deleted:    rc5.d/S01mariadb

	deleted:    systemd/system/multi-user.target.wants/mariadb.service

Untracked files:

  (use "git add <file>..." to include in what will be committed)

	rc2.d/K01mariadb

	rc3.d/K01mariadb

	rc4.d/K01mariadb

	rc5.d/K01mariadb

$ sudo systemctl enable mysql

On branch master

Changes not staged for commit:

  (use "git add/rm <file>..." to update what will be committed)

  (use "git restore <file>..." to discard changes in working directory)

	deleted:    rc2.d/S01mariadb

	deleted:    rc3.d/S01mariadb

	deleted:    rc4.d/S01mariadb

	deleted:    rc5.d/S01mariadb

Untracked files:

  (use "git add <file>..." to include in what will be committed)

	rc2.d/K01mariadb

	rc3.d/K01mariadb

	rc4.d/K01mariadb

	rc5.d/K01mariadb

$ sudo reboot

$ sudo systemctl status mariadb

● mariadb.service - MariaDB 10.5.4 database server

     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)

    Drop-In: /etc/systemd/system/mariadb.service.d

             └─migrated-from-my.cnf-settings.conf

     Active: active (running) since Thu 2020-05-21 11:34:42 CEST; 25min ago

...

Systemd is starting mariadb even if sysv runlevel files were not synchronized by systemctl enable mysql.
But this is a already known behavior (https://github.com/systemd/systemd/issues/15394).

Faustin Lammler added a comment - 2020-05-21 10:07 Hi otto ! I have tested the latest upstream version (10.5.4) against buster and sid. I am not able to reproduce the issue, and synchronization (between sysvinit and systemd) seems very sane to me, also disable works (and persists after reboot) in both cases (mariadb or mysql). The unique "not so sane" but not problematic scenario is the following (change on /etc are tracked with etckeeper ): $ cd /etc $ sudo apt install etckeeper $ sudo systemctl disable mariadb $ sudo git status On branch master Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) deleted: rc2.d/S01mariadb deleted: rc3.d/S01mariadb deleted: rc4.d/S01mariadb deleted: rc5.d/S01mariadb deleted: systemd/system/multi-user.target.wants/mariadb.service Untracked files: (use "git add <file>..." to include in what will be committed) rc2.d/K01mariadb rc3.d/K01mariadb rc4.d/K01mariadb rc5.d/K01mariadb $ sudo systemctl enable mysql On branch master Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) deleted: rc2.d/S01mariadb deleted: rc3.d/S01mariadb deleted: rc4.d/S01mariadb deleted: rc5.d/S01mariadb Untracked files: (use "git add <file>..." to include in what will be committed) rc2.d/K01mariadb rc3.d/K01mariadb rc4.d/K01mariadb rc5.d/K01mariadb $ sudo reboot $ sudo systemctl status mariadb ● mariadb.service - MariaDB 10.5.4 database server Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: active (running) since Thu 2020-05-21 11:34:42 CEST; 25min ago ... Systemd is starting mariadb even if sysv runlevel files were not synchronized by systemctl enable mysql . But this is a already known behavior ( https://github.com/systemd/systemd/issues/15394 ).

Otto Kekäläinen added a comment - 2020-05-21 19:52

Thanks for verifying we are all good now.

Since in systemd the 'mysql' is a alias for 'mariadb', it is correct that 'sudo systemctl enable mysql' will enable MariaDB and MariaDB will be running after the next reboot. It does not matter what state the rcXd./XXmariadb files are in, since if the system has systemd (and systemctl), it will do what systemd says ignoring rcXd./XXmariadb status.

Otto Kekäläinen added a comment - 2020-05-21 19:52 Thanks for verifying we are all good now. Since in systemd the 'mysql' is a alias for 'mariadb', it is correct that 'sudo systemctl enable mysql' will enable MariaDB and MariaDB will be running after the next reboot. It does not matter what state the rcXd./XXmariadb files are in, since if the system has systemd (and systemctl), it will do what systemd says ignoring rcXd./XXmariadb status.

MariaDB Server

SysV init service deployed file '/etc/init.d/mysql' prevents systemctl disable command to work correctly (mariadb|mysql naming support) (debian/ubuntu)

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration