Any news? This is quite blocking as issue.

Thx!

It might be worth checking to see if the fix for MDEV-20614 solves your problem. If so, then this bug might be a duplicate.

Hi Geoff,

The second suggested modification (moving the default variable at the first position) does not change anything. Same error.

Thx!

venizia03 The cause of the error is determined - "mysqld" is added again as the group name, although this part of the line should be excluded from the string passed through --defaults-group-suffix. Patch is being tested.

Thx Julius. But when I try to launch the same command without the mysqld, it does crash too.

venizia03 This patch: https://github.com/MariaDB/server/commit/748c97bc19b1e27e968a6eea684cc8930b634fa3 fixes the problem in mysqld_multi, after verification I will push it to the head revision. I note that it assumes that the previous patch (MDEV-20614) is already applied. If it is applied then we no longer have the error "[ERROR] /usr/sbin/mysqld: unknown variable 'defaults-group-suffix=mysqld1'" but without new fix not all parameters can be read from the configuration files. Please tell me which command line exactly leads to an error without "mysqld"? And can you give me a fragment of the log file with this error? The fact is that if there is some kind of error when starting without "--defaults-group-suffix", then most likely this is no longer related to the regressions after MDEV-18863 and I need to study this issue separately.

Julius,

I applied your changed on my server and still not working. The command launched by systemctl is the following:

/bin/sh /usr/bin/mysqld_safe --defaults-group-suffix=2 --user=mysql --pid-file=/var/run/mysqld/mysqld2.pid --socket=/var/run/mysqld/mysqld2.sock --port=3307 --datadir=/var/lib/mysql2 --server-id=2 --bind-address=0.0.0.0 --log-error=/var/log/mysqld2.log --symbolic-links=0

As you can see the default group suffix is in front and has only the GNR as value.

Here is the log:

2019-10-07 16:15:08 0 [Note] InnoDB: Using Linux native AIO
2019-10-07 16:15:08 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-07 16:15:08 0 [Note] InnoDB: Uses event mutexes
2019-10-07 16:15:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-10-07 16:15:08 0 [Note] InnoDB: Number of pools: 1
2019-10-07 16:15:08 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-10-07 16:15:08 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-10-07 16:15:08 0 [Note] InnoDB: Completed initialization of buffer pool
2019-10-07 16:15:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-10-07 16:15:08 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-10-07 16:15:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-10-07 16:15:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-10-07 16:15:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-10-07 16:15:08 0 [Note] InnoDB: Waiting for purge to start
2019-10-07 16:15:08 0 [Note] InnoDB: 10.4.8 started; log sequence number 22175603; transaction id 3511
2019-10-07 16:15:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql2/ib_buffer_pool
2019-10-07 16:15:08 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-10-07 16:15:08 0 [ERROR] /usr/sbin/mysqld: unknown variable 'defaults-group-suffix=2'
2019-10-07 16:15:08 0 [ERROR] Aborting

Hope this will help.

Log when I comment the defaults-group-suffix line in mysqld_multi (if this could help):

2019-10-07 16:17:21 0 [Note] InnoDB: Using Linux native AIO
2019-10-07 16:17:21 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-07 16:17:21 0 [Note] InnoDB: Uses event mutexes
2019-10-07 16:17:21 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-10-07 16:17:21 0 [Note] InnoDB: Number of pools: 1
2019-10-07 16:17:21 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-10-07 16:17:21 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-10-07 16:17:21 0 [Note] InnoDB: Completed initialization of buffer pool
2019-10-07 16:17:21 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-10-07 16:17:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-10-07 16:17:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-10-07 16:17:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-10-07 16:17:21 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-10-07 16:17:21 0 [Note] InnoDB: Waiting for purge to start
2019-10-07 16:17:21 0 [Note] InnoDB: 10.4.8 started; log sequence number 22176114; transaction id 3511
2019-10-07 16:17:21 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql2/ib_buffer_pool
2019-10-07 16:17:21 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-10-07 16:17:21 0 [Note] Server socket created on IP: '0.0.0.0'.
2019-10-07 16:17:21 0 [Note] InnoDB: Buffer pool(s) load completed at 191007 16:17:21
2019-10-07 16:17:21 0 [Note] Reading of all Master_info entries succeeded
2019-10-07 16:17:21 0 [Note] Added new Master_info '' to hash table
2019-10-07 16:17:21 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.4.8-MariaDB' socket: '/var/run/mysqld/mysqld2.sock' port: 3307 MariaDB Server

venizia03 Thank you very much for the fragment from the log. Apparently MDEV-18863 showed some kind of error in mysqld_safe. Although, at first glance, and judging by the text of the script, everything was done correctly there for passing --defaults-group-suffix as the first mysqld parameter. But at the same time, we see that mysqld behaves as if other options precede this option - as if it were not the first. Now it comes to my mind to ask you what is the value of the MYSQLD environment variable in this system? Maybe it contains some options besides the direct path to the mysqld binary file? If this variable is not set explicitly or if there are no options in it (only the path to the mysqld binary), then I will further deal with mysqld_safe, since then it turns out that this script is not able to process --defaults-group-suffix in the right way , although its source code explicitly provides such an opportunity to the user...

Julius,

I start the process via systemd. I checked again this morning and below you will see the commands launched by the systemd (mysqld_safe) and the mysqld command:

mysql@1.service - MySQL Multi Server for instance 1
Loaded: loaded (/etc/systemd/system/mysql@.service; enabled; vendor preset: disabled)
Active: active (running) since mar 2019-10-08 07:38:43 CEST; 621ms ago
Process: 6944 ExecStop=/usr/bin/mysqld_multi stop %i (code=exited, status=0/SUCCESS)
Process: 6953 ExecStart=/usr/bin/mysqld_multi start %i (code=exited, status=0/SUCCESS)
Main PID: 6958 (mysqld_safe)
CGroup: /system.slice/system-mysql.slice/mysql@1.service
├─6958 /bin/sh /usr/bin/mysqld_safe --defaults-group-suffix=1 --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --datadir=/var/lib/mysql --server-id=1 --bind-address=0.0.0.0 --max_allowed_packet=64M --log-error=/var/log/mysqld1.log --symbolic-links=0
└─7070 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --defaults-group-suffix=1 --server-id=1 --bind-address=0.0.0.0 --max_allowed_packet=64M --symbolic-links=0 --log-error=/var/log/mysqld1.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

As you can see, the defaults-group-suffix variable is not the first variable in mysqld command anymore.
When I start the mysqld process by hand (without mysqld_safe) with the defaults-group-suffix variable at first position, it works. So the issue seems to be located in the way mysqld_safe launches mysqld.

Julius,

Have you made any progress on that issue?
BTW it's not only related to 10.4 but also to 10.3.

Thx!

venizia03 Hi! Please tell me, what does the fragment of the mysqld_safe.sh file look like (or just mysqld_safe, without the .sh extension) if you look in it for the first substring of the form "--basedir=$MY_BASEDIR_VERSION"?

In our GitHub repositories, all current versions (CS) look like this:

for i in  "$ledir/$MYSQLD" "$defaults_group_suffix" "$defaults" "--basedir=$MY_BASEDIR_VERSION" \

  "--datadir=$DATADIR" "--plugin-dir=$plugin_dir" "$USER_OPTION"

do

  cmd="$cmd "`shell_quote_string "$i"`

done

And this is the correct code.

But enterprise versions contain the wrong (actually just outdated) code:

for i in  "$ledir/$MYSQLD" "$defaults" "--basedir=$MY_BASEDIR_VERSION" \

  "--datadir=$DATADIR" "--plugin-dir=$plugin_dir" "$USER_OPTION"

do

  cmd="$cmd "`shell_quote_string "$i"`

done

Your launch is more in line with the ES version than with the open-source one. The fact is that this flaw has already been fixed in head revisions on GitHub, so I can not reproduce it on 10.3/10.4 (CS, head revision). But I reproduced it using ES editions and now I made patch for our ES repositories as well - the patch is currently under the mtr testing.

I am very thankful to you for the detailed information that helps to find the source of the problem!

Hi Julius,

The code I have:

for i in "$ledir/$MYSQLD" "$defaults" "--basedir=$MY_BASEDIR_VERSION" \
"-datadir=$DATADIR" "-plugin-dir=$plugin_dir" "$USER_OPTION"
do
cmd="$cmd "`shell_quote_string "$i"`
done

I so try to add the missing parameter in front but without success. The launched command is still the same and the error message too.

venizia03Hi! This change is not enough - other changes were made there to parse the value of the corresponding parameter. I am attaching the correct versions of mysqld_safe.sh and mysqld_multi.sh. This is the version for the CS edition. If you use Enterprise, I will update them soon too - I am now waiting for the readiness of our updated patch testing system (for ES) to verify patch. mysqld_multi.sh mysqld_safe.sh

Fixed, https://github.com/MariaDB/server/commit/748c97bc19b1e27e968a6eea684cc8930b634fa3

Julius,

Is there a wat to know when this fix will be available in the community version?

Thx!