[MDEV-26612] Two different ways to start MariaDB service can cause data corruption Created: 2021-09-15  Updated: 2022-03-13  Resolved: 2022-03-13

Status: Closed
Project: MariaDB Server
Component/s: Packaging
Affects Version/s: 10.2, 10.3
Fix Version/s: 10.2.41, 10.3.32, 10.4.22, 10.5.13, 10.6.5

Type: Bug Priority: Major
Reporter: Timofey Turenko Assignee: Timofey Turenko
Resolution: Fixed Votes: 0
Labels: None
Environment:

PRM base distribution



 Description   

Server can be started in two ways:

service mariadb start 

and

serivce mysqld start

These two ways are not equal: first way is defined (probably) by post-install scripts, the second - by file from the package /etc/init.d/mysql

The data corruption can happen if we start first `service mysqld start` and then try to restart with `service mariadb ....`

As example:

sudo service mysqld start; sudo service mariadb start
echo 'CREATE DATABASE test1' | sudo mysql
echo 'CREATE TABLE t1 (i INT)' | sudo mysql test1
echo 'INSERT INTO t1 VALUES(1)' | sudo mysql test1
mkdir -p /var/tmp/backup/preupgrade_backup
sudo mariabackup --backup --user=root --target-dir=/var/tmp/backup/preupgrade_backup

results in:

[00] 2021-09-15 17:15:01 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set
[00] 2021-09-15 17:15:01 Using server version 10.3.31-MariaDB
mariabackup based on MariaDB server 10.3.31-MariaDB Linux (x86_64)
[00] 2021-09-15 17:15:01 uses posix_fadvise().
[00] 2021-09-15 17:15:01 cd to /var/lib/mysql/
[00] 2021-09-15 17:15:01 open files limit requested 0, set to 1024
[00] 2021-09-15 17:15:01 mariabackup: using the following InnoDB configuration:
[00] 2021-09-15 17:15:01 innodb_data_home_dir = 
[00] 2021-09-15 17:15:01 innodb_data_file_path = ibdata1:12M:autoextend
[00] 2021-09-15 17:15:01 innodb_log_group_home_dir = ./
[00] 2021-09-15 17:15:01 InnoDB: Using Linux native AIO
2021-09-15 17:15:01 0 [Note] InnoDB: Number of pools: 1
[00] 2021-09-15 17:15:01 mariabackup: Generating a list of tablespaces
2021-09-15 17:15:01 0 [Warning] InnoDB: Allocated tablespace ID 1 for mysql/innodb_table_stats, old maximum was 0
Skipping db: ./#rocksdb
2021-09-15 17:15:01 0 [ERROR] InnoDB: ############### CORRUPT LOG RECORD FOUND ##################
2021-09-15 17:15:01 0 [Note] InnoDB: Log record type 116, page 0:1. Log parsing proceeded successfully up to 1625479. Previous log record type 128, is multi 0 Recv offset 0, prev 0
2021-09-15 17:15:01 0 [Note] InnoDB: Hex dump starting 0 bytes before and ending 100 bytes after the corrupted record:
 len 100; hex 74657374312f74312e69626400370500000f2e2f74657374312f74312e696264001f3b0500020500001808040500002605040500002a00040500002e04040500003200040500003621040500003a00040500003e000405000042f0ffffffff0205000046; asc test1/t1.ibd 7    ./test1/t1.ibd  ;            &     *     .     2     6!    :     >     B         F;
2021-09-15 17:15:01 0 [Note] InnoDB: Set innodb_force_recovery to ignore this error.
[00] 2021-09-15 17:15:01 Error: copying the log failed
[00] FATAL ERROR: 2021-09-15 17:15:01 xtrabackup_copy_logfile() failed: corrupt log.



 Comments   
Comment by Elena Stepanova [ 2021-09-15 ]

The root cause of InnoDB corruption is the lack of protection against double-start, it is discussed in MDEV-24393 and is out of the scope of this issue.
The question here is why mariadb service and mysql service run the server in a different way, the latter uses the init script with mysqld_safe (ATTN: I think it's mysql, not mysqld, but please double check).
Please check from the code/history whether it's intentional, and if not, it probably needs to be fixed.

Comment by Alexey Bychko (Inactive) [ 2021-09-22 ]

the bug is confirmed on RHEL-based systems

Comment by Alexey Bychko (Inactive) [ 2021-09-23 ]

debians are not affected because they use different /etc/init.d/mysql script

Comment by Alexey Bychko (Inactive) [ 2021-09-23 ]

the fix is 23bfbba97959dbeba2e7f1d7046d04dbf911cbc4
I didn't see any problems with that, but requested review just to make sure it will not create another issues

Comment by Alexey Bychko (Inactive) [ 2021-09-24 ]

the fix is pushed.
tturenko, could you please take a look at possibly remaining issues I described above?

Generated at Thu Feb 08 09:46:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.