[MDEV-31349] test maria.maria-purge failed on 'aria_log.00000002 not found' Created: 2023-05-26  Updated: 2023-10-09  Resolved: 2023-10-09

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.11.3, 10.9, 10.10, 10.11, 11.0
Fix Version/s: 10.5.23, 10.6.16, 10.9.8, 10.10.7, 10.11.6, 11.0.4, 11.1.3

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: Michael Widenius
Resolution: Fixed Votes: 0
Labels: None


 Description   

After uploading MariaDB 10.11.3 to Launchpad builders several of them failed on test maria.maria-purge:

maria.maria-purge                        w4 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2023-05-11 23:29:58
 
line
 
2023-05-11 23:29:48 0 [ERROR] mariadbd: File '/<<PKGBUILDDIR>>/builddir/mysql-test/var/4/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory")
 
2023-05-11 23:29:48 0 [ERROR] mariadbd: Aria engine: log purging failed

Examples from amd64 and arm64 builds:
https://launchpadlibrarian.net/665736950/buildlog_ubuntu-mantic-arm64.mariadb_1%3A10.11.3-1~ubuntu23.10.1~1683836249.0a0f09bbe32.dev.otto_BUILDING.txt.gz
https://launchpadlibrarian.net/665721775/buildlog_ubuntu-mantic-amd64.mariadb_1%3A10.11.3-1~ubuntu23.10.1~1683836249.0a0f09bbe32.dev.otto_BUILDING.txt.gz

 Comments   
Comment by Alice Sherepa [ 2023-06-23 ]

10.9 e1a631fecc364ce1268b13a81

 
alice@:~/git/10.10/mysql-test$ ./mtr maria.maria-big  maria.maria-purge
 
Logging: ./mtr  maria.maria-big maria.maria-purge
 
 
 
==============================================================================
 
 
 
TEST                                      RESULT   TIME (ms) or COMMENT
 
--------------------------------------------------------------------------
 
 
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
 
maria.maria-big                          [ pass ]   8664
 
maria.maria-purge                        [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2023-06-23 21:02:01
 
line
 
2023-06-23 21:01:51 0 [ERROR] mariadbd: File '/10.10/mysql-test/var/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory")
 
2023-06-23 21:01:51 0 [ERROR] mariadbd: Aria engine: log purging failed
 
^ Found warnings in /10.10/mysql-test/var/log/mysqld.1.err
 
ok
 
--------------------------------------------------------------------------
 
The servers were restarted 0 times
 
Spent 8.664 of 26 seconds executing testcases

Comment by Otto Kekäläinen [ 2023-10-05 ]

I am seeing this same error now on Salsa CI with 10.11.5 builds, and also when I run build+MTR inside a Docker container on my own laptop:

maria.maria-purge                        w1 [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2023-10-05 02:57:14
 
line
 
2023-10-05  2:57:11 0 [ERROR] mariadbd: File '/builds/otto/mariadb-server/debian/output/source_dir/builddir/mysql-test/var/1/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory")
 
2023-10-05  2:57:11 0 [ERROR] mariadbd: Aria engine: log purging failed
 
^ Found warnings in /builds/otto/mariadb-server/debian/output/source_dir/builddir/mysql-test/var/1/log/mysqld.1.err

Examples:

This is not a sporadic failure but repeats ever time in the build+MTR situation. However, if I run MTR outside of the built sources this test passes, example from https://salsa.debian.org/otto/mariadb-server/-/jobs/4767530/raw:

maria.maria-purge                        w2 [ pass ]  11707

Thus there is something weird going on specifically when the test is run with freshly built binaries in the source code directory which does not occur when running test in standalone mode.

Comment by Otto Kekäläinen [ 2023-10-05 ]

I also saw this on official Debian CI, see log at https://ci.debian.net/data/autopkgtest/unstable/amd64/m/mariadb/38571769/log.gz.

Comment by Otto Kekäläinen [ 2023-10-05 ]

I can reproduce this also at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.11/+builds?build_text=&build_state=all on basically all architectures.

Comment by Otto Kekäläinen [ 2023-10-07 ]

I can confirm that when MTR runs outside of the source directory as pure test system (e.g. installed via mariadb-test package as described in https://optimizedbyotto.com/post/grokking-mariadb-test-run-mtr/ or at https://ci.debian.net/packages/m/mariadb/) this test does pass:

maria.maria-purge                        w1 [ pass ]   4795

eg. https://ci.debian.net/data/autopkgtest/testing/ppc64el/m/mariadb/38598140/log.gz, https://ci.debian.net/data/autopkgtest/testing/i386/m/mariadb/38691874/log.gz, https://ci.debian.net/data/autopkgtest/testing/s390x/m/mariadb/38691392/log.gz

However, when MTR runs in source tree directory after a build the test always fails on all platforms I tested.

Comment by Michael Widenius [ 2023-10-08 ]

The problem was that maria_empty_logs.inc deleted aria log files before
the server was properly shutdown.
Fixed by waiting for pid file to disappear before starting to delete log files.

The test failed if there was a lot of old aria log files from previous tests.

Comment by Otto Kekäläinen [ 2023-10-09 ]

I confirm https://github.com/MariaDB/server/commit/185591c1c0dfd16bc6f74999a23d5aa5d7cfb12b fixed this test failure - it is no longer visible on any of my test builds on Launchpad.

Generated at Thu Feb 08 10:23:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.