Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31349

test maria.maria-purge failed on 'aria_log.00000002 not found'

Details

    Description

      After uploading MariaDB 10.11.3 to Launchpad builders several of them failed on test maria.maria-purge:

      maria.maria-purge                        w4 [ fail ]  Found warnings/errors in server log file!
              Test ended at 2023-05-11 23:29:58
      line
      2023-05-11 23:29:48 0 [ERROR] mariadbd: File '/<<PKGBUILDDIR>>/builddir/mysql-test/var/4/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory")
      2023-05-11 23:29:48 0 [ERROR] mariadbd: Aria engine: log purging failed
      

      Examples from amd64 and arm64 builds:
      https://launchpadlibrarian.net/665736950/buildlog_ubuntu-mantic-arm64.mariadb_1%3A10.11.3-1~ubuntu23.10.1~1683836249.0a0f09bbe32.dev.otto_BUILDING.txt.gz
      https://launchpadlibrarian.net/665721775/buildlog_ubuntu-mantic-amd64.mariadb_1%3A10.11.3-1~ubuntu23.10.1~1683836249.0a0f09bbe32.dev.otto_BUILDING.txt.gz

      Attachments

        Activity

          alice Alice Sherepa added a comment -

          10.9 e1a631fecc364ce1268b13a81

          alice@:~/git/10.10/mysql-test$ ./mtr maria.maria-big  maria.maria-purge
          Logging: ./mtr  maria.maria-big maria.maria-purge
           
          ==============================================================================
           
          TEST                                      RESULT   TIME (ms) or COMMENT
          --------------------------------------------------------------------------
           
          worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
          maria.maria-big                          [ pass ]   8664
          maria.maria-purge                        [ fail ]  Found warnings/errors in server log file!
                  Test ended at 2023-06-23 21:02:01
          line
          2023-06-23 21:01:51 0 [ERROR] mariadbd: File '/10.10/mysql-test/var/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory")
          2023-06-23 21:01:51 0 [ERROR] mariadbd: Aria engine: log purging failed
          ^ Found warnings in /10.10/mysql-test/var/log/mysqld.1.err
          ok
          --------------------------------------------------------------------------
          The servers were restarted 0 times
          Spent 8.664 of 26 seconds executing testcases
          

          alice Alice Sherepa added a comment - 10.9 e1a631fecc364ce1268b13a81 alice@:~/git/10.10/mysql-test$ ./mtr maria.maria-big maria.maria-purge Logging: ./mtr maria.maria-big maria.maria-purge   ==============================================================================   TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 maria.maria-big [ pass ] 8664 maria.maria-purge [ fail ] Found warnings/errors in server log file! Test ended at 2023-06-23 21:02:01 line 2023-06-23 21:01:51 0 [ERROR] mariadbd: File '/10.10/mysql-test/var/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory") 2023-06-23 21:01:51 0 [ERROR] mariadbd: Aria engine: log purging failed ^ Found warnings in /10.10/mysql-test/var/log/mysqld.1.err ok -------------------------------------------------------------------------- The servers were restarted 0 times Spent 8.664 of 26 seconds executing testcases

          I am seeing this same error now on Salsa CI with 10.11.5 builds, and also when I run build+MTR inside a Docker container on my own laptop:

          maria.maria-purge                        w1 [ fail ]  Found warnings/errors in server log file!
                  Test ended at 2023-10-05 02:57:14
          line
          2023-10-05  2:57:11 0 [ERROR] mariadbd: File '/builds/otto/mariadb-server/debian/output/source_dir/builddir/mysql-test/var/1/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory")
          2023-10-05  2:57:11 0 [ERROR] mariadbd: Aria engine: log purging failed
          ^ Found warnings in /builds/otto/mariadb-server/debian/output/source_dir/builddir/mysql-test/var/1/log/mysqld.1.err
          

          Examples:

          This is not a sporadic failure but repeats ever time in the build+MTR situation. However, if I run MTR outside of the built sources this test passes, example from https://salsa.debian.org/otto/mariadb-server/-/jobs/4767530/raw:

          maria.maria-purge                        w2 [ pass ]  11707
          

          Thus there is something weird going on specifically when the test is run with freshly built binaries in the source code directory which does not occur when running test in standalone mode.

          otto Otto Kekäläinen added a comment - I am seeing this same error now on Salsa CI with 10.11.5 builds, and also when I run build+MTR inside a Docker container on my own laptop: maria.maria-purge w1 [ fail ] Found warnings/errors in server log file! Test ended at 2023-10-05 02:57:14 line 2023-10-05 2:57:11 0 [ERROR] mariadbd: File '/builds/otto/mariadb-server/debian/output/source_dir/builddir/mysql-test/var/1/mysqld.1/data/aria_log.00000002' not found (Errcode: 2 "No such file or directory") 2023-10-05 2:57:11 0 [ERROR] mariadbd: Aria engine: log purging failed ^ Found warnings in /builds/otto/mariadb-server/debian/output/source_dir/builddir/mysql-test/var/1/log/mysqld.1.err Examples: https://salsa.debian.org/otto/mariadb-server/-/jobs/4770929/raw https://salsa.debian.org/otto/mariadb-server/-/jobs/4770925/raw This is not a sporadic failure but repeats ever time in the build+MTR situation. However, if I run MTR outside of the built sources this test passes, example from https://salsa.debian.org/otto/mariadb-server/-/jobs/4767530/raw: maria.maria-purge w2 [ pass ] 11707 Thus there is something weird going on specifically when the test is run with freshly built binaries in the source code directory which does not occur when running test in standalone mode.
          otto Otto Kekäläinen added a comment - I also saw this on official Debian CI, see log at https://ci.debian.net/data/autopkgtest/unstable/amd64/m/mariadb/38571769/log.gz .
          otto Otto Kekäläinen added a comment - I can reproduce this also at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.11/+builds?build_text=&build_state=all on basically all architectures.
          otto Otto Kekäläinen added a comment - - edited

          I can confirm that when MTR runs outside of the source directory as pure test system (e.g. installed via mariadb-test package as described in https://optimizedbyotto.com/post/grokking-mariadb-test-run-mtr/ or at https://ci.debian.net/packages/m/mariadb/) this test does pass:

          maria.maria-purge                        w1 [ pass ]   4795
          

          eg. https://ci.debian.net/data/autopkgtest/testing/ppc64el/m/mariadb/38598140/log.gz, https://ci.debian.net/data/autopkgtest/testing/i386/m/mariadb/38691874/log.gz, https://ci.debian.net/data/autopkgtest/testing/s390x/m/mariadb/38691392/log.gz

          However, when MTR runs in source tree directory after a build the test always fails on all platforms I tested.

          otto Otto Kekäläinen added a comment - - edited I can confirm that when MTR runs outside of the source directory as pure test system (e.g. installed via mariadb-test package as described in https://optimizedbyotto.com/post/grokking-mariadb-test-run-mtr/ or at https://ci.debian.net/packages/m/mariadb/ ) this test does pass: maria.maria-purge w1 [ pass ] 4795 eg. https://ci.debian.net/data/autopkgtest/testing/ppc64el/m/mariadb/38598140/log.gz , https://ci.debian.net/data/autopkgtest/testing/i386/m/mariadb/38691874/log.gz , https://ci.debian.net/data/autopkgtest/testing/s390x/m/mariadb/38691392/log.gz However, when MTR runs in source tree directory after a build the test always fails on all platforms I tested.

          The problem was that maria_empty_logs.inc deleted aria log files before
          the server was properly shutdown.
          Fixed by waiting for pid file to disappear before starting to delete log files.

          The test failed if there was a lot of old aria log files from previous tests.

          monty Michael Widenius added a comment - The problem was that maria_empty_logs.inc deleted aria log files before the server was properly shutdown. Fixed by waiting for pid file to disappear before starting to delete log files. The test failed if there was a lot of old aria log files from previous tests.

          I confirm https://github.com/MariaDB/server/commit/185591c1c0dfd16bc6f74999a23d5aa5d7cfb12b fixed this test failure - it is no longer visible on any of my test builds on Launchpad.

          otto Otto Kekäläinen added a comment - I confirm https://github.com/MariaDB/server/commit/185591c1c0dfd16bc6f74999a23d5aa5d7cfb12b fixed this test failure - it is no longer visible on any of my test builds on Launchpad.

          People

            monty Michael Widenius
            otto Otto Kekäläinen
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.