Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27044

Test main.mysql_upgrade seems unstable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.6.5
    • 10.6
    • Tests
    • None

    Description

      During normal build on Gentoo Linux, main.mysql_upgrade is often failing. But when running test manually, it is passing:

      main.mysql_upgrade 'innodb'              w24 [ fail ]
              Test ended at 2021-11-14 19:49:22
       
      CURRENT_TEST: main.mysql_upgrade
      --- /var/tmp/portage/dev-db/mariadb-10.6.5/work/mysql/mysql-test/main/mysql_upgrade.result      2021-11-05 21:03:29.000000000 +0100
      +++ /var/tmp/portage/dev-db/mariadb-10.6.5/work/mysql/mysql-test/main/mysql_upgrade.reject      2021-11-14 19:49:21.729287406 +0100
      @@ -1706,7 +1706,6 @@
       root   N
       root   N
       root   N
      -root   N
       aRole  Y
       DROP ROLE aRole;
       #
      @@ -1961,7 +1960,7 @@
       # mysql_upgrade --force --silent 2>&1
       select count(*) from mysql.global_priv;
       count(*)
      -5
      +4
       drop table mysql.global_priv;
       rename table mysql.global_priv_bak to mysql.global_priv;
       #
       
      mysqltest: Result length mismatch
       
       - saving '/var/tmp/portage/dev-db/mariadb-10.6.5/temp/var-tests/24/log/main.mysql_upgrade-innodb/' to '/var/tmp/portage/dev-db/mariadb-10.6.5/temp/var-tests/log/main.mysql_upgrade-innodb/'
      ***Warnings generated in error logs during shutdown after running tests: main.mysql_upgrade
       
      2021-11-14 19:49:17 88 [ERROR] Column count of mysql.proc is wrong. Expected 21, found 20. The table is probably corrupted
      2021-11-14 19:49:17 104 [Warning] InnoDB: Table mysql.innodb_table_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
      2021-11-14 19:49:17 104 [Warning] InnoDB: Table mysql.innodb_index_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
      2021-11-14 19:49:18 124 [Warning] InnoDB: Table mysql.innodb_table_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
      2021-11-14 19:49:18 124 [Warning] InnoDB: Table mysql.innodb_index_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
      

      Attachments

        1. build.log
          2.87 MB
        2. var-log.tar.xz
          1.34 MB

        Issue Links

          Activity

            Two things we can try.

            • what command do I need to run to repeat what you're doing?
            • can you add --verbose-restart to the mysql-test-run invocation in the ebuild and capture the complete output? Like with script -c 'ebuild mariadb-10.5.12.ebuild test' ?
            serg Sergei Golubchik added a comment - Two things we can try. what command do I need to run to repeat what you're doing? can you add --verbose-restart to the mysql-test-run invocation in the ebuild and capture the complete output? Like with script -c 'ebuild mariadb-10.5.12.ebuild test' ?

            I attached build.log from a run using --verbose-restart and also attached the test directory.
            Please see the following guide how to setup a Gentoo chroot to try on your own:

            # Setup Gentoo chroot using stage3
            cd /tmp
            mkdir /tmp/gentoo
            wget http://distfiles.gentoo.org/releases/amd64/autobuilds/20211114T170549Z/stage3-amd64-openrc-20211114T170549Z.tar.xz
            tar -xaf stage3-amd64-openrc-20211114T170549Z.tar.xz -C /tmp/gentoo
            mount -t proc /proc gentoo/proc
            mount --rbind /sys gentoo/sys
            mount --make-rslave gentoo/sys
            mount --rbind /dev gentoo/dev
            mount --make-rslave gentoo/dev
             
            # Make sure that fs.aio-max-nr is set to 250000 at minimum!
            sysctl fs.aio-max-nr
             
            # Chroot into Gentoo
            env -i HOME=/root TERM=$TERM $(which chroot) /tmp/gentoo /bin/bash --login
             
            # In Gentoo chroot...
            export PS1="(chroot) $PS1"
             
            # Make sure name resolution works...
            echo 'nameserver 8.8.8.8' > /etc/resolv.conf
             
            # Get latest daily portage snapshot
            emerge-webrsync
             
            # Tell portage we want FEATURES=test for dev-db/mariadb
            mkdir /etc/portage/env
            echo 'FEATURES="test"' > /etc/portage/env/run-tests.conf
            echo 'dev-db/mariadb run-tests.conf' >> /etc/portage/package.env
             
            # Because this is Gentoo stable and dev-db/mariadb-10.6 has no keywords at the moment we have to keyword version on our own
            echo '>=dev-db/mariadb-10.6 **' >> /etc/portage/package.accept_keywords
             
            # Normally you would use `emerge ...` to install package and all deps,
            # but because we want to make changes and debug, we have to do it manually:
            # First, we need to install all dependencies:
            emerge -1 --onlydeps '>=dev-db/mariadb-10.6'
             
            # Chdir to mariadb ebuild
            cd $(portageq get_repo_path / gentoo)/dev-db/mariadb
             
            # Check available ebuilds using `ls -1` command.
            # As of writing this guide, mariadb-10.6.5.ebuild is current.
             
            # Because the test is currently failing, I have disabled that test in our ebuild.
            # To debug this test failure, we first have to ensure that it will no longer get
            # disabled (of course you can also use an editor like `nano` to make changes or
            # install vim (`emerge -a app-editors/vim`)):
            sed -e 's/disabled_tests+=( "main.mysql_upgrade;/#&/' -i mariadb-10.6.5.ebuild
             
            # Now we can use `ebuild` command to execute the ebuild. Because
            # we changed ebuild, we need to use "--skip-manifest" parameter to
            # skip Manifest checks:
            ebuild --skip-manifest mariadb-10.6.5.ebuild clean test
             
            # In case you want to run a single test, you have to do that as "portage" user:
            su portage --shell /bin/bash
             
            # As portage use, chdir into mysql-tests and run desired tests
            cd $(portageq envvar PORTAGE_TMPDIR)/portage/dev-db/mariadb-10.6.5/work/mariadb-10.6.5_build/mysql-test/
            ./mtr main.mysql_upgrade
            

            whissi Thomas Deutschmann added a comment - I attached build.log from a run using --verbose-restart and also attached the test directory. Please see the following guide how to setup a Gentoo chroot to try on your own: # Setup Gentoo chroot using stage3 cd /tmp mkdir /tmp/gentoo wget http://distfiles.gentoo.org/releases/amd64/autobuilds/20211114T170549Z/stage3-amd64-openrc-20211114T170549Z.tar.xz tar -xaf stage3-amd64-openrc-20211114T170549Z.tar.xz -C /tmp/gentoo mount -t proc /proc gentoo/proc mount --rbind /sys gentoo/sys mount --make-rslave gentoo/sys mount --rbind /dev gentoo/dev mount --make-rslave gentoo/dev   # Make sure that fs.aio-max-nr is set to 250000 at minimum! sysctl fs.aio-max-nr   # Chroot into Gentoo env -i HOME=/root TERM=$TERM $(which chroot) /tmp/gentoo /bin/bash --login   # In Gentoo chroot... export PS1="(chroot) $PS1"   # Make sure name resolution works... echo 'nameserver 8.8.8.8' > /etc/resolv.conf   # Get latest daily portage snapshot emerge-webrsync   # Tell portage we want FEATURES=test for dev-db/mariadb mkdir /etc/portage/env echo 'FEATURES="test"' > /etc/portage/env/run-tests.conf echo 'dev-db/mariadb run-tests.conf' >> /etc/portage/package.env   # Because this is Gentoo stable and dev-db/mariadb-10.6 has no keywords at the moment we have to keyword version on our own echo '>=dev-db/mariadb-10.6 **' >> /etc/portage/package.accept_keywords   # Normally you would use `emerge ...` to install package and all deps, # but because we want to make changes and debug, we have to do it manually: # First, we need to install all dependencies: emerge -1 --onlydeps '>=dev-db/mariadb-10.6'   # Chdir to mariadb ebuild cd $(portageq get_repo_path / gentoo)/dev-db/mariadb   # Check available ebuilds using `ls -1` command. # As of writing this guide, mariadb-10.6.5.ebuild is current.   # Because the test is currently failing, I have disabled that test in our ebuild. # To debug this test failure, we first have to ensure that it will no longer get # disabled (of course you can also use an editor like `nano` to make changes or # install vim (`emerge -a app-editors/vim`)): sed -e 's/disabled_tests+=( "main.mysql_upgrade;/#&/' -i mariadb-10.6.5.ebuild   # Now we can use `ebuild` command to execute the ebuild. Because # we changed ebuild, we need to use "--skip-manifest" parameter to # skip Manifest checks: ebuild --skip-manifest mariadb-10.6.5.ebuild clean test   # In case you want to run a single test, you have to do that as "portage" user: su portage --shell /bin/bash   # As portage use, chdir into mysql-tests and run desired tests cd $(portageq envvar PORTAGE_TMPDIR)/portage/dev-db/mariadb-10.6.5/work/mariadb-10.6.5_build/mysql-test/ ./mtr main.mysql_upgrade

            First thought — you don't need to run ctest, mysql-test-run does it (look for lines that start with ‘unit.’) so it's just wasted time, running same tests twice.

            serg Sergei Golubchik added a comment - First thought — you don't need to run ctest , mysql-test-run does it (look for lines that start with ‘unit.’) so it's just wasted time, running same tests twice.

            Thank you for the suggestion, I updated our ebuilds and removed the manual call for ctest.

            whissi Thomas Deutschmann added a comment - Thank you for the suggestion, I updated our ebuilds and removed the manual call for ctest.

            People

              serg Sergei Golubchik
              whissi Thomas Deutschmann
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.