[MDEV-27044] Test main.mysql_upgrade seems unstable Created: 2021-11-14  Updated: 2022-12-06

Status: Open
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.6.5
Fix Version/s: 10.6

Type: Bug Priority: Major
Reporter: Thomas Deutschmann Assignee: Sergei Golubchik
Resolution: Unresolved Votes: 0
Labels: None
Environment: 

sh bash 5.1_p8
 
ld GNU ld (Gentoo 2.37_p1 p0) 2.37
 
ccache version 4.4.2 [enabled]
 
app-shells/bash:          5.1_p8::gentoo
 
dev-lang/perl:            5.34.0-r3::gentoo
 
dev-lang/python:          3.9.7_p1::gentoo
 
dev-util/ccache:          4.4.2::gentoo
 
dev-util/cmake:           3.20.5::gentoo
 
sys-apps/baselayout:      2.7::gentoo
 
sys-apps/openrc:          0.44.7::gentoo
 
sys-apps/sandbox:         2.25::gentoo
 
sys-devel/autoconf:       2.71-r1::gentoo
 
sys-devel/automake:       1.16.4::gentoo
 
sys-devel/binutils:       2.37_p1::gentoo
 
sys-devel/gcc:            10.3.0-r2::gentoo
 
sys-devel/gcc-config:     2.4::gentoo
 
sys-devel/libtool:        2.4.6-r6::gentoo
 
sys-devel/make:           4.3::gentoo
 
sys-kernel/linux-headers: 5.10::gentoo (virtual/os-headers)
 
sys-libs/glibc:           2.33-r7::gentoo


cmake -C /var/tmp/portage/dev-db/mariadb-10.6.5/work/mariadb-10.6.5_build/gentoo_common_config.cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_C_FLAGS_RELWITHDEBINFO=-DNDEBUG -DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-DNDEBUG -DMYSQL_DATADIR=/var/lib/mysql -DSYSCONFDIR=/etc/mysql -DINSTALL_BINDIR=bin -DINSTALL_DOCDIR=share/doc/mariadb-10.6.5 -DINSTALL_DOCREADMEDIR=share/doc/mariadb-10.6.5 -DINSTALL_INCLUDEDIR=include/mysql -DINSTALL_INFODIR=share/info -DINSTALL_LIBDIR=lib64 -DINSTALL_MANDIR=share/man -DINSTALL_MYSQLSHAREDIR=share/mariadb -DINSTALL_PLUGINDIR=lib64/mariadb/plugin -DINSTALL_SCRIPTDIR=bin -DINSTALL_MYSQLDATADIR=/var/lib/mysql -DINSTALL_SBINDIR=sbin -DINSTALL_SUPPORTFILESDIR=/usr/share/mariadb -DWITH_COMMENT=Gentoo Linux mariadb-10.6.5 -DWITH_UNIT_TESTS=ON -DWITH_LIBEDIT=0 -DWITH_ZLIB=system -DWITHOUT_LIBWRAP=1 -DENABLED_LOCAL_INFILE=1 -DMYSQL_UNIX_ADDR=/var/run/mysqld/mysqld.sock -DINSTALL_UNIX_ADDRDIR=/var/run/mysqld/mysqld.sock -DWITH_DEFAULT_COMPILER_OPTIONS=0 -DWITH_DEFAULT_FEATURE_SET=0 -DINSTALL_SYSTEMD_UNITDIR=/lib/systemd/system -DSTACK_DIRECTION=-1 -DPKG_CONFIG_EXECUTABLE=/usr/bin/x86_64-pc-linux-gnu-pkg-config -DPLUGIN_AUTH_GSSAPI=NO -DAUTH_GSSAPI_PLUGIN_TYPE=OFF -DCONC_WITH_EXTERNAL_ZLIB=YES -DWITH_EXTERNAL_ZLIB=YES -DSUFFIX_INSTALL_DIR= -DWITH_UNITTEST=OFF -DWITHOUT_CLIENTLIBS=YES -DCLIENT_PLUGIN_DIALOG=OFF -DCLIENT_PLUGIN_AUTH_GSSAPI_CLIENT=OFF -DCLIENT_PLUGIN_CLIENT_ED25519=OFF -DCLIENT_PLUGIN_MYSQL_CLEAR_PASSWORD=STATIC -DCLIENT_PLUGIN_CACHING_SHA2_PASSWORD=OFF -DINSTALL_MYSQLTESTDIR=share/mariadb/mysql-test -DWITH_SSL=system -DCLIENT_PLUGIN_SHA256_PASSWORD=STATIC -DWITH_READLINE=0 -DNOT_FOR_DISTRIBUTION=1 -DENABLE_DTRACE=no -DPLUGIN_CONNECT=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DWITH_PCRE=system -DPLUGIN_OQGRAPH=NO -DPLUGIN_SPHINX=NO -DPLUGIN_AUTH_PAM=YES -DPLUGIN_AWS_KEY_MANAGEMENT=NO -DPLUGIN_CRACKLIB_PASSWORD_CHECK=NO -DPLUGIN_SEQUENCE=NO -DPLUGIN_SPIDER=NO -DPLUGIN_S3=NO -DPLUGIN_COLUMNSTORE=NO -DCONNECT_WITH_MYSQL=1 -DCONNECT_WITH_LIBXML2=no -DCONNECT_WITH_ODBC=no -DCONNECT_WITH_JDBC=no -DCONNECT_WITH_MONGO=OFF -DWITH_WSREP=no -DWITH_INNODB_LZ4=OFF -DWITH_INNODB_LZO=OFF -DWITH_INNODB_SNAPPY=OFF -DPLUGIN_MROONGA=NO -DPLUGIN_AUTH_GSSAPI=NO -DWITH_MARIABACKUP=ON -DWITH_LIBARCHIVE=ON -DINSTALL_SQLBENCHDIR= -DPLUGIN_ROCKSDB=NO -DWITH_SYSTEMD=no -DWITH_NUMA=OFF -DSKIP_TESTS=ON -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_unicode_520_ci -DEXTRA_CHARSETS=all -DMYSQL_USER=mysql -DDISABLE_SHARED=NO -DWITH_DEBUG=no -DWITH_EMBEDDED_SERVER=OFF -DWITH_PROFILING=no -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_CSV_STORAGE_ENGINE=1 -DWITH_HEAP_STORAGE_ENGINE=1 -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_MYISAMMRG_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/dev-db/mariadb-10.6.5/work/mariadb-10.6.5_build/gentoo_toolchain.cmake  /var/tmp/portage/dev-db/mariadb-10.6.5/work/mysql


Attachments: Text File build.log     File var-log.tar.xz    
Issue Links:
Relates
relates to MDEV-29128 Possible table (mysql.innodb_table_st... Open

 Description   

During normal build on Gentoo Linux, main.mysql_upgrade is often failing. But when running test manually, it is passing:

main.mysql_upgrade 'innodb'              w24 [ fail ]
 
        Test ended at 2021-11-14 19:49:22
 
 
 
CURRENT_TEST: main.mysql_upgrade
 
--- /var/tmp/portage/dev-db/mariadb-10.6.5/work/mysql/mysql-test/main/mysql_upgrade.result      2021-11-05 21:03:29.000000000 +0100
 
+++ /var/tmp/portage/dev-db/mariadb-10.6.5/work/mysql/mysql-test/main/mysql_upgrade.reject      2021-11-14 19:49:21.729287406 +0100
 
@@ -1706,7 +1706,6 @@
 
 root   N
 
 root   N
 
 root   N
 
-root   N
 
 aRole  Y
 
 DROP ROLE aRole;
 
 #
 
@@ -1961,7 +1960,7 @@
 
 # mysql_upgrade --force --silent 2>&1
 
 select count(*) from mysql.global_priv;
 
 count(*)
 
-5
 
+4
 
 drop table mysql.global_priv;
 
 rename table mysql.global_priv_bak to mysql.global_priv;
 
 #
 
 
 
mysqltest: Result length mismatch
 
 
 
 - saving '/var/tmp/portage/dev-db/mariadb-10.6.5/temp/var-tests/24/log/main.mysql_upgrade-innodb/' to '/var/tmp/portage/dev-db/mariadb-10.6.5/temp/var-tests/log/main.mysql_upgrade-innodb/'
 
***Warnings generated in error logs during shutdown after running tests: main.mysql_upgrade
 
 
 
2021-11-14 19:49:17 88 [ERROR] Column count of mysql.proc is wrong. Expected 21, found 20. The table is probably corrupted
 
2021-11-14 19:49:17 104 [Warning] InnoDB: Table mysql.innodb_table_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
 
2021-11-14 19:49:17 104 [Warning] InnoDB: Table mysql.innodb_index_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
 
2021-11-14 19:49:18 124 [Warning] InnoDB: Table mysql.innodb_table_stats has length mismatch in the column name table_name. Please run mariadb-upgrade
 
2021-11-14 19:49:18 124 [Warning] InnoDB: Table mysql.innodb_index_stats has length mismatch in the column name table_name. Please run mariadb-upgrade



 Comments   
Comment by Sergei Golubchik [ 2021-11-15 ]

Two things we can try.

  • what command do I need to run to repeat what you're doing?
  • can you add --verbose-restart to the mysql-test-run invocation in the ebuild and capture the complete output? Like with script -c 'ebuild mariadb-10.5.12.ebuild test' ?
Comment by Thomas Deutschmann [ 2021-11-16 ]

I attached build.log from a run using --verbose-restart and also attached the test directory.
Please see the following guide how to setup a Gentoo chroot to try on your own:

# Setup Gentoo chroot using stage3
 
cd /tmp
 
mkdir /tmp/gentoo
 
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/20211114T170549Z/stage3-amd64-openrc-20211114T170549Z.tar.xz
 
tar -xaf stage3-amd64-openrc-20211114T170549Z.tar.xz -C /tmp/gentoo
 
mount -t proc /proc gentoo/proc
 
mount --rbind /sys gentoo/sys
 
mount --make-rslave gentoo/sys
 
mount --rbind /dev gentoo/dev
 
mount --make-rslave gentoo/dev
 
 
 
# Make sure that fs.aio-max-nr is set to 250000 at minimum!
 
sysctl fs.aio-max-nr
 
 
 
# Chroot into Gentoo
 
env -i HOME=/root TERM=$TERM $(which chroot) /tmp/gentoo /bin/bash --login
 
 
 
# In Gentoo chroot...
 
export PS1="(chroot) $PS1"
 
 
 
# Make sure name resolution works...
 
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
 
 
 
# Get latest daily portage snapshot
 
emerge-webrsync
 
 
 
# Tell portage we want FEATURES=test for dev-db/mariadb
 
mkdir /etc/portage/env
 
echo 'FEATURES="test"' > /etc/portage/env/run-tests.conf
 
echo 'dev-db/mariadb run-tests.conf' >> /etc/portage/package.env
 
 
 
# Because this is Gentoo stable and dev-db/mariadb-10.6 has no keywords at the moment we have to keyword version on our own
 
echo '>=dev-db/mariadb-10.6 **' >> /etc/portage/package.accept_keywords
 
 
 
# Normally you would use `emerge ...` to install package and all deps,
 
# but because we want to make changes and debug, we have to do it manually:
 
# First, we need to install all dependencies:
 
emerge -1 --onlydeps '>=dev-db/mariadb-10.6'
 
 
 
# Chdir to mariadb ebuild
 
cd $(portageq get_repo_path / gentoo)/dev-db/mariadb
 
 
 
# Check available ebuilds using `ls -1` command.
 
# As of writing this guide, mariadb-10.6.5.ebuild is current.
 
 
 
# Because the test is currently failing, I have disabled that test in our ebuild.
 
# To debug this test failure, we first have to ensure that it will no longer get
 
# disabled (of course you can also use an editor like `nano` to make changes or
 
# install vim (`emerge -a app-editors/vim`)):
 
sed -e 's/disabled_tests+=( "main.mysql_upgrade;/#&/' -i mariadb-10.6.5.ebuild
 
 
 
# Now we can use `ebuild` command to execute the ebuild. Because
 
# we changed ebuild, we need to use "--skip-manifest" parameter to
 
# skip Manifest checks:
 
ebuild --skip-manifest mariadb-10.6.5.ebuild clean test
 
 
 
# In case you want to run a single test, you have to do that as "portage" user:
 
su portage --shell /bin/bash
 
 
 
# As portage use, chdir into mysql-tests and run desired tests
 
cd $(portageq envvar PORTAGE_TMPDIR)/portage/dev-db/mariadb-10.6.5/work/mariadb-10.6.5_build/mysql-test/
 
./mtr main.mysql_upgrade

Comment by Sergei Golubchik [ 2021-11-18 ]

First thought — you don't need to run ctest, mysql-test-run does it (look for lines that start with ‘unit.’) so it's just wasted time, running same tests twice.

Comment by Thomas Deutschmann [ 2021-11-18 ]

Thank you for the suggestion, I updated our ebuilds and removed the manual call for ctest.

Generated at Thu Feb 08 09:49:55 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.