Yuchen Pei added a comment - 2023-10-19 02:36 - edited As some background, this is a peculiar test, because it tests the server start with a flag (--wsrep-recover) that "aborts" the server start. mtr does not like to see its own server dead, so we can't have the flags in an .opt file, or in a restart_parameter for --source include/restart_mysqld.inc . Prior to MDEV-22979 , the test used $MYSQLD_BOOTSTRAP_CMD. For whatever reason it stopped working with the fix of the spider init bugs, but in any case, we should use $MYSQLD_CMD instead.

Yuchen Pei added a comment - 2023-10-19 02:39 - edited Here's an initial attempt to fix this issue: upstream/bb-10.10-mdev-32515 upstream/bb-10.10-all-builders 5bd85cb229f187c7c24c69659ff2caedb99f6366 MDEV-32515 Use the mtr cnf in the spider/bugfix.mdev_30370 mysqld invocation   This makes sure the $MYSQLD_CMD invocation uses the same port, and does not use the default port which may already be in use. Let's wait and see how it works in the CI. That did not work, as we get the same failures: https://buildbot.mariadb.net/buildbot/grid?category=main&branch=bb-10.10-all-builders https://buildbot.mariadb.org/#grid?branch=bb-10.10-mdev-32515

Yuchen Pei added a comment - 2023-10-19 05:28 It is strange that this test fails at --exec $MYSQLD_CMD with "Do you already have another server running on socket" [1] [2] or "Do you already have another server running on port: 3306 ?" [3] [4] let $MYSQLD_DATADIR= ` select @@datadir`; let $PLUGIN_DIR=` select @@plugin_dir`; --source include/kill_mysqld.inc --exec $MYSQLD_CMD --datadir=$MYSQLD_DATADIR --wsrep-recover --plugin-dir=$PLUGIN_DIR --plugin-load-add=ha_spider --source include/start_mysqld.inc --disable_query_log --source ../../include/clean_up_spider.inc but this test passes let $MYSQLD_DATADIR= ` select @@datadir`; let $PLUGIN_DIR=` select @@plugin_dir`; --source include/kill_mysqld.inc --write_file $MYSQLTEST_VARDIR/tmp/mdev_22979.sql drop table if exists foo.bar; EOF --exec $MYSQLD_CMD --datadir=$MYSQLD_DATADIR --bootstrap --plugin-dir=$PLUGIN_DIR --plugin-load-add=ha_spider < $MYSQLTEST_VARDIR/tmp/mdev_22979.sql --source include/start_mysqld.inc --disable_query_log --source ../../include/clean_up_spider.inc The only difference I can see is the --bootstrap flag... It only happens in certain old buildbot CI builders [5] . [1] https://buildbot.mariadb.net/buildbot/builders/kvm-zyp-opensuse150-amd64/builds/10857/steps/mtr/logs/stdio [2] https://buildbot.mariadb.net/buildbot/builders/kvm-zyp-opensuse150-amd64/builds/10857 [3] https://buildbot.mariadb.net/buildbot/builders/kvm-deb-bullseye-aarch64/builds/1585 [4] https://buildbot.mariadb.net/buildbot/builders/kvm-deb-bullseye-aarch64/builds/1585/steps/mtr/logs/stdio [5] https://buildbot.mariadb.net/buildbot/grid?category=main&branch=bb-10.10-all-builders In any case, I created a commit to add --bootstrap. Let's see whether that helps. upstream/bb-10.10-all-builders c93d8c32c97170d63e968da047927ecf0a3b2001 MDEV-32515 [experiment] Add --bootstrap to the $MYSQLD_CMD invocation in mdev_30370   After all, spider/bugfix.mdev_22979 passes, and the only difference that may matter is the --bootstrap flag

Yuchen Pei added a comment - 2023-10-19 23:30 - edited This simple fix seems to work, see [1] for the commit c9e5d725bb8c which is identical except the commit comment. bb-10.10-mdev-32515 85262c138dbdd1e39046571cb87645621fa7baf2 MDEV-32515 Use $MYSQLD_LAST_CMD in spider/bugfix.mdev_30370   $MYSQLD_CMD uses .1 as the defaults-group-suffix, which could cause the use of the default port (3306) or socket, which will fail in environment where these defaults are already in use by another server.   Adding an extra --defaults-group-suffix=.1.1 does not help, because the first flag wins.   So we use $MYSQLD_LAST_CMD instead, which uses the correct suffix.   The extra innodb buffer pool warning is irrelevant to the goal of the test (running --wsrep-recover with --plug-load-add=ha_spider should not cause hang) [1] https://buildbot.mariadb.net/buildbot/grid?category=main&branch=bb-10.10-all-builders

Yuchen Pei added a comment - 2023-10-20 03:33 danblack said "c9e5d725bb8c0d8eb28caf6bc766e946fc0cf8d7 is fine. Reviewed by me complete." Thanks for the review. So I am going to push 85262c138dbdd1e39046571cb87645621fa7baf2 which is the same as c9e5d725bb8c0d8eb28caf6bc766e946fc0cf8d7 except a more elaborate commit message.

Daniel Black added a comment - 2023-10-20 04:35 Yep. Its good.

Daniel Black added a comment - 2023-10-20 04:40 yep. good fix.

Yuchen Pei added a comment - 2023-10-20 06:15 Pushed 057fd528766eba150b9d7a0de8f95a4094f0e460 to 10.10