I will start working on the issue once MDEV-22979 is fixed.

Deleting storage/spider/mysql-test/spider/bugfix/my.cnf fixes the problem.

Deleting my_1_1.cnf from the file result in a hang:

--- a/storage/spider/mysql-test/spider/bugfix/my.cnf

+++ b/storage/spider/mysql-test/spider/bugfix/my.cnf

@@ -1,2 +1 @@

 !include include/default_mysqld.cnf

-!include my_1_1.cnf

Putting the empty file storage/spider/mysql-test/spider/bugfix/my.cnf still cause a hang.

As elenst said (in a private communication), this is due to the non-conventional server numbering of the Spider test suite.

restart_mysqld.inc writes an *.expect file so that MTR wouldn't panic when the server is down. The file has a server number in it. In Spider case, it's written as mysqld.1.expect, while the servers have double-numbering, e.g. 1.1.

A workaround proposed by elenst is manually doing some part of what restart_mysqld.inc does like the following:

--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.1.expect

--shutdown_server

--source include/wait_until_disconnected.inc

--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.1.expect

--enable_reconnect

--source include/wait_until_connected_again.inc

The workaround became storage/spider/mysql-test/spider/bugfix/include/restart_spider.inc.

However the filename is misleading and should better be something like restart_mysqld_spider.inc.

Also, it misses some features from restart_mysqld.inc, including parameters like $restart_parameters or $shutdown_timeout. To fix that, we can have some options:

1. Update the spider version of restart_mysqld.inc. This will result in more code duplication
2. Update the restart_mysqld.inc by adding an expect_file_name parameter which is simpler and cleaner:

diff --git a/mysql-test/include/shutdown_mysqld.inc

    b/mysql-test/include/shutdown_mysqld.inc

index db0cfb82c68..0a5b9f4fb40 100644

--- a/mysql-test/include/shutdown_mysqld.inc

+++ b/mysql-test/include/shutdown_mysqld.inc

@@ -24,6 +24,10 @@ if ($rpl_inited)

 # Write file to make mysql-test-run.pl expect the "crash", but don't

    start it

 --let $_server_id= `SELECT @@server_id`

 --let $_expect_file_name=

    $MYSQLTEST_VARDIR/tmp/mysqld.$_server_id.expect

+if ($expect_file_name)

+{

+  -- let $_expect_file_name=$expect_file_name

+}

 --exec echo "wait" > $_expect_file_name

 # Avoid warnings from connection threads that does not have time to

    exit

Hi serg, elenst suggested that I send this to you for review, and you may suggest someone else to review it. Here's the (tiny) patch:

https://github.com/MariaDB/server/commit/fb7902b116f

The real problem is that shutdown_mysqld.inc (and, likely, other .inc files too) use @@server_id while mtr uses group suffix. If .inc files would be using the correct file name, spider tests wouldn't need to specify the expect file manually.

I suggest the following fix instead

---let $_server_id= `SELECT @@server_id`

---let $_expect_file_name= $MYSQLTEST_VARDIR/tmp/mysqld.$_server_id.expect

+--let $_expect_file_name= `select regexp_replace(@@tmpdir, '^.*/','')`

+--let $_expect_file_name= $MYSQLTEST_VARDIR/tmp/$_expect_file_name.expect

that seems to work correctly for your spider test case and for other test cases too.

Perhaps this should be done in all other include files that use $_expect_file_name ?

Note, after this change innodb/t/temporary_table.test will fail, because it changes tmpdir to /dev/null/nonexistent. Can be fixed with

---exec echo "restart: --tmpdir=/dev/null/nonexistent" > $_expect_file_name

+--exec echo "restart: --tmpdir=/dev/null/$MYSQL_TMP_DIR" > $_expect_file_name

(and also add_suppression() at the top of the file needs to be adjusted)

Thanks for the review serg, PTAL thanks: https://github.com/MariaDB/server/commit/0548a185a1e

0548a185a1e looks good, thanks. Ok to push

I got back to this task today and tested the patch for 10.3-11.1 (hopefully for the last time). I tried to push to 10.3 but got rejected:

remote: error: GH006: Protected branch update failed for refs/heads/10.3.        

remote: error: You're not authorized to push to this branch. Visit https://docs.github.com/articles/about-protected-branches/ for more information.        

To ssh://ssh.github.com/MariaDB/server.git

 ! [remote rejected]         bb-10.3-mdev-27912 -> 10.3 (protected branch hook declined)

error: failed to push some refs to 'ssh://ssh.github.com/MariaDB/server.git'

The CI displays a green checkmark[1]. I wonder whether this means the branch is locked. serg Shall I try pushing to 10.4 instead?
[1] https://github.com/MariaDB/server/commit/88a589d4efa

Pushed to 10.4, and in time it will be merged to newer versions.

10.5, 10.6 have merge conflicts, and reference commits were pushed to bb-10.

-mdev-27912.