mysql system table is marked as crashed and should be repaired after the server crashes or is killed

The problem can be reproduced by creating a table that uses the Spider storage engine. The problem only involves the Spider node, so only that single node is needed to reproduce the problem.

Start with an empty environment and set up the server as follows:

Run mysql_install_db.

Start the server.

Install Spider as follows:

mysql -u root -P 3306 < <path to install_spider.sql>

Execute the following SQL statements:

CREATE OR REPLACE SERVER SrvDataNode1

  FOREIGN DATA WRAPPER mysql 

OPTIONS( 

  HOST '127.0.0.1', PORT 8607,

  DATABASE 'test',

  USER 'root'

);

USE test;

CREATE OR REPLACE TABLE t1(id INT, f FLOAT, PRIMARY KEY(id))

ENGINE=Spider

COMMENT='wrapper "mysql", srv "SrvDataNode1", table "t1"';

Kill the server and then restart it.

In the event log, you will notice error messages like the following during the restart:

 2018-08-24 11:24:08 0 [ERROR] mysqld: Table './mysql/servers' is marked as crashed and should be repaired

 2018-08-24 11:24:08 0 [Warning] Checking table:   './mysql/servers'

 2018-08-24 11:24:08 0 [ERROR] mysql.servers: 1 client is using or hasn't closed the table properly

There are no messages that indicate that these system tables are automatically repaired. If you then kill and restart the server a second time, then no such error messages appear in the event log during the second restart. That might indicate the the system tables were in fact automatically repaired, but there are no messages regarding their automatic repair.

I have also seen this problem reported on Spider system tables. However, with the current 10.3 code, the problem is reported most often on the mysql.servers and/or the mysql.func system tables.