Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.6.28, 10.11.19, 11.4.13, 11.8.9
-
None
-
None
Description
DROP DATABASE can fail with ER_DB_DROP_RMDIR after dropping the tables, routines, events and db.opt. It lists the tables once, before it waits for their metadata locks and drops them.
A table of the database can appear in the engine after the listing without a CREATE TABLE on this server. The first open of that table, by a SELECT for example, takes no metadata lock on the schema, so discovery writes the table's .frm into the directory, and the DROP's rmdir fails.
The directory stays with only the discovered table's .frm. A second DROP DATABASE succeeds. No crash, assertion or corruption was observed.
Any engine that implements discover_table and whose tables can appear this way is exposed too.
Reproduction
The MTR test below reproduces the bug deterministically, without DEBUG_SYNC (11.8 debug build).
--source include/have_s3.inc
|
--source include/have_innodb.inc
|
|
|
# How this test works
|
#
|
# 1. DROP DATABASE drop_race lists the tables of drop_race (only t0), takes |
# the exclusive lock on the database, and waits for the metadata lock on |
# t0, which an open transaction holds. |
# 2. aria_s3_copy puts test.t1 into the bucket as drop_race.t1, as a second |
# server sharing the bucket would.
|
# 3. A SELECT from drop_race.t1 opens the table for the first time: |
# discovery writes t1.frm into the directory of drop_race. The open takes |
# no lock on the database, so it does not wait for the DROP. |
# 4. The transaction commits. DROP DATABASE drops t0 and fails on rmdir with |
# ER_DB_DROP_RMDIR, because t1.frm is in the directory. |
|
|
create database drop_race comment 'set before the DROP'; |
create table drop_race.t0 (a int) engine=innodb; |
create table test.t1 (a int) engine=aria transactional=0 row_format=page; |
flush tables test.t1;
|
let MYSQLD_DATADIR= `select @@datadir`; |
|
|
--echo # Park DROP DATABASE after it has listed the tables of drop_race
|
connect (holder,localhost,root,,drop_race); |
begin; |
select count(*) from t0; |
|
|
connect (dropper,localhost,root,,test); |
--send drop database drop_race
|
|
|
connection default; |
let $wait_condition= select count(*) = 1 from information_schema.processlist |
where state = 'Waiting for table metadata lock' and info = 'drop database drop_race'; |
--source include/wait_condition.inc
|
|
|
--echo # Put test.t1 into the bucket as drop_race.t1, outside the server
|
--perl
|
my $copy = $ENV{MARIA_CHK};
|
$copy =~ s/aria_chk/aria_s3_copy/;
|
my @cmd = (split(' ', $copy), "--no-defaults", "--op=to_s3", "--database=drop_race", |
"--s3-host-name=$ENV{S3_HOST_NAME}", "--s3-port=$ENV{S3_PORT}", |
"--s3-bucket=$ENV{S3_BUCKET}", "--s3-access-key=$ENV{S3_ACCESS_KEY}", |
"--s3-secret-key=$ENV{S3_SECRET_KEY}", "--s3-region=$ENV{S3_REGION}", |
"--s3-protocol-version=$ENV{S3_PROTOCOL_VERSION}", |
($ENV{S3_USE_HTTP} eq "ON" ? ("--s3-use-http") : ()), |
"$ENV{MYSQLD_DATADIR}/test/t1"); |
my $out = `@cmd 2>&1`; |
die "aria_s3_copy failed ($?): $out" if $? != 0; |
EOF
|
|
|
--echo # First open of drop_race.t1 while DROP DATABASE waits
|
connect (opener,localhost,root,,test); |
--send select count(*) from drop_race.t1
|
|
|
--echo # Wait until t1.frm is written; an open that waits for DROP DATABASE never writes it
|
connection default; |
--perl
|
my $f = "$ENV{MYSQLD_DATADIR}/drop_race/t1.frm"; |
for (1 .. 100) { last if -e $f; select(undef, undef, undef, 0.1); } |
EOF
|
--list_files $MYSQLD_DATADIR/drop_race
|
|
|
connection holder; |
commit; |
|
|
connection dropper; |
--error ER_DB_DROP_RMDIR
|
--reap
|
|
|
connection opener; |
--reap
|
|
|
--echo # Left: t0 and the database options are gone, the directory stays with t1.frm
|
connection default; |
--list_files $MYSQLD_DATADIR/drop_race
|
select schema_name, schema_comment from information_schema.schemata where schema_name = 'drop_race'; |
show tables from drop_race; |
|
|
--echo # A second DROP DATABASE lists t1 and succeeds
|
drop database drop_race; |
drop table test.t1; |
disconnect holder;
|
disconnect dropper;
|
disconnect opener;
|
Results:
connection dropper;
|
ERROR HY000: Error dropping database (can't rmdir './drop_race', errno: 39 "Directory not empty")
|
...
|
# Left: t0 and the database options are gone, the directory stays with t1.frm
|
connection default;
|
t1.frm
|
select schema_name, schema_comment from information_schema.schemata where schema_name = 'drop_race';
|
schema_name schema_comment
|
drop_race
|
show tables from drop_race;
|
Tables_in_drop_race
|
t1
|
Why the open does not wait for DROP DATABASE
Statements take MDL_INTENTION_EXCLUSIVE on the database (i.e. schema MDL) only in lock_table_names, which skips table requests weaker than MDL_SHARED_UPGRADABLE.
Why two parallel discoveries don't race: tdc protects against it. Only one winner holds a tdc element and the others wait.
Order of events in the test. The right lane writes t1.frm while the left lane waits.
DROP DATABASE drop_race select count(*) from drop_race.t1
|
mysql_rm_db_internal, sql/sql_db.cc open_table
|
|
|
lock_schema_name
|
MDL_EXCLUSIVE on schema drop_race
|
find_db_tables_and_rm_known_files
|
list: t0
|
lock_table_names
|
waits for MDL on t0
|
| [aria_s3_copy puts t1 into the bucket]
|
| open_table_get_mdl_lock
|
| MDL on t1 only, none on the schema
|
| tdc_acquire_share
|
| ha_discover_table
|
| s3_discover_table
|
| writes drop_race/t1.frm
|
[holder commits: MDL on t0 is free]
|
rm_dir_w_symlink :1214
|
ER_DB_DROP_RMDIR, errno 39 :1511
|