Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-42047

DROP DATABASE fails half done with ER_DB_DROP_RMDIR when another connection discovers a new table of the database

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.6.28, 10.11.19, 11.4.13, 11.8.9
    • 11.4, 11.8, 12.3
    • None
    • None

    Description

      DROP DATABASE can fail with ER_DB_DROP_RMDIR after dropping the tables, routines, events and db.opt. It lists the tables once, before it waits for their metadata locks and drops them.

      A table of the database can appear in the engine after the listing without a CREATE TABLE on this server. The first open of that table, by a SELECT for example, takes no metadata lock on the schema, so discovery writes the table's .frm into the directory, and the DROP's rmdir fails.

      The directory stays with only the discovered table's .frm. A second DROP DATABASE succeeds. No crash, assertion or corruption was observed.

      Any engine that implements discover_table and whose tables can appear this way is exposed too.

      Reproduction

      The MTR test below reproduces the bug deterministically, without DEBUG_SYNC (11.8 debug build).

      --source include/have_s3.inc
      --source include/have_innodb.inc
       
      # How this test works
      #
      # 1. DROP DATABASE drop_race lists the tables of drop_race (only t0), takes
      #    the exclusive lock on the database, and waits for the metadata lock on
      #    t0, which an open transaction holds.
      # 2. aria_s3_copy puts test.t1 into the bucket as drop_race.t1, as a second
      #    server sharing the bucket would.
      # 3. A SELECT from drop_race.t1 opens the table for the first time:
      #    discovery writes t1.frm into the directory of drop_race. The open takes
      #    no lock on the database, so it does not wait for the DROP.
      # 4. The transaction commits. DROP DATABASE drops t0 and fails on rmdir with
      #    ER_DB_DROP_RMDIR, because t1.frm is in the directory.
       
      create database drop_race comment 'set before the DROP';
      create table drop_race.t0 (a int) engine=innodb;
      create table test.t1 (a int) engine=aria transactional=0 row_format=page;
      flush tables test.t1;
      let MYSQLD_DATADIR= `select @@datadir`;
       
      --echo # Park DROP DATABASE after it has listed the tables of drop_race
      connect (holder,localhost,root,,drop_race);
      begin;
      select count(*) from t0;
       
      connect (dropper,localhost,root,,test);
      --send drop database drop_race
       
      connection default;
      let $wait_condition= select count(*) = 1 from information_schema.processlist
        where state = 'Waiting for table metadata lock' and info = 'drop database drop_race';
      --source include/wait_condition.inc
       
      --echo # Put test.t1 into the bucket as drop_race.t1, outside the server
      --perl
      my $copy = $ENV{MARIA_CHK};
      $copy =~ s/aria_chk/aria_s3_copy/;
      my @cmd = (split(' ', $copy), "--no-defaults", "--op=to_s3", "--database=drop_race",
        "--s3-host-name=$ENV{S3_HOST_NAME}", "--s3-port=$ENV{S3_PORT}",
        "--s3-bucket=$ENV{S3_BUCKET}", "--s3-access-key=$ENV{S3_ACCESS_KEY}",
        "--s3-secret-key=$ENV{S3_SECRET_KEY}", "--s3-region=$ENV{S3_REGION}",
        "--s3-protocol-version=$ENV{S3_PROTOCOL_VERSION}",
        ($ENV{S3_USE_HTTP} eq "ON" ? ("--s3-use-http") : ()),
        "$ENV{MYSQLD_DATADIR}/test/t1");
      my $out = `@cmd 2>&1`;
      die "aria_s3_copy failed ($?): $out" if $? != 0;
      EOF
       
      --echo # First open of drop_race.t1 while DROP DATABASE waits
      connect (opener,localhost,root,,test);
      --send select count(*) from drop_race.t1
       
      --echo # Wait until t1.frm is written; an open that waits for DROP DATABASE never writes it
      connection default;
      --perl
      my $f = "$ENV{MYSQLD_DATADIR}/drop_race/t1.frm";
      for (1 .. 100) { last if -e $f; select(undef, undef, undef, 0.1); }
      EOF
      --list_files $MYSQLD_DATADIR/drop_race
       
      connection holder;
      commit;
       
      connection dropper;
      --error ER_DB_DROP_RMDIR
      --reap
       
      connection opener;
      --reap
       
      --echo # Left: t0 and the database options are gone, the directory stays with t1.frm
      connection default;
      --list_files $MYSQLD_DATADIR/drop_race
      select schema_name, schema_comment from information_schema.schemata where schema_name = 'drop_race';
      show tables from drop_race;
       
      --echo # A second DROP DATABASE lists t1 and succeeds
      drop database drop_race;
      drop table test.t1;
      disconnect holder;
      disconnect dropper;
      disconnect opener;
      

      Results:

      connection dropper;
      ERROR HY000: Error dropping database (can't rmdir './drop_race', errno: 39 "Directory not empty")
      ...
      # Left: t0 and the database options are gone, the directory stays with t1.frm
      connection default;
      t1.frm
      select schema_name, schema_comment from information_schema.schemata where schema_name = 'drop_race';
      schema_name	schema_comment
      drop_race	
      show tables from drop_race;
      Tables_in_drop_race
      t1
      

      Why the open does not wait for DROP DATABASE

      Statements take MDL_INTENTION_EXCLUSIVE on the database (i.e. schema MDL) only in lock_table_names, which skips table requests weaker than MDL_SHARED_UPGRADABLE.

      Why two parallel discoveries don't race: tdc protects against it. Only one winner holds a tdc element and the others wait.

      Order of events in the test. The right lane writes t1.frm while the left lane waits.

      DROP DATABASE drop_race                   select count(*) from drop_race.t1
      mysql_rm_db_internal, sql/sql_db.cc       open_table
       
      lock_schema_name                  
        MDL_EXCLUSIVE on schema drop_race
      find_db_tables_and_rm_known_files
        list: t0
      lock_table_names                 
        waits for MDL on t0
        |                                       [aria_s3_copy puts t1 into the bucket]
        |                                       open_table_get_mdl_lock
        |                                         MDL on t1 only, none on the schema
        |                                       tdc_acquire_share
        |                                         ha_discover_table
        |                                         s3_discover_table
        |                                           writes drop_race/t1.frm 
      [holder commits: MDL on t0 is free]
      rm_dir_w_symlink                  :1214
        ER_DB_DROP_RMDIR, errno 39      :1511
      

      Attachments

        Activity

          People

            nikitamalyavin Nikita Malyavin
            nikitamalyavin Nikita Malyavin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.