Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30552

InnoDB recovery crashes when error handling scenario

    XMLWordPrintable

Details

    Description

      InnoDB crashes while handling the error case during recovery. Assert assumes that
      failure shouldn't happen while apply data directory changes, undo and change buffer changes.

      Assert is (!after_apply || found_corrupt_fs || !(blocks).end)
      In our case, there is no corruption found (found_corrupt_fs is false),
      blocks.end will exist (because we read the redo logs from log file)
      after_apply is the key here, InnoDB does multiple scan of redo logs and applies it.
      At the time, we do set recv_sys.after_apply as true.

      Add the following debug sync point in

       
      diff --git a/storage/innobase/ibuf/ibuf0ibuf.cc b/storage/innobase/ibuf/ibuf0ibuf.cc
      index be798a1d808..15816075f38 100644
      --- a/storage/innobase/ibuf/ibuf0ibuf.cc
      +++ b/storage/innobase/ibuf/ibuf0ibuf.cc
      @@ -426,6 +426,10 @@ ibuf_init_at_db_start(void)
                      goto err_exit;
              }
       
      +       DBUG_EXECUTE_IF("ibuf_init_corrupt",
      +                       err = DB_CORRUPTION;
      +                       goto err_exit;);
      +
              if (page_is_comp(root) || fil_page_get_type(root) != FIL_PAGE_INDEX
                  || btr_page_get_index_id(root) != DICT_IBUF_ID_MIN) {
                      err = DB_CORRUPTION;
      

      stack trace:

      (rr) where
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007fc4d27be7f1 in __GI_abort () at abort.c:79
      #2  0x00007fc4d27ae3fa in __assert_fail_base (fmt=0x7fc4d29356c0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x5588a1a547a0 "!after_apply || found_corrupt_fs || !(blocks).end", file=file@entry=0x5588a1a52b60 "/home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/log/log0recv.cc", line=line@entry=1400, function=function@entry=0x5588a1a5b660 <recv_sys_t::clear()::__PRETTY_FUNCTION__> "void recv_sys_t::clear()") at assert.c:92
      #3  0x00007fc4d27ae472 in __GI___assert_fail (assertion=0x5588a1a547a0 "!after_apply || found_corrupt_fs || !(blocks).end", file=0x5588a1a52b60 "/home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/log/log0recv.cc", line=1400, function=0x5588a1a5b660 <recv_sys_t::clear()::__PRETTY_FUNCTION__> "void recv_sys_t::clear()") at assert.c:101
      #4  0x00005588a01a2ed2 in recv_sys_t::clear (this=0x5588a30304a0 <recv_sys>) at /home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/log/log0recv.cc:1400
      #5  0x00005588a017f6d9 in recv_sys_t::close (this=0x5588a30304a0 <recv_sys>) at /home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/log/log0recv.cc:1353
      #6  0x00005588a0172b60 in log_t::close (this=0x5588a3f21a00 <log_sys>) at /home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/log/log0log.cc:1290
      #7  0x00005588a044227f in innodb_shutdown () at /home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/srv/srv0start.cc:1989
      #8  0x000055889ff819c4 in innodb_init (p=0x616000022908) at /home/thiru/mariarepo/server/10.11/10.11-sample/storage/innobase/handler/ha_innodb.cc:4209
      #9  0x000055889f5ce004 in ha_initialize_handlerton (plugin=0x6210000c1f70) at /home/thiru/mariarepo/server/10.11/10.11-sample/sql/handler.cc:649
      #10 0x000055889ed689f6 in plugin_initialize (tmp_root=0x7ffd573aed10, plugin=0x6210000c1f70, argc=0x5588a3630a00 <remaining_argc>, argv=0x61c000001968, options_only=false) at /home/thiru/mariarepo/server/10.11/10.11-sample/sql/sql_plugin.cc:1465
      #11 0x000055889ed6a8e9 in plugin_init (argc=0x5588a3630a00 <remaining_argc>, argv=0x61c000001968, flags=0) at /home/thiru/mariarepo/server/10.11/10.11-sample/sql/sql_plugin.cc:1758
      #12 0x000055889e939129 in init_server_components () at /home/thiru/mariarepo/server/10.11/10.11-sample/sql/mysqld.cc:5211
      #13 0x000055889e93b39c in mysqld_main (argc=163, argv=0x61c000001968) at /home/thiru/mariarepo/server/10.11/10.11-sample/sql/mysqld.cc:5829
      #14 0x000055889e923875 in main (argc=30, argv=0x7ffd573b1448) at /home/thiru/mariarepo/server/10.11/10.11-sample/sql/main.cc:34
      
      

      Test case to repeat the scenario:

      --source include/have_innodb.inc
      --source include/have_sequence.inc
      set debug_dbug="+d,ib_log_checkpoint_avoid_hard";
      create table t1(f1 int not null)engine=innodb;
      insert into t1 select * from seq_1_to_65536;
      let $shutdown_timeout=0;
      let $restart_parameters=--debug_dbug="+d,ibuf_init_corrupt";
      --source include/restart_mysqld.inc
      select count(*) from t1;
      drop table t1;
      

      Run the above test case with --innodb_buffer_pool_size=5M

      ib_log_checkpoint_avoid_hard is added to avoid any checkpoint, so that InnoDB can ran out of memory during recovery.

      I think assert is very strict. Fix should be removal of after_apply from assert. More over, InnoDB should throw more messages if we encountered the
      change buffer corruption during upgrade or downgrade.
      Crash is happening only in debug builds

      Attachments

        Activity

          People

            thiru Thirunarayanan Balathandayuthapani
            thiru Thirunarayanan Balathandayuthapani
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.