Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34455

innodb_read_only=ON fails to imply innodb_doublewrite=OFF

Details

    Description

      Server crashes when the server is started with innodb-force-recovery=6 and the innodb_buf_flush_list_now and innodb_doublewrite variables are enabled.

      Please make innodb_doublewrite a read-only variable in 11.4+ builds. Now it is dynamic variable in 11.4+ builds.

      -- source include/have_innodb.inc
       
      --let $restart_parameters= --innodb-force-recovery=6
      --source include/restart_mysqld.inc
       
      SET GLOBAL innodb_doublewrite=1;
      SET GLOBAL innodb_buf_flush_list_now=1;
      

      Leads to

      11.6.0 42294b8cd2cbb72c1d5da6058dd6f0c55669def7 (Debug)

      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x0000153490f70859 in __GI_abort () at abort.c:79
      #2  0x0000153490f70729 in __assert_fail_base (fmt=0x153491106588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55ae63627c44 "!srv_read_only_mode", file=0x55ae63691e10 "/test/mtest/MDEV-33856/11.6_dbg/storage/innobase/buf/buf0dblwr.cc", line=732, function=<optimized out>) at assert.c:92
      #3  0x0000153490f81fd6 in __GI___assert_fail (assertion=assertion@entry=0x55ae63627c44 "!srv_read_only_mode", file=file@entry=0x55ae63691e10 "/test/mtest/MDEV-33856/11.6_dbg/storage/innobase/buf/buf0dblwr.cc", line=line@entry=732, function=function@entry=0x55ae63692150 "void buf_dblwr_t::flush_buffered_writes()") at assert.c:101
      #4  0x000055ae63008e65 in buf_dblwr_t::flush_buffered_writes (this=0x55ae64723e20 <buf_dblwr>) at /test/mtest/MDEV-33856/11.6_dbg/storage/innobase/buf/buf0dblwr.cc:732
      #5  0x000055ae630174d9 in buf_flush_list_space (space=space@entry=0x55ae663b7ba8, n_flushed=n_flushed@entry=0x0) at /test/mtest/MDEV-33856/11.6_dbg/storage/innobase/buf/buf0flu.cc:1689
      #6  0x000055ae62d36154 in buf_flush_list_now_set (save=<optimized out>) at /test/mtest/MDEV-33856/11.6_dbg/storage/innobase/handler/ha_innodb.cc:18227
      #7  0x000055ae62695d87 in sys_var_pluginvar::global_update (this=0x55ae662a72a0, thd=0x153478000d48, var=0x153478018e50) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_plugin.cc:3687
      #8  0x000055ae62576024 in sys_var::update (this=0x55ae662a72a0, thd=0x153478000d48, var=0x153478018e50) at /test/mtest/MDEV-33856/11.6_dbg/sql/set_var.cc:210
      #9  0x000055ae62576567 in set_var::update (this=<optimized out>, thd=<optimized out>) at /test/mtest/MDEV-33856/11.6_dbg/sql/set_var.cc:868
      #10 0x000055ae62577922 in sql_set_variables (thd=thd@entry=0x153478000d48, var_list=var_list@entry=0x1534780061d8, free=free@entry=true) at /test/mtest/MDEV-33856/11.6_dbg/sql/set_var.cc:750
      #11 0x000055ae6267371a in mysql_execute_command (thd=thd@entry=0x153478000d48, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_parse.cc:4841
      #12 0x000055ae6266210c in mysql_parse (thd=thd@entry=0x153478000d48, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x1534897ef270) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_parse.cc:7868
      #13 0x000055ae6267916e in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x153478000d48, packet=packet@entry=0x15347800b2d9 "SET GLOBAL innodb_buf_flush_list_now=1", packet_length=packet_length@entry=38, blocking=blocking@entry=true) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_class.h:1638
      #14 0x000055ae6267bc71 in do_command (thd=0x153478000d48, blocking=blocking@entry=true) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_parse.cc:1405
      #15 0x000055ae6280492d in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55ae667c3f08, put_in_cache=put_in_cache@entry=true) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_connect.cc:1448
      #16 0x000055ae62804eee in handle_one_connection (arg=arg@entry=0x55ae667c3f08) at /test/mtest/MDEV-33856/11.6_dbg/sql/sql_connect.cc:1350
      #17 0x000055ae62c8d286 in pfs_spawn_thread (arg=0x55ae66743de8) at /test/mtest/MDEV-33856/11.6_dbg/storage/perfschema/pfs.cc:2201
      #18 0x0000153491481609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #19 0x000015349106d133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      Attachments

        Issue Links

          Activity

            The parameter innodb_doublewrite was made settable at runtime in MDEV-33545. Why would you want to revert that?

            The crash occurs due to a missing condition related to updating the debug-only variable innodb_buf_flush_list_now. That we could fix easily:

            diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc
            index cd460b20c1a..a8acf2e4f2f 100644
            --- a/storage/innobase/handler/ha_innodb.cc
            +++ b/storage/innobase/handler/ha_innodb.cc
            @@ -18408,7 +18408,7 @@ static
             void
             buf_flush_list_now_set(THD*, st_mysql_sys_var*, void*, const void* save)
             {
            -  if (!*(my_bool*) save)
            +  if (!*(my_bool*) save || high_level_read_only)
                 return;
               const uint s= srv_fil_make_page_dirty_debug;
               mysql_mutex_unlock(&LOCK_global_system_variables);
            

            The above patch is for 10.5 (the earliest supported version). But, 10.5 does not crash because of the following:

            void buf_dblwr_t::flush_buffered_writes()
            {
              if (!is_initialised() || !srv_use_doublewrite_buf)
              {
                fil_flush_file_spaces();
                return;
              }
            

            and the following in srv_start() (innodb_force_recovery=6 will also imply innodb_read_only=ON):

            	if (srv_force_recovery) {
            		ib::info() << "!!! innodb_force_recovery is set to "
            			<< srv_force_recovery << " !!!";
            		if (srv_force_recovery == SRV_FORCE_NO_LOG_REDO) {
            			srv_read_only_mode = true;
            		}
            	}
             
            	if (srv_read_only_mode) {
            		sql_print_information("InnoDB: Started in read only mode");
            		srv_use_doublewrite_buf = false;
            	}
            

            In MDEV-33545, the last part was changed to the following:

            diff --git a/storage/innobase/srv/srv0start.cc b/storage/innobase/srv/srv0start.cc
            index 3c90186374c..875dbbe8d57 100644
            --- a/storage/innobase/srv/srv0start.cc
            +++ b/storage/innobase/srv/srv0start.cc
            @@ -1193,7 +1193,7 @@ dberr_t srv_start(bool create_new_db)
             
             	if (srv_read_only_mode) {
             		sql_print_information("InnoDB: Started in read only mode");
            -		srv_use_doublewrite_buf = false;
            +		buf_dblwr.use = buf_dblwr.USE_NO;
             	}
             
             	high_level_read_only = srv_read_only_mode
            

            I will apply the above patch fix 10.5, even though I can confirm that this regression was introduced only in 11.0.6. The setting innodb_flush_list_now only exists in debug instrumented builds for the purpose of testing that the doublewrite buffer can heal some corruption.

            marko Marko Mäkelä added a comment - The parameter innodb_doublewrite was made settable at runtime in MDEV-33545 . Why would you want to revert that? The crash occurs due to a missing condition related to updating the debug-only variable innodb_buf_flush_list_now . That we could fix easily: diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index cd460b20c1a..a8acf2e4f2f 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -18408,7 +18408,7 @@ static void buf_flush_list_now_set(THD*, st_mysql_sys_var*, void*, const void* save) { - if (!*(my_bool*) save) + if (!*(my_bool*) save || high_level_read_only) return; const uint s= srv_fil_make_page_dirty_debug; mysql_mutex_unlock(&LOCK_global_system_variables); The above patch is for 10.5 (the earliest supported version). But, 10.5 does not crash because of the following: void buf_dblwr_t::flush_buffered_writes() { if (!is_initialised() || !srv_use_doublewrite_buf) { fil_flush_file_spaces(); return ; } and the following in srv_start() ( innodb_force_recovery=6 will also imply innodb_read_only=ON ): if (srv_force_recovery) { ib::info() << "!!! innodb_force_recovery is set to " << srv_force_recovery << " !!!" ; if (srv_force_recovery == SRV_FORCE_NO_LOG_REDO) { srv_read_only_mode = true ; } }   if (srv_read_only_mode) { sql_print_information( "InnoDB: Started in read only mode" ); srv_use_doublewrite_buf = false ; } In MDEV-33545 , the last part was changed to the following: diff --git a/storage/innobase/srv/srv0start.cc b/storage/innobase/srv/srv0start.cc index 3c90186374c..875dbbe8d57 100644 --- a/storage/innobase/srv/srv0start.cc +++ b/storage/innobase/srv/srv0start.cc @@ -1193,7 +1193,7 @@ dberr_t srv_start( bool create_new_db) if (srv_read_only_mode) { sql_print_information( "InnoDB: Started in read only mode" ); - srv_use_doublewrite_buf = false ; + buf_dblwr.use = buf_dblwr.USE_NO; } high_level_read_only = srv_read_only_mode I will apply the above patch fix 10.5, even though I can confirm that this regression was introduced only in 11.0.6. The setting innodb_flush_list_now only exists in debug instrumented builds for the purpose of testing that the doublewrite buffer can heal some corruption.

            I realized that innodb_buf_flush_list_now is not the actual culprit, but the SET GLOBAL innodb_doublewrite is. The following fixes this:

            diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc
            index f83120f4076..2723a3890fb 100644
            --- a/storage/innobase/handler/ha_innodb.cc
            +++ b/storage/innobase/handler/ha_innodb.cc
            @@ -18400,7 +18400,8 @@ static void innodb_data_file_write_through_update(THD *, st_mysql_sys_var*,
             static void innodb_doublewrite_update(THD *, st_mysql_sys_var*,
                                                   void *, const void *save)
             {
            -  fil_system.set_use_doublewrite(*static_cast<const ulong*>(save));
            +  if (!srv_read_only_mode)
            +    fil_system.set_use_doublewrite(*static_cast<const ulong*>(save));
             }
             
             static void innodb_log_file_size_update(THD *thd, st_mysql_sys_var*,
            

            Here is a little better test case (not using the dangerous setting innodb_force_recovery):

            --source include/have_innodb.inc
            --source include/have_debug.inc
            --let $restart_parameters=--innodb-read-only
            --source include/restart_mysqld.inc
            SET GLOBAL innodb_doublewrite=1,innodb_buf_flush_list_now=1;
            

            marko Marko Mäkelä added a comment - I realized that innodb_buf_flush_list_now is not the actual culprit, but the SET GLOBAL innodb_doublewrite is. The following fixes this: diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index f83120f4076..2723a3890fb 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -18400,7 +18400,8 @@ static void innodb_data_file_write_through_update(THD *, st_mysql_sys_var*, static void innodb_doublewrite_update(THD *, st_mysql_sys_var*, void *, const void *save) { - fil_system.set_use_doublewrite(*static_cast<const ulong*>(save)); + if (!srv_read_only_mode) + fil_system.set_use_doublewrite(*static_cast<const ulong*>(save)); } static void innodb_log_file_size_update(THD *thd, st_mysql_sys_var*, Here is a little better test case (not using the dangerous setting innodb_force_recovery ): --source include/have_innodb.inc --source include/have_debug.inc --let $restart_parameters=--innodb-read-only --source include/restart_mysqld.inc SET GLOBAL innodb_doublewrite=1,innodb_buf_flush_list_now=1;
            ramesh Ramesh Sivaraman added a comment - - edited

            I did not notice MDEV-33545. innodb_doublewrite was read-only in my local builds (on older versions), and I haven't seen any changes in the documentation. So I thought it was a bug. Thank you for the better MTR test case
            https://mariadb.com/kb/en/innodb-system-variables/#innodb_doublewrite

            ramesh Ramesh Sivaraman added a comment - - edited I did not notice MDEV-33545 . innodb_doublewrite was read-only in my local builds (on older versions), and I haven't seen any changes in the documentation. So I thought it was a bug. Thank you for the better MTR test case https://mariadb.com/kb/en/innodb-system-variables/#innodb_doublewrite

            Thanks for the patch. Looks good to me.

            debarun Debarun Banerjee added a comment - Thanks for the patch. Looks good to me.

            People

              marko Marko Mäkelä
              ramesh Ramesh Sivaraman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.