Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-37263

Hang or crash when shrinking innodb_buffer_pool_size

    XMLWordPrintable

Details

    • Can result in hang or crash
    • When innodb_buffer_pool_size is being shrunk while there are no data pages cached, InnoDB could hang or crash.

    Description

      CLI test case (sporadic):

      SET tx_isolation='SERIALIZABLE';
      CREATE TABLE t (c1 VARCHAR(1) BINARY CHARACTER SET 'latin1' COLLATE 'latin1_bin',c2 YEAR,c3 DATETIME(1)) PARTITION BY LINEAR HASH ((c2)) PARTITIONS 523;
      XA START 'a';
      SELECT * FROM t;
      SET GLOBAL innodb_buffer_pool_size=+1;
      XA END 'a';
      XA ROLLBACK 'a';
      SET GLOBAL innodb_buffer_pool_size=+1;
      SET tx_isolation='SERIALIZABLE';
      

      Leads to:

      CS 11.4.8 dbeef00562dd7088a9f0163268f9eb5e5ba072fe (Debug, Clang) Build 07/07/2025

      Core was generated by `/test/MD070725-mariadb-11.4.8-linux-x86_64-dbg/bin/mariadbd --no-defaults --max'.
      Program terminated with signal SIGABRT, Aborted.
      Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
      #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
       
      [Current thread is 1 (LWP 1898664)]
      (gdb) bt
      #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
      #1  __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78
      #2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89
      #3  0x000078e9a3a4527e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26
      #4  0x000078e9a3a288ff in __GI_abort () at ./stdlib/abort.c:79
      #5  0x000059642dfc86b1 in buf_LRU_check_size_of_non_data_objects ()at /test/11.4_dbg/storage/innobase/buf/buf0lru.cc:293
      #6  0x000059642dfc7f80 in buf_LRU_get_free_block (get=have_mutex)at /test/11.4_dbg/storage/innobase/buf/buf0lru.cc:354
      #7  0x000059642dfd04ac in buf_read_page (page_id={m_id = 8589934638}, chain=@0x59645c9f2810: {first = 0x0}, unzip=true)at /test/11.4_dbg/storage/innobase/buf/buf0rea.cc:426
      #8  0x000059642df9cf34 in buf_pool_t::page_fix (this=0x59642eddb700 <buf_pool>, id={m_id = 8589934638}, err=0x0, c=buf_pool_t::FIX_WAIT_READ)at /test/11.4_dbg/storage/innobase/buf/buf0buf.cc:2570
      #9  0x000059642deec6eb in buf_pool_t::page_fix (this=0x59642eddb700 <buf_pool>, id={m_id = 8589934638})at include/buf0buf.h:1337
      #10 0x000059642dee6d73 in purge_sys_t::get_page (this=0x59642f83b500 <purge_sys>, id={m_id = 8589934638})at /test/11.4_dbg/storage/innobase/trx/trx0purge.cc:791
      #11 0x000059642dee74e2 in purge_sys_t::choose_next_log (this=0x59642f83b500 <purge_sys>)at /test/11.4_dbg/storage/innobase/trx/trx0purge.cc:907
      #12 0x000059642deef36a in purge_sys_t::fetch_next_rec (this=0x59642f83b500 <purge_sys>)at /test/11.4_dbg/storage/innobase/trx/trx0purge.cc:1024
      #13 0x000059642dee87f8 in trx_purge_attach_undo_recs (thd=0x59645c800368, n_work_items=0x78e61bffe900)at /test/11.4_dbg/storage/innobase/trx/trx0purge.cc:1250
      #14 0x000059642dee7dd0 in trx_purge (n_tasks=4, history_size=78)at /test/11.4_dbg/storage/innobase/trx/trx0purge.cc:1388
      #15 0x000059642ded0106 in purge_coordinator_state::do_purge (this=0x59642f83a6c0 <purge_state>)at /test/11.4_dbg/storage/innobase/srv/srv0srv.cc:1415
      #16 0x000059642deccc45 in purge_coordinator_callback ()at /test/11.4_dbg/storage/innobase/srv/srv0srv.cc:1499
      #17 0x000059642e0e6dda in tpool::task_group::execute (this=0x59642f83a480 <purge_coordinator_task_group>, t=0x59642f83a518 <purge_coordinator_task>)at /test/11.4_dbg/tpool/task_group.cc:70
      #18 0x000059642e0e731c in tpool::task::execute (this=0x59642f83a518 <purge_coordinator_task>)at /test/11.4_dbg/tpool/task.cc:32
      #19 0x000059642e0dff63 in tpool::thread_pool_generic::worker_main (this=0x59645c8bebd0, thread_var=0x59645c8bfe80)at /test/11.4_dbg/tpool/tpool_generic.cc:573
      #20 0x000059642e0e4764 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>(__f=@0x78e94c002ed8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x59642e0dfe40 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x78e94c002ed0: 0x59645c8bebd0, __args=@0x78e94c002ec8: 0x59645c8bfe80)at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74
      #21 0x000059642e0e4685 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x78e94c002ed8: (void (tpool::thread_pool_generic::*)(class tpool::thread_pool_generic * const, struct tpool::worker_data *)) 0x59642e0dfe40 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __args=@0x78e94c002ec8: 0x59645c8bfe80, __args=@0x78e94c002ec8: 0x59645c8bfe80)at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96
      #22 0x000059642e0e4653 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x78e94c002ec8)at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292
      #23 0x000059642e0e4605 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x78e94c002ec8)at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299
      #24 0x000059642e0e4479 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x78e94c002ec0)at /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244
      #25 0x000078e9a3eecdb4 in std::execute_native_thread_routine (__p=0x78e94c002ec0)at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
      #26 0x000078e9a3a9caa4 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447
      #27 0x000078e9a3b29c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
      

      Bug Detection Matrix

          Rel    o/d  Build   Commit                                    UniqueID observed
      CS  10.6   dbg  070725  31aa8b6939ee9326b4145a9cceae7e5a3711d7bf  No bug found
      CS  10.6   opt  070725  31aa8b6939ee9326b4145a9cceae7e5a3711d7bf  No bug found
      CS  10.11  dbg  070725  e79aa9ca382ce6d293edeab880212bf472c33f36  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_page_init_for_read|buf_read_page_low
      CS  10.11  opt  070725  e79aa9ca382ce6d293edeab880212bf472c33f36  MARIADBD_ERROR|mariadbd: innodb_buffer_pool_size change aborted
      CS  11.4   dbg  070725  dbeef00562dd7088a9f0163268f9eb5e5ba072fe  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      CS  11.4   opt  070725  dbeef00562dd7088a9f0163268f9eb5e5ba072fe  MARIADBD_ERROR|mariadbd: innodb_buffer_pool_size change aborted
      CS  11.8   dbg  070725  9a4a30aec003a8efdf9d35d3ac669385286ba4b4  MARIADBD_ERROR|mariadbd: innodb_buffer_pool_size change aborted
      CS  11.8   opt  070725  9a4a30aec003a8efdf9d35d3ac669385286ba4b4  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      CS  12.0   dbg  070725  107291bf980822fcc3c02bd4e01ecbc4db7fd192  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      CS  12.0   opt  070725  107291bf980822fcc3c02bd4e01ecbc4db7fd192  MARIADBD_ERROR|mariadbd: innodb_buffer_pool_size change aborted
      CS  12.1   dbg  070725  d4d0dd00b761b3e94454dff9aa93e2e17232339e  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      CS  12.1   opt  070725  d4d0dd00b761b3e94454dff9aa93e2e17232339e  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      ES  10.5   dbg  070725  70586522eacf09d04d49962072e14325a75d8155  No bug found
      ES  10.5   opt  070725  70586522eacf09d04d49962072e14325a75d8155  No bug found
      ES  10.6   dbg  070725  6879c0c0bcdd1b7a88de65dd581cf86195a2cd11  No bug found
      ES  10.6   opt  070725  6879c0c0bcdd1b7a88de65dd581cf86195a2cd11  No bug found
      ES  11.4   dbg  070725  6ac3c1547652a4391fe0b1210ec241a804293567  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      ES  11.4   opt  070725  6ac3c1547652a4391fe0b1210ec241a804293567  MARIADBD_ERROR|mariadbd: innodb_buffer_pool_size change aborted
      ES  11.8   dbg  070725  e09b21aaddefcf0d7ddd96eca5b1d48c1d688de5  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_page_get_gen
      ES  11.8   opt  070725  e09b21aaddefcf0d7ddd96eca5b1d48c1d688de5  SIGABRT|buf_LRU_check_size_of_non_data_objects|buf_LRU_get_free_block|buf_read_page|buf_pool_t::page_fix
      

      Note that the "mariadbd: innodb_buffer_pool_size change aborted" error is generated when the crash did not sporadically happen.
      Also note there is at least 3 different stacks present (buf_read_page_low/buf_pool_t::page_fix/buf_page_get_gen)

      RR trace for the above stack trace is present on the RR machine:
      /data/saahil_rr_logs/MDEV-37263

      The CLI testcase above is 1-in-2 sporadic. If it does not crash, try adding;

      drop table t ;
      

      at the end of the sql.

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              saahil Saahil Alam
              Marko Mäkelä Marko Mäkelä
              Debarun Banerjee Debarun Banerjee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.