Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18286

Assertion `pagecache->cnt_for_resize_op == 0' failed in check_pagecache_is_cleaned_up on server shutdown

    XMLWordPrintable

    Details

      Description

      Note: The description was updated much later than the comments were added. The comments discuss the initial report, which only had the stack trace, without a test case or any other indication how the problem could be reproduced.

      10.4 88cf6f1c7f with the injection below

      2020-04-25  4:55:50 0 [Note] /data/src/10.4-bug/sql/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
      2020-04-25  4:55:50 0 [Note] Event Scheduler: Purging the queue. 0 events
      mysqld: /data/src/10.4-bug/storage/maria/ma_pagecache.c:1133: check_pagecache_is_cleaned_up: Assertion `pagecache->cnt_for_resize_op == 0' failed.
      200425  4:55:50 [ERROR] mysqld got signal 6 ;
       
      #6  0x00007f355a45840f in __assert_fail_base (fmt=0x7f355a5baee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55a54b2bb2a8 "pagecache->cnt_for_resize_op == 0", file=0x55a54b2baf78 "/data/src/10.4-bug/storage/maria/ma_pagecache.c", line=1133, function=<optimized out>) at assert.c:92
      #7  0x00007f355a466102 in __GI___assert_fail (assertion=0x55a54b2bb2a8 "pagecache->cnt_for_resize_op == 0", file=0x55a54b2baf78 "/data/src/10.4-bug/storage/maria/ma_pagecache.c", line=1133, function=0x55a54b2bd2e0 <__PRETTY_FUNCTION__.18326> "check_pagecache_is_cleaned_up") at assert.c:101
      #8  0x000055a54ac6a97f in check_pagecache_is_cleaned_up (pagecache=0x55a54c287d00 <maria_pagecache_var>) at /data/src/10.4-bug/storage/maria/ma_pagecache.c:1133
      #9  0x000055a54ac6aba0 in end_pagecache (pagecache=0x55a54c287d00 <maria_pagecache_var>, cleanup=1 '\001') at /data/src/10.4-bug/storage/maria/ma_pagecache.c:1175
      #10 0x000055a54ac862ba in maria_end () at /data/src/10.4-bug/storage/maria/ma_init.c:110
      #11 0x000055a54acd2157 in maria_panic (flag=HA_PANIC_CLOSE) at /data/src/10.4-bug/storage/maria/ma_panic.c:135
      #12 0x000055a54ac478c6 in maria_hton_panic (hton=0x55a54dd19590, flag=HA_PANIC_CLOSE) at /data/src/10.4-bug/storage/maria/ha_maria.cc:3347
      #13 0x000055a54a5a8fe9 in ha_finalize_handlerton (plugin=0x55a54dd0ba88) at /data/src/10.4-bug/sql/handler.cc:496
      #14 0x000055a54a271157 in plugin_deinitialize (plugin=0x55a54dd0ba88, ref_check=true) at /data/src/10.4-bug/sql/sql_plugin.cc:1241
      #15 0x000055a54a2716c1 in reap_plugins () at /data/src/10.4-bug/sql/sql_plugin.cc:1317
      #16 0x000055a54a273ae0 in plugin_shutdown () at /data/src/10.4-bug/sql/sql_plugin.cc:1986
      #17 0x000055a54a115e20 in clean_up (print_message=true) at /data/src/10.4-bug/sql/mysqld.cc:1988
      #18 0x000055a54a11e4ed in mysqld_main (argc=130, argv=0x55a54dbd98d0) at /data/src/10.4-bug/sql/mysqld.cc:5922
      #19 0x000055a54a112275 in main (argc=6, argv=0x7fffa3e800b8) at /data/src/10.4-bug/sql/main.cc:25
      

      Debug injection for 10.4, e.g. 88cf6f1c7f

      diff --git a/storage/maria/ma_pagecache.c b/storage/maria/ma_pagecache.c
      index f486d8c704f..9f02355b44f 100644
      --- a/storage/maria/ma_pagecache.c
      +++ b/storage/maria/ma_pagecache.c
      @@ -4416,6 +4416,7 @@ static int flush_cached_blocks(PAGECACHE *pagecache,
             @todo change argument of functions to be File.
           */
           pagecache_pthread_mutex_unlock(&pagecache->cache_lock);
      +    DBUG_EXECUTE_IF("sleep_mutex",{while(pagecache->cache_lock.m_mutex.thread == 0) my_sleep(100000);};);
           error= pagecache_fwrite(pagecache, &block->hash_link->file,
                                   block->buffer,
                                   block->hash_link->pageno,
      
      

      Even with the injection above, the test case is still non-deterministic, run with --repeat=N if it doesn't fail right away. Currently it fails for me roughly every other time.

      Test case to be run with the injection above

      --source include/restart_mysqld.inc
       
      CREATE TABLE t1 (pk INT, a VARCHAR(8), PRIMARY KEY(pk)) ENGINE=Aria TRANSACTIONAL=1;
       
      --connect (con1,localhost,root,,test)
      BACKUP STAGE START;
      SET debug_dbug= '+d,sleep_mutex';
      --send
        BACKUP STAGE BLOCK_COMMIT;
       
      --connection default
      INSERT INTO t1 VALUES (1,'a');
      INSERT INTO t1 VALUES (2,'b');
      INSERT INTO t1 VALUES (3,'c');
      INSERT INTO t1 VALUES (4,'c');
      INSERT INTO t1 VALUES (5,'c');
      INSERT INTO t1 VALUES (6,'c');
      INSERT INTO t1 VALUES (7,'c');
      INSERT INTO t1 VALUES (8,'c');
      INSERT INTO t1 VALUES (9,'c');
       
      --connection con1
      --reap
       
      --sleep 1
      --source include/restart_mysqld.inc
       
      DROP TABLE t1;
      

        Attachments

          Activity

            People

            Assignee:
            monty Michael Widenius
            Reporter:
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: