Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6083

Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)' fails in intern_sys_var_ptr on server shutdown after uninstalling TokuDB plugin at runtime

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 5.5.36
    • 5.5.38
    • None
    • None
    • serg-test-sprint-1

    Description

      On 10.0, the assertion failure doesn't happen, but there is some uncleanness on the shutdown still (e.g. seen with valgrind), so I suppose it might be affected also, thus filing both for 5.5 and 10.0.

      # It's not necessary to install TokuDB at runtime, 
      # instead the server can be run with --plugin-load=ha_tokudb
       
      INSTALL SONAME 'ha_tokudb';
      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a INT) ENGINE=TokuDB;
      INSERT INTO t1 VALUES (1),(2);
      UNINSTALL SONAME 'ha_tokudb';
       
      # Let server to shutdown (with MTR, it happens on its own)

      5.5/sql/sql_plugin.cc:2903: uchar* intern_sys_var_ptr(THD*, int, bool): Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pthread_self(), (&(&LOCK_open)->m_mutex)->thread)' failed.
      140412 20:09:03 [ERROR] mysqld got signal 6 ;

      #6  0x00007f6db3ad1621 in *__GI___assert_fail (assertion=0xd8d6c0 "! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pthread_self(), (&(&LOCK_open)->m_mutex)->thread)", file=<optimized out>, line=2903, function=0xd8e500 "uchar* intern_sys_var_ptr(THD*, int, bool)") at assert.c:81
      #7  0x00000000006476df in intern_sys_var_ptr (thd=0x0, offset=136, global_lock=false) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:2903
      #8  0x00000000006482ce in sys_var_pluginvar::real_value_ptr (this=0x7f6db23fe248, thd=0x0, type=OPT_GLOBAL) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3220
      #9  0x000000000064809a in plugin_vars_free_values (vars=0x7f6db33fb090) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3156
      #10 0x0000000000642f11 in plugin_del (plugin=0x7f6db33f53a8) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1202
      #11 0x0000000000643161 in reap_plugins () at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1247
      #12 0x000000000064342a in plugin_unlock (thd=0x0, plugin=0x7f6d9e01d460) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1314
      #13 0x00000000006f08a4 in TABLE_SHARE::destroy (this=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:426
      #14 0x00000000006f0aa2 in free_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:480
      #15 0x00000000005cedb8 in table_def_free_entry (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:368
      #16 0x0000000000cb57d2 in my_hash_delete (hash=0x15877a0, record=0x7f6daf67ad78 "\002") at /home/elenst/bzr/5.5/mysys/hash.c:604
      #17 0x00000000005cfb57 in release_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:794
      #18 0x00000000006f7452 in closefrm (table=0x7f6d9e1e7c60, free_share=true) at /home/elenst/bzr/5.5/sql/table.cc:2761
      #19 0x00000000005d013a in intern_close_table (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:928
      #20 0x00000000005d01a7 in free_cache_entry (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:951
      #21 0x00000000005d0525 in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /home/elenst/bzr/5.5/sql/sql_base.cc:1058
      #22 0x00000000005ceef1 in table_def_start_shutdown () at /home/elenst/bzr/5.5/sql/sql_base.cc:410
      #23 0x00000000005845e4 in clean_up (print_message=true) at /home/elenst/bzr/5.5/sql/mysqld.cc:1821
      #24 0x00000000005843d0 in unireg_end () at /home/elenst/bzr/5.5/sql/mysqld.cc:1734
      #25 0x00000000005842f2 in kill_server (sig_ptr=0x0) at /home/elenst/bzr/5.5/sql/mysqld.cc:1662
      #26 0x000000000058430f in kill_server_thread (arg=0x7f6d9dbfee48) at /home/elenst/bzr/5.5/sql/mysqld.cc:1685
      #27 0x0000000000b61755 in pfs_spawn_thread (arg=0x7f6d9dc271c0) at /home/elenst/bzr/5.5/storage/perfschema/pfs.cc:1015
      #28 0x00007f6db5031b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
      #29 0x00007f6db3b80a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

      Stack trace from:

      revision-id: knielsen@knielsen-hq.org-20140411084611-oygkym1990hgbuhe
      date: 2014-04-11 10:46:11 +0200
      build-date: 2014-04-12 21:42:44 +0400
      revno: 4133
      branch-nick: 5.5

      Valgrind warnings with 10.0:

      ==2806== 212 bytes in 1 blocks are still reachable in loss record 9 of 10
      ==2806==    at 0x4C28CCE: malloc (vg_replace_malloc.c:292)
      ==2806==    by 0xE3853F: sf_malloc (safemalloc.c:115)
      ==2806==    by 0xE276A6: my_malloc (my_malloc.c:100)
      ==2806==    by 0xD63B40C: ???
      ==2806==    by 0xD63EEA2: ???
      ==2806==    by 0xD64F7DE: ???
      ==2806==    by 0x869CC7: handler::ha_external_lock(THD*, int) (handler.cc:5843)
      ==2806==    by 0x93F723: lock_external(THD*, TABLE**, unsigned int) (lock.cc:357)
      ==2806==    by 0x93F4A3: mysql_lock_tables(THD*, st_mysql_lock*, unsigned int) (lock.cc:309)
      ==2806==    by 0x93F3B4: mysql_lock_tables(THD*, TABLE**, unsigned int, unsigned int) (lock.cc:276)
      ==2806==    by 0x61937F: lock_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) (sql_base.cc:5310)
      ==2806==    by 0x618B3A: open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) (sql_base.cc:5081)
      ==2806==    by 0x60D1E2: open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int) (sql_base.h:485)
      ==2806==    by 0x655BE1: mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) (sql_insert.cc:722)
      ==2806==    by 0x676262: mysql_execute_command(THD*) (sql_parse.cc:3471)
      ==2806==    by 0x67E48E: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6452)

      Stack trace from:

      revision-id: elenst@montyprogram.com-20140410221003-dea2djgc31gbu3lu
      date: 2014-04-11 02:10:03 +0400
      build-date: 2014-04-12 21:43:05 +0400
      revno: 4148
      branch-nick: 10.0

      Attachments

        Issue Links

          Activity

            This is not specific to TokuDB, any engine would do.

            serg Sergei Golubchik added a comment - This is not specific to TokuDB, any engine would do.

            Assertion failure in 5.5 and memory leak in 10.0 are different problems.

            Assertion failure must not affect 10.0 because:
            1. there is no such assertion in 10.0
            2. there is no LOCK_open in 10.0
            3. even per-share mutex is not held while calling intern_sys_var_ptr.

            As for the valgrind warning: it looks like a memory leak in TokuDB.

            svoj Sergey Vojtovich added a comment - Assertion failure in 5.5 and memory leak in 10.0 are different problems. Assertion failure must not affect 10.0 because: 1. there is no such assertion in 10.0 2. there is no LOCK_open in 10.0 3. even per-share mutex is not held while calling intern_sys_var_ptr. As for the valgrind warning: it looks like a memory leak in TokuDB.

            Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.

            svoj Sergey Vojtovich added a comment - Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.

            Reported TokuDB memory leak as MDEV-6212.

            svoj Sergey Vojtovich added a comment - Reported TokuDB memory leak as MDEV-6212 .

            Thanks! Ok to push!

            serg Sergei Golubchik added a comment - Thanks! Ok to push!

            Fixed in 5.5.38:

            revno: 4194
            revision-id: svoj@mariadb.org-20140506081303-81bm6s8jvazblaqo
            parent: sanja@askmonty.org-20140527064501-rkk8nxwgchdzy1gx
            committer: Sergey Vojtovich <svoj@mariadb.org>
            branch nick: 5.5-mdev6083
            timestamp: Tue 2014-05-06 12:13:03 +0400
            message:
              MDEV-6083 - Assertion `! (&(&LOCK_open)->m_mutex)->count ||
                          ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)'
                          fails in intern_sys_var_ptr on server shutdown after uninstalling
                          TokuDB plugin at runtime
             
              This assertion was introduced by patch for MDEV-5089 to ensure proper lock order
              among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held
              while acquiring LOCK_global_system_variables.
             
              intern_sys_var_ptr() may be called while freeing storage engine variables with
              PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was
              uninstalled). In this case LOCK_open is held, which is harmless because we need
              global value pointer and thus won't acquire LOCK_global_system_variables.
             
              Relaxed assertion so it is valid only for session variables.

            svoj Sergey Vojtovich added a comment - Fixed in 5.5.38: revno: 4194 revision-id: svoj@mariadb.org-20140506081303-81bm6s8jvazblaqo parent: sanja@askmonty.org-20140527064501-rkk8nxwgchdzy1gx committer: Sergey Vojtovich <svoj@mariadb.org> branch nick: 5.5-mdev6083 timestamp: Tue 2014-05-06 12:13:03 +0400 message: MDEV-6083 - Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)' fails in intern_sys_var_ptr on server shutdown after uninstalling TokuDB plugin at runtime   This assertion was introduced by patch for MDEV-5089 to ensure proper lock order among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held while acquiring LOCK_global_system_variables.   intern_sys_var_ptr() may be called while freeing storage engine variables with PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was uninstalled). In this case LOCK_open is held, which is harmless because we need global value pointer and thus won't acquire LOCK_global_system_variables.   Relaxed assertion so it is valid only for session variables.

            People

              svoj Sergey Vojtovich
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.