[MDEV-6083] Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)' fails in intern_sys_var_ptr on server shutdown after uninstalling TokuDB plugin at runtime Created: 2014-04-12  Updated: 2014-05-28  Resolved: 2014-05-28

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 5.5.36
Fix Version/s: 5.5.38

Type: Bug Priority: Minor
Reporter: Elena Stepanova Assignee: Sergey Vojtovich
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-6212 TokuDB: memory leak in external_lock Open
Sprint: serg-test-sprint-1

 Description   

On 10.0, the assertion failure doesn't happen, but there is some uncleanness on the shutdown still (e.g. seen with valgrind), so I suppose it might be affected also, thus filing both for 5.5 and 10.0.

# It's not necessary to install TokuDB at runtime, 
# instead the server can be run with --plugin-load=ha_tokudb
 
INSTALL SONAME 'ha_tokudb';
DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a INT) ENGINE=TokuDB;
INSERT INTO t1 VALUES (1),(2);
UNINSTALL SONAME 'ha_tokudb';
 
# Let server to shutdown (with MTR, it happens on its own)

5.5/sql/sql_plugin.cc:2903: uchar* intern_sys_var_ptr(THD*, int, bool): Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pthread_self(), (&(&LOCK_open)->m_mutex)->thread)' failed.
140412 20:09:03 [ERROR] mysqld got signal 6 ;

#6  0x00007f6db3ad1621 in *__GI___assert_fail (assertion=0xd8d6c0 "! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pthread_self(), (&(&LOCK_open)->m_mutex)->thread)", file=<optimized out>, line=2903, function=0xd8e500 "uchar* intern_sys_var_ptr(THD*, int, bool)") at assert.c:81
#7  0x00000000006476df in intern_sys_var_ptr (thd=0x0, offset=136, global_lock=false) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:2903
#8  0x00000000006482ce in sys_var_pluginvar::real_value_ptr (this=0x7f6db23fe248, thd=0x0, type=OPT_GLOBAL) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3220
#9  0x000000000064809a in plugin_vars_free_values (vars=0x7f6db33fb090) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3156
#10 0x0000000000642f11 in plugin_del (plugin=0x7f6db33f53a8) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1202
#11 0x0000000000643161 in reap_plugins () at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1247
#12 0x000000000064342a in plugin_unlock (thd=0x0, plugin=0x7f6d9e01d460) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1314
#13 0x00000000006f08a4 in TABLE_SHARE::destroy (this=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:426
#14 0x00000000006f0aa2 in free_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:480
#15 0x00000000005cedb8 in table_def_free_entry (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:368
#16 0x0000000000cb57d2 in my_hash_delete (hash=0x15877a0, record=0x7f6daf67ad78 "\002") at /home/elenst/bzr/5.5/mysys/hash.c:604
#17 0x00000000005cfb57 in release_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:794
#18 0x00000000006f7452 in closefrm (table=0x7f6d9e1e7c60, free_share=true) at /home/elenst/bzr/5.5/sql/table.cc:2761
#19 0x00000000005d013a in intern_close_table (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:928
#20 0x00000000005d01a7 in free_cache_entry (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:951
#21 0x00000000005d0525 in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /home/elenst/bzr/5.5/sql/sql_base.cc:1058
#22 0x00000000005ceef1 in table_def_start_shutdown () at /home/elenst/bzr/5.5/sql/sql_base.cc:410
#23 0x00000000005845e4 in clean_up (print_message=true) at /home/elenst/bzr/5.5/sql/mysqld.cc:1821
#24 0x00000000005843d0 in unireg_end () at /home/elenst/bzr/5.5/sql/mysqld.cc:1734
#25 0x00000000005842f2 in kill_server (sig_ptr=0x0) at /home/elenst/bzr/5.5/sql/mysqld.cc:1662
#26 0x000000000058430f in kill_server_thread (arg=0x7f6d9dbfee48) at /home/elenst/bzr/5.5/sql/mysqld.cc:1685
#27 0x0000000000b61755 in pfs_spawn_thread (arg=0x7f6d9dc271c0) at /home/elenst/bzr/5.5/storage/perfschema/pfs.cc:1015
#28 0x00007f6db5031b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#29 0x00007f6db3b80a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

Stack trace from:

revision-id: knielsen@knielsen-hq.org-20140411084611-oygkym1990hgbuhe
date: 2014-04-11 10:46:11 +0200
build-date: 2014-04-12 21:42:44 +0400
revno: 4133
branch-nick: 5.5

Valgrind warnings with 10.0:

==2806== 212 bytes in 1 blocks are still reachable in loss record 9 of 10
==2806==    at 0x4C28CCE: malloc (vg_replace_malloc.c:292)
==2806==    by 0xE3853F: sf_malloc (safemalloc.c:115)
==2806==    by 0xE276A6: my_malloc (my_malloc.c:100)
==2806==    by 0xD63B40C: ???
==2806==    by 0xD63EEA2: ???
==2806==    by 0xD64F7DE: ???
==2806==    by 0x869CC7: handler::ha_external_lock(THD*, int) (handler.cc:5843)
==2806==    by 0x93F723: lock_external(THD*, TABLE**, unsigned int) (lock.cc:357)
==2806==    by 0x93F4A3: mysql_lock_tables(THD*, st_mysql_lock*, unsigned int) (lock.cc:309)
==2806==    by 0x93F3B4: mysql_lock_tables(THD*, TABLE**, unsigned int, unsigned int) (lock.cc:276)
==2806==    by 0x61937F: lock_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) (sql_base.cc:5310)
==2806==    by 0x618B3A: open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) (sql_base.cc:5081)
==2806==    by 0x60D1E2: open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int) (sql_base.h:485)
==2806==    by 0x655BE1: mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) (sql_insert.cc:722)
==2806==    by 0x676262: mysql_execute_command(THD*) (sql_parse.cc:3471)
==2806==    by 0x67E48E: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6452)

Stack trace from:

revision-id: elenst@montyprogram.com-20140410221003-dea2djgc31gbu3lu
date: 2014-04-11 02:10:03 +0400
build-date: 2014-04-12 21:43:05 +0400
revno: 4148
branch-nick: 10.0



 Comments   
Comment by Sergei Golubchik [ 2014-05-05 ]

This is not specific to TokuDB, any engine would do.

Comment by Sergey Vojtovich [ 2014-05-06 ]

Assertion failure in 5.5 and memory leak in 10.0 are different problems.

Assertion failure must not affect 10.0 because:
1. there is no such assertion in 10.0
2. there is no LOCK_open in 10.0
3. even per-share mutex is not held while calling intern_sys_var_ptr.

As for the valgrind warning: it looks like a memory leak in TokuDB.

Comment by Sergey Vojtovich [ 2014-05-06 ]

Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.

Comment by Sergey Vojtovich [ 2014-05-06 ]

Reported TokuDB memory leak as MDEV-6212.

Comment by Sergei Golubchik [ 2014-05-28 ]

Thanks! Ok to push!

Comment by Sergey Vojtovich [ 2014-05-28 ]

Fixed in 5.5.38:

revno: 4194
revision-id: svoj@mariadb.org-20140506081303-81bm6s8jvazblaqo
parent: sanja@askmonty.org-20140527064501-rkk8nxwgchdzy1gx
committer: Sergey Vojtovich <svoj@mariadb.org>
branch nick: 5.5-mdev6083
timestamp: Tue 2014-05-06 12:13:03 +0400
message:
  MDEV-6083 - Assertion `! (&(&LOCK_open)->m_mutex)->count ||
              ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)'
              fails in intern_sys_var_ptr on server shutdown after uninstalling
              TokuDB plugin at runtime
 
  This assertion was introduced by patch for MDEV-5089 to ensure proper lock order
  among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held
  while acquiring LOCK_global_system_variables.
 
  intern_sys_var_ptr() may be called while freeing storage engine variables with
  PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was
  uninstalled). In this case LOCK_open is held, which is harmless because we need
  global value pointer and thus won't acquire LOCK_global_system_variables.
 
  Relaxed assertion so it is valid only for session variables.

Generated at Thu Feb 08 07:09:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.