Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)' fails in intern_sys_var_ptr on server shutdown after uninstalling TokuDB plugin at runtime
On 10.0, the assertion failure doesn't happen, but there is some uncleanness on the shutdown still (e.g. seen with valgrind), so I suppose it might be affected also, thus filing both for 5.5 and 10.0.
# It's not necessary to install TokuDB at runtime,
# instead the server can be run with --plugin-load=ha_tokudb
INSTALL SONAME 'ha_tokudb';
DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a INT) ENGINE=TokuDB;
INSERT INTO t1 VALUES (1),(2);
UNINSTALL SONAME 'ha_tokudb';
# Let server to shutdown (with MTR, it happens on its own)
#7 0x00000000006476df in intern_sys_var_ptr (thd=0x0, offset=136, global_lock=false) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:2903
#8 0x00000000006482ce in sys_var_pluginvar::real_value_ptr (this=0x7f6db23fe248, thd=0x0, type=OPT_GLOBAL) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3220
#9 0x000000000064809a in plugin_vars_free_values (vars=0x7f6db33fb090) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3156
#10 0x0000000000642f11 in plugin_del (plugin=0x7f6db33f53a8) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1202
#11 0x0000000000643161 in reap_plugins () at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1247
#12 0x000000000064342a in plugin_unlock (thd=0x0, plugin=0x7f6d9e01d460) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1314
#13 0x00000000006f08a4 in TABLE_SHARE::destroy (this=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:426
#14 0x00000000006f0aa2 in free_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:480
#15 0x00000000005cedb8 in table_def_free_entry (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:368
#16 0x0000000000cb57d2 in my_hash_delete (hash=0x15877a0, record=0x7f6daf67ad78 "\002") at /home/elenst/bzr/5.5/mysys/hash.c:604
#17 0x00000000005cfb57 in release_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:794
#18 0x00000000006f7452 in closefrm (table=0x7f6d9e1e7c60, free_share=true) at /home/elenst/bzr/5.5/sql/table.cc:2761
#19 0x00000000005d013a in intern_close_table (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:928
#20 0x00000000005d01a7 in free_cache_entry (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:951
#21 0x00000000005d0525 in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /home/elenst/bzr/5.5/sql/sql_base.cc:1058
#22 0x00000000005ceef1 in table_def_start_shutdown () at /home/elenst/bzr/5.5/sql/sql_base.cc:410
#23 0x00000000005845e4 in clean_up (print_message=true) at /home/elenst/bzr/5.5/sql/mysqld.cc:1821
#24 0x00000000005843d0 in unireg_end () at /home/elenst/bzr/5.5/sql/mysqld.cc:1734
#25 0x00000000005842f2 in kill_server (sig_ptr=0x0) at /home/elenst/bzr/5.5/sql/mysqld.cc:1662
#26 0x000000000058430f in kill_server_thread (arg=0x7f6d9dbfee48) at /home/elenst/bzr/5.5/sql/mysqld.cc:1685
#27 0x0000000000b61755 in pfs_spawn_thread (arg=0x7f6d9dc271c0) at /home/elenst/bzr/5.5/storage/perfschema/pfs.cc:1015
#28 0x00007f6db5031b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#29 0x00007f6db3b80a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
Assertion failure in 5.5 and memory leak in 10.0 are different problems.
Assertion failure must not affect 10.0 because:
1. there is no such assertion in 10.0
2. there is no LOCK_open in 10.0
3. even per-share mutex is not held while calling intern_sys_var_ptr.
As for the valgrind warning: it looks like a memory leak in TokuDB.
Sergey Vojtovich
added a comment - Assertion failure in 5.5 and memory leak in 10.0 are different problems.
Assertion failure must not affect 10.0 because:
1. there is no such assertion in 10.0
2. there is no LOCK_open in 10.0
3. even per-share mutex is not held while calling intern_sys_var_ptr.
As for the valgrind warning: it looks like a memory leak in TokuDB.
Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.
Sergey Vojtovich
added a comment - Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.
fails in intern_sys_var_ptr on server shutdown after uninstalling
TokuDB plugin at runtime
This assertion was introduced by patch for MDEV-5089 to ensure proper lock order
among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held
while acquiring LOCK_global_system_variables.
intern_sys_var_ptr() may be called while freeing storage engine variables with
PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was
uninstalled). In this case LOCK_open is held, which is harmless because we need
global value pointer and thus won't acquire LOCK_global_system_variables.
Relaxed assertion so it is valid only for session variables.
Sergey Vojtovich
added a comment - Fixed in 5.5.38:
revno: 4194
revision-id: svoj@mariadb.org-20140506081303-81bm6s8jvazblaqo
parent: sanja@askmonty.org-20140527064501-rkk8nxwgchdzy1gx
committer: Sergey Vojtovich <svoj@mariadb.org>
branch nick: 5.5-mdev6083
timestamp: Tue 2014-05-06 12:13:03 +0400
message:
MDEV-6083 - Assertion `! (&(&LOCK_open)->m_mutex)->count ||
! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)'
fails in intern_sys_var_ptr on server shutdown after uninstalling
TokuDB plugin at runtime
This assertion was introduced by patch for MDEV-5089 to ensure proper lock order
among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held
while acquiring LOCK_global_system_variables.
intern_sys_var_ptr() may be called while freeing storage engine variables with
PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was
uninstalled). In this case LOCK_open is held, which is harmless because we need
global value pointer and thus won't acquire LOCK_global_system_variables.
Relaxed assertion so it is valid only for session variables.
People
Sergey Vojtovich
Elena Stepanova
Votes:
0Vote for this issue
Watchers:
4Start watching this issue
Dates
Created:
Updated:
Resolved:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":717.7000000476837,"ttfb":168.60000014305115,"pageVisibility":"visible","entityId":35028,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"4ce96aa5-ef3e-4210-a00e-b52db05df2bf","navigationType":0,"readyForUser":841.9000000953674,"redirectCount":0,"resourceLoadedEnd":447.60000014305115,"resourceLoadedStart":176.10000014305115,"resourceTiming":[{"duration":3,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":176.10000014305115,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":176.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":179.10000014305115,"responseStart":0,"secureConnectionStart":0},{"duration":3.8000001907348633,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":176.29999995231628,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":176.29999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":180.10000014305115,"responseStart":0,"secureConnectionStart":0},{"duration":159.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":176.40000009536743,"connectEnd":176.40000009536743,"connectStart":176.40000009536743,"domainLookupEnd":176.40000009536743,"domainLookupStart":176.40000009536743,"fetchStart":176.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":182.10000014305115,"responseEnd":335.5,"responseStart":193.79999995231628,"secureConnectionStart":176.40000009536743},{"duration":271.10000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":176.5,"connectEnd":176.5,"connectStart":176.5,"domainLookupEnd":176.5,"domainLookupStart":176.5,"fetchStart":176.5,"redirectEnd":0,"redirectStart":0,"requestStart":183.29999995231628,"responseEnd":447.60000014305115,"responseStart":204.20000004768372,"secureConnectionStart":176.5},{"duration":30.199999809265137,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":176.60000014305115,"connectEnd":176.60000014305115,"connectStart":176.60000014305115,"domainLookupEnd":176.60000014305115,"domainLookupStart":176.60000014305115,"fetchStart":176.60000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":183.5,"responseEnd":206.79999995231628,"responseStart":202.5,"secureConnectionStart":176.60000014305115},{"duration":19.700000047683716,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":176.70000004768372,"connectEnd":176.70000004768372,"connectStart":176.70000004768372,"domainLookupEnd":176.70000004768372,"domainLookupStart":176.70000004768372,"fetchStart":176.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":183.79999995231628,"responseEnd":196.40000009536743,"responseStart":195.79999995231628,"secureConnectionStart":176.70000004768372},{"duration":40.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":176.70000004768372,"connectEnd":176.70000004768372,"connectStart":176.70000004768372,"domainLookupEnd":176.70000004768372,"domainLookupStart":176.70000004768372,"fetchStart":176.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":185,"responseEnd":216.79999995231628,"responseStart":213.79999995231628,"secureConnectionStart":176.70000004768372},{"duration":7.700000047683716,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":176.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":176.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":184.60000014305115,"responseStart":0,"secureConnectionStart":0},{"duration":56.799999952316284,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":177,"connectEnd":177,"connectStart":177,"domainLookupEnd":177,"domainLookupStart":177,"fetchStart":177,"redirectEnd":0,"redirectStart":0,"requestStart":186.79999995231628,"responseEnd":233.79999995231628,"responseStart":231.79999995231628,"secureConnectionStart":177},{"duration":8.599999904632568,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":177.10000014305115,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":177.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":185.70000004768372,"responseStart":0,"secureConnectionStart":0},{"duration":40.200000047683716,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":177.20000004768372,"connectEnd":177.20000004768372,"connectStart":177.20000004768372,"domainLookupEnd":177.20000004768372,"domainLookupStart":177.20000004768372,"fetchStart":177.20000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":189.40000009536743,"responseEnd":217.40000009536743,"responseStart":215.40000009536743,"secureConnectionStart":177.20000004768372},{"duration":2.5999999046325684,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":197.70000004768372,"connectEnd":197.70000004768372,"connectStart":197.70000004768372,"domainLookupEnd":197.70000004768372,"domainLookupStart":197.70000004768372,"fetchStart":197.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":197.70000004768372,"responseEnd":200.29999995231628,"responseStart":200.29999995231628,"secureConnectionStart":197.70000004768372},{"duration":10.299999952316284,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":204.10000014305115,"connectEnd":204.10000014305115,"connectStart":204.10000014305115,"domainLookupEnd":204.10000014305115,"domainLookupStart":204.10000014305115,"fetchStart":204.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":204.10000014305115,"responseEnd":214.40000009536743,"responseStart":214.40000009536743,"secureConnectionStart":204.10000014305115},{"duration":48.30000019073486,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":481.7999999523163,"connectEnd":481.7999999523163,"connectStart":481.7999999523163,"domainLookupEnd":481.7999999523163,"domainLookupStart":481.7999999523163,"fetchStart":481.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":500.40000009536743,"responseEnd":530.1000001430511,"responseStart":529.4000000953674,"secureConnectionStart":481.7999999523163},{"duration":40.299999952316284,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":711.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":711.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":751.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":124.70000004768372,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":717.9000000953674,"connectEnd":717.9000000953674,"connectStart":717.9000000953674,"domainLookupEnd":717.9000000953674,"domainLookupStart":717.9000000953674,"fetchStart":717.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":765.2999999523163,"responseEnd":842.6000001430511,"responseStart":842.1000001430511,"secureConnectionStart":717.9000000953674}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":24,"responseStart":169,"responseEnd":204,"domLoading":172,"domInteractive":912,"domContentLoadedEventStart":912,"domContentLoadedEventEnd":961,"domComplete":1106,"loadEventStart":1106,"loadEventEnd":1106,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":888.5},{"name":"bigPipe.sidebar-id.end","time":889.2000000476837},{"name":"bigPipe.activity-panel-pipe-id.start","time":889.2999999523163},{"name":"bigPipe.activity-panel-pipe-id.end","time":891.5},{"name":"activityTabFullyLoaded","time":980.2000000476837}],"measures":[],"correlationId":"603f3b0728205d","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":79,"dbReadsTimeInMs":12,"dbConnsTimeInMs":18,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
This is not specific to TokuDB, any engine would do.