[MDEV-32477] safe_mutex: Found wrong usage of mutex 'LOCK_thd_data' and 'LOCK_plugin' Created: 2023-10-16  Updated: 2024-01-31

Status: Confirmed
Project: MariaDB Server
Component/s: None
Affects Version/s: 11.3
Fix Version/s: 11.3

Type: Bug Priority: Major
Reporter: Roel Van de Paar Assignee: Yuchen Pei
Resolution: Unresolved Votes: 0
Labels: mutex, race

Issue Links:
Relates

 Description   

Issue initially observed during testing of MDEV-28856, but present in trunk also:

preview-11.3-preview a4031e4c051d4fc69f466a5678ceb5d3598bf525 (Debug)

 
safe_mutex: Found wrong usage of mutex 'LOCK_thd_data' and 'LOCK_plugin'
 
Mutex currently locked (in reverse order):
 
LOCK_plugin                       /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_plugin.cc  line 2481
 
LOCK_thd_data                     /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_class.h  line 3967
 
LOCK_thd_kill                     /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_class.h  line 3966

The issue can be seen either during normal operation, or during SHUTDOWN. Possibly these are different issues leading to the same outcome.
The issues is highly sporadic, yet regularly seen. Testcase reduction has proven to be very difficult. However, a single occurrence reduced to this SHUTDOWN-based testcase:

INSTALL PLUGIN spider SONAME 'ha_spider.so';
 
CREATE TABLE t (c INT) ENGINE=Spider;
 
SET max_session_mem_used=8192;
 
UNINSTALL SONAME 'ha_spider';
 
SHUTDOWN;

This testcase may not be representative of all issues and should not be taken as a typical example, as this issue is during SHUTDOWN and reduced readily compared with the other sporadic occurrences, which do not reduce. Still, it may provide an indication in the code as to the actual bug(s).

 Comments   
Comment by Roel Van de Paar [ 2023-10-16 ]

The issue is also present in trunk 11.3 with different line numbers:

11.3.0 905c3d61e18ae6222d0d195c43d335046eec65d9 (Debug)

 
2023-10-16 21:42:34 0 [Note] /test/MD270923-mariadb-11.3.0-linux-x86_64-dbg/bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
 
safe_mutex: Found wrong usage of mutex 'LOCK_thd_data' and 'LOCK_plugin'
 
Mutex currently locked (in reverse order):
 
LOCK_plugin                       /test/11.3_dbg/sql/sql_plugin.cc  line 2526
 
LOCK_thd_data                     /test/11.3_dbg/sql/sql_class.h  line 3979
 
LOCK_thd_kill                     /test/11.3_dbg/sql/sql_class.h  line 3978

Comment by Yuchen Pei [ 2023-10-18 ]

Given this involves LOCK_plugin and the testcase has INSTALL and
UNINSTALL a few statements apart, I suggest we first observe whether
it happens in branches containing the fixes to spider init bugs
MDEV-22979.

Comment by Roel Van de Paar [ 2023-11-11 ]

Found a testcase which consistently (i.e. non-sporadically) reproduces the issue:

SET @@max_session_mem_used=500000;  # Set smaller if needed
 
SET @@session.query_alloc_block_size=655536;
 
UNINSTALL PLUGIN IF EXISTS example;
 
SHUTDOWN;

Leads to:

11.3.0 126157061b4376496c034a809ea4943e863d1465 (Debug)

 
2023-11-11 15:19:08 0 [Note] /test/MD311023-mariadb-11.3.0-linux-x86_64-dbg/bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
 
safe_mutex: Found wrong usage of mutex 'LOCK_thd_data' and 'LOCK_plugin'
 
Mutex currently locked (in reverse order):
 
LOCK_plugin                       /test/11.3_dbg/sql/sql_plugin.cc  line 2526
 
LOCK_thd_data                     /test/11.3_dbg/sql/sql_class.h  line 4016
 
LOCK_thd_kill                     /test/11.3_dbg/sql/sql_class.h  line 4015

Testcase is MTR and CLI compatible. Present in 10.6+ however 10.6 needs a smaller max_session_mem_used set: 

SET max_session_mem_used=8192;

Comment by Yuchen Pei [ 2024-01-31 ]

I cannot reproduce either the case in the description or the one in the previous comment, at 11.3.0 126157061b4376496c034a809ea4943e863d1465 or a recent 11.3 2ba8a70658d0fd0df68273bbe26d95c1e82e2cca

Generated at Thu Feb 08 10:31:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.