Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.8(EOL)
-
None
Description
There are two failures here. They might turn out to be completely different bugs, but I can't separate them from each other, they happen interchangeable in the same scenario, which, apparently, involves a race condition upon installation of server_audit plugin in two different threads.
To reproduce:
git clone https://github.com/MariaDB/randgen --branch mdev22647 rqg-mdev22647
|
cd rqg-mdev22647
|
perl ./runall-new.pl --basedir=<basedir> --grammar=mdev22647.yy --threads=2 --skip-gendata --vardir=/dev/shm/var_mdev22647 --duration=30
|
Remember to modify the command line to point at the real basedir.
|
10.2 2c4a2f20 |
safe_mutex: Trying to lock mutex at /data/src/10.2/sql/sql_plugin.cc, line 2998, when the mutex was already locked at /data/src/10.2/sql/sys_vars_shared.h, line 56 in thread
|
T@20
|
200520 21:27:19 [ERROR] mysqld got signal 6 ;
|
|
|
#5 0x00007f36556d742a in __GI_abort () at abort.c:89
|
#6 0x0000564ea6f3240d in safe_mutex_lock (mp=0x564ea7af0e80 <LOCK_global_system_variables>, my_flags=0, file=0x564ea6ff1760 "/data/src/10.2/sql/sql_plugin.cc", line=2998) at /data/src/10.2/mysys/thr_mutex.c:264
|
#7 0x0000564ea6581266 in inline_mysql_mutex_lock (that=0x564ea7af0e80 <LOCK_global_system_variables>, src_file=0x564ea6ff1760 "/data/src/10.2/sql/sql_plugin.cc", src_line=2998) at /data/src/10.2/include/mysql/psi/mysql_thread.h:688
|
#8 0x0000564ea65895d9 in sync_dynamic_session_variables (thd=0x7f35dc000af0, global_lock=true) at /data/src/10.2/sql/sql_plugin.cc:2998
|
#9 0x0000564ea658989f in intern_sys_var_ptr (thd=0x7f35dc000af0, offset=104, global_lock=true) at /data/src/10.2/sql/sql_plugin.cc:3065
|
#10 0x0000564ea6589a0a in mysql_sys_var_str (thd=0x7f35dc000af0, offset=104) at /data/src/10.2/sql/sql_plugin.cc:3112
|
#11 0x00007f365439f173 in get_loc_info (thd=0x7f35dc000af0) at /data/src/10.2/plugin/server_audit/server_audit.c:950
|
#12 0x00007f36543a2fde in mark_always_logged (thd=0x7f35dc000af0) at /data/src/10.2/plugin/server_audit/server_audit.c:2592
|
#13 0x00007f36543a3b25 in update_logging (thd=0x7f35dc000af0, var=0x7f36545a7460 <mysql_sysvar_logging>, var_ptr=0x7f36545a8594 <logging>, save=0x7f35dc0111d0) at /data/src/10.2/plugin/server_audit/server_audit.c:2862
|
#14 0x0000564ea658ada4 in sys_var_pluginvar::global_update (this=0x7f35e80d4670, thd=0x7f35dc000af0, var=0x7f35dc0111b0) at /data/src/10.2/sql/sql_plugin.cc:3516
|
#15 0x0000564ea649e2af in sys_var::update (this=0x7f35e80d4670, thd=0x7f35dc000af0, var=0x7f35dc0111b0) at /data/src/10.2/sql/set_var.cc:208
|
#16 0x0000564ea64a0030 in set_var::update (this=0x7f35dc0111b0, thd=0x7f35dc000af0) at /data/src/10.2/sql/set_var.cc:837
|
#17 0x0000564ea649fc12 in sql_set_variables (thd=0x7f35dc000af0, var_list=0x7f35dc005470, free=true) at /data/src/10.2/sql/set_var.cc:738
|
#18 0x0000564ea657005e in mysql_execute_command (thd=0x7f35dc000af0) at /data/src/10.2/sql/sql_parse.cc:4591
|
#19 0x0000564ea6579e6f in mysql_parse (thd=0x7f35dc000af0, rawbuf=0x7f35dc011018 "SET /* QNO 13 CON_ID 13 */ GLOBAL SERVER_AUDIT_LOGGING = ON", length=59, parser_state=0x7f36545f4660, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:7740
|
#20 0x0000564ea65681c2 in dispatch_command (command=COM_QUERY, thd=0x7f35dc000af0, packet=0x7f35dc0087a1 "SET /* QNO 13 CON_ID 13 */ GLOBAL SERVER_AUDIT_LOGGING = ON", packet_length=59, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:1832
|
#21 0x0000564ea6566c3d in do_command (thd=0x7f35dc000af0) at /data/src/10.2/sql/sql_parse.cc:1385
|
#22 0x0000564ea66bbac9 in do_handle_one_connection (connect=0x564ea9058280) at /data/src/10.2/sql/sql_connect.cc:1336
|
#23 0x0000564ea66bb834 in handle_one_connection (arg=0x564ea9058280) at /data/src/10.2/sql/sql_connect.cc:1241
|
#24 0x00007f36576574a4 in start_thread (arg=0x7f36545f5700) at pthread_create.c:456
|
#25 0x00007f365578bd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
|
All threads' stack trace attached as mdev22647_all_threads_mutex
|
10.2 2c4a2f20 |
mysqld: /data/src/10.2/sql/sql_audit.cc:115: void mysql_audit_acquire_plugins(THD*, ulong*): Assertion `!check_audit_mask(mysql_global_audit_mask, event_class_mask)' failed.
|
200520 21:45:53 [ERROR] mysqld got signal 6 ;
|
|
|
#7 0x00007fabcf613f12 in __GI___assert_fail (assertion=0x559c33648c68 "!check_audit_mask(mysql_global_audit_mask, event_class_mask)", file=0x559c33648c28 "/data/src/10.2/sql/sql_audit.cc", line=115, function=0x559c33648da0 <mysql_audit_acquire_plugins(THD*, unsigned long*)::__PRETTY_FUNCTION__> "void mysql_audit_acquire_plugins(THD*, ulong*)") at assert.c:101
|
#8 0x0000559c32ce313c in mysql_audit_acquire_plugins (thd=0x7fab58000af0, event_class_mask=0x7fabccddca20) at /data/src/10.2/sql/sql_audit.cc:115
|
#9 0x0000559c32ce3838 in mysql_audit_notify (thd=0x7fab58000af0, event_class=0, event=0x7fabccddca70) at /data/src/10.2/sql/sql_audit.cc:432
|
#10 0x0000559c32b49b62 in mysql_audit_general (thd=0x7fab58000af0, event_subtype=1, error_code=1193, msg=0x7fabccddce40 "Unknown system variable 'SERVER_AUDIT_LOGGING'") at /data/src/10.2/sql/sql_audit.h:197
|
#11 0x0000559c32b4c5f6 in THD::raise_condition (this=0x7fab58000af0, sql_errno=1193, sqlstate=0x559c3362eb25 "HY000", level=Sql_condition::WARN_LEVEL_ERROR, msg=0x7fabccddce40 "Unknown system variable 'SERVER_AUDIT_LOGGING'") at /data/src/10.2/sql/sql_class.cc:1047
|
#12 0x0000559c32aa5015 in my_message_sql (error=1193, str=0x7fabccddce40 "Unknown system variable 'SERVER_AUDIT_LOGGING'", MyFlags=0) at /data/src/10.2/sql/mysqld.cc:3639
|
#13 0x0000559c33548db6 in my_error (nr=1193, MyFlags=0) at /data/src/10.2/mysys/my_error.c:125
|
#14 0x0000559c32bb3f05 in find_sys_var_ex (thd=0x7fab58000af0, str=0x7fab58011108 "SERVER_AUDIT_LOGGING", length=20, throw_error=false, locked=false) at /data/src/10.2/sql/sql_plugin.cc:2840
|
#15 0x0000559c32bb3f75 in find_sys_var (thd=0x7fab58000af0, str=0x7fab58011108 "SERVER_AUDIT_LOGGING", length=20) at /data/src/10.2/sql/sql_plugin.cc:2847
|
#16 0x0000559c32d8efc1 in find_sys_var_null_base (thd=0x7fab58000af0, tmp=0x7fabccdde720) at /data/src/10.2/sql/sql_yacc.yy:394
|
#17 0x0000559c32db7052 in MYSQLparse (thd=0x7fab58000af0) at /data/src/10.2/sql/sql_yacc.yy:15373
|
#18 0x0000559c32ba9e2e in parse_sql (thd=0x7fab58000af0, parser_state=0x7fabccddf660, creation_ctx=0x0, do_pfs_digest=true) at /data/src/10.2/sql/sql_parse.cc:10029
|
#19 0x0000559c32ba4ce1 in mysql_parse (thd=0x7fab58000af0, rawbuf=0x7fab58011018 "SET /* QNO 765 CON_ID 13 */ GLOBAL SERVER_AUDIT_LOGGING = ON", length=60, parser_state=0x7fabccddf660, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:7693
|
#20 0x0000559c32b931c2 in dispatch_command (command=COM_QUERY, thd=0x7fab58000af0, packet=0x7fab580087a1 "SET /* QNO 765 CON_ID 13 */ GLOBAL SERVER_AUDIT_LOGGING = ON", packet_length=60, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:1832
|
#21 0x0000559c32b91c3d in do_command (thd=0x7fab58000af0) at /data/src/10.2/sql/sql_parse.cc:1385
|
#22 0x0000559c32ce6ac9 in do_handle_one_connection (connect=0x559c35714280) at /data/src/10.2/sql/sql_connect.cc:1336
|
#23 0x0000559c32ce6834 in handle_one_connection (arg=0x559c35714280) at /data/src/10.2/sql/sql_connect.cc:1241
|
#24 0x00007fabd159c4a4 in start_thread (arg=0x7fabccde0700) at pthread_create.c:456
|
#25 0x00007fabcf6d0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
|
Reproducible on 10.2-10.5.
Couldn't reproduce on 10.1.
Didn't observe immediate effects on a non-debug build (no hang, either).
Attachments
Issue Links
- relates to
-
-
- Open
-
-
-
- Open
-
-
-
- Confirmed
-
Activity
Besides the 2 failures from the description, I get also :
|
10.5 8b1155035638edaf1 |
# 2021-03-25T13:07:11 [1729733] #2 0x00005644b6280d1f in handle_fatal_signal (sig=11) at /10.5/sql/signal_handler.cc:331
|
# 2021-03-25T13:07:11 [1729733] #3 <signal handler called>
|
# 2021-03-25T13:07:11 [1729733] #4 strxnmov (dst=0x7f27b0092223 "\260'\177", len=66, src=0x8f8f8f8f8f8f8f90 <error: Cannot access memory at address 0x8f8f8f8f8f8f8f90>) at /10.5/strings/strxnmov.c:66
|
# 2021-03-25T13:07:11 [1729733] #5 0x00005644b5df0f7d in sys_var::do_deprecated_warning (this=0x7f27544abbb8, thd=0x7f2758000db8) at /10.5/sql/set_var.cc:425
|
# 2021-03-25T13:07:11 [1729733] #6 0x00005644b5df18d3 in set_var::check (this=0x7f27580b0850, thd=0x7f2758000db8) at /10.5/sql/set_var.cc:776
|
# 2021-03-25T13:07:11 [1729733] #7 0x00005644b5df1751 in sql_set_variables (thd=0x7f2758000db8, var_list=0x7f27580afa90, free=true) at /10.5/sql/set_var.cc:739
|
# 2021-03-25T13:07:11 [1729733] #8 0x00005644b5f1d880 in mysql_execute_command (thd=0x7f2758000db8) at /10.5/sql/sql_parse.cc:5093
|
# 2021-03-25T13:07:11 [1729733] #9 0x00005644b5f487fd in Prepared_statement::execute (this=0x7f27580baee8, expanded_query=0x7f27b0092b10, open_cursor=false) at /10.5/sql/sql_prepare.cc:5023
|
# 2021-03-25T13:07:11 [1729733] #10 0x00005644b5f46b13 in Prepared_statement::execute_loop (this=0x7f27580baee8, expanded_query=0x7f27b0092b10, open_cursor=false, packet=0x0, packet_end=0x0) at /10.5/sql/sql_prepare.cc:4486
|
# 2021-03-25T13:07:11 [1729733] #11 0x00005644b5f48e45 in Prepared_statement::execute_immediate (this=0x7f27580baee8, query=0x7f27580149e8 " SET GLOBAL SERVER_AUDIT_LOGGING = OFF ", query_len=39) at /10.5/sql/sql_prepare.cc:5151
|
# 2021-03-25T13:07:11 [1729733] #12 0x00005644b5f42e06 in mysql_sql_stmt_execute_immediate (thd=0x7f2758000db8) at /10.5/sql/sql_prepare.cc:2989
|
# 2021-03-25T13:07:11 [1729733] #13 0x00005644b5f198c5 in mysql_execute_command (thd=0x7f2758000db8) at /10.5/sql/sql_parse.cc:4009
|
# 2021-03-25T13:07:11 [1729733] #14 0x00005644b5f274fc in mysql_parse (thd=0x7f2758000db8, rawbuf=0x7f27580148e0 "EXECUTE /* QNO 181 CON_ID 8 */ IMMEDIATE ' SET GLOBAL SERVER_AUDIT_LOGGING = OFF '", length=82, parser_state=0x7f27b0093490, is_com_multi=false, is_next_command=false) at /10.5/sql/sql_parse.cc:8093
|
# 2021-03-25T13:07:11 [1729733] #15 0x00005644b5f133c7 in dispatch_command (command=COM_QUERY, thd=0x7f2758000db8, packet=0x7f275800b499 "EXECUTE /* QNO 181 CON_ID 8 */ IMMEDIATE ' SET GLOBAL SERVER_AUDIT_LOGGING = OFF '", packet_length=82, is_com_multi=false, is_next_command=false) at /10.5/sql/sql_parse.cc:1889
|
# 2021-03-25T13:07:11 [1729733] #16 0x00005644b5f11bbb in do_command (thd=0x7f2758000db8) at /10.5/sql/sql_parse.cc:1370
|
# 2021-03-25T13:07:11 [1729733] #17 0x00005644b60c0375 in do_handle_one_connection (connect=0x5644b9ad43d8, put_in_cache=true) at /10.5/sql/sql_connect.cc:1410
|
# 2021-03-25T13:07:11 [1729733] #18 0x00005644b60c00d8 in handle_one_connection (arg=0x5644b9ad43d8) at /10.5/sql/sql_connect.cc:1312
|
# 2021-03-25T13:07:11 [1729733] #19 0x00005644b6621d51 in pfs_spawn_thread (arg=0x5644b99b9bf8) at /10.5/storage/perfschema/pfs.cc:2201
|
# 2021-03-25T13:07:11 [1729733] #20 0x00007f27b8c81609 in start_thread (arg=<optimized out>) at pthread_create.c:477
|
I used the same grammar, but with ps:
query:
|
EXECUTE IMMEDIATE ' SET GLOBAL SERVER_AUDIT_LOGGING = ON ' |
| EXECUTE IMMEDIATE ' SET GLOBAL SERVER_AUDIT_LOGGING = OFF ' |
;
|
I got this locally on 10.6:
|
10.6 f078788e293c0eed416e1bff0a4a3437d4085abb |
sys_vars.sysvars_star w27 [ fail ] Found warnings/errors in server log file!
|
Test ended at 2021-05-28 16:20:20
|
line
|
mariadbd: /mariadb/10.6/sql/sql_audit.cc:116: void mysql_audit_acquire_plugins(THD *, ulong *): Assertion `!check_audit_mask(mysql_global_audit_mask, event_class_mask)' failed.
|
…
|
stdlib/abort.c:81(__GI_abort)[0x7f838a377537]
|
intl/loadmsgcat.c:509(get_sysdep_segment_value)[0x7f838a37740f]
|
Sorry, I do not have a better stack trace. This was just a one-time occurrence after running dozens of
nice ./mtr --parallel=60 --big-test --force
|
on a system where nproc reports 40. The preceding tests that were run on that worker were as follows:
binlog_encryption.rpl_parallel_rollback_assert 'innodb,mix' w27 [ pass ] 480
|
binlog_encryption.rpl_binlog_errors 'innodb,row' w27 [ pass ] 17935
|
rpl.rpl_loaddata_s 'row' w27 [ skipped ] Neither MIXED nor STATEMENT binlog format
|
rpl.rpl_slave_load_tmpdir_not_exist 'row' w27 [ pass ] 108
|
rpl.rpl_multi_update2 'row' w27 [ pass ] 266
|
rpl.rpl_innodb_bug30888 'innodb,stmt' w27 [ skipped ] Neither MIXED nor ROW binlog format
|
rpl.rpl_parallel_optimistic_xa_lsu_off 'innodb,stmt' w27 [ skipped ] Need performance schema compiled in
|
rpl.rpl_loaddata_map 'stmt' w27 [ pass ] 898
|
rpl.rpl_init_slave 'stmt' w27 [ pass ] 385
|
rpl.rpl_000011 'stmt' w27 [ pass ] 223
|
rpl.rpl_gtid_basic 'innodb,mix' w27 [ pass ] 4175
|
sys_vars.sysvars_star w27 [ fail ] Found warnings/errors in server log file!
|
Possibly related:
|
10.7 8c5e5e1be9353b90f341aad73cb3d2ed7c405a96 |
sys_vars.sysvars_star w25 [ fail ] Found warnings/errors in server log file!
|
Test ended at 2021-10-13 10:04:24
|
line
|
mariadbd: /mariadb/10.7/sql/sql_audit.cc:116: void mysql_audit_acquire_plugins(THD*, ulong*): Assertion `!check_audit_mask(mysql_global_audit_mask, event_class_mask)' failed.
|
There was no stack trace, because this was an ASAN build, and ASAN disables core dumps by default.
This might be repeatable with the following (which I did not try):
./mtr --verbose-restart --no-reorder binlog_encryption.rpl_corruption encryption.innodb_encryption-page-compression encryption.innodb_encryption_tables rpl.rpl_special_charset rpl.mdev_17588 rpl.myisam_external_lock rpl.rpl_multi_delete2 rpl.rpl_sporadic_master rpl.rpl_flush_logs rpl.rpl_sp_effects rpl.rpl_multi_update4 rpl.rpl_gtid_basic sys_vars.sysvars_star
|
The test sys_vars.sysvars_star was introduced in 10.5.
Possibly related:
|
10.8 32d741b5b0087d6322cee0e8a7d34aa694e30d69 |
sys_vars.sysvars_star w59 [ fail ] Found warnings/errors in server log file!
|
Test ended at 2022-03-01 11:13:12
|
line
|
mariadbd: /mariadb/10.8/sql/sql_audit.cc:116: void mysql_audit_acquire_plugins(THD*, ulong*): Assertion `!check_audit_mask(mysql_global_audit_mask, event_class_mask)' failed.
|
Attempting backtrace. You can use the following information to find out
|
Neither a core dump nor a stack trace of the current thread were produced.
Note only Assertion `!check_audit_mask was fixed, I wasn't able to repeat safe_mutex: Trying to lock mutex ... when the mutex was already locked error. Please, report a new bug if you see that again.
I got this today on latest 10.5 code while running the full mtr test suite.
mariadbd: /my/maria-10.5/sql/sql_audit.cc:116: void mysql_audit_acquire_plugins(THD*, ulong*): Assertion `!check_audit_mask(mysql_global_audit_mask, event_class_mask)' failed.