[MDEV-21211] plugin loading error deadlock with server_audit enabled Created: 2019-12-03  Updated: 2021-04-19  Resolved: 2021-02-02

Status: Closed
Project: MariaDB Server
Component/s: Plugin - Audit, Plugins
Affects Version/s: 10.4.8, 10.5
Fix Version/s: 10.4.14, 10.5.7

Type: Bug Priority: Critical
Reporter: Hartmut Holzgraefe Assignee: Alexey Botchkov
Resolution: Fixed Votes: 3
Labels: None

Issue Links:
Blocks
Relates
relates to MDEV-19918 Server hangs or crashes while trying ... Closed

 Description   

When having the audit plugin installed with plugin-load-add, a failure to load a plugin that was installed with INSTALL; not plugin-load-add, leads to a server startup deadlock.

Problem seems to be with the audit plugins error handling callback.

I can't reproduce startup hang when having both plugins loaded with plugin-load-add, or both with INSTALL, only with the combination described above.

gdb shows the following backtrace:

#0  0x00007fa79a5474ed in __lll_lock_wait () from /lib64/libpthread.so.0
 
#1  0x00007fa79a542e01 in _L_lock_1093 () from /lib64/libpthread.so.0
 
#2  0x00007fa79a542da2 in pthread_mutex_lock () from /lib64/libpthread.so.0
 
#3  0x0000556bb62a09e0 in inline_mysql_mutex_lock (src_line=src_line@entry=2405, 
    src_file=0x556bb6ac1f30 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/sql/sql_plugin.cc", 
    that=0x556bb7431f90 <LOCK_plugin>) at /usr/src/debug/MariaDB-10.4.8/src_0/include/mysql/psi/mysql_thread.h:712
 
#4  0x0000556bb62a3c4d in plugin_foreach_with_mask (thd=thd@entry=0x556bb9eeeb88, func=func@entry=0x556bb63793f0 <acquire_plugins(THD*, plugin_ref, void*)>, 
    type=type@entry=5, state_mask=state_mask@entry=8, arg=arg@entry=0x7fff15818af0) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_plugin.cc:2405
 
#5  0x0000556bb63794a2 in mysql_audit_acquire_plugins (thd=thd@entry=0x556bb9eeeb88, event_class_mask=event_class_mask@entry=0x7fff15818af0)
 
    at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_audit.cc:120
 
#6  0x0000556bb63799d3 in mysql_audit_notify (thd=0x556bb9eeeb88, event_class=<optimized out>, event=<optimized out>)
 
    at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_audit.cc:433
 
#7  0x0000556bb6243b77 in THD::raise_condition (this=this@entry=0x556bb9eeeb88, sql_errno=sql_errno@entry=1126, sqlstate=0x556bb6ac94b0 "HY000", 
    sqlstate@entry=0x0, level=Sql_state_errno_level::WARN_LEVEL_ERROR, ucid=..., 
    msg=msg@entry=0x7fff158190c0 "Can't open shared library '/usr/lib64/mysql/plugin/ha_blackhole.so' (errno: 0, cannot open shared object file: No such file or directory)") at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_audit.h:192
 
#8  0x0000556bb61cb2bf in raise_condition (
 
    msg=0x7fff158190c0 "Can't open shared library '/usr/lib64/mysql/plugin/ha_blackhole.so' (errno: 0, cannot open shared object file: No such file or directory)", level=<optimized out>, sqlstate=0x0, sql_errno=1126, this=0x556bb9eeeb88) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_class.h:4356
 
#9  my_message_sql (error=1126, 
    str=0x7fff158190c0 "Can't open shared library '/usr/lib64/mysql/plugin/ha_blackhole.so' (errno: 0, cannot open shared object file: No such file or directory)", MyFlags=64) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/mysqld.cc:3371
 
#10 0x0000556bb6a000c0 in my_error (nr=1126, MyFlags=64) at /usr/src/debug/MariaDB-10.4.8/src_0/mysys/my_error.c:125
 
#11 0x0000556bb62a1491 in plugin_dl_add (dl=dl@entry=0x7fff15819c80, MyFlags=MyFlags@entry=64) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_plugin.cc:797
 
#12 0x0000556bb62a1c61 in plugin_add (tmp_root=tmp_root@entry=0x7fff1581a700, if_not_exists=if_not_exists@entry=false, name=name@entry=0x7fff15819c70, 
    dl=dl@entry=0x7fff15819c80, MyFlags=MyFlags@entry=64) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_plugin.cc:1101
 
#13 0x0000556bb62a2828 in plugin_load (tmp_root=tmp_root@entry=0x7fff1581a700) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_plugin.cc:1859
 
#14 0x0000556bb62a77b9 in plugin_init (argc=argc@entry=0x556bb742dc70 <remaining_argc>, argv=0x556bb95fa760, flags=0)
 
    at /usr/src/debug/MariaDB-10.4.8/src_0/sql/sql_plugin.cc:1735
 
#15 0x0000556bb61cf9c8 in init_server_components () at /usr/src/debug/MariaDB-10.4.8/src_0/sql/mysqld.cc:5205
 
#16 0x0000556bb61d5e91 in mysqld_main (argc=3, argv=0x556bb95fa760) at /usr/src/debug/MariaDB-10.4.8/src_0/sql/mysqld.cc:5726
 
#17 0x00007fa7988083d5 in __libc_start_main () from /lib64/libc.so.6
 
#18 0x0000556bb61c9bd4 in _start ()

How to reproduce:

  • load server audit plugin from my.cnf with plugin-load-add
  • install another plugin, e.g. ha_blackhole, with INSTALL PLUGIN or INSTALL SONAME
  • stop the mariadb server
  • remove the ha_blackhole.so file from the plugin directory
  • restart mariadb server

Server startup will now hang, startup will never finish

 Comments   
Comment by Jeff Dyke [ 2020-06-08 ]

A bit of extra information if you build a server from scratch, you can't have configuration for sql_error_log, until you run INSTALL SONAME 'sql_errlog'; This may be well known, but just in case. I ran into this b/c all servers are built with Salt, so i commented out those lines, installed manually and then reran the saltstate, restarted and all was fine.

Thanks!

Comment by Geoff Montee (Inactive) [ 2020-06-08 ]

Hi ekydfejj,

A bit of extra information if you build a server from scratch, you can't have configuration for sql_error_log, until you run INSTALL SONAME 'sql_errlog';

You can work around that by using the loose prefix:

https://mariadb.com/kb/en/mysqld-options/#-loose-

e.g.:

[mariadb]
 
loose_sql_error_log_filename=/path/to/file.log
 
loose_sql_error_log_rotate=ON

Comment by Jeff Dyke [ 2020-06-08 ]

Wow, how did i not know about `loose`. Thanks! @Geoff Montee!

Comment by Alexey Botchkov [ 2020-08-10 ]

https://github.com/MariaDB/server/commit/19bdb03e97d6f760699cd718e11f71e01c268f5e

Comment by Hartmut Holzgraefe [ 2020-10-05 ]

This actually seems to be fixed in 10.4.14 in 10.5.5?

Comment by Hartmut Holzgraefe [ 2020-10-15 ]

I can't reproduce this in 10.4.14 or 10.5.5 anymore. So can this be considered fixed now, or what am I missing?

Comment by Sergei Golubchik [ 2020-10-15 ]

The symptom was fixed with a solution very specific to audit plugins failing during installation.

The new patch should generally prevent all kinds of deadlocks on LOCK_plugin mutex.

Generated at Thu Feb 08 09:05:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.