Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19918

Server hangs or crashes while trying to lock mutex when the mutex was already locked upon startup with server_audit and orphan records in mysql.plugin

Details

    Description

      When the server is started with --plugin-load-add=server_audit and it has records in mysql.plugin referring non-existent libraries, it hangs on a release build or crashes on a debug build.

      10.4 9e0ed0fa

      		 
      Trying to lock mutex at /data/src/10.4/sql/sql_plugin.cc, line 2399, when the mutex was alread
      		 
      y locked at /data/src/10.4/sql/sql_plugin.cc, line 1854 in thread T@1
      		 
      190701 20:58:06 [ERROR] mysqld got signal 6 ;
      		 
       
      		 
      #5  0x00007f1adb13042a in __GI_abort () at abort.c:89
      		 
      #6  0x00005620a7381932 in safe_mutex_lock (mp=0x5620a80e95e0 <LOCK_plugin>, my_flags=0, file=0x5620a747d330 "/data/src/10.4/sql/sql_plugin.cc", line=2399) at /data/src/10.4/mysys/thr_mutex.c:265
      		 
      #7  0x00005620a6889ef5 in inline_mysql_mutex_lock (that=0x5620a80e95e0 <LOCK_plugin>, src_file=0x5620a747d330 "/data/src/10.4/sql/sql_plugin.cc", src_line=2399) at /data/src/10.4/include/mysql/psi/mysql_thread.h:710
      		 
      #8  0x00005620a6890a99 in plugin_foreach_with_mask (thd=0x5620ab294120, func=0x5620a69e269f <acquire_plugins(THD*, plugin_ref, void*)>, type=5, state_mask=8, arg=0x7ffee2db8700) at /data/src/10.4/sql/sql_plugin.cc:2399
      		 
      #9  0x00005620a69e28a1 in mysql_audit_acquire_plugins (thd=0x5620ab294120, event_class_mask=0x7ffee2db8700) at /data/src/10.4/sql/sql_audit.cc:120
      		 
      #10 0x00005620a69e2f19 in mysql_audit_notify (thd=0x5620ab294120, event_class=0, event=0x7ffee2db8770) at /data/src/10.4/sql/sql_audit.cc:417
      		 
      #11 0x00005620a67fd6d7 in mysql_audit_general (thd=0x5620ab294120, event_subtype=1, error_code=1126, msg=0x7ffee2db8fb0 "Can't open shared library '/data/bld/10.4-debug/lib/plugin/bar.so' (errno: 0, cannot open shared object file: No such file or directory)") at /data/src/10.4/sql/sql_audit.h:191
      		 
      #12 0x00005620a68004e1 in THD::raise_condition (this=0x5620ab294120, sql_errno=1126, sqlstate=0x5620a74935dd "HY000", level=Sql_state_errno_level::WARN_LEVEL_ERROR, ucid=..., msg=0x7ffee2db8fb0 "Can't open shared library '/data/bld/10.4-debug/lib/plugin/bar.so' (errno: 0, cannot open shared object file: No such file or directory)") at /data/src/10.4/sql/sql_class.cc:1076
      		 
      #13 0x00005620a674e464 in THD::raise_condition (this=0x5620ab294120, sql_errno=1126, sqlstate=0x0, level=Sql_state_errno_level::WARN_LEVEL_ERROR, msg=0x7ffee2db8fb0 "Can't open shared library '/data/bld/10.4-debug/lib/plugin/bar.so' (errno: 0, cannot open shared object file: No such file or directory)") at /data/src/10.4/sql/sql_class.h:4354
      		 
      #14 0x00005620a67413ff in my_message_sql (error=1126, str=0x7ffee2db8fb0 "Can't open shared library '/data/bld/10.4-debug/lib/plugin/bar.so' (errno: 0, cannot open shared object file: No such file or directory)", MyFlags=64) at /data/src/10.4/sql/mysqld.cc:3371
      		 
      #15 0x00005620a736d933 in my_error (nr=1126, MyFlags=64) at /data/src/10.4/mysys/my_error.c:125
      		 
      #16 0x00005620a688b4ce in plugin_dl_add (dl=0x7ffee2db9b50, MyFlags=64) at /data/src/10.4/sql/sql_plugin.cc:753
      		 
      #17 0x00005620a688c4ff in plugin_add (tmp_root=0x7ffee2dba5c0, if_not_exists=false, name=0x7ffee2db9b40, dl=0x7ffee2db9b50, MyFlags=64) at /data/src/10.4/sql/sql_plugin.cc:1100
      		 
      #18 0x00005620a688ed03 in plugin_load (tmp_root=0x7ffee2dba5c0) at /data/src/10.4/sql/sql_plugin.cc:1855
      		 
      #19 0x00005620a688e641 in plugin_init (argc=0x5620a80e5fd8 <remaining_argc>, argv=0x5620aa5f3a48, flags=0) at /data/src/10.4/sql/sql_plugin.cc:1734
      		 
      #20 0x00005620a6744cce in init_server_components () at /data/src/10.4/sql/mysqld.cc:5202
      		 
      #21 0x00005620a6745e99 in mysqld_main (argc=12, argv=0x5620aa5f3a48) at /data/src/10.4/sql/mysqld.cc:5725
      		 
      #22 0x00005620a673a730 in main (argc=12, argv=0x7ffee2dbb008) at /data/src/10.4/sql/main.cc:25

      The test case below is artificial, but there is a quite real-life scenario which represents the problem.

      • on Debian/Ubuntu, install mariadb-server and mariadb-plugin-spider packages;
      • add plugin-load-add=server_audit at the end of /etc/mysql/my.cnf;
      • uninstall mariadb-plugin-spider
      • restart the service => it hangs upon restart

      Run the test case below with --plugin-load-add=server_audit
      It seems important, installing it via INSTALL SONAME doesn't cause the same effect.

      insert into mysql.plugin values ('foo','bar.so');
      		 
       
      		 
      --source include/restart_mysqld.inc
      		 
       
      		 
      # Cleanup
      		 
      delete from mysql.plugin where name = 'foo';

      Couldn't reproduce on 10.2-10.3.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20442 Failed to upgrade from 10.3.9 to 10.4.7

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25864 After upgrading from 10.2.14 to 10.4.12, wait almost 24 hours can not open database.

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19917 Spider doesn't remove itself from mysql.plugin upon uninstallation

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22406 Galera node freezes when query_response_time plugin loading

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21211 plugin loading error deadlock with server_audit enabled

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Transition Time In Source Status Execution Times
            Alexey Botchkov made transition -
            Open In Progress
            		24d 12h 24m 1
            Alexey Botchkov made transition -
            In Progress Stalled
            		3d 22h 41m 1
            Alexey Botchkov made transition -
            Stalled In Progress
            		14d 2h 29m 1
            Alexey Botchkov made transition -
            In Progress Closed
            		348d 6h 1

            People

              holyfoot Alexey Botchkov
              elenst Elena Stepanova
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.