Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15686

Loading MyRocks plugin back after it has been unloaded causes a crash

Details

    Description

      Running the following sequence of commands

      INSTALL SONAME 'ha_rocksdb';
      UNINSTALL SONAME 'ha_rocksdb';
      INSTALL SONAME 'ha_rocksdb';
      

      causes an assertion failure:

      mysqld: /home/psergey/dev-git/10.2-r5-debug/storage/rocksdb/rdb_threads.cc:46: 
      void myrocks::Rdb_thread::init(PSI_mutex_key, PSI_cond_key): Assertion `!m_run_once' failed.
      

      Looking at the code there, indeed MyRocks code did not intend to allow the plugin to be started after it was stopped.

      Attachments

        Issue Links

          Activity

            Ok, running this under ASAN produces a clear report:

            ==28053==ERROR: AddressSanitizer: heap-use-after-free on address 0x61500002e980 at pc 0x5622f1709b0d bp 0x7fe1c2bf02e0 sp 0x7fe1c2bf02d0
            WRITE of size 1 at 0x61500002e980 thread T6
                #0 0x5622f1709b0c in init_one_value /home/ubuntu/mariadb-10.2/mysys/my_getopt.c:1265
                #1 0x5622f170a3a5 in init_variables /home/ubuntu/mariadb-10.2/mysys/my_getopt.c:1387
                #2 0x5622f1703a3e in handle_options /home/ubuntu/mariadb-10.2/mysys/my_getopt.c:216
                #3 0x5622f01c4043 in test_plugin_options /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:4073
                #4 0x5622f01b169f in plugin_initialize /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:1403
                #5 0x5622f01b536b in finalize_install /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:2080
                #6 0x5622f01b62c2 in mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:2187
                #7 0x5622f018c1da in mysql_execute_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:6130
                #8 0x5622f0196b6f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:7914
                #9 0x5622f0172840 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1815
                #10 0x5622f016f9f4 in do_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1369
                #11 0x5622f048c824 in do_handle_one_connection(CONNECT*) /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1335
                #12 0x5622f048c22c in handle_one_connection /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1241
                #13 0x5622f151ae19 in pfs_spawn_thread /home/ubuntu/mariadb-10.2/storage/perfschema/pfs.cc:1862
                #14 0x7fe1cd2716b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
                #15 0x7fe1cc70641c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
            

            0x61500002e980 is located 0 bytes inside of 504-byte region [0x61500002e980,0x61500002eb78)
            freed by thread T6 here:
                #0 0x7fe1cea55b2a in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99b2a)
                #1 0x7fe1c18a726d in std::default_delete<rocksdb::DBOptions>::operator()(rocksdb::DBOptions*) const (/home/ubuntu/mariadb-10.2/mysql-test/var/plugins/ha_rocksdb.so+0xaa326d)
                #2 0x7fe1c18ab6ee in std::unique_ptr<rocksdb::DBOptions, std::default_delete<rocksdb::DBOptions> >::reset(rocksdb::DBOptions*) (/home/ubuntu/mariadb-10.2/mysql-test/var/plugins/ha_rocksdb.so+0xaa76ee)
                #3 0x7fe1c18a0110 in std::unique_ptr<rocksdb::DBOptions, std::default_delete<rocksdb::DBOptions> >::operator=(decltype(nullptr)) (/home/ubuntu/mariadb-10.2/mysql-test/var/plugins/ha_rocksdb.so+0xa9c110)
                #4 0x7fe1c183a510 in rocksdb_done_func /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:4508
                #5 0x5622f072799e in ha_finalize_handlerton(st_plugin_int*) /home/ubuntu/mariadb-10.2/sql/handler.cc:468
                #6 0x5622f01affb1 in plugin_deinitialize /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:1219
                #7 0x5622f01b0ab4 in reap_plugins /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:1295
                #8 0x5622f01b73c9 in mysql_uninstall_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:2333
                #9 0x5622f018c276 in mysql_execute_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:6135
                #10 0x5622f0196b6f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:7914
                #11 0x5622f0172840 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1815
                #12 0x5622f016f9f4 in do_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1369
                #13 0x5622f048c824 in do_handle_one_connection(CONNECT*) /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1335
                #14 0x5622f048c22c in handle_one_connection /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1241
                #15 0x5622f151ae19 in pfs_spawn_thread /home/ubuntu/mariadb-10.2/storage/perfschema/pfs.cc:1862
                #16 0x7fe1cd2716b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
            

            previously allocated by thread T6 here:
                #0 0x7fe1cea55532 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99532)
                #1 0x7fe1c183026d in rdb_init_rocksdb_db_options /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:507
                #2 0x7fe1c1875ba9 in __static_initialization_and_destruction_0 /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:523
                #3 0x7fe1c1878e39 in _GLOBAL__sub_I_ha_rocksdb.cc /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:12591
                #4 0x7fe1cf9396b9  (/lib64/ld-linux-x86-64.so.2+0x106b9)
            

            psergei Sergei Petrunia added a comment - Ok, running this under ASAN produces a clear report: ==28053==ERROR: AddressSanitizer: heap-use-after-free on address 0x61500002e980 at pc 0x5622f1709b0d bp 0x7fe1c2bf02e0 sp 0x7fe1c2bf02d0 WRITE of size 1 at 0x61500002e980 thread T6 #0 0x5622f1709b0c in init_one_value /home/ubuntu/mariadb-10.2/mysys/my_getopt.c:1265 #1 0x5622f170a3a5 in init_variables /home/ubuntu/mariadb-10.2/mysys/my_getopt.c:1387 #2 0x5622f1703a3e in handle_options /home/ubuntu/mariadb-10.2/mysys/my_getopt.c:216 #3 0x5622f01c4043 in test_plugin_options /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:4073 #4 0x5622f01b169f in plugin_initialize /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:1403 #5 0x5622f01b536b in finalize_install /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:2080 #6 0x5622f01b62c2 in mysql_install_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:2187 #7 0x5622f018c1da in mysql_execute_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:6130 #8 0x5622f0196b6f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:7914 #9 0x5622f0172840 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1815 #10 0x5622f016f9f4 in do_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1369 #11 0x5622f048c824 in do_handle_one_connection(CONNECT*) /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1335 #12 0x5622f048c22c in handle_one_connection /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1241 #13 0x5622f151ae19 in pfs_spawn_thread /home/ubuntu/mariadb-10.2/storage/perfschema/pfs.cc:1862 #14 0x7fe1cd2716b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) #15 0x7fe1cc70641c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c) 0x61500002e980 is located 0 bytes inside of 504-byte region [0x61500002e980,0x61500002eb78) freed by thread T6 here: #0 0x7fe1cea55b2a in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99b2a) #1 0x7fe1c18a726d in std::default_delete<rocksdb::DBOptions>::operator()(rocksdb::DBOptions*) const (/home/ubuntu/mariadb-10.2/mysql-test/var/plugins/ha_rocksdb.so+0xaa326d) #2 0x7fe1c18ab6ee in std::unique_ptr<rocksdb::DBOptions, std::default_delete<rocksdb::DBOptions> >::reset(rocksdb::DBOptions*) (/home/ubuntu/mariadb-10.2/mysql-test/var/plugins/ha_rocksdb.so+0xaa76ee) #3 0x7fe1c18a0110 in std::unique_ptr<rocksdb::DBOptions, std::default_delete<rocksdb::DBOptions> >::operator=(decltype(nullptr)) (/home/ubuntu/mariadb-10.2/mysql-test/var/plugins/ha_rocksdb.so+0xa9c110) #4 0x7fe1c183a510 in rocksdb_done_func /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:4508 #5 0x5622f072799e in ha_finalize_handlerton(st_plugin_int*) /home/ubuntu/mariadb-10.2/sql/handler.cc:468 #6 0x5622f01affb1 in plugin_deinitialize /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:1219 #7 0x5622f01b0ab4 in reap_plugins /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:1295 #8 0x5622f01b73c9 in mysql_uninstall_plugin(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*) /home/ubuntu/mariadb-10.2/sql/sql_plugin.cc:2333 #9 0x5622f018c276 in mysql_execute_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:6135 #10 0x5622f0196b6f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:7914 #11 0x5622f0172840 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1815 #12 0x5622f016f9f4 in do_command(THD*) /home/ubuntu/mariadb-10.2/sql/sql_parse.cc:1369 #13 0x5622f048c824 in do_handle_one_connection(CONNECT*) /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1335 #14 0x5622f048c22c in handle_one_connection /home/ubuntu/mariadb-10.2/sql/sql_connect.cc:1241 #15 0x5622f151ae19 in pfs_spawn_thread /home/ubuntu/mariadb-10.2/storage/perfschema/pfs.cc:1862 #16 0x7fe1cd2716b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) previously allocated by thread T6 here: #0 0x7fe1cea55532 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99532) #1 0x7fe1c183026d in rdb_init_rocksdb_db_options /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:507 #2 0x7fe1c1875ba9 in __static_initialization_and_destruction_0 /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:523 #3 0x7fe1c1878e39 in _GLOBAL__sub_I_ha_rocksdb.cc /home/ubuntu/mariadb-10.2/storage/rocksdb/ha_rocksdb.cc:12591 #4 0x7fe1cf9396b9 (/lib64/ld-linux-x86-64.so.2+0x106b9)

            The above was fixed by not freeing rocksdb_db_options and rocksdb_tbl_options. Plugin's system variables point to the objects behind these two smart pointers, so they must not be freed until there's a chance that they are used.

            psergei Sergei Petrunia added a comment - The above was fixed by not freeing rocksdb_db_options and rocksdb_tbl_options . Plugin's system variables point to the objects behind these two smart pointers, so they must not be freed until there's a chance that they are used.

            http://buildbot.askmonty.org/buildbot/builders/mac-1012-bintar/builds/3153/steps/test/logs/stdio

            rocksdb.mariadb_plugin                   [ fail ]
                    Test ended at 2018-03-28 21:43:11
             
            CURRENT_TEST: rocksdb.mariadb_plugin
            mysqltest: At line 38: query 'INSTALL SONAME 'ha_rocksdb'' succeeded - should have failed with errno 1815...
             
            The result from queries just before the failure was:
            < snip >
            # MDEV-14843: Assertion `s_tx_list.size() == 0' failed in myrocks::Rdb_transaction::term_mutex
            #
            INSTALL SONAME 'ha_rocksdb';
            CREATE TABLE t1 (i INT) ENGINE=RocksDB;
            insert into t1 values (1);
            connect  con1,localhost,root,,;
            connection con1;
            insert into test.t1 values (1);
            connection default;
            DROP TABLE t1;
            UNINSTALL SONAME 'ha_rocksdb';
            #
            # MDEV-15686: Loading MyRocks plugin back after it has been unloaded causes a crash
            #
            call mtr.add_suppression("Plugin 'ROCKSDB.*' init function returned error.");
            call mtr.add_suppression("Plugin 'ROCKSDB.*' registration as a INFORMATION SCHEMA failed.");
            call mtr.add_suppression("Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed");
            INSTALL SONAME 'ha_rocksdb';
            
            

            psergei Sergei Petrunia added a comment - http://buildbot.askmonty.org/buildbot/builders/mac-1012-bintar/builds/3153/steps/test/logs/stdio rocksdb.mariadb_plugin [ fail ] Test ended at 2018-03-28 21:43:11   CURRENT_TEST: rocksdb.mariadb_plugin mysqltest: At line 38: query 'INSTALL SONAME 'ha_rocksdb'' succeeded - should have failed with errno 1815...   The result from queries just before the failure was: < snip > # MDEV-14843: Assertion `s_tx_list.size() == 0' failed in myrocks::Rdb_transaction::term_mutex # INSTALL SONAME 'ha_rocksdb'; CREATE TABLE t1 (i INT) ENGINE=RocksDB; insert into t1 values (1); connect con1,localhost,root,,; connection con1; insert into test.t1 values (1); connection default; DROP TABLE t1; UNINSTALL SONAME 'ha_rocksdb'; # # MDEV-15686: Loading MyRocks plugin back after it has been unloaded causes a crash # call mtr.add_suppression("Plugin 'ROCKSDB.*' init function returned error."); call mtr.add_suppression("Plugin 'ROCKSDB.*' registration as a INFORMATION SCHEMA failed."); call mtr.add_suppression("Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed"); INSTALL SONAME 'ha_rocksdb';

            Apparently on mac, unloading the plugin means that plugin's internal global variables are reset as well (that is, .so is unmapped?). In this case, loading it back succeeds but that is ok because its state is completely reset.

            psergei Sergei Petrunia added a comment - Apparently on mac, unloading the plugin means that plugin's internal global variables are reset as well (that is, .so is unmapped?). In this case, loading it back succeeds but that is ok because its state is completely reset.

            Pushed another cset to address this.

            psergei Sergei Petrunia added a comment - Pushed another cset to address this.

            People

              psergei Sergei Petrunia
              psergei Sergei Petrunia
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.