Nayuta Yanagisawa (Inactive) added a comment - 2022-01-24 05:15 I've confirmed that the bug is reproducible on the preview branch, but it is also reproducible on 10.8 HEAD (e222e44). So, this is not a regression. Further, I think that the server should not allow setting Spider as the default temporary table storage engine because it won't definitely work. cc: Roel

Nayuta Yanagisawa (Inactive) added a comment - 2022-01-24 05:29 The server crashes on 10.2-10.8 except 10.3. On 10.3, the server hangs during SHUTDOWN.

Nayuta Yanagisawa (Inactive) added a comment - 2022-01-24 05:55 - edited The server raises an error when one tries to set the storage engine which has the HTON_TEMPORARY_NOT_SUPPORTED flag. However, the check is only available on 10.7+ and it is unlikely for users to set Spider as the default tmp SE. So, I will fix the bug on 10.7+ only.

Nayuta Yanagisawa (Inactive) added a comment - 2022-01-24 06:06 holyfoot Please review: https://github.com/MariaDB/server/commit/327ab418e823b1a58eaabe80f2b0233a245794c8

Roel Van de Paar added a comment - 2022-01-28 04:38 Thank you nayuta-yanagisawa . Btw, I confirmed that RocksDB does not crash on 10.6. INSTALL SONAME 'ha_rocksdb' ; SET GLOBAL default_tmp_storage_engine=RocksDB; SHUTDOWN;

Nayuta Yanagisawa (Inactive) added a comment - 2022-01-28 09:04 Hmm. Now, I will see if it is possible to avoid crashes in 10.6 and below.

Roel Van de Paar added a comment - 2022-01-29 00:54 - edited Discovered that the testcase above fails on 10.8 with: 10.8.1 0c5d1342ae6b5ab3256848be7a83e5c3b1f21566 (Optimized) 10.8.1-opt>SET GLOBAL default_tmp_storage_engine=RocksDB; ERROR 1478 (HY000): Table storage engine 'ROCKSDB' does not support the create option 'TEMPORARY' However not on 10.6: 10.6.6 bd03c0e51629e1c3969a171137712a6bb854c232 (Optimized) 10.6.6-dbg>SET GLOBAL default_tmp_storage_engine=RocksDB; Query OK, 0 rows affected (0.000 sec) Yet this one does not crash (tried a number of times)

Roel Van de Paar added a comment - 2022-01-29 00:58 This bug can also be observed, sporadically, with this more generic testcase: INSTALL PLUGIN spider SONAME 'ha_spider.so' ; SET GLOBAL storage_engine=Spider; SHUTDOWN; 10.8.1 0c5d1342ae6b5ab3256848be7a83e5c3b1f21566 (Optimized) Core was generated by `/test/MD290122-mariadb-10.8.1-linux-x86_64-opt/bin/mysqld --no-defaults --core-'. Program terminated with signal SIGSEGV, Segmentation fault. #0 intern_plugin_lock (state_mask=14, rc=0x0, lex=0x0) at /test/10.8_opt/sql/sql_plugin.cc:973 973 if (pi->state & state_mask) [Current thread is 1 (Thread 0x148647add800 (LWP 64708))] (gdb) bt #0 intern_plugin_lock (state_mask=14, rc=0x0, lex=0x0) at /test/10.8_opt/sql/sql_plugin.cc:973 #1 plugin_thdvar_init (thd=0x55eb2f2a57f8) at /test/10.8_opt/sql/sql_plugin.cc:3242 #2 0x000055eb2c2761c7 in THD::init (this=0x55eb2f2a57f8) at /test/10.8_opt/sql/sql_class.cc:1234 #3 0x000055eb2c27bab6 in THD::THD (this=0x55eb2f2a57f8, id=<optimized out>, is_wsrep_applier=<optimized out>) at /test/10.8_opt/sql/sql_class.cc:849 #4 0x0000148624162cd0 in spider_create_thd () at /test/10.8_opt/sql/sql_list.h:680 #5 spider_db_done (p=<optimized out>) at /test/10.8_opt/storage/spider/spd_table.cc:6683 #6 0x000055eb2c51bb7e in ha_finalize_handlerton (plugin=0x55eb2ef56e10) at /test/10.8_opt/sql/handler.cc:599 #7 0x000055eb2c2e5f5c in plugin_deinitialize (plugin=0x55eb2ef56e10, ref_check=ref_check@entry=true) at /test/10.8_opt/sql/sql_plugin.cc:1267 #8 0x000055eb2c2ea54e in reap_plugins () at /test/10.8_opt/sql/sql_plugin.cc:1341 #9 0x000055eb2c2eb065 in plugin_shutdown () at /test/10.8_opt/sql/sql_plugin.cc:2049 #10 0x000055eb2c1c5e27 in clean_up (print_message=print_message@entry=true) at /test/10.8_opt/sql/mysqld.cc:1958 #11 0x000055eb2c1d0b6a in clean_up (print_message=true) at /test/10.8_opt/sql/mysqld.cc:5895 #12 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.8_opt/sql/mysqld.cc:5895 #13 0x0000148647cbc0b3 in __libc_start_main (main=0x55eb2c1940e0 <main(int, char**)>, argc=10, argv=0x7fff824dc078, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff824dc068) at ../csu/libc-start.c:308 #14 0x000055eb2c1c47de in _start () at /test/10.8_opt/sql/mysqld.cc:4515 10.8.1 0c5d1342ae6b5ab3256848be7a83e5c3b1f21566 (Debug) Core was generated by `/test/MD290122-mariadb-10.8.1-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'. Program terminated with signal SIGSEGV, Segmentation fault. #0 intern_plugin_lock (lex=lex@entry=0x0, rc=<optimized out>, state_mask=state_mask@entry=14) at /test/10.8_dbg/sql/sql_plugin.cc:973 973 if (pi->state & state_mask) [Current thread is 1 (Thread 0x151d857bc800 (LWP 820319))] (gdb) bt #0 intern_plugin_lock (lex=lex@entry=0x0, rc=<optimized out>, state_mask=state_mask@entry=14) at /test/10.8_dbg/sql/sql_plugin.cc:973 #1 0x000056099c1a0d0b in plugin_thdvar_init (thd=thd@entry=0x56099f856308) at /test/10.8_dbg/sql/sql_plugin.cc:3241 #2 0x000056099c1091ea in THD::init (this=this@entry=0x56099f856308) at /test/10.8_dbg/sql/sql_class.cc:1234 #3 0x000056099c1125bf in THD::THD (this=0x56099f856308, id=<optimized out>, is_wsrep_applier=<optimized out>) at /test/10.8_dbg/sql/sql_class.cc:849 #4 0x0000151d7010e296 in spider_create_thd () at /test/10.8_dbg/sql/sql_list.h:680 #5 spider_db_done (p=<optimized out>) at /test/10.8_dbg/storage/spider/spd_table.cc:6683 #6 0x000056099c489139 in ha_finalize_handlerton (plugin=0x56099f1f55a0) at /test/10.8_dbg/sql/handler.cc:599 #7 0x000056099c19cc38 in plugin_deinitialize (plugin=0x56099f1f55a0, ref_check=ref_check@entry=true) at /test/10.8_dbg/sql/sql_plugin.cc:1267 #8 0x000056099c1a1428 in reap_plugins () at /test/10.8_dbg/sql/sql_plugin.cc:1341 #9 0x000056099c1a23bd in plugin_shutdown () at /test/10.8_dbg/sql/sql_plugin.cc:2049 #10 0x000056099c0284ae in clean_up (print_message=print_message@entry=true) at /test/10.8_dbg/sql/mysqld.cc:1958 #11 0x000056099c0364c6 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.8_dbg/sql/mysqld.cc:5895 #12 0x000056099c026b56 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.8_dbg/sql/main.cc:34 10.6.6 bd03c0e51629e1c3969a171137712a6bb854c232 (Optimized) Core was generated by `/test/MD190122-mariadb-10.6.6-linux-x86_64-opt/bin/mysqld --no-defaults --core-'. Program terminated with signal SIGSEGV, Segmentation fault. #0 intern_plugin_lock (state_mask=14, rc=0x0, lex=0x0) at /test/10.6_opt/sql/sql_plugin.cc:973 [Current thread is 1 (Thread 0x1474abad1800 (LWP 1081841))] (gdb) bt #0 intern_plugin_lock (state_mask=14, rc=0x0, lex=0x0) at /test/10.6_opt/sql/sql_plugin.cc:973 #1 plugin_thdvar_init (thd=0x5641280dabc8) at /test/10.6_opt/sql/sql_plugin.cc:3243 #2 0x0000564125eba4d7 in THD::init (this=0x5641280dabc8) at /test/10.6_opt/sql/sql_class.cc:1247 #3 0x0000564125ebfdc6 in THD::THD (this=0x5641280dabc8, id=<optimized out>, is_wsrep_applier=<optimized out>) at /test/10.6_opt/sql/sql_class.cc:850 #4 0x000014747715f460 in spider_create_thd () at /test/10.6_opt/sql/sql_list.h:680 #5 spider_db_done (p=<optimized out>) at /test/10.6_opt/storage/spider/spd_table.cc:7121 #6 0x0000564126145585 in ha_finalize_handlerton (plugin=0x564127c385e0) at /test/10.6_opt/sql/handler.cc:604 #7 0x0000564125f18c7c in plugin_deinitialize (plugin=0x564127c385e0, ref_check=ref_check@entry=true) at /test/10.6_opt/sql/sql_plugin.cc:1266 #8 0x0000564125f1c68e in reap_plugins () at /test/10.6_opt/sql/sql_plugin.cc:1342 #9 0x0000564125f1d1c5 in plugin_shutdown () at /test/10.6_opt/sql/sql_plugin.cc:2050 #10 0x0000564125e338b7 in clean_up (print_message=print_message@entry=true) at /test/10.6_opt/sql/mysqld.cc:1925 #11 0x0000564125e3e732 in clean_up (print_message=true) at /test/10.6_opt/sql/mysqld.cc:5849 #12 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_opt/sql/mysqld.cc:5849 #13 0x00001474abcb00b3 in __libc_start_main (main=0x564125e01fe0 <main(int, char**)>, argc=10, argv=0x7ffcf3c69258, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcf3c69248) at ../csu/libc-start.c:308 #14 0x0000564125e3247e in _start () at /test/10.6_opt/sql/mysqld.cc:4477 Given this new more generic testcase it may indeed make sense to try and avoid this crash in 10.6 (and earlier) also.

Roel Van de Paar added a comment - 2022-01-29 04:54 - edited I have increased the priority given how; Generic the last testcase is and, That the issue happens on optimized builds, and that it is Delayed on shutdown, and that it is Sporadic at that i.e. it would be very hard for any customer running into this to find out the real source for a common setting change.

Nayuta Yanagisawa (Inactive) added a comment - 2022-02-22 08:28 I don't know why but --source include/restart_mysqld.inc results in a test failure in the Spider test suites. This makes me difficult to test.

Roel Van de Paar added a comment - 2022-02-22 20:13 nayuta-yanagisawa What is the output? Any relevant logs?

Nayuta Yanagisawa (Inactive) added a comment - 2022-02-23 06:51 - edited Roel No output from mysqltest. MTR showed server logs but I found nothing meaningful. The problem will be handled by MDEV-27912 . The Spider hangs on the server startup on 10.4+ ( MDEV-22979 ) and thus the failure was expected on 10.4+. However, even on 10.2, the test fails. This is possibly due to a different reason.

Sergei Golubchik added a comment - 2022-02-28 09:53 Likely, not a spider specific problem. Please see if it happens with a different dynamically loaded storage engine

Roel Van de Paar added a comment - 2022-03-03 01:42 - edited serg Thanks. We already had this discussion (and test) here and also note here . I also retested 10.5 and 10.6 with RocksDB to be sure. No crashes with RocksDB (whilst the sql statements are all accepted there). I also tested, in as far as such tests may be deemed reliable or not, 10.5 and 10.6 with ha_sphinx , ha_archive and ha_mroonga . None of them crashed either. Thus far only Spider has been known to crash and the testcase still crashes all versions 10.2-10.9.

Nayuta Yanagisawa (Inactive) added a comment - 2022-03-07 06:19 Roel I also guess that the crash is not due to Spider. Could you check whether it is reproducible on https://github.com/MariaDB/server/compare/bb-10.2-MDEV-27575?

Roel Van de Paar added a comment - 2022-03-07 08:58 nayuta-yanagisawa The issue does not reproduce in bb-10.2- MDEV-27575 and your patch seems to fix the crash. Not sure why it is only Spider that crashes, but nice work on the patch in generic plugin code. You rock, as usual.

Nayuta Yanagisawa (Inactive) added a comment - 2022-03-07 12:51 Roel Thank you for your confirmation! I'm not yet sure why it happens only with Spider. I will investigate the reason, but I will not pursue it too deeply since the surrounding code seems to assume that pi could be NULL. serg If you have any idea the reason, I would be grateful for your comments.

Sergei Golubchik added a comment - 2022-03-07 14:11 Server is shutting down. It needs to unload all plugins. First it tries to unload everything that's possible, Spider cannot be unloaded, because it's referenced by @@global.default_storage_engine . Then server unlocks plugins mentioned by global variables (default_storage_engine, tmp_storage_engine, enforce_storage_engine), sets those variables to NULL, and unloads more plugins that can be unloaded — namely, Spider. You can see the rest in the stack trace — spider_db_done() creates a new THD, new THD needs to lock its default_storage_engine , etc, and it does not expect that the server is almost shut down and default_storage_engine is NULL. I'd say, Spider should not create a new THD that late in the server lifecycle.

Nayuta Yanagisawa (Inactive) added a comment - 2022-03-08 04:15 Thank you very much for your analysis. The THD creation in spider_db_done() is introduced by the following commit: https://github.com/MariaDB/server/commit/ab9d420df37d76a1ff68e6fd2d5bf53a797c4102 I will try to remove that.

Roel Van de Paar added a comment - 2022-03-08 05:21 Thank you very much for your analysis++;

Nayuta Yanagisawa (Inactive) added a comment - 2022-04-03 13:39 The server raises an error when one tries to set the storage engine which has the HTON_TEMPORARY_NOT_SUPPORTED flag. However, the check is only available on 10.7+ and it is unlikely for users to set Spider as the default tmp SE. So, I will fix the bug on 10.7+ only. The above is not true. HTON_TEMPORARY_NOT_SUPPORTED exists even on 10.2. I misunderstood something.

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-20 11:18 holyfoot https://github.com/MariaDB/server/commit/03935c9bd1f57f6d72b09be4f384bac8bffbe99a

Alexey Botchkov added a comment - 2022-06-27 12:19 ok to push.

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-27 16:21 I noticed that the newly added test failed on some buildbots. Checking... http://buildbot.askmonty.org/ci/reports/cross_reference#branch=&revision=03935c9bd1f57f6d72b09be4f384bac8bffbe99a&platform=&fail_name=&fail_variant=&fail_info_full=&typ=&info=&dt=&limit=100&fail_info_short=

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-28 02:37 I believe that the failures above are related to problems that lay in Spider's plugin initialization. So, I will resume working on the present issue once MDEV-27233 has been fixed.

Roel Van de Paar added a comment - 2022-08-31 01:04 - edited During a Spider run, I just noticed that a lot of trials end in this assert (and are thus filtered), so this is blocking testing significantly.

Roel Van de Paar added a comment - 2022-10-07 02:51 Please also test with INSTALL PLUGIN Spider SONAME 'ha_spider.so' ; SET GLOBAL default_storage_engine=Spider; SELECT SLEEP (1); SHUTDOWN;

Roel Van de Paar added a comment - 2023-05-20 00:39 - edited UBSAN also report a member access within null pointer of type 'struct st_plugin_int' in sql/sql_plugin.cc for this: INSTALL PLUGIN Spider SONAME 'ha_spider.so' ; SET GLOBAL default_storage_engine=Spider; SHUTDOWN; Leads to: 11.0.2 368dd22a816f3b437bccd0b9ff28b9de9b1abf0a (Optimized, UBASAN) 2023-05-20 10:36:08 0 [Note] InnoDB: Shutdown completed; log sequence number 47139; transaction id 15 /test/11.0_opt_san/sql/sql_plugin.cc:976:11: runtime error: member access within null pointer of type 'struct st_plugin_int' #0 0x5577507300b7 in intern_plugin_lock /test/11.0_opt_san/sql/sql_plugin.cc:976 #1 0x5577507300b7 in plugin_thdvar_init(THD*) /test/11.0_opt_san/sql/sql_plugin.cc:3248 #2 0x557750329b07 in THD::init() /test/11.0_opt_san/sql/sql_class.cc:1231 #3 0x55775035da6b in THD::THD(unsigned long long, bool) /test/11.0_opt_san/sql/sql_class.cc:851 #4 0x14d0271e6d54 in spider_create_thd() /test/11.0_opt_san/storage/spider/spd_table.cc:96 #5 0x14d0271e6d54 in spider_db_done(void*) /test/11.0_opt_san/storage/spider/spd_table.cc:6012 #6 0x557751b5fae9 in ha_finalize_handlerton(st_plugin_int*) /test/11.0_opt_san/sql/handler.cc:601 #7 0x55775071c09c in plugin_deinitialize /test/11.0_opt_san/sql/sql_plugin.cc:1273 #8 0x55775071f5c5 in reap_plugins /test/11.0_opt_san/sql/sql_plugin.cc:1344 #9 0x557750727f83 in plugin_shutdown() /test/11.0_opt_san/sql/sql_plugin.cc:2055 #10 0x55774fe094c6 in clean_up /test/11.0_opt_san/sql/mysqld.cc:1999 #11 0x55774fe094c6 in clean_up /test/11.0_opt_san/sql/mysqld.cc:1962 #12 0x55774fe2d7a0 in mysqld_main(int, char**) /test/11.0_opt_san/sql/mysqld.cc:6051 #13 0x14d049c29d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #14 0x14d049c29e3f in __libc_start_main_impl ../csu/libc-start.c:392 #15 0x55774fd2cde4 in _start (/test/UBASAN_MD120523-mariadb-11.0.2-linux-x86_64-opt/bin/mariadbd+0x798ede4)   230520 10:36:10 [ERROR] mysqld got signal 11 ; Same on debug. Confirmed bug present in 10.4 and 11.0

Yuchen Pei added a comment - 2023-11-03 07:04 An initial patch, based on 10.10. Still need to check for 10.4 upstream/bb-10.10-mdev-27575 fee173053887ba3914e7575bc7d41162280972a3 MDEV-27575 Remove thd from spider_db_done   It is unused, and causing segfaults

Yuchen Pei added a comment - 2023-11-08 06:53 Hi holyfoot , ptal thanks (based on 10.10) [Revision fee173053887ba3914e7575bc7d41162280972a3] Author: Yuchen Pei <ycp@mariadb.com> Date: 2023-11-03 Fri 18:00:51 AEDT   MDEV-27575 Remove thd from spider_db_done   It is unused, and causing segfaults There's also a 10.4 version at e3141826794fea9fec771407e8c36feb12cf6f6b, where the SET GLOBAL default_tmp_storage_engine=spider; does not cause an error, presumably because of commit f7216fa63d69448c3de1532a1dd197d0f28faefd which is included in 10.7+

Alexey Botchkov added a comment - 2023-11-20 21:23 ok to push.

Yuchen Pei added a comment - 2023-11-21 01:14 Thanks for the review. Pushing 9656573376516807b41066dd5f0ff7fa316946fc to 10.4. There's no conflict when cherry-picked to higher versions, but 10.11 requires a slightly different patch (see my previous comment).