|
sanja, who should be assigned to this?
|
|
According to traces it is something spider waits.
Can be something in the server, but first analyzis shoud be done in spider.
|
|
sanja, Spider create threads at db_init() and these threads wait finishing starting up server using LOCK_server_started and COND_server_started, because Spider uses Aria tables. But this case, I think spider_db_done() is called before finishing start up, right?
Should Spider wait starting up using other way?
|
|
I have no idea, better ask our engine interface specialists, serg, psergei ?
|
|
I have no clue.
these threads wait finishing starting up server using LOCK_server_started and COND_server_started, because Spider uses Aria tables.
MyRocks doesn't have anything like this so I haven't encountered this issue.
|
|
Kentoku, mysqld_server_started is never set during a bootstrap. At all.
But perhaps you simply don't need to wait, because Aria is a mandatory engine and it's always started before Spider.
If you keep the wait, it would be a good idea to have a way to interrupt it — this is not only a bootstrap problem, if the server encounters an error during the startup it'll also try to shutdown without ever setting mysqld_server_started=1.
Why do you need to lock thread->mutex over the whole long server startup wait?
|
|
serg, I tried to it without waiting, Spider got an error by uninitialized Aria. And also Spider system tables can be changed to other engines.
Anyway, I'll fix it soon.
|
|
Hi GeoffMontee, I fixed and pushed. 71e8183 Would you please check it?
|
|
I will check the fix. However, shouldn't it go to 10.4? It is affected as well. I've updated affects/fix versions.
|
|
Yes, it should be fixed 10.4 too.
|
|
Still hangs on 10.5 baeeb9ba45 which includes the revision mentioned above.
|
10.5 baeeb9ba45
|
$ bin/mysqld --help --verbose | grep revision
|
version-source-revision baeeb9ba4575ae1e8770b09f0e3648943e75db7f
|
$ rm -rf data/*
|
$ scripts/mysql_install_db --no-defaults --plugin-load-add=ha_spider
|
Installing MariaDB/MySQL system tables in './data' ...
|
2020-06-29 16:20:06 0 [Warning] Plugin 'SPIDER' is of maturity level gamma while the server is stable
|
|
<hangs here>
|
Thread 1 (Thread 0x7f12801f2f40 (LWP 10751)):
|
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
|
#1 0x00007f127fddece6 in __GI___pthread_mutex_lock (mutex=0x564809b22e08) at ../nptl/pthread_mutex_lock.c:135
|
#2 0x0000564806f151df in safe_mutex_lock (mp=0x564809b22de0, my_flags=0, file=0x7f127c479bf8 "/data/src/10.5/storage/spider/spd_table.cc", line=10164) at /data/src/10.5/mysys/thr_mutex.c:290
|
#3 0x00007f127c381f2b in inline_mysql_mutex_lock (that=0x564809b22de0, src_file=0x7f127c479bf8 "/data/src/10.5/storage/spider/spd_table.cc", src_line=10164) at /data/src/10.5/include/mysql/psi/mysql_thread.h:763
|
#4 0x00007f127c3b59e4 in spider_free_sts_threads (spider_thread=0x564809b22d88) at /data/src/10.5/storage/spider/spd_table.cc:10164
|
#5 0x00007f127c3abd31 in spider_db_done (p=0x0) at /data/src/10.5/storage/spider/spd_table.cc:7025
|
#6 0x00005648065ce244 in ha_finalize_handlerton (plugin=0x564809afe910) at /data/src/10.5/sql/handler.cc:571
|
#7 0x000056480629041a in plugin_deinitialize (plugin=0x564809afe910, ref_check=true) at /data/src/10.5/sql/sql_plugin.cc:1262
|
#8 0x0000564806290978 in reap_plugins () at /data/src/10.5/sql/sql_plugin.cc:1338
|
#9 0x0000564806292e21 in plugin_shutdown () at /data/src/10.5/sql/sql_plugin.cc:2019
|
#10 0x000056480612b5e3 in clean_up (print_message=false) at /data/src/10.5/sql/mysqld.cc:1984
|
#11 0x000056480612b38b in unireg_abort (exit_code=0) at /data/src/10.5/sql/mysqld.cc:1906
|
#12 0x0000564806132e6e in mysqld_main (argc=14, argv=0x564809ab7238) at /data/src/10.5/sql/mysqld.cc:5584
|
#13 0x0000564806127d10 in main (argc=13, argv=0x7ffc19453dd8) at /data/src/10.5/sql/main.cc:25
|
|
|
elenst, sorry, I just fixed again and pushed it to bb-10.5-MDEV-22979 tree. Would you please check it?
|
|
Please remember it should go to 10.4.
|
|
elenst, I just pushed it to bb-10.4-MDEV-22979.
|
|
Sorry I forgot to add a comment. It doesn't seem to hang anymore on bb-10.4-MDEV-22979, but you of course know this, it's very easy to verify. I suppose the main check here is a review.
|
|
Kentoku, how does your fix work? You set thread->init_command, but how do you interrupt a condition wait?
You've changed it to a timed wait in 10.5, but this second fix is for 10.4.
|
|
this issue still seems to exist in 10.5.9
|
|
Let me pick this up. The fixes proposed by Kentoku are the following:
https://github.com/MariaDB/server/commit/71e8183e412d7e935fe20d98b82486f15c9fdcee
https://github.com/MariaDB/server/commit/22a0097727ffc5d893ef56fe4470a5c61985feb8
https://github.com/MariaDB/server/commit/75b3589aef35a9b9dbc56042457291755e513560
|
|
I have confirmed that the bug still reproduces on 10.4, 10.5, and 10.6. Servers built from different branches hang in different places but anyway they all hang.
10.4 865e5b6405d163c5591b400689717af80e99427f
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
0x00007f3f6d092110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
|
(gdb) bt
|
#0 0x00007f3f6d092110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#1 0x00007f3f6d08a235 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#2 0x000055e276bdbaed in safe_mutex_lock (mp=0x55e278f12b28, my_flags=0, file=0x7f3f5ffb1900 "/home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc", line=9886)
|
at /home/vagrant/repo/mariadb-server/mysys/thr_mutex.c:293
|
#3 0x00007f3f5feae450 in inline_mysql_mutex_lock (that=0x55e278f12b28, src_file=0x7f3f5ffb1900 "/home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc", src_line=9886)
|
at /home/vagrant/repo/mariadb-server/include/mysql/psi/mysql_thread.h:717
|
#4 0x00007f3f5fee036b in spider_free_sts_threads (spider_thread=0x55e278f12ad0) at /home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc:9886
|
#5 0x00007f3f5fed681f in spider_db_done (p=0x0) at /home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc:6707
|
#6 0x000055e27631de3f in ha_finalize_handlerton (plugin=0x55e278ef2688) at /home/vagrant/repo/mariadb-server/sql/handler.cc:522
|
#7 0x000055e275fc598f in plugin_deinitialize (plugin=0x55e278ef2688, ref_check=true) at /home/vagrant/repo/mariadb-server/sql/sql_plugin.cc:1241
|
#8 0x000055e275fc5f56 in reap_plugins () at /home/vagrant/repo/mariadb-server/sql/sql_plugin.cc:1317
|
#9 0x000055e275fc845d in plugin_shutdown () at /home/vagrant/repo/mariadb-server/sql/sql_plugin.cc:2013
|
#10 0x000055e275e61293 in clean_up (print_message=false) at /home/vagrant/repo/mariadb-server/sql/mysqld.cc:1966
|
#11 0x000055e275e60fe8 in unireg_abort (exit_code=0) at /home/vagrant/repo/mariadb-server/sql/mysqld.cc:1879
|
#12 0x000055e275e695f5 in mysqld_main (argc=12, argv=0x55e278eabbe8) at /home/vagrant/repo/mariadb-server/sql/mysqld.cc:5819
|
#13 0x000055e275e5d6bd in main (argc=10, argv=0x7ffe0a6dcac8) at /home/vagrant/repo/mariadb-server/sql/main.cc:25
|
10.5 10f08aff159f916358cf9902fc5f6a976b8faeda
(gdb) bt
|
#0 0x00007f0d6a166110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#1 0x00007f0d6a15e235 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#2 0x000055deffb84689 in safe_mutex_lock (mp=0x55df0245f860, my_flags=0, file=0x7f0d5d702278 "/home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc", line=6906)
|
at /home/vagrant/repo/mariadb-server/mysys/thr_mutex.c:290
|
#3 0x00007f0d5d5f4520 in inline_mysql_mutex_lock (that=0x55df0245f860, src_file=0x7f0d5d702278 "/home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc", src_line=6906)
|
at /home/vagrant/repo/mariadb-server/include/mysql/psi/mysql_thread.h:765
|
#4 0x00007f0d5d61df44 in spider_create_handler (hton=0x55df02454f08, table=0x0, mem_root=0x55df02446688) at /home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc:6906
|
#5 0x000055deff259eab in get_new_handler (share=0x0, alloc=0x55df02446688, db_type=0x55df02454f08) at /home/vagrant/repo/mariadb-server/sql/handler.cc:375
|
#6 0x000055deff25a9ab in hton_drop_table (hton=0x55df02454f08, path=0x7ffe477feef0 "./mysql/ndb_binlog_index") at /home/vagrant/repo/mariadb-server/sql/handler.cc:560
|
#7 0x000055deff26023d in ha_delete_table (thd=0x55df02440a08, hton=0x55df02454f08, path=0x7ffe477feef0 "./mysql/ndb_binlog_index", db=0x7ffe477fed70, alias=0x7ffe477fed80,
|
generate_warning=false) at /home/vagrant/repo/mariadb-server/sql/handler.cc:2784
|
#8 0x000055deff268c0b in delete_table_force (thd=0x55df02440a08, plugin=0x55df02ff5618, arg=0x7ffe477fec60) at /home/vagrant/repo/mariadb-server/sql/handler.cc:5041
|
#9 0x000055defef09515 in plugin_foreach_with_mask (thd=0x55df02440a08, func=0x55deff268b5b <delete_table_force(THD*, plugin_ref, void*)>, type=1, state_mask=8, arg=0x7ffe477fec60)
|
at /home/vagrant/repo/mariadb-server/sql/sql_plugin.cc:2505
|
#10 0x000055deff268d7d in ha_delete_table_force (thd=0x55df02440a08, path=0x7ffe477feef0 "./mysql/ndb_binlog_index", db=0x7ffe477fed70, alias=0x7ffe477fed80)
|
at /home/vagrant/repo/mariadb-server/sql/handler.cc:5087
|
#11 0x000055defefdea62 in mysql_rm_table_no_locks (thd=0x55df02440a08, tables=0x55df02f24608, if_exists=true, drop_temporary=false, drop_view=false, drop_sequence=false,
|
dont_log_query=false, dont_free_locks=false) at /home/vagrant/repo/mariadb-server/sql/sql_table.cc:2576
|
#12 0x000055defefdd6e6 in mysql_rm_table (thd=0x55df02440a08, tables=0x55df02f24608, if_exists=true, drop_temporary=false, drop_sequence=false, dont_log_query=false)
|
at /home/vagrant/repo/mariadb-server/sql/sql_table.cc:2139
|
#13 0x000055defeeefe90 in mysql_execute_command (thd=0x55df02440a08) at /home/vagrant/repo/mariadb-server/sql/sql_parse.cc:5008
|
#14 0x000055defeef9fb4 in mysql_parse (thd=0x55df02440a08, rawbuf=0x55df02f244f0 "drop table if exists mysql.ndb_binlog_index;", length=44, parser_state=0x7ffe477ffa40,
|
is_com_multi=false, is_next_command=false) at /home/vagrant/repo/mariadb-server/sql/sql_parse.cc:8100
|
#15 0x000055defeee3b07 in bootstrap (file=0x55df0111aa10 <instrumented_stdin>) at /home/vagrant/repo/mariadb-server/sql/sql_parse.cc:1087
|
#16 0x000055defed9dd22 in mysqld_main (argc=15, argv=0x55df023f12d8) at /home/vagrant/repo/mariadb-server/sql/mysqld.cc:5574
|
#17 0x000055defed9299d in main (argc=10, argv=0x7ffe47804be8) at /home/vagrant/repo/mariadb-server/sql/main.cc:25
|
10.6 ee39757f3c91e04a0ccbb5424fba7dd56167ad93
(gdb) bt
|
#0 0x00007fd38b65b110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#1 0x00007fd38b653235 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#2 0x000055f069f6b689 in safe_mutex_lock (mp=0x55f06d281860, my_flags=0, file=0x7fd389649278 "/home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc", line=6906)
|
at /home/vagrant/repo/mariadb-server/mysys/thr_mutex.c:290
|
#3 0x00007fd38953b520 in inline_mysql_mutex_lock (that=0x55f06d281860, src_file=0x7fd389649278 "/home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc", src_line=6906)
|
at /home/vagrant/repo/mariadb-server/include/mysql/psi/mysql_thread.h:765
|
#4 0x00007fd389564f44 in spider_create_handler (hton=0x55f06d276f08, table=0x0, mem_root=0x55f06d268688) at /home/vagrant/repo/mariadb-server/storage/spider/spd_table.cc:6906
|
#5 0x000055f069640eab in get_new_handler (share=0x0, alloc=0x55f06d268688, db_type=0x55f06d276f08) at /home/vagrant/repo/mariadb-server/sql/handler.cc:375
|
#6 0x000055f0696419ab in hton_drop_table (hton=0x55f06d276f08, path=0x7ffe7b720fb0 "./mysql/ndb_binlog_index") at /home/vagrant/repo/mariadb-server/sql/handler.cc:560
|
#7 0x000055f06964723d in ha_delete_table (thd=0x55f06d262a08, hton=0x55f06d276f08, path=0x7ffe7b720fb0 "./mysql/ndb_binlog_index", db=0x7ffe7b720e30, alias=0x7ffe7b720e40,
|
generate_warning=false) at /home/vagrant/repo/mariadb-server/sql/handler.cc:2784
|
#8 0x000055f06964fc0b in delete_table_force (thd=0x55f06d262a08, plugin=0x55f06de17618, arg=0x7ffe7b720d20) at /home/vagrant/repo/mariadb-server/sql/handler.cc:5041
|
#9 0x000055f0692f0515 in plugin_foreach_with_mask (thd=0x55f06d262a08, func=0x55f06964fb5b <delete_table_force(THD*, plugin_ref, void*)>, type=1, state_mask=8, arg=0x7ffe7b720d20)
|
at /home/vagrant/repo/mariadb-server/sql/sql_plugin.cc:2505
|
#10 0x000055f06964fd7d in ha_delete_table_force (thd=0x55f06d262a08, path=0x7ffe7b720fb0 "./mysql/ndb_binlog_index", db=0x7ffe7b720e30, alias=0x7ffe7b720e40)
|
at /home/vagrant/repo/mariadb-server/sql/handler.cc:5087
|
#11 0x000055f0693c5a62 in mysql_rm_table_no_locks (thd=0x55f06d262a08, tables=0x55f06dd46608, if_exists=true, drop_temporary=false, drop_view=false, drop_sequence=false,
|
dont_log_query=false, dont_free_locks=false) at /home/vagrant/repo/mariadb-server/sql/sql_table.cc:2576
|
#12 0x000055f0693c46e6 in mysql_rm_table (thd=0x55f06d262a08, tables=0x55f06dd46608, if_exists=true, drop_temporary=false, drop_sequence=false, dont_log_query=false)
|
at /home/vagrant/repo/mariadb-server/sql/sql_table.cc:2139
|
#13 0x000055f0692d6e90 in mysql_execute_command (thd=0x55f06d262a08) at /home/vagrant/repo/mariadb-server/sql/sql_parse.cc:5008
|
#14 0x000055f0692e0fb4 in mysql_parse (thd=0x55f06d262a08, rawbuf=0x55f06dd464f0 "drop table if exists mysql.ndb_binlog_index;", length=44, parser_state=0x7ffe7b721b00,
|
is_com_multi=false, is_next_command=false) at /home/vagrant/repo/mariadb-server/sql/sql_parse.cc:8100
|
#15 0x000055f0692cab07 in bootstrap (file=0x55f06b501a10 <instrumented_stdin>) at /home/vagrant/repo/mariadb-server/sql/sql_parse.cc:1087
|
#16 0x000055f069184d22 in mysqld_main (argc=15, argv=0x55f06d2132d8) at /home/vagrant/repo/mariadb-server/sql/mysqld.cc:5574
|
#17 0x000055f06917999d in main (argc=10, argv=0x7ffe7b726ca8) at /home/vagrant/repo/mariadb-server/sql/main.cc:25
|
|
|
On 10.4, the bug seems to be introduced by https://github.com/MariaDB/server/commit/b428b09997d172f29fc201b9ab05c160ef4cbc39.
However, even before the commit, the Spider plugin initialization fails due to the following error:
[ERROR] Unknown database 'mysql'
|
2021-12-24 23:55:26 0 [ERROR] Plugin 'SPIDER' init function returned error.
|
2021-12-24 23:55:26 0 [ERROR] Plugin 'SPIDER' registration as a STORAGE ENGINE failed.
|
|
|
I applied the unmerged patch for MDEV-27233. Then, the hang is resolved but the server crashes...
|
|
Server version: 10.4.23-MariaDB-debug
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=153
|
thread_count=20
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467873 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f980c0032f0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f981d13acf8 thread_stack 0x49000
|
addr2line: './bld/sql/mysqld': No such file
|
./bld/sql/mysqld(my_print_stacktrace+0x44)[0x556bb35a6bb8]
|
./bld/sql/mysqld(handle_fatal_signal+0x3d7)[0x556bb2cbcd3f]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x141f0)[0x7f981fc141f0]
|
addr2line: './bld/sql/mysqld': No such file
|
./bld/sql/mysqld(+0xddbf96)[0x556bb2f15f96]
|
./bld/sql/mysqld(_Z24ha_maria_implicit_commitP3THDb+0x3c)[0x556bb2cbf879]
|
./bld/sql/mysqld(_Z21mysql_execute_commandP3THD+0x9edb)[0x556bb2986a60]
|
./bld/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2c0)[0x556bb298b7c4]
|
./bld/sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x10e1)[0x556bb2977aca]
|
spider/spd_table.cc:10034(spider_table_bg_sts_action(void*))[0x7f981d234395]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9450)[0x7f981fc09450]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f981f7a0d53]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f980c016aa8): create table if not exists mysql.spider_xa( format_id int not null default 0, gtrid_length int not null default 0, bqual_length int not null de
|
fault 0, data char(128) charset binary not null default '', status char(8) not null default '', primary key (data, format_id, gtrid_length), key idx1 (status)) engine
|
=MyISAM default charset=utf8 collate=utf8_bin
|
|
Connection ID (thread ID): 1
|
Status: NOT_KILLED
|
|
|
I will resume working on the issue once MDEV-27233 is fixed.
|
|
The same bug can be reproduced in 10.9 with the following Dockerfile:
FROM mariadb:10.9
|
ENV DEBIAN_FRONTEND=noninteractive
|
RUN apt-get update && \
|
apt-get -y install unixodbc odbc-postgresql odbc-mariadb mariadb-plugin-spider && \
|
apt -y clean
|
Build the image with:
docker build -t testing-spider
|
and start it:
$ docker run -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=Y -ti --rm testing-spider
|
2022-10-10 09:48:23+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.9.3+maria~ubu2204 started.
|
2022-10-10 09:48:24+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
|
2022-10-10 09:48:24+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.9.3+maria~ubu2204 started.
|
2022-10-10 09:48:24+00:00 [Note] [Entrypoint]: Initializing database files
|
|
#
|
# This is where I did a "kill -11" on the main mariadbd process.
|
#
|
|
^C221010 9:49:52 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.9.3-MariaDB-1:10.9.3+maria~ubu2204
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=153
|
thread_count=21
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468006 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x5642d04a5ce8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7ffc47426bb8 thread_stack 0x49000
|
Printing to addr2line failed
|
/usr/sbin/mariadbd(my_print_stacktrace+0x32)[0x5642cdce6342]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x478)[0x5642cd7b78e8]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f17d5442520]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x91340)[0x7f17d5491340]
|
/lib/x86_64-linux-gnu/libc.so.6(pthread_mutex_lock+0x277)[0x7f17d54981e7]
|
/usr/lib/mysql/plugin/ha_spider.so(_Z21spider_create_handlerP10handlertonP11TABLE_SHAREP11st_mem_root+0xb5)[0x7f17d4f13085]
|
/usr/sbin/mariadbd(_Z15get_new_handlerP11TABLE_SHAREP11st_mem_rootP10handlerton+0x2e)[0x5642cd7ba0fe]
|
/usr/sbin/mariadbd(+0xa5072b)[0x5642cd7bc72b]
|
/usr/sbin/mariadbd(_Z15ha_delete_tableP3THDP10handlertonPKcPK25st_mysql_const_lex_stringS7_b+0xc9)[0x5642cd7c3069]
|
/usr/sbin/mariadbd(+0xa57353)[0x5642cd7c3353]
|
/usr/sbin/mariadbd(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1a9)[0x5642cd57fb99]
|
/usr/sbin/mariadbd(_Z21ha_delete_table_forceP3THDPKcPK25st_mysql_const_lex_stringS5_+0xc4)[0x5642cd7c19a4]
|
/usr/sbin/mariadbd(_Z23mysql_rm_table_no_locksP3THDP10TABLE_LISTPK25st_mysql_const_lex_stringP16st_ddl_log_statebbbbbb+0x11a5)[0x5642cd60d255]
|
/usr/sbin/mariadbd(_Z14mysql_rm_tableP3THDP10TABLE_LISTbbbb+0x1a4)[0x5642cd60e664]
|
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x188b)[0x5642cd5640ab]
|
/usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x5642cd568697]
|
/usr/sbin/mariadbd(_Z9bootstrapP13st_mysql_file+0x22f)[0x5642cd568abf]
|
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0xb77)[0x5642cd45c0b7]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f17d5429d90]
|
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f17d5429e40]
|
/usr/sbin/mariadbd(_start+0x25)[0x5642cd450395]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x5642d04c7570): drop table if exists mysql.ndb_binlog_index;
|
|
Connection ID (thread ID): 21
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes unlimited unlimited processes
|
Max open files 1024 1024 files
|
Max locked memory 8388608 8388608 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 127100 127100 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
|
|
Kernel version: Linux version 5.19.14-200.fc36.x86_64 (mockbuild@bkernel02.iad2.fedoraproject.org) (gcc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2), GNU ld version 2.37-36.fc36) #1 SMP PREEMPT_DYNAMIC Wed Oct 5 21:31:17 UTC 2022
|
|
|
To reprod after a build (assuming previously was on src dir and currently in build dir), simply run
rm -rf /tmp/${PWD##*/}-datadir && scripts/mysql_install_db --no-defaults --srcdir=$OLDPWD --builddir=$PWD --datadir=/tmp/${PWD##*/}-datadir --plugin-dir=${PWD}/storage/spider/ --plugin-load-add=ha_spider
|
|
I could reprod this in 11.0 8e55d7ea4a2f94ae3f38fdd8785778612d4b1203 too.
As traces in previous indicate, the statement that spider is allergic to is drop table.
A simple way to reprod this bug while pwd is the build dir would be:
mkdir -p /tmp/data && ./sql/mariadbd --bootstrap --datadir=/tmp/data --plugin-dir=./storage/spider --plugin-load-add=ha_spider <<< "drop table if exists foo.bar;"
|
In mtr one could use $MYSQLD_BOOTSTRAP_CMD, but I am not sure how debuggable this would be:
--echo MDEV-22979
|
--exec $MYSQLD_BOOTSTRAP_CMD --plugin-load-add=ha_spider < /tmp/bs
|
where tmp/bs has the drop table statement.
drop table if exists foo.bar;
|
|
|
Assuming 1) reading from stdin is equivalent to -init-file and 2) the bug is equivalent with or without -bootstrap, we have the following mtr testcase
|
mdev_22979.test
|
--echo MDEV-22979
|
|
mdev_22979.opt
|
--plugin-load-add=ha_spider
|
--init-file=$MYSQL_TEST_DIR/../storage/spider/mysql-test/spider/bugfix/t/mdev_22979.sql
|
|
mdev_22979.sql
|
drop table if exists foo.bar;
|
It is the same as MDEV-27233 except the init query is different (CREATE TABLE t (c INT) ENGINE=SPIDER; instead of drop table if exists foo.bar;).
Also, removing 10.4 from affected/fixversions as it has been fixed for 10.4 in MDEV-30370.
|
|
Looks like I have to be more careful about choosing what testcase to debug, as running mariadbd --bootstrap and running mtr with a .opt and a .sql file are NOT equivalent, as the following experiment[1] shows.
[1] https://github.com/MariaDB/server/commit/5fb5286efd9
If we break spider by removing the wait for init queries in spider_create_handler(), and devise three tests:
# mdev_22979.test
|
--echo MDEV-22979 with --plugin-load-add in opt
|
# mdev_22979.opt
|
--plugin-load-add=ha_spider
|
--init-file=$MYSQL_TEST_DIR/../storage/spider/mysql-test/spider/bugfix/t/mdev_22979.sql
|
# mdev_22979.sql
|
drop table if exists foo.bar;
|
# mdev_22979_2.test
|
--echo MDEV-22979 with install soname ha_spider in sql
|
# mdev_22979_2.opt
|
--init-file=$MYSQL_TEST_DIR/../storage/spider/mysql-test/spider/bugfix/t/mdev_22979.sql
|
# mdev_22979_2.sql
|
install soname ha_spider;
|
drop table if exists foo.bar;
|
# mdev_22979_bootstrap.test
|
--echo MDEV-22979 with --bootstrap
|
|
let $MYSQLD_DATADIR= `select @@datadir`;
|
let $PLUGIN_DIR=`select @@plugin_dir`;
|
--source include/kill_mysqld.inc
|
--exec $MYSQLD_CMD --datadir=$MYSQLD_DATADIR --bootstrap --plugin-dir=$PLUGIN_DIR --plugin-load-add=ha_spider < $MYSQL_TEST_DIR/../storage/spider/mysql-test/spider/bugfix/t/mdev_22979_bootstrap.sql
|
--source include/start_mysqld.inc
|
# mdev_22979_bootstrap.sql
|
drop table if exists foo.bar;
|
The three tests have different results:
- mdev_22979: pass with failed post-test check
- mdev_22979_2: pass
- mdev_22979_bootstrap: ERROR: 12524 Can't open system table mysql.spider_table_sts
Only mdev_22979_bootstrap has the same result as running mariadbd directly:
mkdir -p /tmp/data && ./sql/mariadbd --bootstrap --datadir=/tmp/data --plugin-dir=./storage/spider --plugin-load-add=ha_spider <<< "drop table if exists foo.bar;"
So before figuring out why they have different results, I guess the moral of the story is to always use --exec with the actual command when writing tests for bugs involving --bootstrap. For debugging, just do directly rr ./sql/mariadbd ....
|
|
Hi holyfoot, PTAL thanks
https://github.com/MariaDB/server/commit/02212b6723a
This commit is based on 10.9, as it uses sql service (10.7+) and 10.7 is EOL and 10.8 EOL soon. Once I get an approval for this commit, I will either use pre-sql-service mysql_real_query() to issue the queries for 10.4-10.6, or wait for MDEV-27595 that backports sql serivce.
The patch is also based on the commits for MDEV-27095.
The idea is described in [1], see also analysis at [2] for further rationale. It fixes both the present ticket and MDEV-27233. As such, I am marking the present ticket blocking MDEV-27233. Because it defers the spider init to after the bootstrap, mysqld --bootstrap or mysql_install_db with plugin-load-add=ha_spider will not install spider, but subsequent server startups will.
[1] https://jira.mariadb.org/browse/MDEV-27095?focusedCommentId=257580&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-257580
[2] https://jira.mariadb.org/browse/MDEV-27095?focusedCommentId=257540&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-257540
|
|
Following up on a discussion with holyfoot, I took another look into my patch and I still think it is a better solution compared to some alternatives, because
1. The execution of the temp init file is right before where the normal init file is executed, so there's less chance of things breaking down
2. The deferment to temp init file uses existing mechanisms for loading plugin in a simple and minimal way. Granted it does not exactly go through the lines in plugin_init() where plugins from plugin-load-add is initialised, but the alternative of replacing read_init_file(temp_init_file) with lines copied from plugin_init() makes it harder to maintain, because now we have two snippets of code doing the same thing. The better approach would be to refactor plugin_init() to extract a routine that is called both by plugin_init() for non-deferred plugins and mysqld_main() for deferred plugins. But that is a bigger and more involved change, and I am not sure if it is doable in a reasonable span of time, or worth the trouble compared to the temp init file solution
Any thoughts holyfoot?
|
|
In a related note to a comment made by serg in MDEV-30576[1], I wonder what we can do with the hardcoded reference to spider in the patch. A natural idea is to add something to the plugin declaration instructing the server to defer the initialisation. The declaration seems to go into constructing a st_maria_plugin, but I do not see any existing field in that struct that can be used for our purpose. In the st_plugin_int which is used in plugin_init(), there's the field load_option which seems the natural candidate. However, we still need to extend maria_declare_plugin and st_maria_plugin, so that st_maria_plugin has an load_option field that is used to override the field of the same name in st_plugin_int.
I am not sure if this is a good idea. For example, st_maria_plugin has not been changed for 10+ years and there must be a reason, right.
What are your thoughts on this holyfoot? Is there a third way other than keeping the hardcoded string or the idea above?
[1] https://jira.mariadb.org/browse/MDEV-30576?focusedCommentId=259410&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-259410
|
|
Correct, there should be no hardcoded reference to spider in the server
|
|
serg OK, then do you think my idea in the previous comment[1] that avoids such hardcode is a reasonable one, or do you have other suggestions w.r.t. telling the server to defer the initialisation of a plugin?
[1] https://jira.mariadb.org/browse/MDEV-22979?focusedCommentId=259519&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-259519
|
|
Given the problem with fully synchronous initialisation mentioned above, and following up on discussions with holyfoot, I had a new idea that adopts a mixed approach, preferencing synchronous initialisation when possible, but still using a background thread if necessary. It avoids hardcoded reference to spider, and involves less dramatic changes to the server. On the flip side it is more complex than the fully synchronous solution. More detailed description and analysis can be found in commit message:
https://github.com/MariaDB/server/commit/cbeab05a145f6894daac8e9695e11378d6b1021b
Can you take a look holyfoot? If this is a better idea then I'll work on polishing the patch.
|
|
is the problem, that you need to create spider tables after Aria is initialized but before init-file/bootstrap scripts?
we can finally implement plugin startup dependencies (using the same approach we use for shutdown dependencies).
MDEV-19850 can also help, but it's more tricky to get right
|
|
serg, that is not the only problem. If you make some quick changes to the init order in plugin_init() so that spider is the last to initialise you'll run into other unsatisfied requirements, including udf_init() (needed for creating udfs) and the existence of mysql database (and perhaps more).
udf_init() is called in mysqld_main() before the bootstrap, and mysql database is created during bootstrap, thus the earliest safe point to initialise spider is after bootstrap. Given spider tables can be created in --init_file files, an initialisation that is requested early (with plugin-load-add) should happen before the call to read_init_file():
|
10.9 d8997f875e2d78300999876e25d348cf6ad3f73e mysqld_main()
|
if (init_server_components()) // <---- loads plugins, including ones from --plugin-load-add
|
unireg_abort(1);
|
|
... // we omit many lines here
|
|
udf_init();
|
|
... // we omit many lines here
|
|
if (opt_bootstrap)
|
{
|
select_thread_in_use= 0; // Allow 'kill' to work
|
int bootstrap_error= bootstrap(mysql_stdin);
|
if (!abort_loop)
|
unireg_abort(bootstrap_error);
|
else
|
{
|
sleep(2); // Wait for kill
|
exit(0);
|
}
|
}
|
// <------------------- can only init spider after this
|
|
/* Copy default global rpl_filter to global_rpl_filter */
|
copy_filter_setting(global_rpl_filter, get_or_create_rpl_filter("", 0));
|
|
/*
|
init_slave() must be called after the thread keys are created.
|
Some parts of the code (e.g. SHOW STATUS LIKE 'slave_running' and other
|
places) assume that active_mi != 0, so let's fail if it's 0 (out of
|
memory); a message has already been printed.
|
*/
|
if (init_slave() && !active_mi)
|
{
|
unireg_abort(1);
|
}
|
|
// <------------------- init spider before this if it has already been requested
|
if (opt_init_file && *opt_init_file)
|
{
|
if (read_init_file(opt_init_file))
|
unireg_abort(1);
|
}
|
Both of my current solutions involve deferring the spider initialisation requested in inig_server_components() to just before the call to read_init_file(): the fully synchronous* solution calls read_init_file() on a temporary file with INSTALL ... SONAME "ha_spider" followed by DELETE statements to emulate plugin-load-add; whereas the mixed solution broadcast a conditional variable called COND_server_query_ready which the spider background init thread waits on to proceed with the initialisation.
The mixed solution also flips a bit called mysqld_server_query_ready just before the mysqld_main() call to read_init_file(), so that if spider init is requested after this point (e.g. with an INSTALL statement), it will initialise synchronously without a background thread.
* Note that I may have been using the word "synchronous" in a different way from Nayuta. By "synchronous" I simply mean doing things without a background thread.
|
|
Still, same two solutions could help:
- plugin startup dependencies would be sufficient, you don't need to wait for udf_init() if you will directly insert into mysql.func instead of using CREATE FUNCTION.
- MDEV-19850 alone will also be sufficient, because it'll run your startup code after udf_init().
|
|
Hi serg,
> * plugin startup dependencies would be sufficient, you don't need to wait for udf_init() if you will directly insert into mysql.func instead of using CREATE FUNCTION.
I tried this idea by updating the init queries by inserting into mysql.func. Turns out this is not sufficient to emulate CREATE FUNCTION, which also inserts the function into the udf registry udf_hash, among other things. Because of this if we just insert into mysql.func, when calling the udf, we get something like ER_SP_DOES_NOT_EXIST (1305): 'FUNCTION test.SPIDER_DIRECT_SQL does not exist'. I suspect emulating udf creation will require a lot more (much more than emulating plugin-load-add with an INSTALL followed by a DELETE), not to mention the pre-bootstrap mysql.func non-existence issue, and so IMO this option is not the right path to fix the spider init bugs
> * MDEV-19850 alone will also be sufficient, because it'll run your startup code after udf_init().
I read the description and comments of the ticket, and it is not clear to me: how and when would the server run the init queries, when spider is being loaded with plugin-load-add? If it does it before the bootstrap step in mysqld_main() it will cause the same issues. So I suppose the server will somehow understand that it needs to run some scripts to run after the bootstrap step, but how does server know about this? Without knowing sepcifics of MDEV-19850, it sounds to me that it is similar the idea in https://jira.mariadb.org/browse/MDEV-22979?focusedCommentId=259519&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-259519
Also, do you have any opinion about my second proposal above at https://jira.mariadb.org/browse/MDEV-22979?focusedCommentId=259770&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-259770?
And more generally what do you suggest we proceed with these init bugs, would you like to continue with the current review (holyfoot has been working on it), or block this with MDEV-19850?
|
|
right, CREATE FUNCTION updates the table and in-memory data structures. When they exist. But if you run your sql before udf_init, there's nothing in memory to update, so you simply insert rows into mysql.func and then later udf_init will read them from there and create proper in-memory objects for everything. Perhaps we could fix CREATE FUNCTION to work before udf_init by letting it skip in-memory changes. It'd be easier to use in other scriptlets too.
as for MDEV-19850, I didn't really give it much thought. Supposedly, it can run plugin scriptlets after the server is completely up but before it starts listening for incoming connections.
Still, plugin dependencies is by far the easiest approach. "Add a flag" is kind of like a manual dependency flag which is rather spider specific, if two plugins are using it the ordering problem comes back.
|
|
Hi serg,
> right, CREATE FUNCTION updates the table and in-memory data structures. When they exist. But if you run your sql before udf_init, there's nothing in memory to update, so you simply insert rows into mysql.func and then later udf_init will read them from there and create proper in-memory objects for everything.
OK you are right on this. I didn't realise that in my previous experiment the failure was caused by the row insertion after udf_init(). I have a poc commit that shows how a combination of initialising spider last + inserting into mysql.func could work: https://github.com/MariaDB/server/commit/4fc8dea5efa.
> Perhaps we could fix CREATE FUNCTION to work before udf_init by letting it skip in-memory changes. It'd be easier to use in other scriptlets too.
I have created MDEV-31401 for this.
> Still, plugin dependencies is by far the easiest approach.
I don't see how it is the easiest, but the outcome may be better than the complexity introduced by initialising in a background thread. I have opened MDEV-31400 for this. I marked it as causing the present issue and I'm not sure I should mark it as blocking because 1. the init bugs are critical and I think the patch with the mixed approach under review are reasonable enough at least as a temporary measure and 2. I don't see an obvious way of doing MDEV-31400, so it will need some design first and may take a while to finish. Maybe holyfoot can chip in the discussion.
What is your idea of implementing the plugin dependencies? Shall we extend maria_declare_plugin and st_maria_plugin with such information?
> "Add a flag" is kind of like a manual dependency flag which is rather spider specific, if two plugins are using it the ordering problem comes back.
I do not think this flag is spider specific, because it simply means the server is ready to take init queries. What do you mean the ordering problem comes back if two plugins use it? Another plugin that would rely on this flag to initialise may depend on the builtin plugins, but it probably will not depend on spider, or be depended upon by spider, so there's no problem.
|
|
> I have created MDEV-31401 for this.
right. without it a plugin would need something like "if udf_init was already run, use CREATE FUNCTION otherwise use INSERT" logic. Which is totally doable, but I think we'd better avoid forcing this complexity on sql scriptlet writers.
> What is your idea of implementing the plugin dependencies? Shall we extend maria_declare_plugin and st_maria_plugin with such information?
That's what I always thought it'll be and this might be the reason why we don't have plugin dependencies yet. It was never exactly clear how to specify these dependencies in a way that's robust and generic, but not too complex. It'll need building a dependency graph to resolve initialization order, cycle detection, etc.
But in my comments above I meant something else. I've just realized that we already have plugin shutdown dependencies, in a way. The server has a list of plugins to deinitialize, it walks the list and deinitializes plugins one by one, then it walks the list again, deinitializing those plugins that failed to deinitialize earlier. And so on, until nothing succeeded to deinitialize. We can easily do the same with initialization.
|
|
> But in my comments above I meant something else. I've just realized that we already have plugin shutdown dependencies, in a way. The server has a list of plugins to deinitialize, it walks the list and deinitializes plugins one by one, then it walks the list again, deinitializing those plugins that failed to deinitialize earlier. And so on, until nothing succeeded to deinitialize. We can easily do the same with initialization.
This sounds simpler indeed - I will give it a try. The only spider issue it will not fix is bootstrapping the server with plugin_load_add=ha_spider. The plugin init query requires mysql database to exist to create its system tables, which is yet to be created by the bootstrap script. It is a minor issue as the failure/error has a reasonable explanation and only happens once (during bootstrap). Note that I have not considered upgrading and I assume we can test upgrading after fixing these init bugs first.
|
|
See plugin_shutdown(), "reap" code.
|
|
serg, given your actions on the ticket yesterday I assume you would like MDEV-31400 to block the present ticket and you want to be the reviewer of both tickets.
|
|
Hi holyfoot, could you please continue with the review? The commits are based on the patch for MDEV-31400 but have not diverged much from the ones you have been reviewing. Thank you
For more context see the review on MDEV-31400 at https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thread/IQKT52MKQCNQXODD2GV7GKEEVEFYYP6H/
|
|
Discussed on Slack about udf_initizlied / initialized variables.
otherwise looks ok to push.
|
|
Thanks for the review holyfoot. I updated my commit accordingly: https://github.com/MariaDB/server/commit/74afa221807
OTOH, serg suggested[1] another approach where instead of making initialized a global variable, we use a declare handler for creating udf functions. I tried out the suggestion and it seems to work. Could you review the updated patch[2]? The only difference compared to the the patch you already approved are:
- No more changes in any sql/* files
- Updated the queries in spd_init_query.h
- Simplified logic in spider_init_system_tables()
[1] https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/message/QG3URVTXYZBVJJVSDEU5NHJJNKE6IYAX/
[2] https://github.com/MariaDB/server/commit/189962d816c
|
|
ok to push.
|
|
Thanks for the review holyfoot. Will push this once the blocking
issue MDEV-31400 is done.
|
|
The commit for the blocking issue MDEV-31400 has been pushed to 10.4.
Now because of the SQL service, we need to wait for either the change
to bubble to 10.9 or (better yet) holyfoot to fix MDEV-27595 before
pushing the commit for this ticket.
|
|
The push is blocked by the unstable test spider/bugfix.mdev_27240:
https://buildbot.mariadb.org/#/builders/221/builds/26142
https://buildbot.mariadb.org/#/builders/576/builds/3104
so I'm marking MDEV-32046 as blocking this issue.
|
|
Pushed the following to 10.10
ef14d6d6a42 @ upstream/bb-10.10-mdev-22979 upstream/10.10 MDEV-32046 Adding ER_NET_READ_ERROR to spider/bugfix.mdev_27240
|
a60cf9c7ae4 MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs
|
c6ba81d6bf6 MDEV-27095 clean up spd_init_query.h
|
fc2548c862b MDEV-27095 installing one spider plugin should not trigger others
|
Re-opened MDEV-29870 for 10.4-10.6
|