[MDEV-29421] Thread (10.6+) and server hangs (10.4/10.5) in 'Opening tables' (on optimized builds) and SIGABRT in safe_mutex_lock (on debug) on I_S read when using Spider - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.8(EOL), 10.9(EOL), 10.10(EOL), 10.11, 11.1(EOL)
Fix Version/s: 10.4.33, 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4, 11.2.3
Component/s: Locking, Storage Engine - Spider
Labels:

Description

Please replace '../socket.sock' with the actual socket location.
The issue is not sporadic. On debug builds, 10.4 to 10.11, we will see the crash below.

INSTALL PLUGIN Spider SONAME 'ha_spider.so';

CREATE USER Spider@localhost IDENTIFIED BY 'PWD123';

CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET '../socket.sock', DATABASE 'test', USER 'Spider', PASSWORD 'PWD123');

CREATE TABLE t (c INT);

CREATE TABLE t_s (c INT) ENGINE=Spider COMMENT='WRAPPER "mysql", srv "srv", TABLE "t"';

SET GLOBAL table_open_cache=10;

CREATE TABLE t1 (a INT) ENGINE=Spider;

SELECT * FROM t1;

SELECT * FROM information_schema.tables;

Leads to:

10.11.0 bc563f1a4b0b38de3b41fd0f0d3d8b7f1aacbd8b (Debug)
Core was generated by `/test/MD190822-mariadb-10.11.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x145d84b68700 (LWP 4144754))]
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000145d9e0c5859 in __GI_abort () at abort.c:79
#2 0x000055ccfc7d8b62 in safe_mutex_lock (mp=mp@entry=0x145d84b17ca0 <spider_tbl_mutex>, my_flags=my_flags@entry=0, file=file@entry=0x145d84ae7110 "/test/10.11_dbg/storage/spider/spd_table.cc", line=line@entry=5602) at /test/10.11_dbg/mysys/thr_mutex.c:262
#3 0x0000145d84a623f3 in inline_mysql_mutex_lock (src_line=5602, src_file=0x145d84ae7110 "/test/10.11_dbg/storage/spider/spd_table.cc", that=0x145d84b17ca0 <spider_tbl_mutex>) at /test/10.11_dbg/include/mysql/psi/mysql_thread.h:750
#4 spider_free_share (share=0x145d3407de58) at /test/10.11_dbg/storage/spider/spd_table.cc:5602
#5 0x0000145d84a86927 in ha_spider::close (this=0x145d3407c160) at /test/10.11_dbg/storage/spider/ha_spider.cc:577
#6 0x000055ccfc09ab30 in handler::ha_close (this=0x145d3407c160) at /test/10.11_dbg/sql/handler.cc:3401
#7 0x000055ccfbec25a1 in closefrm (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table.cc:4534
#8 0x000055ccfbfeec17 in intern_close_table (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table_cache.cc:225
#9 0x000055ccfbfeed0a in tc_remove_table (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table_cache.cc:263
#10 0x000055ccfbfeee9c in tc_add_table (thd=thd@entry=0x145d34000db8, table=0x145d340dba18) at /test/10.11_dbg/sql/table_cache.cc:363
#11 0x000055ccfbd21a83 in open_table (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d84b61fe0, ot_ctx=ot_ctx@entry=0x145d84b61f00) at /test/10.11_dbg/sql/sql_base.cc:2208
#12 0x000055ccfbd24307 in open_ltable (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d84b61fe0, lock_type=TL_WRITE, lock_flags=lock_flags@entry=2075) at /test/10.11_dbg/sql/sql_base.cc:5468
#13 0x0000145d84a01b1d in spider_sys_open_table (thd=thd@entry=0x145d34000db8, tables=tables@entry=0x145d84b61fe0, open_tables_backup=open_tables_backup@entry=0x145d84b62738) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:587
#14 0x0000145d84a01db0 in spider_open_sys_table (thd=thd@entry=0x145d34000db8, table_name=<optimized out>, table_name@entry=0x145d84ae138a "spider_table_sts", table_name_length=table_name_length@entry=16, write=write@entry=true, open_tables_backup=open_tables_backup@entry=0x145d84b62738, need_lock=need_lock@entry=false, error_num=0x145d84b62734) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:263
#15 0x0000145d84a0a65e in spider_sys_insert_or_update_table_sts (thd=thd@entry=0x145d34000db8, name=0x145d34013910 "./test/t_s", name_length=10, stat=stat@entry=0x145d3411e620, need_lock=need_lock@entry=false) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:3240
#16 0x0000145d84a625c2 in spider_free_share (share=0x145d3411dde8) at /test/10.11_dbg/storage/spider/spd_table.cc:5633
#17 0x0000145d84a86927 in ha_spider::close (this=0x145d3411d230) at /test/10.11_dbg/storage/spider/ha_spider.cc:577
#18 0x000055ccfc09ab30 in handler::ha_close (this=0x145d3411d230) at /test/10.11_dbg/sql/handler.cc:3401
#19 0x000055ccfbec25a1 in closefrm (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table.cc:4534
#20 0x000055ccfbfeec17 in intern_close_table (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table_cache.cc:225
#21 0x000055ccfbfeed0a in tc_remove_table (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table_cache.cc:263
#22 0x000055ccfbfeee9c in tc_add_table (thd=thd@entry=0x145d34000db8, table=0x145d340b6a88) at /test/10.11_dbg/sql/table_cache.cc:363
#23 0x000055ccfbd21a83 in open_table (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d3419c230, ot_ctx=ot_ctx@entry=0x145d84b62fa0) at /test/10.11_dbg/sql/sql_base.cc:2208
#24 0x000055ccfbd261b5 in open_and_process_table (ot_ctx=0x145d84b62fa0, has_prelocking_list=false, prelocking_strategy=0x145d84b63038, flags=1090, counter=0x145d84b63034, tables=0x145d3419c230, thd=0x145d34000db8) at /test/10.11_dbg/sql/sql_base.cc:4106
#25 open_tables (thd=thd@entry=0x145d34000db8, options=@0x145d84b64758: {m_options = DDL_options_st::OPT_NONE}, start=start@entry=0x145d84b63028, counter=counter@entry=0x145d84b63034, flags=1090, prelocking_strategy=prelocking_strategy@entry=0x145d84b63038) at /test/10.11_dbg/sql/sql_base.cc:4593
#26 0x000055ccfbd2732d in open_tables (prelocking_strategy=0x145d84b63038, flags=<optimized out>, counter=0x145d84b63034, tables=0x145d84b63028, thd=0x145d34000db8) at /test/10.11_dbg/sql/sql_base.h:266
#27 open_normal_and_derived_tables (thd=thd@entry=0x145d34000db8, tables=<optimized out>, tables@entry=0x145d3419c230, flags=<optimized out>, dt_phases=dt_phases@entry=3) at /test/10.11_dbg/sql/sql_base.cc:5631
#28 0x000055ccfbd27420 in open_tables_only_view_structure (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d3419c230, can_deadlock=can_deadlock@entry=false) at /test/10.11_dbg/sql/sql_base.cc:5681
#29 0x000055ccfbe41f0c in fill_schema_table_by_open (thd=thd@entry=0x145d34000db8, mem_root=mem_root@entry=0x145d84b64f30, is_show_fields_or_keys=is_show_fields_or_keys@entry=false, table=table@entry=0x145d341026d0, schema_table=schema_table@entry=0x55ccfd26dc80 <schema_tables+2368>, orig_db_name=orig_db_name@entry=0x145d34111f58, orig_table_name=0x145d34112198, open_tables_state_backup=0x145d84b64f70, can_deadlock=false) at /test/10.11_dbg/sql/sql_show.cc:4718
#30 0x000055ccfbe6573e in get_all_tables (thd=0x145d34000db8, tables=0x145d340142d8, cond=<optimized out>) at /test/10.11_dbg/sql/sql_show.cc:5389
#31 0x000055ccfbe66cd0 in get_schema_tables_result (join=join@entry=0x145d34015e28, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC) at /test/10.11_dbg/sql/sql_show.cc:8943
#32 0x000055ccfbe3b1ed in JOIN::exec_inner (this=this@entry=0x145d34015e28) at /test/10.11_dbg/sql/sql_string.h:687
#33 0x000055ccfbe3be28 in JOIN::exec (this=this@entry=0x145d34015e28) at /test/10.11_dbg/sql/sql_select.cc:4590
#34 0x000055ccfbe39bac in mysql_select (thd=thd@entry=0x145d34000db8, tables=0x145d340142d8, fields=@0x145d34013f88: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x145d34014278, last = 0x145d3410f658, elements = 23}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2701396736, result=0x145d34015e00, unit=0x145d34004ff0, select_lex=0x145d34013ce8) at /test/10.11_dbg/sql/sql_select.cc:5070
#35 0x000055ccfbe3a3a2 in handle_select (thd=thd@entry=0x145d34000db8, lex=lex@entry=0x145d34004f18, result=result@entry=0x145d34015e00, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.11_dbg/sql/sql_select.cc:581
#36 0x000055ccfbda45a6 in execute_sqlcom_select (thd=thd@entry=0x145d34000db8, all_tables=0x145d340142d8) at /test/10.11_dbg/sql/sql_parse.cc:6261
#37 0x000055ccfbdb08c7 in mysql_execute_command (thd=thd@entry=0x145d34000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.11_dbg/sql/sql_parse.cc:3945
#38 0x000055ccfbd9e882 in mysql_parse (thd=thd@entry=0x145d34000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x145d84b67330) at /test/10.11_dbg/sql/sql_parse.cc:8035
#39 0x000055ccfbdabe6a in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x145d34000db8, packet=packet@entry=0x145d3400b6e9 "SELECT * FROM information_schema.tables", packet_length=packet_length@entry=39, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_class.h:1339
#40 0x000055ccfbdae574 in do_command (thd=0x145d34000db8, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_parse.cc:1407
#41 0x000055ccfbf101da in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55ccfedb1628, put_in_cache=put_in_cache@entry=true) at /test/10.11_dbg/sql/sql_connect.cc:1418
#42 0x000055ccfbf106e3 in handle_one_connection (arg=0x55ccfedb1628) at /test/10.11_dbg/sql/sql_connect.cc:1312
#43 0x0000145d9e5d6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#44 0x0000145d9e1c2133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Optimized builds will thread-hang with:

10.10.2 87e8463e0454a04c2bbaa38d44227c491fb07dc1 (Optimized)
MariaDB [test]> SHOW PROCESSLIST;
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
\| Id \| User \| Host \| db \| Command \| Time \| State \| Info \| Progress \|
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
\| 4 \| root \| localhost \| test \| Query \| 1289 \| Opening tables \| SELECT * FROM information_schema.tables \| 0.000 \|
\| 5 \| system user \| \| NULL \| Sleep \| 1289 \| Reset for next command \| NULL \| 0.000 \|
\| 6 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 7 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 8 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 11 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 12 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 14 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 15 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 16 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 18 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 9 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 10 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 13 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 17 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background statistics action handler \| NULL \| 0.000 \|
\| 19 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 20 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 21 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 22 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 23 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 24 \| system user \| \| NULL \| Daemon \| NULL \| Spider table background cardinality action handler \| NULL \| 0.000 \|
\| 25 \| Spider \| localhost \| test \| Sleep \| 1288 \| \| NULL \| 0.000 \|
\| 26 \| root \| localhost \| test \| Query \| 0 \| starting \| SHOW PROCESSLIST \| 0.000 \|
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
23 rows in set (0.001 sec)

A shutdown is still possible for 10.6 to 10.11 in this case. However, 10.4 and 10.5 will fully hang during shutdown.

Bug confirmed present in:
MariaDB: 10.4.27 (dbg), 10.4.27 (opt), 10.5.18 (dbg), 10.5.18 (opt), 10.6.10 (dbg), 10.6.10 (opt), 10.7.6 (dbg), 10.7.6 (opt), 10.8.5 (dbg), 10.8.5 (opt), 10.9.2 (dbg), 10.9.2 (opt), 10.10.2 (dbg), 10.10.2 (opt), 10.11.0 (dbg), 10.11.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.38 (dbg), 5.7.38 (opt), 8.0.29 (dbg), 8.0.29 (opt)

SIGABRT|safe_mutex_lock|inline_mysql_mutex_lock|spider_free_share|ha_spider::close

Attachments

Issue Links

is blocked by

MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache

Closed

is part of

MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache

Closed

relates to

MDEV-32807 Delete spider sts and crd system tables

Open

Activity

Transition	Time In Source Status	Execution Times

Ramesh Sivaraman made transition - 2022-08-31 09:13

Open

Confirmed

29m 58s

Nayuta Yanagisawa (Inactive) made transition - 2022-08-31 11:36

Confirmed

In Progress

2h 22m

Nayuta Yanagisawa (Inactive) made transition - 2022-09-06 13:45

In Progress

In Review

6d 2h 9m

Alexey Botchkov made transition - 2022-09-27 12:07

In Review

Stalled

20d 22h 22m

Nayuta Yanagisawa (Inactive) made transition - 2022-11-29 23:31

In Progress

Stalled

63d 10h 12m

Yuchen Pei made transition - 2023-11-09 06:37

Stalled

In Progress

344d 8h 17m

Yuchen Pei made transition - 2023-11-24 06:13

In Progress

Closed

14d 23h 36m

People

Assignee:: Yuchen Pei

Reporter:: Roel Van de Paar

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2022-08-31 08:43

Updated:: 2023-11-24 06:13

Resolved:: 2023-11-24 06:13

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration