Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Fix
-
10.5
Description
SET innodb_compression_default=1; |
CREATE TEMPORARY TABLE t (c INT); |
SET GLOBAL innodb_compression_level=0; |
TRUNCATE t; |
DROP TABLE t; |
Leads to:
|
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Debug) |
10.5.27-dbg>TRUNCATE t;
|
ERROR 1030 (HY000): Got error 140 "Wrong create options" from storage engine InnoDB
|
|
|
10.5.27-dbg>DROP TABLE t;
|
ERROR 2013 (HY000): Lost connection to MySQL server during query
|
On optimized we see:
|
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Optimized) |
10.5.27-opt>TRUNCATE t;
|
ERROR 1030 (HY000): Got error 140 "Wrong create options" from storage engine InnoDB
|
|
|
10.5.27-opt>DROP TABLE t;
|
ERROR 1051 (42S02): Unknown table 'test.t'
|
Interestingly, on optimized, SHOW CREATE TABLE sees the table, DROP TABLE does not (though the DROP still happens):
|
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Optimized) |
10.5.27-opt>TRUNCATE t;
|
ERROR 1030 (HY000): Got error 140 "Wrong create options" from storage engine InnoDB
|
|
|
10.5.27-opt>SHOW CREATE TABLE t;
|
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Table | Create Table |
|
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------+
|
| t | CREATE TEMPORARY TABLE `t` (
|
`c` int(11) DEFAULT NULL
|
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci `PAGE_COMPRESSED`='ON' |
|
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------+
|
1 row in set (0.000 sec)
|
|
|
10.5.27-opt>DROP TABLE t;
|
ERROR 1051 (42S02): Unknown table 'test.t'
|
Debug assertion:
|
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Debug) |
mariadbd: /test/10.5_dbg/sql/sql_table.cc:2445: int mysql_rm_table_no_locks(THD*, TABLE_LIST*, bool, bool, bool, bool, bool, bool): Assertion `thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)' failed.
|
|
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Debug) |
Core was generated by `/test/MD130824-mariadb-10.5.27-linux-x86_64-dbg/bin/mariadbd --no-defaults --ma'.
|
Program terminated with signal SIGABRT, Aborted.
|
Download failed: Invalid argument. Continuing without source file ./nptl/./nptl/pthread_kill.c.
|
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
|
|
|
warning: 44 ./nptl/pthread_kill.c: No such file or directory
|
[Current thread is 1 (LWP 567478)]
|
(gdb) bt
|
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
|
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78
|
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89
|
#3 0x0000143067e4526e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26
|
#4 0x0000143067e288ff in __GI_abort () at ./stdlib/abort.c:79
|
#5 0x0000143067e2881b in __assert_fail_base (fmt=0x143067fd01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x5eba1369c418 "thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)", file=file@entry=0x5eba1369bb58 "/test/10.5_dbg/sql/sql_table.cc", line=line@entry=2445, function=function@entry=0x5eba1369c330 "int mysql_rm_table_no_locks(THD*, TABLE_LIST*, bool, bool, bool, bool, bool, bool)") at ./assert/assert.c:94
|
#6 0x0000143067e3b507 in __assert_fail (assertion=0x5eba1369c418 "thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)", file=0x5eba1369bb58 "/test/10.5_dbg/sql/sql_table.cc", line=2445, function=0x5eba1369c330 "int mysql_rm_table_no_locks(THD*, TABLE_LIST*, bool, bool, bool, bool, bool, bool)") at ./assert/assert.c:103
|
#7 0x00005eba12a58a95 in mysql_rm_table_no_locks (thd=thd@entry=0x143000000d58, tables=tables@entry=0x143000012f50, if_exists=if_exists@entry=false, drop_temporary=drop_temporary@entry=false, drop_view=drop_view@entry=false, drop_sequence=drop_sequence@entry=false, dont_log_query=false, dont_free_locks=false)at /test/10.5_dbg/sql/sql_table.cc:2445
|
#8 0x00005eba12a5a8fa in mysql_rm_table (thd=thd@entry=0x143000000d58, tables=tables@entry=0x143000012f50, if_exists=<optimized out>, drop_temporary=<optimized out>, drop_sequence=<optimized out>, dont_log_query=dont_log_query@entry=false)at /test/10.5_dbg/sql/sql_table.cc:2164
|
#9 0x00005eba129a58b8 in mysql_execute_command (thd=thd@entry=0x143000000d58)at /test/10.5_dbg/sql/sql_parse.cc:5079
|
#10 0x00005eba129aa479 in mysql_parse (thd=thd@entry=0x143000000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x143065b012a0, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false)at /test/10.5_dbg/sql/sql_parse.cc:8229
|
#11 0x00005eba129ac9f9 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x143000000d58, packet=packet@entry=0x14300000aaf9 "DROP TABLE t", packet_length=packet_length@entry=12, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false)at /test/10.5_dbg/sql/sql_class.h:238
|
#12 0x00005eba129af1cd in do_command (thd=0x143000000d58)at /test/10.5_dbg/sql/sql_parse.cc:1376
|
#13 0x00005eba12ae26ac in do_handle_one_connection (connect=<optimized out>, connect@entry=0x5eba15ebeef8, put_in_cache=put_in_cache@entry=true)at /test/10.5_dbg/sql/sql_connect.cc:1417
|
#14 0x00005eba12ae29d2 in handle_one_connection (arg=arg@entry=0x5eba15ebeef8)at /test/10.5_dbg/sql/sql_connect.cc:1319
|
#15 0x00005eba12f32f78 in pfs_spawn_thread (arg=0x5eba15eb3268)at /test/10.5_dbg/storage/perfschema/pfs.cc:2201
|
#16 0x0000143067e9ca94 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447
|
#17 0x0000143067f29c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
|
Bug confirmed present in:
MariaDB: 10.5.27 (dbg), 10.5.27 (opt)
Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.6.20 (dbg), 10.6.20 (opt), 10.11.10 (dbg), 10.11.10 (opt), 11.1.7 (dbg), 11.1.7 (opt), 11.2.6 (dbg), 11.2.6 (opt), 11.4.4 (dbg), 11.4.4 (opt), 11.5.2 (dbg), 11.5.2 (opt), 11.6.1 (dbg), 11.6.1 (opt)
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
MDEV-18518 Implement atomic multi-table (or multi-partition) CREATE TABLE for InnoDB
-
- Closed
-
-
-
- Closed
-
Activity
Optimized is also affected, but differently:
|
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Optimized) |
10.5.27-opt>TRUNCATE t;
|
ERROR 1030 (HY000): Got error 140 "Wrong create options" from storage engine InnoDB
|
10.5.27-opt>DROP TABLE t;
|
ERROR 1051 (42S02): Unknown table 'test.t'
|
For comparison, 10.6 opt:
|
10.6.20 2e580dc2a8da4aaf3a7f1b3cfb4f897dbb5f7089 (Optimized) |
10.6.20-opt>TRUNCATE t;
|
Query OK, 0 rows affected (0.000 sec)
|
|
|
10.6.20-opt>DROP TABLE t;
|
Query OK, 0 rows affected (0.000 sec)
|
TRUNCATE TABLE was rewritten in MariaDB Server 10.6 in order to have crash-safe and mostly atomic DDL operations. The most substantial changes were MDEV-18518 and MDEV-25506, but there were others.
Given that MariaDB Server 10.5 will reach its end of life already in about a year and the bug does not exist in later major versions, I am very reluctant to fix this.
I observe the same SEGFAULTS on 11.4.3-MariaDB-debug source revision 5ab81ffe0097a22a774957df28c5223cf0201de3 very frequently generally during running multiply mariadb-install-db instances concurrently on MBP M3Max:
2024-08-30 15:45:10 1 [Warning] Could not remove temporary table: '/var/folders/yj/8g2l7_c972s_kwvct7hqtlfw0000gn/T/#sql-temptable-13703-1-4', error: 2
|
Assertion failed: (thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)), function <unknown>, file sql_table.cc, line 1516.
|
|
|
240830 15:45:10 [ERROR] mysqld got signal 6 ;
|
Sorry, we probably made a mistake, and this is a bug.
|
|
|
Your assistance in bug reporting will enable us to fix this for the next release.
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 11.4.3-MariaDB-debug source revision: 5ab81ffe0097a22a774957df28c5223cf0201de3
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=153
|
thread_count=1
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468221 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x128812288
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x16d701940 thread_stack 0x49000
|
Printing to addr2line failed
|
0 mariadbd 0x00000001038af614 my_print_stacktrace + 64
|
0 mariadbd 0x00000001027fd834 handle_fatal_signal + 1164
|
0 libsystem_platform.dylib 0x0000000188867584 _sigtramp + 56
|
0 libsystem_pthread.dylib 0x0000000188836c20 pthread_kill + 288
|
2024-08-30 15:45:10 0 [Note] InnoDB: Data file .//undo001 did not exist: new to be created
|
2024-08-30 15:45:10 0 [Note] InnoDB: Setting file .//undo001 size to 10.000MiB
|
2024-08-30 15:45:10 0 [Note] InnoDB: Database physically writes the file full: wait...
|
0 libsystem_c.dylib 0x0000000188743a30 abort + 180
|
0 libsystem_c.dylib 0x0000000188742d20 err + 0
|
0 mariadbd 0x0000000102da5b28 _Z23mysql_rm_table_no_locksP3THDP10TABLE_LISTPK25st_mysql_const_lex_stringP16st_ddl_log_statebbbbbb + 2644
|
0 mariadbd 0x0000000102da4f74 _Z14mysql_rm_tableP3THDP10TABLE_LISTbbbb + 1196
|
0 mariadbd 0x0000000102c71c20 _Z21mysql_execute_commandP3THDb + 22516
|
0 mariadbd 0x0000000102c64728 _Z11mysql_parseP3THDPcjP12Parser_state + 1028
|
0 mariadbd 0x0000000102c63fdc _Z9bootstrapP13st_mysql_file + 1732
|
0 mariadbd 0x0000000102a608e8 _Z11mysqld_mainiPPc + 2832
|
0 mariadbd 0x00000001026fe164 main + 36
|
0 dyld 0x00000001884ae0e0 start + 2360
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x12881eaa0): DROP TABLE tmp_user_sys;
|
|
|
Connection ID (thread ID): 1
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=on,cset_narrowing=off,sargable_casefold=on
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
|
information that should help you find out what is causing the crash.
|
Core pattern: /cores/core.%P
|
|
|
scripts/mariadb-install-db: line 567: 79619 Abort trap: 6 "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K
|
sitano Hi! Thank you for your input. Interesting, we do not see the issue in 10.6+. Looking at the demangled stack (cat error.log | c++filt) and the query;
0x0000000102da5b28 mysql_rm_table_no_locks(THD*, TABLE_LIST*, st_mysql_const_lex_string const*, st_ddl_log_state*, bool, bool, bool, bool, bool, bool) + 2644
|
0x0000000102da4f74 mysql_rm_table(THD*, TABLE_LIST*, bool, bool, bool, bool) + 1196
|
0x0000000102c71c20 mysql_execute_command(THD*, bool) + 22516
|
0x0000000102c64728 mysql_parse(THD*, char*, unsigned int, Parser_state*) + 1028
|
0x0000000102c63fdc bootstrap(st_mysql_file*) + 1732
|
0x0000000102a608e8 mysqld_main(int, char**) + 2832
|
DROP TABLE tmp_user_sys; |
It indeed looks related. However, the "Could not remove temporary table" warning, which is likely related to the assert that follows, shows a somewhat different scenario. I strongly suspect the issue is not MBM specific/related.
Interestingly, while your crash is in sql/sql_table.cc on line 1516 (11.4), and mine in the same file on line 2445 (10.5), the related code is the same and has likely been moved:
else if (!drop_temporary) |
{
|
non_temp_tables_count++;
|
|
|
DBUG_ASSERT(thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str,
|
table_name.str, MDL_SHARED)); # <<< Assert location
|
|
|
alias= (lower_case_table_names == 2) ? table->alias : table_name;
|
/* remove .frm file and engine files */ |
path_length= build_table_filename(path, sizeof(path) - 1, db.str, |
alias.str, reg_ext, 0);
|
path_end= path + path_length - reg_ext_length;
|
}
|
It seems to point to a locking issue around dropping temporary tables.
Could you somehow make an SQL trace which leads to the issue please? I suspect that your issue will be very reproducible once we have an SQL trace. You can look at all things in connection with creating, updating and finally dropping tmp_user_sys and put it into a .sql trace, then test with SOURCE at the mariadb command line, on a test server. Use these command line options to ensure higher reproducibility when replaying with SOURCE:
bin/mariadb -A -uroot --force --binary-mode
|
I also checked mariadb-install-db in 11.4 for 'tmp_user_sys' but there is no such term within it. You can check it yourself if you like, it is a shell script.
Once you have a crashing SQL trace, even if sporadic, we can reduce it; just upload it to this issue.
Thank you!
Some other things;
1) I see 5ab81ffe0097a22a774957df28c5223cf0201de3 is included in Community Server and Enterprise Server, which one are you using please?
2) your comment about mariadb-install-db may be onto something. In the past I have regularly observed this indentical stack;
SIGABRT|mysql_rm_table_no_locks|mysql_rm_table|mysql_execute_command|mysql_parse
|
While mass-parallel setting up MariaDB instances accross versions. I will see if I can somehow make a testcase for that.
I also saw this stack on ES 11.4.2 on startup a number of times:
|
ES 11.4.2-1 97f04821b26e5d6660c1a6401996b8eec51b131c (Debug) |
Core was generated by `/test/EMD230724-mariadb-11.4.2-1-linux-x86_64-dbg/bin/mariadbd --no-defaults --'.
|
Program terminated with signal SIGABRT, Aborted.
|
Download failed: Invalid argument. Continuing without source file ./nptl/./nptl/pthread_kill.c.
|
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
|
|
|
[Current thread is 1 (LWP 3135)]
|
(gdb) bt
|
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
|
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78
|
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89
|
#3 0x000010c97e64526e in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26
|
#4 0x000010c97e6288ff in __GI_abort () at ./stdlib/abort.c:79
|
#5 0x000010c97e62881b in __assert_fail_base (fmt=0x10c97e7d01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x59c80a968860 "thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)", file=file@entry=0x59c80a968330 "/test/11.4-enterprise_dbg/sql/sql_table.cc", line=line@entry=1525, function=function@entry=0x59c80a968758 "int mysql_rm_table_no_locks(THD*, TABLE_LIST*, const LEX_CSTRING*, DDL_LOG_STATE*, bool, bool, bool, bool, bool, bool)") at ./assert/assert.c:94
|
#6 0x000010c97e63b507 in __assert_fail (assertion=0x59c80a968860 "thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)", file=0x59c80a968330 "/test/11.4-enterprise_dbg/sql/sql_table.cc", line=1525, function=0x59c80a968758 "int mysql_rm_table_no_locks(THD*, TABLE_LIST*, const LEX_CSTRING*, DDL_LOG_STATE*, bool, bool, bool, bool, bool, bool)")at ./assert/assert.c:103
|
#7 0x000059c809cdfa0f in mysql_rm_table_no_locks (thd=thd@entry=0x59c80db3b248, tables=tables@entry=0x59c80db63d78, current_db=current_db@entry=0x59c80db3b2e8, ddl_log_state=0x7ffe79c34550, ddl_log_state@entry=0x0, if_exists=if_exists@entry=false, drop_temporary=drop_temporary@entry=false, drop_view=false, drop_sequence=false, dont_log_query=false, dont_free_locks=false)at /test/11.4-enterprise_dbg/sql/sql_table.cc:1525
|
#8 0x000059c809ce2138 in mysql_rm_table (thd=thd@entry=0x59c80db3b248, tables=tables@entry=0x59c80db63d78, if_exists=<optimized out>, drop_temporary=<optimized out>, drop_sequence=<optimized out>, dont_log_query=dont_log_query@entry=false)at /test/11.4-enterprise_dbg/sql/sql_table.cc:1211
|
#9 0x000059c809c085f3 in mysql_execute_command (thd=thd@entry=0x59c80db3b248, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false)at /test/11.4-enterprise_dbg/sql/sql_parse.cc:4765
|
#10 0x000059c809c0caa2 in mysql_parse (thd=0x59c80db3b248, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7ffe79c352c0)at /test/11.4-enterprise_dbg/sql/sql_parse.cc:7890
|
#11 0x000059c809c0d08d in bootstrap (file=0x59c80bd249d0 <instrumented_stdin>)at /test/11.4-enterprise_dbg/sql/sql_class.h:262
|
#12 0x000059c809ad9f0c in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/11.4-enterprise_dbg/sql/mysqld.cc:5964
|
#13 0x000059c809acbd86 in main (argc=<optimized out>, argv=<optimized out>)at /test/11.4-enterprise_dbg/sql/main.cc:34
|
> Could you somehow make an SQL trace which leads to the issue please?
@Roel Van de Paar this is the trace that you asked for: mariadb.95.0.trace
. I also have a coredump but its built on Arch so it is hard to load on other systems. I used `--debug=d:A,/tmp/mariadb.{}.{}.trace`. If you need something more ping me back.
> I also checked mariadb-install-db in 11.4 for 'tmp_user_sys' but there is no such term within it. You can check it yourself if you like, it is a shell script.
mariadbd in bootstrap mode seems to execute content from scripts/mariadb_system_tables.sql.
> 1) I see 5ab81ffe0097a22a774957df28c5223cf0201de3 is included in Community Server and Enterprise Server, which one are you using please?
I use community server from github.
> 2) your comment about mariadb-install-db may be onto something. In the past I have regularly observed this indentical stack;
I don't know why I never saw it when running only a single instance of this, but not sure process parallelism is anyhow related.
The trace ends with:
35024:handle_bootstrap: query: CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv;
|
36267:close_thread_tables: tcache: table: 'global_priv' query_id: 0
|
36321:handle_bootstrap: query: INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}');
|
36741:handle_bootstrap: query: INSERT IGNORE INTO global_priv SELECT * FROM tmp_user_sys WHERE 0 <> @need_sys_user_creation;
|
38421:check_integrity: THD::enter_stage: Waiting for query cache lock at local-mariadb/artifacts/sources/mariadb-mariadb-11.4.3/sql/sql_cache.cc:692
|
38447:close_thread_tables: tcache: table: 'global_priv' query_id: 18
|
38510:handle_bootstrap: query: DROP TABLE tmp_user_sys;
|
btw, it fails on my Linux as well.
sitano Hi again! Thank you for the feedback and for confirming this is community server.
For the trace, I was looking for an SQL trace, i.e. the SQL commands executed against the MariaDB server, which led to the crash at hand. Or alternatively the commands (bash + sql, if any) you use in your Linux instance to reproduce the issue. If the issue is always on startup/init only, then the command or script used there.
The trace provided is interesting, especially this:
fn_format: enter: name: /tmp/#sql-temptable-3da6b-1-3 dir: extension: .MAI flag: 4
|
dirname_part: enter: '/tmp/#sql-temptable-3da6b-1-3'
|
dirname_part: enter: '/tmp/'
|
cleanup_dirname: enter: from: '/tmp/'
|
cleanup_dirname: exit: to: '/tmp/'
|
my_realpath: info: executing realpath
|
my_load_path: enter: path: /tmp/#sql-temptable-3da6b-1-3.MAI prefix:
|
my_load_path: exit: to: /tmp/#sql-temptable-3da6b-1-3.MAI
|
fn_format: enter: name: /tmp/#sql-temptable-3da6b-1-3 dir: extension: .MAI flag: 260
|
dirname_part: enter: '/tmp/#sql-temptable-3da6b-1-3'
|
dirname_part: enter: '/tmp/'
|
cleanup_dirname: enter: from: '/tmp/'
|
cleanup_dirname: exit: to: '/tmp/'
|
my_delete: my: name /tmp/#sql-temptable-3da6b-1-3.MAI MyFlags 16
|
my_error: my: nr: 6 MyFlags: 4 errno: 2
|
my_message_sql: error: error: 6 message: 'Error on delete of '/tmp/#sql-temptable-3da6b-1-3.MAI' (Errcode: 2 "No such file or directory")' Flag: 4
|
fn_format: enter: name: /tmp/#sql-temptable-3da6b-1-3 dir: extension: .MAD flag: 260
|
dirname_part: enter: '/tmp/#sql-temptable-3da6b-1-3'
|
dirname_part: enter: '/tmp/'
|
cleanup_dirname: enter: from: '/tmp/'
|
cleanup_dirname: exit: to: '/tmp/'
|
my_delete: my: name /tmp/#sql-temptable-3da6b-1-3.MAD MyFlags 16
|
my_error: my: nr: 6 MyFlags: 4 errno: 2
|
my_message_sql: error: error: 6 message: 'Error on delete of '/tmp/#sql-temptable-3da6b-1-3.MAD' (Errcode: 2 "No such file or directory")' Flag: 4
|
fn_format: enter: name: /tmp/#sql-temptable-3da6b-1-3 dir: extension: .TMD flag: 260
|
dirname_part: enter: '/tmp/#sql-temptable-3da6b-1-3'
|
dirname_part: enter: '/tmp/'
|
cleanup_dirname: enter: from: '/tmp/'
|
cleanup_dirname: exit: to: '/tmp/'
|
my_delete: my: name /tmp/#sql-temptable-3da6b-1-3.TMD MyFlags 0
|
print_buffer_to_file: enter: buffer: Could not remove temporary table: '/tmp/#sql-temptable-3da6b-1-3', error: 2
|
print_buffer_to_file: mutex: LOCK_error_log (0x591cc9bbdf20) locking
|
print_buffer_to_file: mutex: LOCK_error_log (0x591cc9bbdf20) locked
|
print_buffer_to_file: mutex: LOCK_error_log (0x591cc9bbdf20) unlocking
|
my_delete: my: name /tmp/#sql-temptable-3da6b-1-3.frm MyFlags 48
|
free_table_share: enter: table: mysql.tmp_user_sys
|
TABLE_SHARE::destroy: info: db: mysql table: tmp_user_sys
|
my_hash_free: enter: hash:0x76644cf5c090 elements: 0
|
plugin_unlock: mutex: LOCK_plugin (0x591cc9bc60c0) locking
|
plugin_unlock: mutex: LOCK_plugin (0x591cc9bc60c0) locked
|
my_free: my: ptr: 0x7664268f5168
|
my_free: info: global thd memory_used: 258032 size: -32
|
intern_plugin_unlock: lock: thd: 0x76643bde3b88 plugin: "Aria" UNLOCK ref_count: 2
|
plugin_unlock: mutex: LOCK_plugin (0x591cc9bc60c0) unlocking
|
free_root: enter: root: 0x7ffc67e6a6c0 flags: 0
|
my_free: my: ptr: 0x766426979c88
|
my_free: info: thd memory_used: 69720 size: -1040
|
my_free: my: ptr: 0x766426a06a88
|
my_free: info: thd memory_used: 68680 size: -2112
|
my_free: my: ptr: 0x76643b87dc88
|
my_free: info: thd memory_used: 66568 size: -1216
|
my_free: my: ptr: 0x766426979788
|
my_free: info: thd memory_used: 65352 size: -1040
|
my_free: my: ptr: 0x76644cf5c088
|
my_free: info: global thd memory_used: 258000 size: -1768
|
mysql_rm_table_no_locks: error: handling fatal signal
|
Note the final line and the temporary table errors leading up to it.
Also good catch on the tmp_user_sys in scripts/mariadb_system_tables.sql (or share/mariadb_system_tables.sql on tar builds);
CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; |
INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); |
INSERT IGNORE INTO global_priv SELECT * FROM tmp_user_sys WHERE 0 <> @need_sys_user_creation; |
DROP TABLE tmp_user_sys; |
This provides helpful context.
> For the trace, I was looking for an SQL trace, i.e. the SQL commands executed against the MariaDB server, which led to the crash at hand. Or alternatively the commands (bash + sql, if any) you use in your Linux instance to reproduce the issue. If the issue is always on startup/init only, then the command or script used there.
Basically the only thing that I am doing is running a few mariadb-install-db scripts in parallel. I will try to see on a weekend if it exactly helps to reproduce.
Ok, I was able to reproduce the issue 2 out 3 times by just running this:
[mariadb]
|
log-bin
|
server-id=100
|
binlog_format=row
|
binlog_row_image=minimal
|
innodb_autoinc_lock_mode=2
|
|
|
# This makes `show engine innodb status` output current locks, helpful for debugging locks
|
innodb-status-output-locks=on
|
and
rm -rf data{1,2,3,4} |
PATH=~/Projects/mariadb-server/build_local/install/bin:$PATH ~/Projects/mariadb-server/build_local/install/scripts/mariadb-install-db --defaults-file=./primary.cnf --datadir=./data1 >> 1.log 2>&1 & |
PATH=~/Projects/mariadb-server/build_local/install/bin:$PATH ~/Projects/mariadb-server/build_local/install/scripts/mariadb-install-db --defaults-file=./primary.cnf --datadir=./data2 >> 2.log 2>&1 & |
PATH=~/Projects/mariadb-server/build_local/install/bin:$PATH ~/Projects/mariadb-server/build_local/install/scripts/mariadb-install-db --defaults-file=./primary.cnf --datadir=./data3 >> 3.log 2>&1 & |
PATH=~/Projects/mariadb-server/build_local/install/bin:$PATH ~/Projects/mariadb-server/build_local/install/scripts/mariadb-install-db --defaults-file=./primary.cnf --datadir=./data4 >> 4.log 2>&1 & |
Even for the 1st try I have got: 4.log
sitano Excellent, and I was able to reproduce it on my end also. Thank you for your kind assistance.
I made a full looping test script as it at times takes 30 retries. No cnf file needed:
while true; do |
rm -rf data[0-9] [0-9].log |
for((i=0;i<10;i++)){ |
./scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=${PWD} --datadir=./data${i} >> ${i}.log 2>&1 & |
}
|
wait %1 %2 %3 %4 %5 %6 %7 %8 %9 %10 2>/dev/null |
if grep -qi 'mysql_rm_table_no_locks' [0-9].log; then |
break |
fi |
done
|
grep 'mysql_rm_table_no_locks' [0-9].log |
It halts as soon as the issue is detected and displays it.
Present in 10.5-11.7, debug builds, using the script in previous comment. Commonly reproduces in between 1 to 30 tries.
It is possible that, while the same assert, there are two issues at play here. marko Your thoughts? Whom should this (or these?) go to? Thank you
I tried the following single-threaded loop, but it failed to trigger the bug on my system:
while $BUILD/scripts/mariadb-install-db --builddir=$BUILD --srcdir=$SRC --defaults-file=/dev/shm/primary.cnf --datadir=/dev/shm/data |
do
|
rm -fr /dev/shm/data |
done |
I think that we would need a patch to the script in order to run the server bootstrap under rr record, and assign _RR_TRACE_DIR to point to a directory that would be removed at the end of each successful run, to conserve space.
@@ -561,7 +561,7 @@
|
mysqld_bootstrap="${MYSQLD_BOOTSTRAP-$mysqld}"
|
mysqld_install_cmd_line()
|
{
|
- "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup\
|
+ rr record "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup\
|
"--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" \
|
"--plugin-dir=${plugindir}" \
|
$args --max_allowed_packet=8M \ |
Another option could be to point the environment variable MYSQLD_BOOTSTRAP to a script like the following:
#!/bin/sh
|
exec rr record /path/to/mariadbd "$@"
|
marko Hi! The issue is deemed to be in the starting of multiple instances of mariadb-install-db at the same time. Can you please try the script as posted here? Thank you
Example run:
|
CS 11.2.6 e91a79945822def1452787f825e6047c6a64dbd9 (Debug) |
/test/MD090924-mariadb-11.2.6-linux-x86_64-dbg$ while true; do
|
rm -rf data[0-9] [0-9].log
|
for((i=0;i<10;i++)){
|
./scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=${PWD} --datadir=./data${i} >> ${i}.log 2>&1 &
|
}
|
wait %1 %2 %3 %4 %5 %6 %7 %8 %9 %10 2>/dev/null
|
if grep -qi 'mysql_rm_table_no_locks' [0-9].log; then
|
break
|
fi
|
done
|
grep 'mysql_rm_table_no_locks' [0-9].log
|
[1] 219032
|
[2] 219033
|
[3] 219034
|
[4] 219035
|
[5] 219037
|
[6] 219040
|
[7] 219042
|
[8] 219047
|
[9] 219049
|
[10] 219057
|
[1] 223819
|
[2] 223821
|
[3] 223822
|
[4] 223824
|
[5] 223826
|
[6] 223830
|
[7] 223833
|
[8] 223836
|
[9] 223845
|
[10] 223851
|
3.log:mariadbd: /test/11.2_dbg/sql/sql_table.cc:1524: int mysql_rm_table_no_locks(THD*, TABLE_LIST*, const LEX_CSTRING*, DDL_LOG_STATE*, bool, bool, bool, bool, bool, bool): Assertion `thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)' failed.
|
Roel, can you try with --external-locking to work around the logic change of MDEV-24393?
marko Yes, same outcome:
|
CS 11.2.6 e91a79945822def1452787f825e6047c6a64dbd9 (Debug) |
/test/MD090924-mariadb-11.2.6-linux-x86_64-dbg$ while true; do
|
rm -rf data[0-9] [0-9].log
|
for((i=0;i<10;i++)){
|
./scripts/mariadb-install-db --no-defaults --force --external-locking --auth-root-authentication-method=normal --basedir=${PWD} --datadir=./data${i} >> ${i}.log 2>&1 &
|
}
|
wait %1 %2 %3 %4 %5 %6 %7 %8 %9 %10 2>/dev/null
|
if grep -qi 'mysql_rm_table_no_locks' [0-9].log; then
|
break
|
fi
|
done
|
grep 'mysql_rm_table_no_locks' [0-9].log
|
[1] 1702720
|
[2] 1702721
|
[3] 1702722
|
[4] 1702724
|
[5] 1702727
|
[6] 1702730
|
[7] 1702732
|
[8] 1702737
|
[9] 1702740
|
[10] 1702751
|
[1] 1704662
|
[2] 1704663
|
[3] 1704664
|
[4] 1704665
|
[5] 1704667
|
[6] 1704669
|
[7] 1704672
|
[8] 1704675
|
[9] 1704680
|
[10] 1704686
|
5.log:mariadbd: /test/11.2_dbg/sql/sql_table.cc:1524: int mysql_rm_table_no_locks(THD*, TABLE_LIST*, const LEX_CSTRING*, DDL_LOG_STATE*, bool, bool, bool, bool, bool, bool): Assertion `thd->mdl_context.is_lock_owner(MDL_key::TABLE, db.str, table_name.str, MDL_SHARED)' failed.
|
4.log shows a crash on DROP TABLE tmp_user_sys, completely outside InnoDB:
|
scripts/mysql_system_tables.sql |
CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; |
INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); |
INSERT IGNORE INTO global_priv SELECT * FROM tmp_user_sys WHERE 0 <> @need_sys_user_creation; |
DROP TABLE tmp_user_sys; |
The cmake -DWITH_DBUG_TRACE=ON trace in mariadb.95.0.trace does not tell me anything, but hopefully it is useful to someone who can fix this non-InnoDB bug.
Would the crash be avoided if each mariadb-install-db used their own tmpdir?
Excellent. Thank you marko; setting an individual tmpdir fixes the problem.
while true; do |
rm -rf data[0-9] tmp[0-9] [0-9].log |
for((i=0;i<10;i++)){ |
mkdir tmp${i} |
}
|
for((i=0;i<10;i++)){ |
scripts/mariadb-install-db --no-defaults --force --auth-root-authentication-method=normal --basedir=${PWD} --datadir=${PWD}/data${i} --tmpdir=${PWD}/tmp${i} >> ${i}.log 2>&1 & |
}
|
wait %1 %2 %3 %4 %5 %6 %7 %8 %9 %10 2>/dev/null |
if grep -qi 'mysql_rm_table_no_locks' [0-9].log; then |
break |
fi |
done
|
grep 'mysql_rm_table_no_locks' [0-9].log |
Does not crash/end. This remind me of the days of creating reducer.sh where so many little tweaks - just like this - were required to make issues reproducible 95%+ of the time.
sitano Hi! Please see the discussion above and please check from your end also.
So while the secondary issue with mariadb-install-db on all versions looks to be resolved, the original testcase issue (10.5 only) stands as-is, and as discussed/agreed earlier in the ticket that one won't be fixed. I have removed any 10.6+ versions and will close the bug as 'Won't fix' for the original issue.
Note: even with an addition of --source include/have_innodb.inc the testcase does not crash in MTR. Please use the CLI.
Also tested:
And it similarly fails. In the CLI however we see:
10.5.27 0e27351028a4888b0da271e2089bf1f847620396 (Debug)
10.5.27-dbg>SET innodb_compression_default=1;Query OK, 0 rows affected (0.000 sec)10.5.27-dbg>CREATE TEMPORARY TABLE t (c INT);Query OK, 0 rows affected (0.001 sec)10.5.27-dbg>SET GLOBAL innodb_compression_level=0;Query OK, 0 rows affected (0.000 sec)10.5.27-dbg>TRUNCATE t;ERROR 1030 (HY000): Got error 140 "Wrong create options" from storage engine InnoDB10.5.27-dbg>DROP TABLE t;ERROR 2013 (HY000): Lost connection to MySQL server during query