[MDEV-21329] InnoDB: Failing assertion: lock->lock_word.load(std::memory_order_relaxed) == X_LOCK_DECR upon server shutdown Created: 2019-12-16  Updated: 2020-11-02  Resolved: 2020-10-08

Status: Closed
Project: MariaDB Server
Component/s: Full-text Search, Storage Engine - InnoDB, Tests
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: 10.2.35, 10.3.26, 10.4.16, 10.5.7

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Thirunarayanan Balathandayuthapani
Resolution: Fixed Votes: 0
Labels: rr-profile

Issue Links:
Relates
relates to MDEV-23637 binlog_encryption.encrypted_master f... Closed
relates to MDEV-24090 Optimize buf_page_optimistic_get() Open
relates to MDEV-23693 Failing assertion: my_atomic_load32_e... Closed

 Description   

10.4 2cff807d

 
2020-01-03 14:39:50 0x7f9534216700  InnoDB: Assertion failure in file /data/src/10.4/storage/innobase/sync/sync0rw.cc line 255
 
InnoDB: Failing assertion: lock->lock_word.load(std::memory_order_relaxed) == X_LOCK_DECR
 
 
 
Query (0x7f94c0011d68): ALTER TABLE t1 CHANGE IF EXISTS b a INT, ALGORITHM=COPY /* QNO 169 CON_ID 12 */
 
Connection ID (thread ID): 12
 
Status: KILL_SERVER


#6  0x00005653d0678a6a in ut_dbg_assertion_failed (expr=0x5653d0e03f18 "lock->lock_word.load(std::memory_order_relaxed) == X_LOCK_DECR", file=0x5653d0e03e90 "/data/src/10.4/storage/innobase/sync/sync0rw.cc", line=255) at /data/src/10.4/storage/innobase/ut/ut0dbg.cc:60
 
#7  0x00005653d061c65f in rw_lock_free_func (lock=0x7f94c00b1e18) at /data/src/10.4/storage/innobase/sync/sync0rw.cc:255
 
#8  0x00005653d07d3a7e in pfs_rw_lock_free_func (lock=0x7f94c00b1e18) at /data/src/10.4/storage/innobase/include/sync0rw.ic:617
 
#9  0x00005653d07d7d89 in fts_cache_destroy (cache=0x7f94c00b1e18) at /data/src/10.4/storage/innobase/fts/fts0fts.cc:287
 
#10 0x00005653d07e2fb4 in fts_t::~fts_t (this=0x7f94c00844d8, __in_chrg=<optimized out>) at /data/src/10.4/storage/innobase/fts/fts0fts.cc:5349
 
#11 0x00005653d07e3094 in fts_free (table=0x7f94c043d0a8) at /data/src/10.4/storage/innobase/fts/fts0fts.cc:5386
 
#12 0x00005653d059a3e4 in row_drop_ancillary_fts_tables (table=0x7f94c043d0a8, trx=0x7f95218e4830) at /data/src/10.4/storage/innobase/row/row0mysql.cc:3248
 
#13 0x00005653d059b7fb in row_drop_table_for_mysql (name=0x7f9534210e80 "test/#sql-73c7_c", trx=0x7f95218e4830, sqlcom=SQLCOM_ALTER_TABLE, create_failed=false, nonatomic=true) at /data/src/10.4/storage/innobase/row/row0mysql.cc:3728
 
#14 0x00005653d0413245 in ha_innobase::delete_table (this=0x7f94c0013a10, name=0x7f94c00c7058 "./test/#sql-73c7_c", sqlcom=SQLCOM_ALTER_TABLE) at /data/src/10.4/storage/innobase/handler/ha_innodb.cc:13184
 
#15 0x00005653d03fc780 in ha_innobase::delete_table (this=0x7f94c0013a10, name=0x7f94c00c7058 "./test/#sql-73c7_c") at /data/src/10.4/storage/innobase/handler/ha_innodb.cc:13309
 
#16 0x00005653d01aed5c in handler::ha_delete_table (this=0x7f94c0013a10, name=0x7f94c00c7058 "./test/#sql-73c7_c") at /data/src/10.4/sql/handler.cc:4695
 
#17 0x00005653d00d3783 in THD::rm_temporary_table (this=0x7f94c0000af0, base=0x5653d284a5c0, path=0x7f94c00c7058 "./test/#sql-73c7_c") at /data/src/10.4/sql/temporary_tables.cc:701
 
#18 0x00005653d00d5663 in THD::free_tmp_table_share (this=0x7f94c0000af0, share=0x7f94c00c6ac0, delete_table=true) at /data/src/10.4/sql/temporary_tables.cc:1464
 
#19 0x00005653d00d35bb in THD::drop_temporary_table (this=0x7f94c0000af0, table=0x7f94c043aa80, is_trans=0x0, delete_table=true) at /data/src/10.4/sql/temporary_tables.cc:664
 
#20 0x00005653cff3810b in mysql_alter_table (thd=0x7f94c0000af0, new_db=0x7f94c00052b0, new_name=0x7f94c00056b8, create_info=0x7f9534213d40, table_list=0x7f94c0011eb0, alter_info=0x7f9534213c80, order_num=0, order=0x0, ignore=false) at /data/src/10.4/sql/sql_table.cc:10470
 
#21 0x00005653cffcf0cf in Sql_cmd_alter_table::execute (this=0x7f94c00126a8, thd=0x7f94c0000af0) at /data/src/10.4/sql/sql_alter.cc:508
 
#22 0x00005653cfe4db75 in mysql_execute_command (thd=0x7f94c0000af0) at /data/src/10.4/sql/sql_parse.cc:6102
 
#23 0x00005653cfe53237 in mysql_parse (thd=0x7f94c0000af0, rawbuf=0x7f94c0011d68 "ALTER TABLE t1 CHANGE IF EXISTS b a INT, ALGORITHM=COPY /* QNO 169 CON_ID 12 */", length=79, parser_state=0x7f95342151b0, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:7901
 
#24 0x00005653cfe3e3dc in dispatch_command (command=COM_QUERY, thd=0x7f94c0000af0, packet=0x7f94c037ebf1 "ALTER TABLE t1 CHANGE IF EXISTS b a INT, ALGORITHM=COPY /* QNO 169 CON_ID 12 */ ", packet_length=80, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1842
 
#25 0x00005653cfe3ca69 in do_command (thd=0x7f94c0000af0) at /data/src/10.4/sql/sql_parse.cc:1360
 
#26 0x00005653cffc59df in do_handle_one_connection (connect=0x5653d3484930) at /data/src/10.4/sql/sql_connect.cc:1412
 
#27 0x00005653cffc572e in handle_one_connection (arg=0x5653d3484930) at /data/src/10.4/sql/sql_connect.cc:1316
 
#28 0x00007f95391754a4 in start_thread (arg=0x7f9534216700) at pthread_create.c:456
 
#29 0x00007f95372a9d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Reproducible on 10.2-10.4.
Both debug and non-debug are affected.
Couldn't reproduce on 10.1 and 10.5.

To reproduce:

git clone https://github.com/MariaDB/randgen --branch mdev21329 rqg-mdev21329
 
cd rqg-mdev21329
 
perl runall-new.pl --basedir=<your basedir>  --grammar=mdev21329.yy --duration=600 --threads=2 --scenario=Restart --skip-gendata --seed=1 --scenario-restart-interval=20 --vardir=/dev/shm/vardir-mdev21329

Remember to set the path to the basedir on the command line.

The test runs the following two statements concurrently on a table with fulltext index:

    INSERT INTO t1 () VALUES () |
 
    ALTER TABLE t1 CHANGE IF EXISTS <field> <field> INT, ALGORITHM=COPY;

and restarts the server every 20 seconds.

Currently it takes a few minutes to fail on my machine, but it can vary on different machines and builds.

Variations of the assertion representation:

10.2 ef1e488b

 
2020-01-03 14:49:39 0x7f8f59057700  InnoDB: Assertion failure in file /data/src/10.2/storage/innobase/sync/sync0rw.cc line 259
 
InnoDB: Failing assertion: lock->lock_word == X_LOCK_DECR

10.3 02e30069

 
2020-01-03 14:48:58 0x7fbc04232700  InnoDB: Assertion failure in file /data/src/10.3/storage/innobase/sync/sync0rw.cc line 258
 
InnoDB: Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR



 Comments   
Comment by Elena Stepanova [ 2020-06-19 ]

MTR test seem to be failing on the same assertion:
http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos5-x86/builds/13719

10.2 e835881c474ec1c43ab3c24a9d4a6e26

 
innodb_zip.create_options '16k,innodb'   w1 [ fail ]
 
        Test ended at 2020-06-11 15:26:51
 
 
 
CURRENT_TEST: innodb_zip.create_options
 
mysqltest: At line 354: query 'DROP TABLE t1' failed: 2013: Lost connection to MySQL server during query
 
 
 
The result from queries just before the failure was:
 
< snip >
 
Warning	1478	InnoDB: assuming ROW_FORMAT=DYNAMIC.
 
SHOW WARNINGS;
 
Level	Code	Message
 
Warning	1478	InnoDB: assuming ROW_FORMAT=DYNAMIC.
 
SELECT TABLE_NAME,ROW_FORMAT,CREATE_OPTIONS FROM information_schema.tables WHERE TABLE_NAME = 't1';
 
TABLE_NAME	ROW_FORMAT	CREATE_OPTIONS
 
t1	Dynamic	row_format=FIXED
 
# Test 10) StrictMode=OFF, CREATE with each ROW_FORMAT & a valid KEY_BLOCK_SIZE
 
#          KEY_BLOCK_SIZE is ignored with COMPACT, REDUNDANT, & DYNAMIC
 
DROP TABLE t1;
 
CREATE TABLE t1 ( i INT ) ROW_FORMAT=COMPACT KEY_BLOCK_SIZE=1;
 
Warnings:
 
Warning	1478	InnoDB: ignoring KEY_BLOCK_SIZE=1 unless ROW_FORMAT=COMPRESSED.
 
SHOW WARNINGS;
 
Level	Code	Message
 
Warning	1478	InnoDB: ignoring KEY_BLOCK_SIZE=1 unless ROW_FORMAT=COMPRESSED.
 
SELECT TABLE_NAME,ROW_FORMAT,CREATE_OPTIONS FROM information_schema.tables WHERE TABLE_NAME = 't1';
 
TABLE_NAME	ROW_FORMAT	CREATE_OPTIONS
 
t1	Compact	row_format=COMPACT key_block_size=1
 
DROP TABLE t1;
 
 
 
More results from queries before failure can be found in /usr/local/mariadb-10.2.33-linux-i686/mysql-test/var/1/log/create_options.log
 
 
 
 
 
Server [mysqld.1 - pid: 16564, winpid: 16564, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
2020-06-11 15:26:51 0xb06ffb70  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc line 259
 
InnoDB: Failing assertion: lock->lock_word == X_LOCK_DECR
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
200611 15:26:51 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.33-MariaDB-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=2
 
max_threads=153
 
thread_count=8
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61844 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0xa53048d8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0xb06ff314 thread_stack 0x49000
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(my_print_stacktrace+0x2d)[0x8a9ce2d]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(handle_fatal_signal+0x624)[0x84370e4]
 
addr2line: '': No such file
 
[0xdc6400]
 
/lib/libc.so.6(abort+0x17a)[0x42c3ca]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x87f678b]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x87b50cf]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x8880b6f]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88a4c3a]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x882f6dc]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88327b0]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x886a843]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x886c5c4]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x886cc2f]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x8851051]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88da593]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88e066d]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88e25cc]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88e3855]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x87f27f5]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x87d3888]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x88166aa]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x879391f]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x8794886]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x8795e41]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x872b224]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x872be94]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x8762101]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x869a86a]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x868eedd]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z15ha_delete_tableP3THDP10handlertonPKcS4_S4_b+0x116)[0x843f2b6]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z23mysql_rm_table_no_locksP3THDP10TABLE_LISTbbbbb+0x5a0)[0x82cad90]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z14mysql_rm_tableP3THDP10TABLE_LISTcc+0x17b)[0x82d23fb]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1da7)[0x82281b7]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x33f)[0x8231d2f]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x19a9)[0x8234079]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z10do_commandP3THD+0x129)[0x8236199]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x273)[0x8328a93]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld(handle_one_connection+0x3b)[0x8328c8b]
 
/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld[0x8654e88]
 
pthread_create.c:0(start_thread)[0x685a09]
 
/lib/libc.so.6(clone+0x5e)[0x4dd43e]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0xa5312e40): DROP TABLE t1
 
Connection ID (thread ID): 38
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /dev/shm/var_auto_ksyx/1/mysqld.1/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             1024                 unlimited            processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       47566                47566                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
 
 
----------SERVER LOG END-------------
 
 
 
 
 
 - found 'core.16565' (0/0)
 
 
 
Trying 'dbx' to get a backtrace
 
 
 
Trying 'gdb' to get a backtrace from coredump /usr/local/mariadb-10.2.33-linux-i686/mysql-test/var/1/log/innodb_zip.create_options-16k,innodb/mysqld.1/data/core.16565
 
Core generated by '/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
[New Thread 16617]
 
[New Thread 16569]
 
[New Thread 17000]
 
[New Thread 16582]
 
[New Thread 16583]
 
[New Thread 16584]
 
[New Thread 16585]
 
[New Thread 16595]
 
[New Thread 16586]
 
[New Thread 16588]
 
[New Thread 16599]
 
[New Thread 16587]
 
[New Thread 16593]
 
[New Thread 16606]
 
[New Thread 16591]
 
[New Thread 16607]
 
[New Thread 16605]
 
[New Thread 16594]
 
[New Thread 16615]
 
[New Thread 16596]
 
[New Thread 16616]
 
[New Thread 16598]
 
[New Thread 16604]
 
[New Thread 16597]
 
[New Thread 16592]
 
[New Thread 16590]
 
[New Thread 16581]
 
[New Thread 16565]
 
[Thread debugging using libthread_db enabled]
 
Core was generated by `/usr/local/mariadb-10.2.33-linux-i686/bin/mysqld --defaults-group-suffix=.1 --d'.
 
Program terminated with signal 6, Aborted.
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068a403 in pthread_kill () from /lib/libpthread.so.0
 
#2  0x08a9cdeb in my_write_core (sig=6) at /home/buildbot/buildbot/build/mysys/stacktrace.c:477
 
#3  0x08437087 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:343
 
#4  <signal handler called>
 
#5  0x00dc6424 in __kernel_vsyscall ()
 
#6  0x0042aaf1 in raise () from /lib/libc.so.6
 
#7  0x0042c3ca in abort () from /lib/libc.so.6
 
#8  0x087f678b in ut_dbg_assertion_failed (expr=0x8c83888 "lock->lock_word == X_LOCK_DECR", file=0x8c83848 "/home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc", line=259) at /home/buildbot/buildbot/build/storage/innobase/ut/ut0dbg.cc:60
 
#9  0x087b50cf in rw_lock_free_func (lock=0xa53731a0) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:259
 
#10 0x08880b6f in pfs_rw_lock_free_func (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:603
 
#11 dict_table_stats_latch_free (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/dict/dict0dict.cc:300
 
#12 dict_table_stats_latch_destroy (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/dict/dict0dict.cc:337
 
#13 0x088a4c3a in dict_mem_table_free (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/dict/dict0mem.cc:205
 
#14 0x0882f6dc in btr_search_lazy_free (index=0xa53516f0) at /home/buildbot/buildbot/build/storage/innobase/btr/btr0sea.cc:265
 
#15 0x088327b0 in btr_search_drop_page_hash_index (block=0xb0aa1cd0) at /home/buildbot/buildbot/build/storage/innobase/btr/btr0sea.cc:1272
 
#16 0x0886a843 in buf_LRU_free_page (bpage=0xb0aa1cd0, zip=true) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:1616
 
#17 0x0886c5c4 in buf_LRU_free_from_common_LRU_list (buf_pool=0xb998210, scan_all=false) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:698
 
#18 buf_LRU_scan_and_free_block (buf_pool=0xb998210, scan_all=false) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:740
 
#19 0x0886cc2f in buf_LRU_get_free_block (buf_pool=0xb998210) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:971
 
#20 0x08851051 in buf_page_create (page_id=..., page_size=..., mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0buf.cc:5593
 
#21 0x088da593 in fsp_page_create (space=<value optimized out>, offset=<value optimized out>, page_size=..., rw_latch=RW_SX_LATCH, mtr=0xb06f978c, init_mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:1309
 
#22 0x088e066d in fsp_alloc_free_page (space=0xba45b18, page_size=..., hint=<value optimized out>, rw_latch=RW_SX_LATCH, mtr=0xb06f978c, init_mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:1455
 
#23 0x088e25cc in fseg_alloc_free_page_low (space=0xba45b18, page_size=..., seg_inode=0xb1295832 "", hint=0, direction=111 'o', rw_latch=RW_SX_LATCH, mtr=0xb06f978c, init_mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:2519
 
#24 0x088e3855 in fseg_create_general (space_id=0, page=0, byte_offset=60, has_done_reservation=1, mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:2105
 
#25 0x087f27f5 in trx_undo_create (trx=0xb12cb93c, rseg=0xba7f248, undo=0xb12cc558, type=2) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0undo.cc:451
 
#26 trx_undo_assign_undo (trx=0xb12cb93c, rseg=0xba7f248, undo=0xb12cc558, type=2) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0undo.cc:1610
 
#27 0x087d3888 in trx_undo_report_row_operation (thr=0xa5366560, index=0xba6c268, clust_entry=0xa53adf80, update=0x0, cmpl_info=0, rec=0xb10adda9 "", offsets=0xb06faa1c, roll_ptr=0xb06fa290) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0rec.cc:2073
 
#28 0x088166aa in btr_cur_del_mark_set_clust_rec (block=0xb0aaaa30, rec=0xb10adda9 "", index=0xba6c268, offsets=0xb06faa1c, thr=0xa5366560, entry=0xa53adf80, mtr=0xb06fa364) at /home/buildbot/buildbot/build/storage/innobase/btr/btr0cur.cc:4865
 
#29 0x0879391f in row_upd_del_mark_clust_rec (node=0xa5365538, thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:2997
 
#30 row_upd_clust_step (node=0xa5365538, thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:3169
 
#31 0x08794886 in row_upd (node=0xa5365538, thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:3290
 
#32 0x08795e41 in row_upd_step (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:3436
 
#33 0x0872b224 in que_thr_step (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1024
 
#34 que_run_threads_low (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1104
 
#35 que_run_threads (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1144
 
#36 0x0872be94 in que_eval_sql (info=0xa53aaf10, sql=0x8c79bec "PROCEDURE DROP_TABLE_PROC () IS\ntid CHAR;\niid CHAR;\nDECLARE CURSOR cur_idx IS\nSELECT ID FROM SYS_INDEXES\nWHERE TABLE_ID = tid FOR UPDATE;\nBEGIN\nSELECT ID INTO tid FROM SYS_TABLES\nWHERE NAME = :name FO"..., reserve_dict_mutex=0, trx=0xb12cb93c) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1221
 
#37 0x08762101 in row_drop_table_for_mysql (name=0xb06fc61c "test/t1", trx=0xb12cb93c, sqlcom=SQLCOM_DROP_TABLE, create_failed=false, nonatomic=true) at /home/buildbot/buildbot/build/storage/innobase/row/row0mysql.cc:3665
 
#38 0x0869a86a in ha_innobase::delete_table (this=0xa5313510, name=0xb06fd78f "./test/t1", sqlcom=SQLCOM_DROP_TABLE) at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:13357
 
#39 0x0868eedd in ha_innobase::delete_table (this=0xa5313510, name=0xb06fd78f "./test/t1") at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:13482
 
#40 0x0843f2b6 in ha_delete_table (thd=0xa53048d8, table_type=0xb84bc20, path=0xb06fd78f "./test/t1", db=0xa5313320 "test", alias=0xa5312eb0 "t1", generate_warning=true) at /home/buildbot/buildbot/build/sql/handler.cc:4473
 
#41 ha_delete_table (thd=0xa53048d8, table_type=0xb84bc20, path=0xb06fd78f "./test/t1", db=0xa5313320 "test", alias=0xa5312eb0 "t1", generate_warning=true) at /home/buildbot/buildbot/build/sql/handler.cc:2442
 
#42 0x082cad90 in mysql_rm_table_no_locks (thd=0xa53048d8, tables=0xa5312ed8, if_exists=false, drop_temporary=false, drop_view=false, dont_log_query=false, dont_free_locks=false) at /home/buildbot/buildbot/build/sql/sql_table.cc:2447
 
#43 0x082d23fb in mysql_rm_table (thd=0xa53048d8, tables=0xa5312ed8, if_exists=0 '\000', drop_temporary=0 '\000') at /home/buildbot/buildbot/build/sql/sql_table.cc:2090
 
#44 0x082281b7 in mysql_execute_command (thd=0xa53048d8) at /home/buildbot/buildbot/build/sql/sql_parse.cc:4506
 
#45 0x08231d2f in mysql_parse (thd=0xa53048d8, rawbuf=0xa5312e40 "DROP TABLE t1", length=13, parser_state=0xb06ff0c4, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/sql/sql_parse.cc:7741
 
#46 0x08234079 in dispatch_command (command=COM_QUERY, thd=0xa53048d8, packet=0xa5308b81 "DROP TABLE t1", packet_length=13, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1832
 
#47 0x08236199 in do_command (thd=0xa53048d8) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1386
 
#48 0x08328a93 in do_handle_one_connection (connect=0xbafa180) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1336
 
#49 0x08328c8b in handle_one_connection (arg=0xbafa180) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241
 
#50 0x08654e88 in pfs_spawn_thread (arg=0xbafa358) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#51 0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#52 0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 28 (Thread 0xb78239f0 (LWP 16565)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x004d25f6 in poll () from /lib/libc.so.6
 
#2  0x08172db4 in handle_connections_sockets () at /home/buildbot/buildbot/build/sql/mysqld.cc:6667
 
#3  0x081747c6 in mysqld_main (argc=145, argv=0xb6fae98) at /home/buildbot/buildbot/build/sql/mysqld.cc:6133
 
#4  0x08166602 in main (argc=Cannot access memory at address 0x2
 
) at /home/buildbot/buildbot/build/sql/main.cc:25
 
 
 
Thread 27 (Thread 0xb20cbb70 (LWP 16581)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x08970217 in inline_mysql_cond_timedwait (control=0x9035298, sleep_time=29000000000) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1182
 
#3  my_service_thread_sleep (control=0x9035298, sleep_time=29000000000) at /home/buildbot/buildbot/build/storage/maria/ma_servicethread.c:115
 
#4  0x08968032 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/storage/maria/ma_checkpoint.c:709
 
#5  0x08654e88 in pfs_spawn_thread (arg=0xb84b908) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 26 (Thread 0xab0f9b70 (LWP 16590)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870cf8e in os_event::timed_wait (this=0xba1d4e0, abstime=0xab0f9270) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
 
#3  0x0870d651 in os_event::wait_time_low (this=0xba1d4e0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
 
#4  0x086e4466 in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/storage/innobase/lock/lock0wait.cc:525
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 25 (Thread 0xabafab70 (LWP 16592)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870cf8e in os_event::timed_wait (this=0xb8deef8, abstime=0xabafa2a0) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
 
#3  0x0870d651 in os_event::wait_time_low (this=0xb8deef8, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
 
#4  0x087a3d0c in srv_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1751
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 24 (Thread 0xa74f3b70 (LWP 16597)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870d3c5 in wait (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
 
#3  wait_low (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325
 
#4  os_event_wait_low (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
 
#5  0x087b320a in sync_array_wait_event (arr=0xb8c4cd8, cell=@0xa74f2c24) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0arr.cc:471
 
#6  0x087b47ef in rw_lock_s_lock_spin (lock=0x90fc0a0, pass=0, file_name=0x8c7abc0 "/home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc", line=982) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:373
 
#7  0x087719b2 in row_purge_parse_undo_rec (node=0xba72690, undo_rec=0xa5507900 "", thr=0xba72628) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:290
 
#8  row_purge (node=0xba72690, undo_rec=0xa5507900 "", thr=0xba72628) at /home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc:1157
 
#9  0x0877314a in row_purge_step (thr=0xba72628) at /home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc:1219
 
#10 0x0872b7bc in que_thr_step (thr=0xba72628) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1042
 
#11 que_run_threads_low (thr=0xba72628) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1104
 
#12 que_run_threads (thr=0xba72628) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1144
 
#13 0x087a4af5 in srv_task_execute (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2571
 
#14 srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2618
 
#15 0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#16 0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 23 (Thread 0xa52ffb70 (LWP 16604)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870d3c5 in wait (event=0xb8def50, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
 
#3  wait_low (event=0xb8def50, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325
 
#4  os_event_wait_low (event=0xb8def50, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
 
#5  0x0885bb67 in buf_dump_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0dump.cc:806
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 22 (Thread 0xa6af2b70 (LWP 16598)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870d3c5 in wait (event=0xb8dee48, reset_sig_count=566) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
 
#3  wait_low (event=0xb8dee48, reset_sig_count=566) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325
 
#4  os_event_wait_low (event=0xb8dee48, reset_sig_count=566) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
 
#5  0x087a4842 in srv_resume_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940
 
#6  srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2616
 
#7  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#8  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 21 (Thread 0xb0849b70 (LWP 16616)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x08194bda in inline_mysql_cond_wait (arg=0x0) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
 
#3  handle_slave_background (arg=0x0) at /home/buildbot/buildbot/build/sql/slave.cc:337
 
#4  0x08654e88 in pfs_spawn_thread (arg=0xbad5338) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 20 (Thread 0xa7ef4b70 (LWP 16596)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870d3c5 in wait (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
 
#3  wait_low (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325
 
#4  os_event_wait_low (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
 
#5  0x087b320a in sync_array_wait_event (arr=0xb8c4cd8, cell=@0xa7ef36c4) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0arr.cc:471
 
#6  0x087b47ef in rw_lock_s_lock_spin (lock=0x90fc0a0, pass=0, file_name=0x8c7abc0 "/home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc", line=982) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:373
 
#7  0x087719b2 in row_purge_parse_undo_rec (node=0xba72b08, undo_rec=0xa550bb50 "", thr=0xba72aa0) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:290
 
#8  row_purge (node=0xba72b08, undo_rec=0xa550bb50 "", thr=0xba72aa0) at /home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc:1157
 
#9  0x0877314a in row_purge_step (thr=0xba72aa0) at /home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc:1219
 
#10 0x0872b7bc in que_thr_step (thr=0xba72aa0) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1042
 
#11 que_run_threads_low (thr=0xba72aa0) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1104
 
#12 que_run_threads (thr=0xba72aa0) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1144
 
#13 0x087c61d2 in trx_purge (n_purge_threads=3, batch_size=300, truncate=false) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0purge.cc:1738
 
#14 0x0879fd82 in srv_do_purge (n_total_purged=0xa7ef4368) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2733
 
#15 0x087a7c5c in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2879
 
#16 0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#17 0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 19 (Thread 0xb0893b70 (LWP 16615)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068d590 in sigwait () from /lib/libpthread.so.0
 
#2  0x08171b32 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/sql/mysqld.cc:3509
 
#3  0x08654e88 in pfs_spawn_thread (arg=0xba90970) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#4  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#5  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 18 (Thread 0xa92f6b70 (LWP 16594)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870cf8e in os_event::timed_wait (this=0xba46410, abstime=0xa92f6280) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
 
#3  0x0870d651 in os_event::wait_time_low (this=0xba46410, time_in_usec=10000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
 
#4  0x088b2525 in dict_stats_thread () at /home/buildbot/buildbot/build/storage/innobase/dict/dict0stats_bg.cc:474
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 17 (Thread 0xa48feb70 (LWP 16605)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068cc26 in nanosleep () from /lib/libpthread.so.0
 
#2  0x0870d8bc in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/storage/innobase/os/os0thread.cc:231
 
#3  0x0883b1cc in btr_defragment_thread () at /home/buildbot/buildbot/build/storage/innobase/btr/btr0defragment.cc:727
 
#4  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#5  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 16 (Thread 0xa34fcb70 (LWP 16607)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0868db22 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1156
 
#3  thd_destructor_proxy () at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:328
 
#4  0x08654e88 in pfs_spawn_thread (arg=0xba8a188) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 15 (Thread 0xaa6f8b70 (LWP 16591)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870cf8e in os_event::timed_wait (this=0xb8deea0, abstime=0xaa6f8190) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
 
#3  0x0870d651 in os_event::wait_time_low (this=0xb8deea0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
 
#4  0x0879ff23 in srv_error_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1915
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 14 (Thread 0xa3efdb70 (LWP 16606)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870d3c5 in wait (event=0xb8df000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
 
#3  wait_low (event=0xb8df000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325
 
#4  os_event_wait_low (event=0xb8df000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
 
#5  0x0885451b in buf_resize_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0buf.cc:3133
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 13 (Thread 0xa9cf7b70 (LWP 16593)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068cc26 in nanosleep () from /lib/libpthread.so.0
 
#2  0x0870d8bc in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/storage/innobase/os/os0thread.cc:231
 
#3  0x087a5aac in srv_master_sleep (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2418
 
#4  srv_master_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2458
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 12 (Thread 0xad48cb70 (LWP 16587)):
 
#0  0x00f0551a in ?? () from /lib/libaio.so.1
 
#1  0x0870aea4 in LinuxAIOHandler::poll (this=0xb92e390, m1=0xad48c33c, m2=0xad48c338, request=0xad48c328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
 
#2  0x0870b5a5 in os_aio_linux_handler (segment=5, m1=0xad48c33c, m2=0xad48c338, request=0xad48c328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
 
#3  os_aio_handler (segment=5, m1=0xad48c33c, m2=0xad48c338, request=0xad48c328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
 
#4  0x088bee40 in fil_aio_wait (segment=5) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5119
 
#5  0x087a8a8a in io_handler_thread (arg=0x90eec74) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 11 (Thread 0xa60f1b70 (LWP 16599)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068927c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870d3c5 in wait (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
 
#3  wait_low (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325
 
#4  os_event_wait_low (event=0xba51dd0, reset_sig_count=29) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
 
#5  0x087b320a in sync_array_wait_event (arr=0xb8c4cd8, cell=@0xa60f0c24) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0arr.cc:471
 
#6  0x087b47ef in rw_lock_s_lock_spin (lock=0x90fc0a0, pass=0, file_name=0x8c7abc0 "/home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc", line=982) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:373
 
#7  0x087719b2 in row_purge_parse_undo_rec (node=0xba729a0, undo_rec=0xa5509e58 "", thr=0xba72938) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:290
 
#8  row_purge (node=0xba729a0, undo_rec=0xa5509e58 "", thr=0xba72938) at /home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc:1157
 
#9  0x0877314a in row_purge_step (thr=0xba72938) at /home/buildbot/buildbot/build/storage/innobase/row/row0purge.cc:1219
 
#10 0x0872b7bc in que_thr_step (thr=0xba72938) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1042
 
#11 que_run_threads_low (thr=0xba72938) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1104
 
#12 que_run_threads (thr=0xba72938) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1144
 
#13 0x087a4af5 in srv_task_execute (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2571
 
#14 srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2618
 
#15 0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#16 0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 10 (Thread 0xaca8bb70 (LWP 16588)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870cf8e in os_event::timed_wait (this=0xb8defa8, abstime=0xaca8b080) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
 
#3  0x0870d651 in os_event::wait_time_low (this=0xb8defa8, time_in_usec=388000, reset_sig_count=6) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
 
#4  0x08866f28 in pc_sleep_if_needed () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:2698
 
#5  buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:3164
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 9 (Thread 0xade8db70 (LWP 16586)):
 
#0  0x00f0551a in ?? () from /lib/libaio.so.1
 
#1  0x0870aea4 in LinuxAIOHandler::poll (this=0xb92c390, m1=0xade8d33c, m2=0xade8d338, request=0xade8d328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
 
#2  0x0870b5a5 in os_aio_linux_handler (segment=4, m1=0xade8d33c, m2=0xade8d338, request=0xade8d328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
 
#3  os_aio_handler (segment=4, m1=0xade8d33c, m2=0xade8d338, request=0xade8d328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
 
#4  0x088bee40 in fil_aio_wait (segment=4) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5119
 
#5  0x087a8a8a in io_handler_thread (arg=0x90eec70) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 8 (Thread 0xa88f5b70 (LWP 16595)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x0870cf8e in os_event::timed_wait (this=0xba89ad0, abstime=0xa88f50b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
 
#3  0x0870d651 in os_event::wait_time_low (this=0xba89ad0, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
 
#4  0x087fbbab in ib_wqueue_timedwait (wq=0xba89a08, wait_in_usecs=5000000) at /home/buildbot/buildbot/build/storage/innobase/ut/ut0wqueue.cc:156
 
#5  0x0890aec4 in fts_optimize_thread (arg=0xba89a08) at /home/buildbot/buildbot/build/storage/innobase/fts/fts0opt.cc:2820
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 7 (Thread 0xae88eb70 (LWP 16585)):
 
#0  0x00f0551a in ?? () from /lib/libaio.so.1
 
#1  0x0870aea4 in LinuxAIOHandler::poll (this=0xb900f28, m1=0xae88e33c, m2=0xae88e338, request=0xae88e328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
 
#2  0x0870b5a5 in os_aio_linux_handler (segment=3, m1=0xae88e33c, m2=0xae88e338, request=0xae88e328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
 
#3  os_aio_handler (segment=3, m1=0xae88e33c, m2=0xae88e338, request=0xae88e328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
 
#4  0x088bee40 in fil_aio_wait (segment=3) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5119
 
#5  0x087a8a8a in io_handler_thread (arg=0x90eec6c) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 6 (Thread 0xaf28fb70 (LWP 16584)):
 
#0  0x00f0551a in ?? () from /lib/libaio.so.1
 
#1  0x0870aea4 in LinuxAIOHandler::poll (this=0xb8fef28, m1=0xaf28f33c, m2=0xaf28f338, request=0xaf28f328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
 
#2  0x0870b5a5 in os_aio_linux_handler (segment=2, m1=0xaf28f33c, m2=0xaf28f338, request=0xaf28f328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
 
#3  os_aio_handler (segment=2, m1=0xaf28f33c, m2=0xaf28f338, request=0xaf28f328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
 
#4  0x088bee40 in fil_aio_wait (segment=2) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5119
 
#5  0x087a8a8a in io_handler_thread (arg=0x90eec68) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 5 (Thread 0xafc90b70 (LWP 16583)):
 
#0  0x00f0551a in ?? () from /lib/libaio.so.1
 
#1  0x0870aea4 in LinuxAIOHandler::poll (this=0xb917a08, m1=0xafc9033c, m2=0xafc90338, request=0xafc90328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
 
#2  0x0870b5a5 in os_aio_linux_handler (segment=1, m1=0xafc9033c, m2=0xafc90338, request=0xafc90328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
 
#3  os_aio_handler (segment=1, m1=0xafc9033c, m2=0xafc90338, request=0xafc90328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
 
#4  0x088bee40 in fil_aio_wait (segment=1) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5119
 
#5  0x087a8a8a in io_handler_thread (arg=0x90eec64) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 4 (Thread 0xb0691b70 (LWP 16582)):
 
#0  0x00f0551a in ?? () from /lib/libaio.so.1
 
#1  0x0870aea4 in LinuxAIOHandler::poll (this=0xb90c480, m1=0xb069133c, m2=0xb0691338, request=0xb0691328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
 
#2  0x0870b5a5 in os_aio_linux_handler (segment=0, m1=0xb069133c, m2=0xb0691338, request=0xb0691328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
 
#3  os_aio_handler (segment=0, m1=0xb069133c, m2=0xb0691338, request=0xb0691328) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5781
 
#4  0x088bee40 in fil_aio_wait (segment=0) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5119
 
#5  0x087a8a8a in io_handler_thread (arg=0x90eec60) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 3 (Thread 0xac08ab70 (LWP 17000)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x081725d7 in one_thread_per_connection_end (thd=0xabe049b0, put_in_cache=true) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1182
 
#3  0x083289d9 in do_handle_one_connection (connect=0xba76218) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1355
 
#4  0x08328c8b in handle_one_connection (arg=0xba76218) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241
 
#5  0x08654e88 in pfs_spawn_thread (arg=0xb9fe850) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#6  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#7  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 2 (Thread 0xb3712b70 (LWP 16569)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x00689624 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
 
#2  0x08aa195a in inline_mysql_cond_timedwait (arg=0x0) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1182
 
#3  timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mysys/thr_timer.c:292
 
#4  0x08654e88 in pfs_spawn_thread (arg=0xb817ac0) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#5  0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#6  0x004dd43e in clone () from /lib/libc.so.6
 
 
 
Thread 1 (Thread 0xb06ffb70 (LWP 16617)):
 
#0  0x00dc6424 in __kernel_vsyscall ()
 
#1  0x0068a403 in pthread_kill () from /lib/libpthread.so.0
 
#2  0x08a9cdeb in my_write_core (sig=6) at /home/buildbot/buildbot/build/mysys/stacktrace.c:477
 
#3  0x08437087 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:343
 
#4  <signal handler called>
 
#5  0x00dc6424 in __kernel_vsyscall ()
 
#6  0x0042aaf1 in raise () from /lib/libc.so.6
 
#7  0x0042c3ca in abort () from /lib/libc.so.6
 
#8  0x087f678b in ut_dbg_assertion_failed (expr=0x8c83888 "lock->lock_word == X_LOCK_DECR", file=0x8c83848 "/home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc", line=259) at /home/buildbot/buildbot/build/storage/innobase/ut/ut0dbg.cc:60
 
#9  0x087b50cf in rw_lock_free_func (lock=0xa53731a0) at /home/buildbot/buildbot/build/storage/innobase/sync/sync0rw.cc:259
 
#10 0x08880b6f in pfs_rw_lock_free_func (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/include/sync0rw.ic:603
 
#11 dict_table_stats_latch_free (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/dict/dict0dict.cc:300
 
#12 dict_table_stats_latch_destroy (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/dict/dict0dict.cc:337
 
#13 0x088a4c3a in dict_mem_table_free (table=0xa534fc70) at /home/buildbot/buildbot/build/storage/innobase/dict/dict0mem.cc:205
 
#14 0x0882f6dc in btr_search_lazy_free (index=0xa53516f0) at /home/buildbot/buildbot/build/storage/innobase/btr/btr0sea.cc:265
 
#15 0x088327b0 in btr_search_drop_page_hash_index (block=0xb0aa1cd0) at /home/buildbot/buildbot/build/storage/innobase/btr/btr0sea.cc:1272
 
#16 0x0886a843 in buf_LRU_free_page (bpage=0xb0aa1cd0, zip=true) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:1616
 
#17 0x0886c5c4 in buf_LRU_free_from_common_LRU_list (buf_pool=0xb998210, scan_all=false) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:698
 
#18 buf_LRU_scan_and_free_block (buf_pool=0xb998210, scan_all=false) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:740
 
#19 0x0886cc2f in buf_LRU_get_free_block (buf_pool=0xb998210) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0lru.cc:971
 
#20 0x08851051 in buf_page_create (page_id=..., page_size=..., mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0buf.cc:5593
 
#21 0x088da593 in fsp_page_create (space=<value optimized out>, offset=<value optimized out>, page_size=..., rw_latch=RW_SX_LATCH, mtr=0xb06f978c, init_mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:1309
 
#22 0x088e066d in fsp_alloc_free_page (space=0xba45b18, page_size=..., hint=<value optimized out>, rw_latch=RW_SX_LATCH, mtr=0xb06f978c, init_mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:1455
 
#23 0x088e25cc in fseg_alloc_free_page_low (space=0xba45b18, page_size=..., seg_inode=0xb1295832 "", hint=0, direction=111 'o', rw_latch=RW_SX_LATCH, mtr=0xb06f978c, init_mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:2519
 
#24 0x088e3855 in fseg_create_general (space_id=0, page=0, byte_offset=60, has_done_reservation=1, mtr=0xb06f978c) at /home/buildbot/buildbot/build/storage/innobase/fsp/fsp0fsp.cc:2105
 
#25 0x087f27f5 in trx_undo_create (trx=0xb12cb93c, rseg=0xba7f248, undo=0xb12cc558, type=2) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0undo.cc:451
 
#26 trx_undo_assign_undo (trx=0xb12cb93c, rseg=0xba7f248, undo=0xb12cc558, type=2) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0undo.cc:1610
 
#27 0x087d3888 in trx_undo_report_row_operation (thr=0xa5366560, index=0xba6c268, clust_entry=0xa53adf80, update=0x0, cmpl_info=0, rec=0xb10adda9 "", offsets=0xb06faa1c, roll_ptr=0xb06fa290) at /home/buildbot/buildbot/build/storage/innobase/trx/trx0rec.cc:2073
 
#28 0x088166aa in btr_cur_del_mark_set_clust_rec (block=0xb0aaaa30, rec=0xb10adda9 "", index=0xba6c268, offsets=0xb06faa1c, thr=0xa5366560, entry=0xa53adf80, mtr=0xb06fa364) at /home/buildbot/buildbot/build/storage/innobase/btr/btr0cur.cc:4865
 
#29 0x0879391f in row_upd_del_mark_clust_rec (node=0xa5365538, thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:2997
 
#30 row_upd_clust_step (node=0xa5365538, thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:3169
 
#31 0x08794886 in row_upd (node=0xa5365538, thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:3290
 
#32 0x08795e41 in row_upd_step (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc:3436
 
#33 0x0872b224 in que_thr_step (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1024
 
#34 que_run_threads_low (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1104
 
#35 que_run_threads (thr=0xa5366560) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1144
 
#36 0x0872be94 in que_eval_sql (info=0xa53aaf10, sql=0x8c79bec "PROCEDURE DROP_TABLE_PROC () IS\ntid CHAR;\niid CHAR;\nDECLARE CURSOR cur_idx IS\nSELECT ID FROM SYS_INDEXES\nWHERE TABLE_ID = tid FOR UPDATE;\nBEGIN\nSELECT ID INTO tid FROM SYS_TABLES\nWHERE NAME = :name FO"..., reserve_dict_mutex=0, trx=0xb12cb93c) at /home/buildbot/buildbot/build/storage/innobase/que/que0que.cc:1221
 
#37 0x08762101 in row_drop_table_for_mysql (name=0xb06fc61c "test/t1", trx=0xb12cb93c, sqlcom=SQLCOM_DROP_TABLE, create_failed=false, nonatomic=true) at /home/buildbot/buildbot/build/storage/innobase/row/row0mysql.cc:3665
 
#38 0x0869a86a in ha_innobase::delete_table (this=0xa5313510, name=0xb06fd78f "./test/t1", sqlcom=SQLCOM_DROP_TABLE) at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:13357
 
#39 0x0868eedd in ha_innobase::delete_table (this=0xa5313510, name=0xb06fd78f "./test/t1") at /home/buildbot/buildbot/build/storage/innobase/handler/ha_innodb.cc:13482
 
#40 0x0843f2b6 in ha_delete_table (thd=0xa53048d8, table_type=0xb84bc20, path=0xb06fd78f "./test/t1", db=0xa5313320 "test", alias=0xa5312eb0 "t1", generate_warning=true) at /home/buildbot/buildbot/build/sql/handler.cc:4473
 
#41 ha_delete_table (thd=0xa53048d8, table_type=0xb84bc20, path=0xb06fd78f "./test/t1", db=0xa5313320 "test", alias=0xa5312eb0 "t1", generate_warning=true) at /home/buildbot/buildbot/build/sql/handler.cc:2442
 
#42 0x082cad90 in mysql_rm_table_no_locks (thd=0xa53048d8, tables=0xa5312ed8, if_exists=false, drop_temporary=false, drop_view=false, dont_log_query=false, dont_free_locks=false) at /home/buildbot/buildbot/build/sql/sql_table.cc:2447
 
#43 0x082d23fb in mysql_rm_table (thd=0xa53048d8, tables=0xa5312ed8, if_exists=0 '\000', drop_temporary=0 '\000') at /home/buildbot/buildbot/build/sql/sql_table.cc:2090
 
#44 0x082281b7 in mysql_execute_command (thd=0xa53048d8) at /home/buildbot/buildbot/build/sql/sql_parse.cc:4506
 
#45 0x08231d2f in mysql_parse (thd=0xa53048d8, rawbuf=0xa5312e40 "DROP TABLE t1", length=13, parser_state=0xb06ff0c4, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/sql/sql_parse.cc:7741
 
#46 0x08234079 in dispatch_command (command=COM_QUERY, thd=0xa53048d8, packet=0xa5308b81 "DROP TABLE t1", packet_length=13, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1832
 
#47 0x08236199 in do_command (thd=0xa53048d8) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1386
 
#48 0x08328a93 in do_handle_one_connection (connect=0xbafa180) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1336
 
#49 0x08328c8b in handle_one_connection (arg=0xbafa180) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241
 
#50 0x08654e88 in pfs_spawn_thread (arg=0xbafa358) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
 
#51 0x00685a09 in start_thread () from /lib/libpthread.so.0
 
#52 0x004dd43e in clone () from /lib/libc.so.6

Comment by Alice Sherepa [ 2020-08-03 ]

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/9877/steps/mtr/logs/stdio

gcol.innodb_virtual_basic 'innodb'       w1 [ fail ]
 
        Test ended at 2020-07-28 03:23:56
 
 
 
CURRENT_TEST: gcol.innodb_virtual_basic
 
mysqltest: In included file "/usr/share/mysql/mysql-test/suite/gcol/inc/innodb_v_large_col.inc": 
included from /usr/share/mysql/mysql-test/suite/gcol/t/innodb_virtual_basic.test at line 736:
 
At line 41: query 'CALL UPDATE_t()' failed: 2013: Lost connection to MySQL server during query
 
 
 
The result from queries just before the failure was:
 
< snip >
 
CREATE INDEX idx ON t(c(100), d(20));
 
UPDATE t SET a = REPEAT(CAST(1 AS CHAR), 2000) WHERE h = 1;
 
CREATE PROCEDURE UPDATE_t()
 
begin
 
DECLARE i INT DEFAULT 1;
 
WHILE (i <= 100) DO
 
UPDATE t SET a = REPEAT(CAST(i AS CHAR), 2000) WHERE h = 1;
 
SET i = i + 1;
 
END WHILE;
 
END|
 
CREATE PROCEDURE DELETE_insert_t()
 
begin
 
DECLARE i INT DEFAULT 1;
 
WHILE (i <= 100) DO
 
DELETE FROM t WHERE h = 1;
 
INSERT INTO t VALUES (REPEAT(CAST(i AS CHAR), 2000) ,  REPEAT('b', 2000), DEFAULT, DEFAULT, DEFAULT, 1);
 
SET i = i + 1;
 
END WHILE;
 
END|
 
CALL UPDATE_t();
 
 
 
More results from queries before failure can be found in /dev/shm/var/1/log/innodb_virtual_basic.log
 
 
 
 
 
Server [mysqld.1 - pid: 11176, winpid: 11176, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
2020-07-28 03:23:55 0x7f4265ffb700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.2.33/storage/innobase/sync/sync0rw.cc line 259
 
InnoDB: Failing assertion: lock->lock_word == X_LOCK_DECR
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
200728  3:23:55 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.33-MariaDB-10.2.33+maria~jessie-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=2
 
max_threads=153
 
thread_count=8
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63019 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f42500009a8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f4265ffad38 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f427e72767e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x7f427e1b5a95]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f427d6cf890]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f427bc67067]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f427bc68448]
 
/usr/sbin/mysqld(+0x43b89a)[0x7f427df3b89a]
 
/usr/sbin/mysqld(+0x9542b2)[0x7f427e4542b2]
 
/usr/sbin/mysqld(+0xa10f8e)[0x7f427e510f8e]
 
/usr/sbin/mysqld(+0xa30e6e)[0x7f427e530e6e]
 
/usr/sbin/mysqld(+0x9c7bcc)[0x7f427e4c7bcc]
 
/usr/sbin/mysqld(+0x9fd15b)[0x7f427e4fd15b]
 
/usr/sbin/mysqld(+0x9ff091)[0x7f427e4ff091]
 
/usr/sbin/mysqld(+0x9ffb7c)[0x7f427e4ffb7c]
 
/usr/sbin/mysqld(+0x9d8fa8)[0x7f427e4d8fa8]
 
/usr/sbin/mysqld(+0xa02f7f)[0x7f427e502f7f]
 
/usr/sbin/mysqld(+0x9d7302)[0x7f427e4d7302]
 
/usr/sbin/mysqld(+0x96202e)[0x7f427e46202e]
 
/usr/sbin/mysqld(+0x9676e8)[0x7f427e4676e8]
 
/usr/sbin/mysqld(+0x949fcf)[0x7f427e449fcf]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f427d6c8064]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f427bd1a62d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.

Comment by Geza Lucz [ 2020-09-06 ]

I'm receiving this error (or something very similar) about once every 3 days. MariaDB crashes every time:

2020-08-26 19:38:25 0x7f065355b700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.24/storage/innobase/sync/sync0rw.cc line 258
InnoDB: Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200826 19:38:25 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.24-MariaDB
key_buffer_size=1048576000
read_buffer_size=33554432
max_used_connections=101
max_threads=102
thread_count=22
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 17737955 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f05fdc2ca28
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f065355ad30 thread_stack 0x49000

      • buffer overflow detected ***: /usr/sbin/mysqld terminated
        ======= Backtrace: =========
        /lib64/libc.so.6(__fortify_fail+0x37)[0x7f07589d9577]
        /lib64/libc.so.6(+0x1166f2)[0x7f07589d76f2]
        /lib64/libc.so.6(+0x1184d7)[0x7f07589d94d7]
        /usr/sbin/mysqld(my_addr_resolve+0xda)[0x5629c874e32a]
        /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x5629c87376b2]
        /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5629c81cca4f]
        /lib64/libpthread.so.0(+0xf630)[0x7f075a627630]
        /lib64/libc.so.6(gsignal+0x37)[0x7f07588f7387]
        /lib64/libc.so.6(abort+0x148)[0x7f07588f8a78]
        /usr/sbin/mysqld(+0x4ed8a0)[0x5629c7f078a0]
        /usr/sbin/mysqld(+0xa657c6)[0x5629c847f7c6]
        /usr/sbin/mysqld(+0xb132a4)[0x5629c852d2a4]
        /usr/sbin/mysqld(+0xb34cbb)[0x5629c854ecbb]
        /usr/sbin/mysqld(+0xad4ddf)[0x5629c84eeddf]
        /usr/sbin/mysqld(+0xb02174)[0x5629c851c174]
        /usr/sbin/mysqld(+0xb03f8b)[0x5629c851df8b]
        /usr/sbin/mysqld(+0xb0480c)[0x5629c851e80c]
        /usr/sbin/mysqld(+0xae0961)[0x5629c84fa961]
        /usr/sbin/mysqld(+0xad16ee)[0x5629c84eb6ee]
        /usr/sbin/mysqld(+0xad6d9c)[0x5629c84f0d9c]
        /usr/sbin/mysqld(+0xac9bf8)[0x5629c84e3bf8]
        /usr/sbin/mysqld(+0xa37fb2)[0x5629c8451fb2]
        /usr/sbin/mysqld(+0xa3b49e)[0x5629c845549e]
        /usr/sbin/mysqld(+0x95be77)[0x5629c8375e77]
        /usr/sbin/mysqld(_ZN7handler18ha_index_next_sameEPhPKhj+0x1c5)[0x5629c81d3225]
        /usr/sbin/mysqld(+0x613fed)[0x5629c802dfed]
        /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x211)[0x5629c8021c21]
        /usr/sbin/mysqld(+0x5fcecc)[0x5629c8016ecc]
        /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1de)[0x5629c8021bee]
        /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa8b)[0x5629c8045b4b]
        /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5629c8045d63]
        /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x5629c804426a]
        /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x5629c8044d7c]
        /usr/sbin/mysqld(+0x4daf06)[0x5629c7ef4f06]
        /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x63ba)[0x5629c7fed8ca]
        /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x36d)[0x5629c7ff063d]
        /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xfe1)[0x5629c7ff1ed1]
        /usr/sbin/mysqld(_Z10do_commandP3THD+0x11b)[0x5629c7ff416b]
        /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1d6)[0x5629c80caa86]
        /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5629c80cab9d]
        /lib64/libpthread.so.0(+0x7ea5)[0x7f075a61fea5]
        /lib64/libc.so.6(clone+0x6d)[0x7f07589bf8dd]
Comment by Ross Campbell [ 2020-09-11 ]

Comment was not relevant for original issue, my bad. Moved to MDEV-23693.

Comment by Geza Lucz [ 2020-09-12 ]

I made the mistake of pasting my error message here, thinking it was the same problem, but yours and my error is a buffer overflow, so I opened a new ticket for it: https://jira.mariadb.org/browse/MDEV-23693

Comment by Elena Stepanova [ 2020-09-29 ]

With rr I'm persistently getting SIGSEGV with the same build and test where I am getting the assertion failure without rr.

10.4 79e32e47a1115 with rr

 
#0  0x0000000070000002 in ?? ()
 
#1  0x00007f30c797bb27 in _raw_syscall () at /home/roc/rr/rr/src/preload/raw_syscall.S:120
 
#2  0x00007f30c7976e7e in traced_raw_syscall (call=<optimized out>) at /home/roc/rr/rr/src/preload/syscallbuf.c:229
 
#3  0x00007f30c797ab55 in syscall_hook_internal (call=0x7f303b7fffa0) at /home/roc/rr/rr/src/preload/syscallbuf.c:2953
 
#4  syscall_hook (call=0x7f303b7fffa0) at /home/roc/rr/rr/src/preload/syscallbuf.c:2987
 
#5  0x00007f30c7976d5a in _syscall_hook_trampoline () at /home/roc/rr/rr/src/preload/syscall_hook.S:282
 
#6  0x00007f30c7976d8a in __morestack () at /home/roc/rr/rr/src/preload/syscall_hook.S:417
 
#7  0x00007f30c7976de3 in _syscall_hook_trampoline_89_c2_f7_da () at /home/roc/rr/rr/src/preload/syscall_hook.S:463
 
#8  0x00007f30c6e9d2d5 in __pthread_kill (threadid=<optimized out>, signo=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
 
#9  0x00005563623f2555 in my_write_core (sig=11) at /home/mdbe/MDEV-21329/10.4/mysys/stacktrace.c:386
 
#10 0x0000556361b5360b in handle_fatal_signal (sig=11) at /home/mdbe/MDEV-21329/10.4/sql/signal_handler.cc:343
 
#11 <signal handler called>
 
#12 0x0000556361dce0d5 in PolicyMutex<TTASEventMutex<GenericPolicy> >::pfs_begin_lock (this=0x0, state=0x7f30940a7640, name=0x55636282f550 "/home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0opt.cc", line=2607)
 
    at /home/mdbe/MDEV-21329/10.4/storage/innobase/include/ib0mutex.h:709
 
#13 0x0000556361dcc499 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x0, n_spins=30, n_delay=4, name=0x55636282f550 "/home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0opt.cc", line=2607)
 
    at /home/mdbe/MDEV-21329/10.4/storage/innobase/include/ib0mutex.h:589
 
#14 0x00005563621a5c44 in fts_optimize_remove_table (table=0x7f3024106db0) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0opt.cc:2607
 
#15 0x0000556361f55b6e in row_drop_table_for_mysql (name=0x7f30940a7f60 "test/#sql2-1419-c", trx=0x7f30af984140, sqlcom=SQLCOM_ALTER_TABLE, create_failed=false, nonatomic=true)
 
    at /home/mdbe/MDEV-21329/10.4/storage/innobase/row/row0mysql.cc:3392
 
#16 0x0000556361dcb190 in ha_innobase::delete_table (this=0x7f3024014b78, name=0x7f30940a9a30 "./test/#sql2-1419-c", sqlcom=SQLCOM_ALTER_TABLE) at /home/mdbe/MDEV-21329/10.4/storage/innobase/handler/ha_innodb.cc:13095
 
#17 0x0000556361db3888 in ha_innobase::delete_table (this=0x7f3024014b78, name=0x7f30940a9a30 "./test/#sql2-1419-c") at /home/mdbe/MDEV-21329/10.4/storage/innobase/handler/ha_innodb.cc:13220
 
#18 0x0000556361b62fc0 in handler::ha_delete_table (this=0x7f3024014b78, name=0x7f30940a9a30 "./test/#sql2-1419-c") at /home/mdbe/MDEV-21329/10.4/sql/handler.cc:4747
 
#19 0x0000556361b5bf39 in ha_delete_table (thd=0x7f3024000d50, table_type=0x556365aacc10, path=0x7f30940a9a30 "./test/#sql2-1419-c", db=0x7f30940ab750, alias=0x7f30940a9e30, generate_warning=false)
 
    at /home/mdbe/MDEV-21329/10.4/sql/handler.cc:2624
 
#20 0x00005563618d8da3 in quick_rm_table (thd=0x7f3024000d50, base=0x556365aacc10, db=0x7f30940ab750, table_name=0x7f30940a9e30, flags=3, table_path=0x0) at /home/mdbe/MDEV-21329/10.4/sql/sql_table.cc:2767
 
#21 0x00005563618eec7f in mysql_alter_table (thd=0x7f3024000d50, new_db=0x7f3024005518, new_name=0x7f3024005920, create_info=0x7f30940ac320, table_list=0x7f3024012120, alter_info=0x7f30940ac260, order_num=0, order=0x0, ignore=false)
 
    at /home/mdbe/MDEV-21329/10.4/sql/sql_table.cc:10422
 
#22 0x0000556361986202 in Sql_cmd_alter_table::execute (this=0x7f3024012918, thd=0x7f3024000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_alter.cc:514
 
#23 0x0000556361802bcb in mysql_execute_command (thd=0x7f3024000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:6098
 
#24 0x00005563618080ad in mysql_parse (thd=0x7f3024000d50, rawbuf=0x7f3024011fd8 "ALTER TABLE t1 CHANGE IF EXISTS b b INT, ALGORITHM=COPY /* QNO 49 CON_ID 12 */", length=78, parser_state=0x7f30940ad5a0, is_com_multi=false, 
    is_next_command=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:7896
 
#25 0x00005563617f45e7 in dispatch_command (command=COM_QUERY, thd=0x7f3024000d50, packet=0x7f30241e7151 "ALTER TABLE t1 CHANGE IF EXISTS b b INT, ALGORITHM=COPY /* QNO 49 CON_ID 12 */ ", packet_length=79, is_com_multi=false, 
    is_next_command=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:1834
 
#26 0x00005563617f2d88 in do_command (thd=0x7f3024000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:1352
 
#27 0x000055636197cc24 in do_handle_one_connection (connect=0x5563666f7ed0) at /home/mdbe/MDEV-21329/10.4/sql/sql_connect.cc:1412
 
#28 0x000055636197c973 in handle_one_connection (arg=0x5563666f7ed0) at /home/mdbe/MDEV-21329/10.4/sql/sql_connect.cc:1316
 
#29 0x00007f30c6e956db in start_thread (arg=0x7f30940ae700) at pthread_create.c:463
 
#30 0x00007f30c5923a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.4 79e32e47a1115 without rr

 
2020-09-29 16:21:06 0x7f2e00182700  InnoDB: Assertion failure in file /home/mdbe/MDEV-21329/10.4/storage/innobase/sync/sync0rw.cc line 252
 
InnoDB: Failing assertion: lock->lock_word == X_LOCK_DECR
 
 
 
#5  0x00007f2e159498b1 in __GI_abort () at abort.c:79
 
#6  0x000055cd30c3833a in ut_dbg_assertion_failed (expr=0x55cd313ab180 "lock->lock_word == X_LOCK_DECR", file=0x55cd313ab0f0 "/home/mdbe/MDEV-21329/10.4/storage/innobase/sync/sync0rw.cc", line=252) at /home/mdbe/MDEV-21329/10.4/storage/innobase/ut/ut0dbg.cc:60
 
#7  0x000055cd30bdf72b in rw_lock_free_func (lock=0x7f2d94298100) at /home/mdbe/MDEV-21329/10.4/storage/innobase/sync/sync0rw.cc:252
 
#8  0x000055cd30d93f7d in pfs_rw_lock_free_func (lock=0x7f2d94298100) at /home/mdbe/MDEV-21329/10.4/storage/innobase/include/sync0rw.ic:605
 
#9  0x000055cd30d99050 in fts_cache_destroy (cache=0x7f2d94298100) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0fts.cc:287
 
#10 0x000055cd30da42ce in fts_t::~fts_t (this=0x7f2d9445eea0, __in_chrg=<optimized out>) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0fts.cc:5317
 
#11 0x000055cd30da43ae in fts_free (table=0x7f2d98079aa0) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0fts.cc:5354
 
#12 0x000055cd30b5f319 in row_drop_ancillary_fts_tables (table=0x7f2d98079aa0, trx=0x7f2e00184140) at /home/mdbe/MDEV-21329/10.4/storage/innobase/row/row0mysql.cc:3241
 
#13 0x000055cd30b60681 in row_drop_table_for_mysql (name=0x7f2e0017d470 "test/#sql-7435_c", trx=0x7f2e00184140, sqlcom=SQLCOM_ALTER_TABLE, create_failed=false, nonatomic=true) at /home/mdbe/MDEV-21329/10.4/storage/innobase/row/row0mysql.cc:3692
 
#14 0x000055cd309d5190 in ha_innobase::delete_table (this=0x7f2d94015e50, name=0x7f2d942d9070 "./test/#sql-7435_c", sqlcom=SQLCOM_ALTER_TABLE) at /home/mdbe/MDEV-21329/10.4/storage/innobase/handler/ha_innodb.cc:13095
 
#15 0x000055cd309bd888 in ha_innobase::delete_table (this=0x7f2d94015e50, name=0x7f2d942d9070 "./test/#sql-7435_c") at /home/mdbe/MDEV-21329/10.4/storage/innobase/handler/ha_innodb.cc:13220
 
#16 0x000055cd3076cfc0 in handler::ha_delete_table (this=0x7f2d94015e50, name=0x7f2d942d9070 "./test/#sql-7435_c") at /home/mdbe/MDEV-21329/10.4/sql/handler.cc:4747
 
#17 0x000055cd3068fc33 in THD::rm_temporary_table (this=0x7f2d94000d50, base=0x55cd33b8fc10, path=0x7f2d942d9070 "./test/#sql-7435_c") at /home/mdbe/MDEV-21329/10.4/sql/temporary_tables.cc:701
 
#18 0x000055cd30691b37 in THD::free_tmp_table_share (this=0x7f2d94000d50, share=0x7f2d942d8ad0, delete_table=true) at /home/mdbe/MDEV-21329/10.4/sql/temporary_tables.cc:1462
 
#19 0x000055cd3068fa6b in THD::drop_temporary_table (this=0x7f2d94000d50, table=0x7f2d943980f0, is_trans=0x0, delete_table=true) at /home/mdbe/MDEV-21329/10.4/sql/temporary_tables.cc:664
 
#20 0x000055cd304f919c in mysql_alter_table (thd=0x7f2d94000d50, new_db=0x7f2d94005518, new_name=0x7f2d94005920, create_info=0x7f2e00180320, table_list=0x7f2d940142e0, alter_info=0x7f2e00180260, order_num=0, order=0x0, ignore=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_table.cc:10508
 
#21 0x000055cd30590202 in Sql_cmd_alter_table::execute (this=0x7f2d94014ad8, thd=0x7f2d94000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_alter.cc:514
 
#22 0x000055cd3040cbcb in mysql_execute_command (thd=0x7f2d94000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:6098
 
#23 0x000055cd304120ad in mysql_parse (thd=0x7f2d94000d50, rawbuf=0x7f2d94014198 "ALTER TABLE t1 CHANGE IF EXISTS b b INT, ALGORITHM=COPY /* QNO 46 CON_ID 12 */", length=78, parser_state=0x7f2e001815a0, is_com_multi=false, is_next_command=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:7896
 
#24 0x000055cd303fe5e7 in dispatch_command (command=COM_QUERY, thd=0x7f2d94000d50, packet=0x7f2d94008661 "ALTER TABLE t1 CHANGE IF EXISTS b b INT, ALGORITHM=COPY /* QNO 46 CON_ID 12 */ ", packet_length=79, is_com_multi=false, is_next_command=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:1834
 
#25 0x000055cd303fcd88 in do_command (thd=0x7f2d94000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:1352
 
#26 0x000055cd30586c24 in do_handle_one_connection (connect=0x55cd347d7c70) at /home/mdbe/MDEV-21329/10.4/sql/sql_connect.cc:1412
 
#27 0x000055cd30586973 in handle_one_connection (arg=0x55cd347d7c70) at /home/mdbe/MDEV-21329/10.4/sql/sql_connect.cc:1316
 
#28 0x00007f2e16f9c6db in start_thread (arg=0x7f2e00182700) at pthread_create.c:463
 
#29 0x00007f2e15a2aa3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Elena Stepanova [ 2020-09-30 ]

An rr profile for lock->lock_word == X_LOCK_DECR assertion failure has been produced on a custom build of 10.4 with the patch for MDEV-23856.

10.4 79e32e47a + patch d6b33ea23

 
(rr) bt
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 
#1  0x00006ffc0eb138b1 in __GI_abort () at abort.c:79
 
#2  0x000055e15793133a in ut_dbg_assertion_failed (expr=0x55e1580a4180 "lock->lock_word == X_LOCK_DECR", file=0x55e1580a40f0 "/home/mdbe/MDEV-21329/10.4/storage/innobase/sync/sync0rw.cc", line=252)
 
    at /home/mdbe/MDEV-21329/10.4/storage/innobase/ut/ut0dbg.cc:60
 
#3  0x000055e1578d872b in rw_lock_free_func (lock=0x7f501c083960) at /home/mdbe/MDEV-21329/10.4/storage/innobase/sync/sync0rw.cc:252
 
#4  0x000055e157a8cf7d in pfs_rw_lock_free_func (lock=0x7f501c083960) at /home/mdbe/MDEV-21329/10.4/storage/innobase/include/sync0rw.ic:605
 
#5  0x000055e157a92050 in fts_cache_destroy (cache=0x7f501c083960) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0fts.cc:287
 
#6  0x000055e157a9d2ce in fts_t::~fts_t (this=0x35fb4c2a0b20, __in_chrg=<optimized out>) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0fts.cc:5317
 
#7  0x000055e157a9d3ae in fts_free (table=0x35fb4c4ebaf0) at /home/mdbe/MDEV-21329/10.4/storage/innobase/fts/fts0fts.cc:5354
 
#8  0x000055e157858319 in row_drop_ancillary_fts_tables (table=0x35fb4c4ebaf0, trx=0x361b3ec644b8) at /home/mdbe/MDEV-21329/10.4/storage/innobase/row/row0mysql.cc:3241
 
#9  0x000055e157859681 in row_drop_table_for_mysql (name=0xe2b713b0470 "test/#sql-71af_d", trx=0x361b3ec644b8, sqlcom=SQLCOM_ALTER_TABLE, create_failed=false, nonatomic=true)
 
    at /home/mdbe/MDEV-21329/10.4/storage/innobase/row/row0mysql.cc:3692
 
#10 0x000055e1576ce190 in ha_innobase::delete_table (this=0x7f501c013c90, name=0x7f501c04dbb0 "./test/#sql-71af_d", sqlcom=SQLCOM_ALTER_TABLE) at /home/mdbe/MDEV-21329/10.4/storage/innobase/handler/ha_innodb.cc:13095
 
#11 0x000055e1576b6888 in ha_innobase::delete_table (this=0x7f501c013c90, name=0x7f501c04dbb0 "./test/#sql-71af_d") at /home/mdbe/MDEV-21329/10.4/storage/innobase/handler/ha_innodb.cc:13220
 
#12 0x000055e157465fc0 in handler::ha_delete_table (this=0x7f501c013c90, name=0x7f501c04dbb0 "./test/#sql-71af_d") at /home/mdbe/MDEV-21329/10.4/sql/handler.cc:4747
 
#13 0x000055e157388c33 in THD::rm_temporary_table (this=0x7f501c000d50, base=0x55e159c56be0, path=0x7f501c04dbb0 "./test/#sql-71af_d") at /home/mdbe/MDEV-21329/10.4/sql/temporary_tables.cc:701
 
#14 0x000055e15738ab37 in THD::free_tmp_table_share (this=0x7f501c000d50, share=0x7f501c04d610, delete_table=true) at /home/mdbe/MDEV-21329/10.4/sql/temporary_tables.cc:1462
 
#15 0x000055e157388a6b in THD::drop_temporary_table (this=0x7f501c000d50, table=0x7f501c08f090, is_trans=0x0, delete_table=true) at /home/mdbe/MDEV-21329/10.4/sql/temporary_tables.cc:664
 
#16 0x000055e1571f219c in mysql_alter_table (thd=0x7f501c000d50, new_db=0x7f501c005518, new_name=0x7f501c005920, create_info=0xe2b713b3320, table_list=0x7f501c012120, alter_info=0xe2b713b3260, order_num=0, order=0x0, ignore=false)
 
    at /home/mdbe/MDEV-21329/10.4/sql/sql_table.cc:10508
 
#17 0x000055e157289202 in Sql_cmd_alter_table::execute (this=0x7f501c012918, thd=0x7f501c000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_alter.cc:514
 
#18 0x000055e157105bcb in mysql_execute_command (thd=0x7f501c000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:6098
 
#19 0x000055e15710b0ad in mysql_parse (thd=0x7f501c000d50, rawbuf=0x7f501c011fd8 "ALTER TABLE t1 CHANGE IF EXISTS b a INT, ALGORITHM=COPY /* QNO 49 CON_ID 13 */", length=78, parser_state=0xe2b713b45a0, is_com_multi=false, 
    is_next_command=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:7896
 
#20 0x000055e1570f75e7 in dispatch_command (command=COM_QUERY, thd=0x7f501c000d50, packet=0x7f501c008661 "ALTER TABLE t1 CHANGE IF EXISTS b a INT, ALGORITHM=COPY /* QNO 49 CON_ID 13 */ ", packet_length=79, is_com_multi=false, 
    is_next_command=false) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:1834
 
#21 0x000055e1570f5d88 in do_command (thd=0x7f501c000d50) at /home/mdbe/MDEV-21329/10.4/sql/sql_parse.cc:1352
 
#22 0x000055e15727fc24 in do_handle_one_connection (connect=0x55e15a7d5a90) at /home/mdbe/MDEV-21329/10.4/sql/sql_connect.cc:1412
 
#23 0x000055e15727f973 in handle_one_connection (arg=0x55e15a7d5a90) at /home/mdbe/MDEV-21329/10.4/sql/sql_connect.cc:1316
 
#24 0x000063347ac7f6db in start_thread (arg=0xe2b713b5700) at pthread_create.c:463
 
#25 0x00006ffc0ebf4a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Marko Mäkelä [ 2020-10-08 ]

The change looks OK to me. Are 10.0 and 10.1 really not affected? We might lack this debug assertion, but shouldn’t the double-free issue be possible there as well?

Please make it clear in the commit message that fts_optimize_remove_table() is executed by a user thread that performs DDL, and that the fts_optimize_wq may be removed if fts_optimize_shutdown() has started executing.

It appears that the assignment to set the flag fts_opt_start_shutdown is protected by dict_sys.mutex in fts_optimize_shutdown(). But we are reading that flag without holding any mutex. Is fts_optimize_remove_table() missing a mutex_own() assertion?

If we are really not holding the dict_system.mutex when we are reading the flag, then I would suggest that in the 10.4 version you change the flag declaration to

static std::atomic<bool> fts_opt_start_shutdown;

and change all reads to use explicit 

if (fts_opt_start_shutdown.load(std::memory_order_consume))

and the assignment to be done without acquiring any extra mutex:

fts_opt_start_shutdown.store(true, std::memory_order_release);

I believe that we would want Release-Consume ordering here.

I would rewrite this logic in MDEV-21452 later to use condition variables instead of a polling loop.

Comment by Marko Mäkelä [ 2020-10-08 ]

Sorry, I missed the note in the page that I linked to:

The specification of release-consume ordering is being revised, and the use of memory_order_consume is temporarily discouraged.

It seems to refer to P0371R1, which states:

All current compilers essentially map it to memory_order_acquire.

Comment by Thirunarayanan Balathandayuthapani [ 2020-10-08 ]

fts_opt_start_shutdown is being read without any mutex protection in fts_optimize_remove_table().

Generated at Thu Feb 08 09:06:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.