[MDEV-11927] InnoDB change buffer is not being merged to tables in the system tablespace Created: 2017-01-27  Updated: 2020-08-25  Resolved: 2017-02-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.3
Fix Version/s: 10.2.5

Type: Bug Priority: Critical
Reporter: Nirbhay Choubey (Inactive) Assignee: Marko Mäkelä
Resolution: Fixed Votes: 1
Labels: None

Attachments: File innodb_many_tables_pk.result     File innodb_many_tables_pk.test    
Issue Links:
Duplicate
duplicates MDEV-11894 Galera crashes after a few minutes - ... Closed
duplicates MDEV-13056 The server keeps crashing Closed
is duplicated by MDEV-11843 MariaDB Server Fail - "Has Gone Away"... Closed
Problem/Incident
is caused by MDEV-11585 Hard-code the shared InnoDB temporary... Closed
Relates
relates to MDEV-11894 Galera crashes after a few minutes - ... Closed
Sprint: 10.0.30

 Description   

Logging: mtr  galera.galera_many_tables_nopk
...
2017-01-27 12:11:30 140121873573632 [Note] WSREP: 1.0 (nirbhay-vbox): State transfer from 0.0 (nirbhay-vbox) complete.
2017-01-27 12:11:30 140121873573632 [Note] WSREP: Member 1.0 (nirbhay-vbox) synced with group.
2017-01-27 12:13:14 140121680615168 [ERROR] InnoDB: tried to purge non-delete-marked record in index `ID_IND` of table `SYS_TABLES`: tuple: TUPLE (info_bits=0, 2 fields): {[8]       1(0x0000000000000301),[9]test/t102(0x040503040F04010002)}, record: RECORD(info_bits=0, 2 fields): {[8]       1(0x0000000000000301),[9]test/t102(0x040503040F04010002)}
2017-01-27 12:13:14 0x7f709eff9700  InnoDB: Assertion failure in file /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc line 498
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170127 12:13:14 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
...
Thread 1 (Thread 0x7f709eff9700 (LWP 18808)):
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x000055b41eec793f in my_write_core (sig=6) at /home/nirbhay/project/git-repo/mariadb/10.2/mysys/stacktrace.c:477
#2  0x000055b41e734bca in handle_fatal_signal (sig=6) at /home/nirbhay/project/git-repo/mariadb/10.2/sql/signal_handler.cc:296
#3  <signal handler called>
#4  0x00007f70b4bbe428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#5  0x00007f70b4bc002a in __GI_abort () at abort.c:89
#6  0x000055b41ecddbe8 in ut_dbg_assertion_failed (expr=0x55b41f1313bf "0", file=0x55b41f131a68 "/home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc", line=498) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/ut/ut0dbg.cc:59
#7  0x000055b41ec239c8 in row_purge_remove_sec_if_poss_leaf (node=0x7f70b03ddc88, index=0x7f70afbf4288, entry=0x7f70a98faa88) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc:498
#8  0x000055b41ec23d08 in row_purge_remove_sec_if_poss (node=0x7f70b03ddc88, index=0x7f70afbf4288, entry=0x7f70a98faa88) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc:587
#9  0x000055b41ec23f16 in row_purge_del_mark (node=0x7f70b03ddc88) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc:657
#10 0x000055b41ec24a85 in row_purge_record_func (node=0x7f70b03ddc88, undo_rec=0x7f70b0bf3218 "\001\270\016\002\001", thr=0x7f70b4087c90, updated_extern=false) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc:975
#11 0x000055b41ec24cdc in row_purge (node=0x7f70b03ddc88, undo_rec=0x7f70b0bf3218 "\001\270\016\002\001", thr=0x7f70b4087c90) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc:1031
#12 0x000055b41ec24f85 in row_purge_step (thr=0x7f70b4087c90) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/row/row0purge.cc:1110
#13 0x000055b41ebaeeca in que_thr_step (thr=0x7f70b4087c90) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/que/que0que.cc:1056
#14 0x000055b41ebaf0cc in que_run_threads_low (thr=0x7f70b4087c90) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/que/que0que.cc:1118
#15 0x000055b41ebaf27e in que_run_threads (thr=0x7f70b4087c90) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/que/que0que.cc:1158
#16 0x000055b41ec74ae3 in srv_task_execute () at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/srv/srv0srv.cc:2655
#17 0x000055b41ec74c61 in srv_worker_thread (arg=0x0) at /home/nirbhay/project/git-repo/mariadb/10.2/storage/innobase/srv/srv0srv.cc:2705
#18 0x00007f70b57fa6ba in start_thread (arg=0x7f709eff9700) at pthread_create.c:333
#19 0x00007f70b4c8f82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
...
Failing test(s): galera.galera_many_tables_nopk

galera.galera_many_tables_pk is affected too.



 Comments   
Comment by Marko Mäkelä [ 2017-01-30 ]

This could be a duplicate of MDEV-9663, with an important observation that change buffering should not be the culprit, except if SET unique_checks=0 was used at some point.

The reason why the change buffer is normally disabled for SYS_TABLES.ID_IND is that this is a unique index. (Side note: I think that we should simply remove the corruption-prone behaviour of InnoDB and disregard the unique_checks=0 setting inside InnoDB. At Oracle it could not be done for some reason that I never understood. Deprecation policy, perhaps?)

I would very much appreciate all data files (including core dump if there is one). Even more, I would appreciate a reasonably repeatable test case. I suppose this takes a fair amount of DDL with the right timing.

Comment by Nirbhay Choubey (Inactive) [ 2017-01-30 ]

marko Yes, its a "big" test and it runs for some time before assertion failure shows up.
The issue can reliably be reproduced by running either galera.galera_many_tables_pk or
galera.galera_many_tables_nopk test.

Comment by Hartmut Holzgraefe [ 2017-02-20 ]

I could reproduce it on first try after re-enabling galera_many_tables_pk

I couldn't run the test with just adding --enable-disabled though, this just changed the tests state from disabled to skipped, I had to remove the entries related to MDEV-11927 from mysql-test/suite/galera/disabled.def to actually execute it.

Also --big-test is needed to actually run this test.

Comment by Hartmut Holzgraefe [ 2017-02-20 ]

--enable-disabled works as expected with mysql-test-run, fails with mysql-test-run.pl

Comment by Hartmut Holzgraefe [ 2017-02-21 ]

The real test issue was that libgalera was not found. For some yet unknown reason this interacts with --enable-disable in some way though that makes the message from the disabled.def file stick instead of the "library not found one"

I'm probably going to remove all related comments about this unrelated problem and create a separate bug report for this later today

Comment by Marko Mäkelä [ 2017-02-21 ]

I can repeat this very quickly in a non-debug build with

./mtr --manual-gdb --enable-disabled galera.galera_many_tables_pk

and setting a breakpoint on ib::error::~error() in both the master and the slave.
Both are reporting the error that they refuse to purge a non-delete-marked record in the unique secondary index of SYS_TABLES.
Neither server is restarted before that, so it should be fairly easy to narrow this down.

Comment by Jan Lindström (Inactive) [ 2017-02-21 ]

10.1 Works as designed:

MariaDB Version 10.1.22-MariaDB-debug
 - SSL connections supported
 - binaries are debug compiled
Sphinx 'indexer' binary not found, sphinx suite will be skipped
Collecting tests...
Installing system database...
 
==============================================================================
 
TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
galera.galera_many_tables_pk 'innodb_plugin' [ pass ]  29257
galera.galera_many_tables_pk 'xtradb'    [ pass ]  31574
--------------------------------------------------------------------------

This means there is something wrong on 10.2 most likely after InnoDB 5.7 merge.

Comment by Jan Lindström (Inactive) [ 2017-02-23 ]

Unfortunately this bug is not only Galera related, I can repeat the bug using standalone server (slightly modified test case attached):

2017-02-23  7:33:10 139741420824320 [ERROR] InnoDB: tried to purge non-delete-marked record in index `ID_IND` of table `SYS_TABLES`: tuple: TUPLE (info_bits=0, 2 fields): {[8]       3(0x0000000000000303),[9]test/t100(0x040503040F04010000)}, record: RECORD(info_bits=0, 2 fields): {[8]       3(0x0000000000000303),[9]test/t100(0x040503040F04010000)}
2017-02-23 07:33:10 0x7f1815bfa700  InnoDB: Assertion failure in file /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc line 498
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170223  7:33:10 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.5-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=8
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63061 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f1812c16070
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f1815bf9e68 thread_stack 0x48c00
/home/jan/mysql/10.2/sql/mysqld(my_print_stacktrace+0x38)[0x55c359dfd441]
/home/jan/mysql/10.2/sql/mysqld(handle_fatal_signal+0x3a3)[0x55c359676fee]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11100)[0x7f1826f07100]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f18246c6fdf]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f18246c840a]
/home/jan/mysql/10.2/sql/mysqld(+0xc3a7c0)[0x55c359aa67c0]
/home/jan/mysql/10.2/sql/mysqld(+0xb810ba)[0x55c3599ed0ba]
mysys/stacktrace.c:267(my_print_stacktrace)[0x55c3599ed3fe]
ut/ut0dbg.cc:60(__static_initialization_and_destruction_0(int, int))[0x55c3599ed610]
row/row0purge.cc:975(row_purge_record_func(purge_node_t*, unsigned char*, que_thr_t const*, bool))[0x55c3599ee17f]
row/row0purge.cc:1031(row_purge(purge_node_t*, unsigned char*, que_thr_t*))[0x55c3599ee386]
row/row0purge.cc:1112(row_purge_step(que_thr_t*))[0x55c3599ee636]
que/que0que.cc:1057(que_thr_step(que_thr_t*))[0x55c359977118]
que/que0que.cc:1119(que_run_threads_low(que_thr_t*))[0x55c35997731c]
que/que0que.cc:1161(que_run_threads(que_thr_t*))[0x55c3599774ce]
srv/srv0srv.cc:2671(srv_task_execute())[0x55c359a3db3f]
srv/srv0srv.cc:2718(srv_worker_thread)[0x55c359a3dcc3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7464)[0x7f1826efd464]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f182477c9df]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): 
Connection ID (thread ID): 3
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------
 
 
 - saving '/dev/shm/log/innodb.galera_many_tables_pk-innodb/' to '/dev/shm/log/innodb.galera_many_tables_pk-innodb/'
 - found 'core' (0/5)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace
Core generated by '/home/jan/mysql/10.2/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 20500]
[New LWP 20486]
[New LWP 20487]
[New LWP 20490]
[New LWP 20495]
[New LWP 20493]
[New LWP 20489]
[New LWP 20488]
[New LWP 20497]
[New LWP 20496]
[New LWP 20498]
[New LWP 20502]
[New LWP 20503]
[New LWP 20505]
[New LWP 20504]
[New LWP 20509]
[New LWP 20507]
[New LWP 20506]
[New LWP 20511]
[New LWP 20508]
[New LWP 20492]
[New LWP 20510]
[New LWP 20514]
[New LWP 20517]
[New LWP 20485]
[New LWP 20501]
[New LWP 20491]
[New LWP 20499]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/jan/mysql/10.2/sql/mysqld --defaults-group-suffix=.1 --defaults-file=/dev'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
62	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f1815bfa700 (LWP 20500))]
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x000055c359dfd533 in my_write_core (sig=6) at /home/jan/mysql/10.2/mysys/stacktrace.c:477
#2  0x000055c359677380 in handle_fatal_signal (sig=6) at /home/jan/mysql/10.2/sql/signal_handler.cc:296
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#5  0x00007f18246c840a in __GI_abort () at abort.c:89
#6  0x000055c359aa67c0 in ut_dbg_assertion_failed (expr=0x55c359fddf5f "0", file=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=498) at /home/jan/mysql/10.2/storage/innobase/ut/ut0dbg.cc:59
#7  0x000055c3599ed0ba in row_purge_remove_sec_if_poss_leaf (node=0x7f1823c674f0, index=0x7f181fbd0488, entry=0x7f1812c1cb88) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:498
#8  0x000055c3599ed3fe in row_purge_remove_sec_if_poss (node=0x7f1823c674f0, index=0x7f181fbd0488, entry=0x7f1812c1cb88) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:587
#9  0x000055c3599ed610 in row_purge_del_mark (node=0x7f1823c674f0) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:657
#10 0x000055c3599ee17f in row_purge_record_func (node=0x7f1823c674f0, undo_rec=0x7f18138db0d0 "\001\355\016\003\001", thr=0x7f1823c67428, updated_extern=false) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:975
#11 0x000055c3599ee386 in row_purge (node=0x7f1823c674f0, undo_rec=0x7f18138db0d0 "\001\355\016\003\001", thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1031
#12 0x000055c3599ee636 in row_purge_step (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1110
#13 0x000055c359977118 in que_thr_step (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1057
#14 0x000055c35997731c in que_run_threads_low (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1119
#15 0x000055c3599774ce in que_run_threads (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1159
#16 0x000055c359a3db3f in srv_task_execute () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2669
#17 0x000055c359a3dcc3 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2718
#18 0x00007f1826efd464 in start_thread (arg=0x7f1815bfa700) at pthread_create.c:333
#19 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 28 (Thread 0x7f18163fb700 (LWP 20499)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c3599440a2 in os_event::wait (this=0x7f181ceed600) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:166
#2  0x000055c359943ace in os_event::wait_low (this=0x7f181ceed600, reset_sig_count=770) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:336
#3  0x000055c359943e4b in os_event_wait_low (event=0x7f181ceed600, reset_sig_count=770) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:535
#4  0x000055c359a4b183 in sync_array_wait_event (arr=0x7f1823c781e0, cell=@0x7f18163fab58: 0x7f181f1ff180) at /home/jan/mysql/10.2/storage/innobase/sync/sync0arr.cc:490
#5  0x000055c359a4e10e in rw_lock_s_lock_spin (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/sync/sync0rw.cc:391
#6  0x000055c3599e8eb5 in rw_lock_s_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:298
#7  0x000055c3599e926f in pfs_rw_lock_s_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:649
#8  0x000055c3599edddf in row_purge_parse_undo_rec (node=0x7f181f7792f0, undo_rec=0x7f18138df988 "\001\202\016\001\003", updated_extern=0x7f18163facde, thr=0x7f181f779228) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:861
#9  0x000055c3599ee367 in row_purge (node=0x7f181f7792f0, undo_rec=0x7f18138df988 "\001\202\016\001\003", thr=0x7f181f779228) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1028
#10 0x000055c3599ee636 in row_purge_step (thr=0x7f181f779228) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1110
#11 0x000055c359977118 in que_thr_step (thr=0x7f181f779228) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1057
#12 0x000055c35997731c in que_run_threads_low (thr=0x7f181f779228) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1119
#13 0x000055c3599774ce in que_run_threads (thr=0x7f181f779228) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1159
#14 0x000055c359a6f21f in trx_purge (n_purge_threads=4, batch_size=300, truncate=false) at /home/jan/mysql/10.2/storage/innobase/trx/trx0purge.cc:1851
#15 0x000055c359a3e001 in srv_do_purge (thd=0x7f1813816070, n_threads=4, n_total_purged=0x7f18163faea0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2827
#16 0x000055c359a3e5bb in srv_purge_coordinator_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2972
#17 0x00007f1826efd464 in start_thread (arg=0x7f18163fb700) at pthread_create.c:333
#18 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 27 (Thread 0x7f181a7fb700 (LWP 20491)):
#0  0x00007f182649b6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055c359937a04 in LinuxAIOHandler::collect (this=0x7f181a7fad90) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:1898
#2  0x000055c359937f3c in LinuxAIOHandler::poll (this=0x7f181a7fad90, m1=0x7f181a7fae30, m2=0x7f181a7fae38, request=0x7f181a7fae40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2044
#3  0x000055c3599380c4 in os_aio_linux_handler (global_segment=4, m1=0x7f181a7fae30, m2=0x7f181a7fae38, request=0x7f181a7fae40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2098
#4  0x000055c35993cc53 in os_aio_handler (segment=4, m1=0x7f181a7fae30, m2=0x7f181a7fae38, request=0x7f181a7fae40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:5669
#5  0x000055c359ba4f70 in fil_aio_wait (segment=4) at /home/jan/mysql/10.2/storage/innobase/fil/fil0fil.cc:5356
#6  0x000055c359a4136a in io_handler_thread (arg=0x55c35a9249c0 <n+32>) at /home/jan/mysql/10.2/storage/innobase/srv/srv0start.cc:327
#7  0x00007f1826efd464 in start_thread (arg=0x7f181a7fb700) at pthread_create.c:333
#8  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 26 (Thread 0x7f18153f9700 (LWP 20501)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c3599440a2 in os_event::wait (this=0x7f181ceed600) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:166
#2  0x000055c359943ace in os_event::wait_low (this=0x7f181ceed600, reset_sig_count=770) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:336
#3  0x000055c359943e4b in os_event_wait_low (event=0x7f181ceed600, reset_sig_count=770) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:535
#4  0x000055c359a4b183 in sync_array_wait_event (arr=0x7f1823c781e0, cell=@0x7f18153f8c18: 0x7f181f1ff040) at /home/jan/mysql/10.2/storage/innobase/sync/sync0arr.cc:490
#5  0x000055c359a4e10e in rw_lock_s_lock_spin (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/sync/sync0rw.cc:391
#6  0x000055c3599e8eb5 in rw_lock_s_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:298
#7  0x000055c3599e926f in pfs_rw_lock_s_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:649
#8  0x000055c3599edddf in row_purge_parse_undo_rec (node=0x7f1823c67288, undo_rec=0x7f18138c9098 "\004\316\016\004\r", updated_extern=0x7f18153f8d9e, thr=0x7f181f7f3660) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:861
#9  0x000055c3599ee367 in row_purge (node=0x7f1823c67288, undo_rec=0x7f18138c9098 "\004\316\016\004\r", thr=0x7f181f7f3660) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1028
#10 0x000055c3599ee636 in row_purge_step (thr=0x7f181f7f3660) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1110
#11 0x000055c359977118 in que_thr_step (thr=0x7f181f7f3660) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1057
#12 0x000055c35997731c in que_run_threads_low (thr=0x7f181f7f3660) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1119
#13 0x000055c3599774ce in que_run_threads (thr=0x7f181f7f3660) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1159
#14 0x000055c359a3db3f in srv_task_execute () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2669
#15 0x000055c359a3dcc3 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2718
#16 0x00007f1826efd464 in start_thread (arg=0x7f18153f9700) at pthread_create.c:333
#17 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 25 (Thread 0x7f1827301780 (LWP 20485)):
#0  0x00007f182477356d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055c35933a6b1 in handle_connections_sockets () at /home/jan/mysql/10.2/sql/mysqld.cc:6570
#2  0x000055c35933997f in mysqld_main (argc=142, argv=0x7f1823c65ac0) at /home/jan/mysql/10.2/sql/mysqld.cc:6036
#3  0x000055c35932e220 in main (argc=22, argv=0x7ffcf04f3d38) at /home/jan/mysql/10.2/sql/main.cc:25
 
Thread 24 (Thread 0x7f1827222300 (LWP 20517)):
#0  0x00007f182477356d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055c359e61b83 in vio_io_wait (vio=0x7f1809bc6af0, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/jan/mysql/10.2/vio/viosocket.c:945
#2  0x000055c359e6038b in vio_socket_io_wait (vio=0x7f1809bc6af0, event=VIO_IO_EVENT_READ) at /home/jan/mysql/10.2/vio/viosocket.c:108
#3  0x000055c359e60542 in vio_read (vio=0x7f1809bc6af0, buf=0x7f1818878070 "\a", size=4) at /home/jan/mysql/10.2/vio/viosocket.c:184
#4  0x000055c35934314a in my_real_read (net=0x7f1818836338, complen=0x7f1827221500, header=1 '\001') at /home/jan/mysql/10.2/sql/net_serv.cc:888
#5  0x000055c359343949 in my_net_read_packet_reallen (net=0x7f1818836338, read_from_server=1 '\001', reallen=0x7f1827221568) at /home/jan/mysql/10.2/sql/net_serv.cc:1158
#6  0x000055c3593438fd in my_net_read_packet (net=0x7f1818836338, read_from_server=1 '\001') at /home/jan/mysql/10.2/sql/net_serv.cc:1142
#7  0x000055c35940a72e in do_command (thd=0x7f1818836070) at /home/jan/mysql/10.2/sql/sql_parse.cc:1244
#8  0x000055c359552711 in do_handle_one_connection (connect=0x7f1823c792f0) at /home/jan/mysql/10.2/sql/sql_connect.cc:1354
#9  0x000055c359552491 in handle_one_connection (arg=0x7f1823c792f0) at /home/jan/mysql/10.2/sql/sql_connect.cc:1260
#10 0x000055c359da4131 in pfs_spawn_thread (arg=0x7f1809bc6d70) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#11 0x00007f1826efd464 in start_thread (arg=0x7f1827222300) at pthread_create.c:333
#12 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 23 (Thread 0x7f182726c300 (LWP 20514)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c3599440a2 in os_event::wait (this=0x7f181ceed680) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:166
#2  0x000055c359943ace in os_event::wait_low (this=0x7f181ceed680, reset_sig_count=530) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:336
#3  0x000055c359943e4b in os_event_wait_low (event=0x7f181ceed680, reset_sig_count=530) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:535
#4  0x000055c359a4b183 in sync_array_wait_event (arr=0x7f1823c781e0, cell=@0x7f1827267238: 0x7f181f1ff0c0) at /home/jan/mysql/10.2/storage/innobase/sync/sync0arr.cc:490
#5  0x000055c359a4e2c1 in rw_lock_x_lock_wait_func (lock=0x7f1823c78c60, pass=0, threshold=0, file_name=0x55c359fd6580 "/home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc", line=3630) at /home/jan/mysql/10.2/storage/innobase/sync/sync0rw.cc:477
#6  0x000055c359a4e45e in rw_lock_x_lock_low (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fd6580 "/home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc", line=3630) at /home/jan/mysql/10.2/storage/innobase/sync/sync0rw.cc:547
#7  0x000055c359a4e8b7 in rw_lock_x_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fd6580 "/home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc", line=3630) at /home/jan/mysql/10.2/storage/innobase/sync/sync0rw.cc:705
#8  0x000055c3599c40cf in pfs_rw_lock_x_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fd6580 "/home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc", line=3630) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:556
#9  0x000055c3599ccb35 in row_mysql_lock_data_dictionary_func (trx=0x7f181dc011c8, file=0x55c359fd6580 "/home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc", line=3630) at /home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc:2388
#10 0x000055c3599cf816 in row_drop_table_for_mysql (name=0x7f1827267e20 "test/t803", trx=0x7f181dc011c8, drop_db=true, create_failed=0, nonatomic=true) at /home/jan/mysql/10.2/storage/innobase/row/row0mysql.cc:3630
#11 0x000055c35988c9c0 in ha_innobase::delete_table (this=0x7f180a36b7d8, name=0x7f1827269960 "./test/t803") at /home/jan/mysql/10.2/storage/innobase/handler/ha_innodb.cc:13815
#12 0x000055c3596820aa in handler::ha_delete_table (this=0x7f180a36b7d8, name=0x7f1827269960 "./test/t803") at /home/jan/mysql/10.2/sql/handler.cc:4326
#13 0x000055c35967caf1 in ha_delete_table (thd=0x7f181cfe7070, table_type=0x7f1823c5ebf0, path=0x7f1827269960 "./test/t803", db=0x7f181863d120 "test", alias=0x7f1818641a88 "t803", generate_warning=false) at /home/jan/mysql/10.2/sql/handler.cc:2351
#14 0x000055c3594cf31d in mysql_rm_table_no_locks (thd=0x7f181cfe7070, tables=0x7f1818642588, if_exists=true, drop_temporary=false, drop_view=true, dont_log_query=true, dont_free_locks=false) at /home/jan/mysql/10.2/sql/sql_table.cc:2469
#15 0x000055c3593ddad6 in mysql_rm_db_internal (thd=0x7f181cfe7070, db=0x7f181863d120 "test", if_exists=false, silent=false) at /home/jan/mysql/10.2/sql/sql_db.cc:918
#16 0x000055c3593de1d4 in mysql_rm_db (thd=0x7f181cfe7070, db=0x7f181863d120 "test", if_exists=false) at /home/jan/mysql/10.2/sql/sql_db.cc:1065
#17 0x000055c359414f57 in mysql_execute_command (thd=0x7f181cfe7070) at /home/jan/mysql/10.2/sql/sql_parse.cc:4941
#18 0x000055c35941e443 in mysql_parse (thd=0x7f181cfe7070, rawbuf=0x7f181863d088 "DROP SCHEMA test", length=16, parser_state=0x7f182726add0, is_com_multi=false, is_next_command=false) at /home/jan/mysql/10.2/sql/sql_parse.cc:7858
#19 0x000055c35940c2b6 in dispatch_command (command=COM_QUERY, thd=0x7f181cfe7070, packet=0x7f1818631071 "DROP SCHEMA test", packet_length=16, is_com_multi=false, is_next_command=false) at /home/jan/mysql/10.2/sql/sql_parse.cc:1812
#20 0x000055c35940ac5f in do_command (thd=0x7f181cfe7070) at /home/jan/mysql/10.2/sql/sql_parse.cc:1361
#21 0x000055c359552711 in do_handle_one_connection (connect=0x7f1823c79130) at /home/jan/mysql/10.2/sql/sql_connect.cc:1354
#22 0x000055c359552491 in handle_one_connection (arg=0x7f1823c79130) at /home/jan/mysql/10.2/sql/sql_connect.cc:1260
#23 0x000055c359da4131 in pfs_spawn_thread (arg=0x7f181cfcf5f0) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#24 0x00007f1826efd464 in start_thread (arg=0x7f182726c300) at pthread_create.c:333
#25 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 22 (Thread 0x7f1827300300 (LWP 20510)):
#0  do_sigwait (set=<optimized out>, set@entry=0x7f18272ff9f0, sig=sig@entry=0x7f18272ff9c0) at ../sysdeps/unix/sysv/linux/sigwait.c:64
#1  0x00007f1826f06d2d in __sigwait (set=0x7f18272ff9f0, sig=0x7f18272ff9c0) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2  0x000055c3593349a9 in signal_hand (arg=0x0) at /home/jan/mysql/10.2/sql/mysqld.cc:3499
#3  0x000055c359da4131 in pfs_spawn_thread (arg=0x7f1823c60770) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#4  0x00007f1826efd464 in start_thread (arg=0x7f1827300300) at pthread_create.c:333
#5  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 21 (Thread 0x7f1819ffa700 (LWP 20492)):
#0  0x00007f182649b6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055c359937a04 in LinuxAIOHandler::collect (this=0x7f1819ff9d90) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:1898
#2  0x000055c359937f3c in LinuxAIOHandler::poll (this=0x7f1819ff9d90, m1=0x7f1819ff9e30, m2=0x7f1819ff9e38, request=0x7f1819ff9e40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2044
#3  0x000055c3599380c4 in os_aio_linux_handler (global_segment=5, m1=0x7f1819ff9e30, m2=0x7f1819ff9e38, request=0x7f1819ff9e40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2098
#4  0x000055c35993cc53 in os_aio_handler (segment=5, m1=0x7f1819ff9e30, m2=0x7f1819ff9e38, request=0x7f1819ff9e40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:5669
#5  0x000055c359ba4f70 in fil_aio_wait (segment=5) at /home/jan/mysql/10.2/storage/innobase/fil/fil0fil.cc:5356
#6  0x000055c359a4136a in io_handler_thread (arg=0x55c35a9249c8 <n+40>) at /home/jan/mysql/10.2/storage/innobase/srv/srv0start.cc:327
#7  0x00007f1826efd464 in start_thread (arg=0x7f1819ffa700) at pthread_create.c:333
#8  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 20 (Thread 0x7f180e3ff700 (LWP 20508)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c359e02a83 in safe_cond_wait (cond=0x7f180e3fed80, mp=0x7f180e3fedc0, file=0x55c359f90ac8 "/home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h", line=1149) at /home/jan/mysql/10.2/mysys/thr_mutex.c:491
#2  0x000055c35986df5c in inline_mysql_cond_wait (that=0x7f180e3fed80, mutex=0x7f180e3fedc0, src_file=0x55c359f92478 "/home/jan/mysql/10.2/storage/innobase/handler/ha_innodb.cc", src_line=338) at /home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h:1149
#3  0x000055c359872f0d in thd_destructor_proxy () at /home/jan/mysql/10.2/storage/innobase/handler/ha_innodb.cc:338
#4  0x000055c359da4131 in pfs_spawn_thread (arg=0x7f1823c5b9f0) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#5  0x00007f1826efd464 in start_thread (arg=0x7f180e3ff700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 19 (Thread 0x7f18272b6300 (LWP 20511)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c359e02a83 in safe_cond_wait (cond=0x55c35a8ba380 <COND_slave_background>, mp=0x55c35a8bd600 <LOCK_slave_background>, file=0x55c359e736a8 "/home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h", line=1138) at /home/jan/mysql/10.2/mysys/thr_mutex.c:491
#2  0x000055c35935afdf in inline_mysql_cond_wait (that=0x55c35a8ba380 <COND_slave_background>, mutex=0x55c35a8bd600 <LOCK_slave_background>, src_file=0x55c359e73f90 "/home/jan/mysql/10.2/sql/slave.cc", src_line=339) at /home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h:1138
#3  0x000055c35935bc1c in handle_slave_background (arg=0x0) at /home/jan/mysql/10.2/sql/slave.cc:339
#4  0x000055c359da4131 in pfs_spawn_thread (arg=0x7f1823c613f0) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#5  0x00007f1826efd464 in start_thread (arg=0x7f18272b6300) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 18 (Thread 0x7f18103fd700 (LWP 20506)):
#0  0x00007f1826f0681d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055c3599443ba in os_thread_sleep (tm=1000000) at /home/jan/mysql/10.2/storage/innobase/os/os0thread.cc:225
#2  0x000055c359afd1dd in btr_defragment_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/btr/btr0defragment.cc:757
#3  0x00007f1826efd464 in start_thread (arg=0x7f18103fd700) at pthread_create.c:333
#4  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 17 (Thread 0x7f180f3ff700 (LWP 20507)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c3599440a2 in os_event::wait (this=0x7f181f223e80) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:166
#2  0x000055c359943ace in os_event::wait_low (this=0x7f181f223e80, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:336
#3  0x000055c359943e4b in os_event_wait_low (event=0x7f181f223e80, reset_sig_count=0) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:535
#4  0x000055c359b0ac02 in buf_resize_thread () at /home/jan/mysql/10.2/storage/innobase/buf/buf0buf.cc:3170
#5  0x00007f1826efd464 in start_thread (arg=0x7f180f3ff700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 16 (Thread 0x7f180c7ff700 (LWP 20509)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359e02d8d in safe_cond_timedwait (cond=0x55c35b134ae0 <COND_checkpoint>, mp=0x55c35b134a20 <LOCK_checkpoint>, abstime=0x7f180c7fedc0, file=0x55c35a068020 "/home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h", line=1175) at /home/jan/mysql/10.2/mysys/thr_mutex.c:545
#2  0x000055c359c84007 in inline_mysql_cond_timedwait (that=0x55c35b134ae0 <COND_checkpoint>, mutex=0x55c35b134a20 <LOCK_checkpoint>, abstime=0x7f180c7fedc0, src_file=0x55c35a068058 "/home/jan/mysql/10.2/storage/maria/ma_servicethread.c", src_line=116) at /home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h:1175
#3  0x000055c359c84509 in my_service_thread_sleep (control=0x55c35a6fc280 <checkpoint_control>, sleep_time=30000000000) at /home/jan/mysql/10.2/storage/maria/ma_servicethread.c:115
#4  0x000055c359c7915b in ma_checkpoint_background (arg=0x1e) at /home/jan/mysql/10.2/storage/maria/ma_checkpoint.c:708
#5  0x000055c359da4131 in pfs_spawn_thread (arg=0x7f1823c60270) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#6  0x00007f1826efd464 in start_thread (arg=0x7f180c7ff700) at pthread_create.c:333
#7  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 15 (Thread 0x7f18113ff700 (LWP 20504)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359943938 in os_event::timed_wait (this=0x7f181f3bf980, abstime=0x7f18113fee60) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:286
#2  0x000055c359943c6c in os_event::wait_time_low (this=0x7f181f3bf980, time_in_usec=10000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:413
#3  0x000055c359943e26 in os_event_wait_time_low (event=0x7f181f3bf980, time_in_usec=10000000, reset_sig_count=0) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:518
#4  0x000055c359b8c1fa in dict_stats_thread () at /home/jan/mysql/10.2/storage/innobase/dict/dict0stats_bg.cc:422
#5  0x00007f1826efd464 in start_thread (arg=0x7f18113ff700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 14 (Thread 0x7f1810bfe700 (LWP 20505)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359943938 in os_event::timed_wait (this=0x7f181cefd100, abstime=0x7f1810bfdc30) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:286
#2  0x000055c359943c6c in os_event::wait_time_low (this=0x7f181cefd100, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:413
#3  0x000055c359943e26 in os_event_wait_time_low (event=0x7f181cefd100, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:518
#4  0x000055c359aaa9e4 in ib_wqueue_timedwait (wq=0x7f181cef7f80, wait_in_usecs=5000000) at /home/jan/mysql/10.2/storage/innobase/ut/ut0wqueue.cc:163
#5  0x000055c359bf18f5 in fts_optimize_thread (arg=0x7f181cef7f80) at /home/jan/mysql/10.2/storage/innobase/fts/fts0opt.cc:3046
#6  0x00007f1826efd464 in start_thread (arg=0x7f1810bfe700) at pthread_create.c:333
#7  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 13 (Thread 0x7f18127ff700 (LWP 20503)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c3599440a2 in os_event::wait (this=0x7f181f223d80) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:166
#2  0x000055c359943ace in os_event::wait_low (this=0x7f181f223d80, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:336
#3  0x000055c359943e4b in os_event_wait_low (event=0x7f181f223d80, reset_sig_count=0) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:535
#4  0x000055c359b227d7 in buf_dump_thread () at /home/jan/mysql/10.2/storage/innobase/buf/buf0dump.cc:799
#5  0x00007f1826efd464 in start_thread (arg=0x7f18127ff700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 12 (Thread 0x7f18147f8700 (LWP 20502)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055c3599440a2 in os_event::wait (this=0x7f181ceed600) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:166
#2  0x000055c359943ace in os_event::wait_low (this=0x7f181ceed600, reset_sig_count=770) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:336
#3  0x000055c359943e4b in os_event_wait_low (event=0x7f181ceed600, reset_sig_count=770) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:535
#4  0x000055c359a4b183 in sync_array_wait_event (arr=0x7f1823c781e0, cell=@0x7f18147f7c18: 0x7f181f1ff000) at /home/jan/mysql/10.2/storage/innobase/sync/sync0arr.cc:490
#5  0x000055c359a4e10e in rw_lock_s_lock_spin (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/sync/sync0rw.cc:391
#6  0x000055c3599e8eb5 in rw_lock_s_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:298
#7  0x000055c3599e926f in pfs_rw_lock_s_lock_func (lock=0x7f1823c78c60, pass=0, file_name=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=861) at /home/jan/mysql/10.2/storage/innobase/include/sync0rw.ic:649
#8  0x000055c3599edddf in row_purge_parse_undo_rec (node=0x7f181f779088, undo_rec=0x7f18138dd820 "\001\267\016\002\002", updated_extern=0x7f18147f7d9e, thr=0x7f1823c67690) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:861
#9  0x000055c3599ee367 in row_purge (node=0x7f181f779088, undo_rec=0x7f18138dd820 "\001\267\016\002\002", thr=0x7f1823c67690) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1028
#10 0x000055c3599ee636 in row_purge_step (thr=0x7f1823c67690) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1110
#11 0x000055c359977118 in que_thr_step (thr=0x7f1823c67690) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1057
#12 0x000055c35997731c in que_run_threads_low (thr=0x7f1823c67690) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1119
#13 0x000055c3599774ce in que_run_threads (thr=0x7f1823c67690) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1159
#14 0x000055c359a3db3f in srv_task_execute () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2669
#15 0x000055c359a3dcc3 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2718
#16 0x00007f1826efd464 in start_thread (arg=0x7f18147f8700) at pthread_create.c:333
#17 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 11 (Thread 0x7f1816bfc700 (LWP 20498)):
#0  0x00007f1826f0681d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055c3599443ba in os_thread_sleep (tm=1000000) at /home/jan/mysql/10.2/storage/innobase/os/os0thread.cc:225
#2  0x000055c359a3d67e in srv_master_sleep () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2523
#3  0x000055c359a3d7ce in srv_master_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2570
#4  0x00007f1826efd464 in start_thread (arg=0x7f1816bfc700) at pthread_create.c:333
#5  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 10 (Thread 0x7f1817bfe700 (LWP 20496)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359943938 in os_event::timed_wait (this=0x7f181f223c80, abstime=0x7f1817bfdc80) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:286
#2  0x000055c359943c6c in os_event::wait_time_low (this=0x7f181f223c80, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:413
#3  0x000055c359943e26 in os_event_wait_time_low (event=0x7f181f223c80, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:518
#4  0x000055c359a3c4c2 in srv_error_monitor_thread () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:1986
#5  0x00007f1826efd464 in start_thread (arg=0x7f1817bfe700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 9 (Thread 0x7f18173fd700 (LWP 20497)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359943938 in os_event::timed_wait (this=0x7f181f223d00, abstime=0x7f18173fce10) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:286
#2  0x000055c359943c6c in os_event::wait_time_low (this=0x7f181f223d00, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:413
#3  0x000055c359943e26 in os_event_wait_time_low (event=0x7f181f223d00, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:518
#4  0x000055c359a3bdf1 in srv_monitor_thread () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:1761
#5  0x00007f1826efd464 in start_thread (arg=0x7f18173fd700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 8 (Thread 0x7f181bffe700 (LWP 20488)):
#0  0x00007f182649b6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055c359937a04 in LinuxAIOHandler::collect (this=0x7f181bffdd90) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:1898
#2  0x000055c359937f3c in LinuxAIOHandler::poll (this=0x7f181bffdd90, m1=0x7f181bffde30, m2=0x7f181bffde38, request=0x7f181bffde40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2044
#3  0x000055c3599380c4 in os_aio_linux_handler (global_segment=1, m1=0x7f181bffde30, m2=0x7f181bffde38, request=0x7f181bffde40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2098
#4  0x000055c35993cc53 in os_aio_handler (segment=1, m1=0x7f181bffde30, m2=0x7f181bffde38, request=0x7f181bffde40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:5669
#5  0x000055c359ba4f70 in fil_aio_wait (segment=1) at /home/jan/mysql/10.2/storage/innobase/fil/fil0fil.cc:5356
#6  0x000055c359a4136a in io_handler_thread (arg=0x55c35a9249a8 <n+8>) at /home/jan/mysql/10.2/storage/innobase/srv/srv0start.cc:327
#7  0x00007f1826efd464 in start_thread (arg=0x7f181bffe700) at pthread_create.c:333
#8  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 7 (Thread 0x7f181b7fd700 (LWP 20489)):
#0  0x00007f182649b6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055c359937a04 in LinuxAIOHandler::collect (this=0x7f181b7fcd90) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:1898
#2  0x000055c359937f3c in LinuxAIOHandler::poll (this=0x7f181b7fcd90, m1=0x7f181b7fce30, m2=0x7f181b7fce38, request=0x7f181b7fce40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2044
#3  0x000055c3599380c4 in os_aio_linux_handler (global_segment=2, m1=0x7f181b7fce30, m2=0x7f181b7fce38, request=0x7f181b7fce40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2098
#4  0x000055c35993cc53 in os_aio_handler (segment=2, m1=0x7f181b7fce30, m2=0x7f181b7fce38, request=0x7f181b7fce40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:5669
#5  0x000055c359ba4f70 in fil_aio_wait (segment=2) at /home/jan/mysql/10.2/storage/innobase/fil/fil0fil.cc:5356
#6  0x000055c359a4136a in io_handler_thread (arg=0x55c35a9249b0 <n+16>) at /home/jan/mysql/10.2/storage/innobase/srv/srv0start.cc:327
#7  0x00007f1826efd464 in start_thread (arg=0x7f181b7fd700) at pthread_create.c:333
#8  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 6 (Thread 0x7f18197f9700 (LWP 20493)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359943938 in os_event::timed_wait (this=0x7f181f223e00, abstime=0x7f18197f8be0) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:286
#2  0x000055c359943c6c in os_event::wait_time_low (this=0x7f181f223e00, time_in_usec=981000, reset_sig_count=112) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:413
#3  0x000055c359943e26 in os_event_wait_time_low (event=0x7f181f223e00, time_in_usec=981000, reset_sig_count=112) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:518
#4  0x000055c359b2bcd3 in pc_sleep_if_needed (next_loop_time=1487827990748, sig_count=112) at /home/jan/mysql/10.2/storage/innobase/buf/buf0flu.cc:2718
#5  0x000055c359b2cdd4 in buf_flush_page_cleaner_coordinator (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/buf/buf0flu.cc:3220
#6  0x00007f1826efd464 in start_thread (arg=0x7f18197f9700) at pthread_create.c:333
#7  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 5 (Thread 0x7f18183ff700 (LWP 20495)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359943938 in os_event::timed_wait (this=0x7f181f3bf100, abstime=0x7f18183fee40) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:286
#2  0x000055c359943c6c in os_event::wait_time_low (this=0x7f181f3bf100, time_in_usec=1000000, reset_sig_count=2) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:413
#3  0x000055c359943e26 in os_event_wait_time_low (event=0x7f181f3bf100, time_in_usec=1000000, reset_sig_count=2) at /home/jan/mysql/10.2/storage/innobase/os/os0event.cc:518
#4  0x000055c3599111bb in lock_wait_timeout_thread () at /home/jan/mysql/10.2/storage/innobase/lock/lock0wait.cc:529
#5  0x00007f1826efd464 in start_thread (arg=0x7f18183ff700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 4 (Thread 0x7f181affc700 (LWP 20490)):
#0  0x00007f182649b6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055c359937a04 in LinuxAIOHandler::collect (this=0x7f181affbd90) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:1898
#2  0x000055c359937f3c in LinuxAIOHandler::poll (this=0x7f181affbd90, m1=0x7f181affbe30, m2=0x7f181affbe38, request=0x7f181affbe40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2044
#3  0x000055c3599380c4 in os_aio_linux_handler (global_segment=3, m1=0x7f181affbe30, m2=0x7f181affbe38, request=0x7f181affbe40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2098
#4  0x000055c35993cc53 in os_aio_handler (segment=3, m1=0x7f181affbe30, m2=0x7f181affbe38, request=0x7f181affbe40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:5669
#5  0x000055c359ba4f70 in fil_aio_wait (segment=3) at /home/jan/mysql/10.2/storage/innobase/fil/fil0fil.cc:5356
#6  0x000055c359a4136a in io_handler_thread (arg=0x55c35a9249b8 <n+24>) at /home/jan/mysql/10.2/storage/innobase/srv/srv0start.cc:327
#7  0x00007f1826efd464 in start_thread (arg=0x7f181affc700) at pthread_create.c:333
#8  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 3 (Thread 0x7f181c7ff700 (LWP 20487)):
#0  0x00007f182649b6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055c359937a04 in LinuxAIOHandler::collect (this=0x7f181c7fed90) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:1898
#2  0x000055c359937f3c in LinuxAIOHandler::poll (this=0x7f181c7fed90, m1=0x7f181c7fee30, m2=0x7f181c7fee38, request=0x7f181c7fee40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2044
#3  0x000055c3599380c4 in os_aio_linux_handler (global_segment=0, m1=0x7f181c7fee30, m2=0x7f181c7fee38, request=0x7f181c7fee40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:2098
#4  0x000055c35993cc53 in os_aio_handler (segment=0, m1=0x7f181c7fee30, m2=0x7f181c7fee38, request=0x7f181c7fee40) at /home/jan/mysql/10.2/storage/innobase/os/os0file.cc:5669
#5  0x000055c359ba4f70 in fil_aio_wait (segment=0) at /home/jan/mysql/10.2/storage/innobase/fil/fil0fil.cc:5356
#6  0x000055c359a4136a in io_handler_thread (arg=0x55c35a9249a0 <n>) at /home/jan/mysql/10.2/storage/innobase/srv/srv0start.cc:327
#7  0x00007f1826efd464 in start_thread (arg=0x7f181c7ff700) at pthread_create.c:333
#8  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 2 (Thread 0x7f181efff700 (LWP 20486)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055c359e02d8d in safe_cond_timedwait (cond=0x55c35b1461e0 <COND_timer>, mp=0x55c35b146120 <LOCK_timer>, abstime=0x7f181effee70, file=0x55c35a0a4d20 "/home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h", line=1175) at /home/jan/mysql/10.2/mysys/thr_mutex.c:545
#2  0x000055c359e03d02 in inline_mysql_cond_timedwait (that=0x55c35b1461e0 <COND_timer>, mutex=0x55c35b146120 <LOCK_timer>, abstime=0x7f181effee70, src_file=0x55c35a0a4d58 "/home/jan/mysql/10.2/mysys/thr_timer.c", src_line=292) at /home/jan/mysql/10.2/include/mysql/psi/mysql_thread.h:1175
#3  0x000055c359e0480f in timer_handler (arg=0x0) at /home/jan/mysql/10.2/mysys/thr_timer.c:292
#4  0x000055c359da4131 in pfs_spawn_thread (arg=0x7f1823c5b9f0) at /home/jan/mysql/10.2/storage/perfschema/pfs.cc:1862
#5  0x00007f1826efd464 in start_thread (arg=0x7f181efff700) at pthread_create.c:333
#6  0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
Thread 1 (Thread 0x7f1815bfa700 (LWP 20500)):
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x000055c359dfd533 in my_write_core (sig=6) at /home/jan/mysql/10.2/mysys/stacktrace.c:477
#2  0x000055c359677380 in handle_fatal_signal (sig=6) at /home/jan/mysql/10.2/sql/signal_handler.cc:296
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#5  0x00007f18246c840a in __GI_abort () at abort.c:89
#6  0x000055c359aa67c0 in ut_dbg_assertion_failed (expr=0x55c359fddf5f "0", file=0x55c359fde538 "/home/jan/mysql/10.2/storage/innobase/row/row0purge.cc", line=498) at /home/jan/mysql/10.2/storage/innobase/ut/ut0dbg.cc:59
#7  0x000055c3599ed0ba in row_purge_remove_sec_if_poss_leaf (node=0x7f1823c674f0, index=0x7f181fbd0488, entry=0x7f1812c1cb88) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:498
#8  0x000055c3599ed3fe in row_purge_remove_sec_if_poss (node=0x7f1823c674f0, index=0x7f181fbd0488, entry=0x7f1812c1cb88) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:587
#9  0x000055c3599ed610 in row_purge_del_mark (node=0x7f1823c674f0) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:657
#10 0x000055c3599ee17f in row_purge_record_func (node=0x7f1823c674f0, undo_rec=0x7f18138db0d0 "\001\355\016\003\001", thr=0x7f1823c67428, updated_extern=false) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:975
#11 0x000055c3599ee386 in row_purge (node=0x7f1823c674f0, undo_rec=0x7f18138db0d0 "\001\355\016\003\001", thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1031
#12 0x000055c3599ee636 in row_purge_step (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/row/row0purge.cc:1110
#13 0x000055c359977118 in que_thr_step (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1057
#14 0x000055c35997731c in que_run_threads_low (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1119
#15 0x000055c3599774ce in que_run_threads (thr=0x7f1823c67428) at /home/jan/mysql/10.2/storage/innobase/que/que0que.cc:1159
#16 0x000055c359a3db3f in srv_task_execute () at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2669
#17 0x000055c359a3dcc3 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2/storage/innobase/srv/srv0srv.cc:2718
#18 0x00007f1826efd464 in start_thread (arg=0x7f1815bfa700) at pthread_create.c:333
#19 0x00007f182477c9df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Comment by Jan Lindström (Inactive) [ 2017-02-23 ]

Update: failure happens at DROP SCHEMA; at the end of test, as not all drop database/drop schema does fail this failure is related to fact that we have ~1024 open files. Bug was introduced by following commit (found by bisecting:

commit 8375a2c1cec3c386f6db797d75f64b8f03c83b37
Author: Marko Mäkelä <marko.makela@mariadb.com>
Date:   Fri Dec 16 16:36:54 2016 +0200
 
    MDEV-11585 Hard-code the shared InnoDB temporary tablespace ID at -1
    
    MySQL 5.7 supports only one shared temporary tablespace.
    
    MariaDB 10.2 does not support any other shared InnoDB tablespaces than
    the two predefined tablespaces: the persistent InnoDB system tablespace
    (default file name ibdata1) and the temporary tablespace
    (default file name ibtmp1).
    
    InnoDB is unnecessarily allocating a tablespace ID for the predefined
    temporary tablespace on every startup, and it is in several places
    testing whether a tablespace ID matches this dynamically generated ID.
    
    We should use a compile-time constant to reduce code size and to avoid
    unnecessary updates to the DICT_HDR page at every startup.
    
    Using a hard-coded tablespace ID will should make it easier to remove the
    TEMPORARY flag from FSP_SPACE_FLAGS in MDEV-11202.

Comment by Jan Lindström (Inactive) [ 2017-02-23 ]

Fix for crash, but please review other conditions:

diff --git a/storage/innobase/include/fsp0space.h b/storage/innobase/include/fsp0space.h
index 6ecca6211ed..827a0e93f7d 100644
--- a/storage/innobase/include/fsp0space.h
+++ b/storage/innobase/include/fsp0space.h
@@ -205,7 +205,7 @@ class Tablespace {
        @return true if undo tablespace */
        static bool is_undo_tablespace(ulint id)
        {
-               return(id <= srv_undo_tablespaces_open);
+               return(id == SRV_TMP_SPACE_ID);
        }
 private:
        /**

Comment by Marko Mäkelä [ 2017-02-24 ]

jplindst, thanks for identifying the cause. The proper patch should be this one:

diff --git a/storage/innobase/include/fsp0space.h b/storage/innobase/include/fsp0space.h
index 6ecca6211ed..eb26addcca5 100644
--- a/storage/innobase/include/fsp0space.h
+++ b/storage/innobase/include/fsp0space.h
@@ -201,11 +201,12 @@ class Tablespace {
 		return(&m_files.front());
 	}
 
-	/** Check if undo tablespace.
-	@return true if undo tablespace */
+	/** Check if this is a dedicated undo tablespace.
+	@param[in]	id	tablespace ID
+	@return whether this is a dedicated undo tablespace */
 	static bool is_undo_tablespace(ulint id)
 	{
-		return(id <= srv_undo_tablespaces_open);
+		return(id != 0 && id <= srv_undo_tablespaces_open);
 	}
 private:
 	/**

I will investigate a bit more to understand why the wrong implementation was causing the issue. It currently seems to me that this would affect any table that is stored in the InnoDB system tablespace. I would expect that most users (and tests) would create their tables in separate .ibd files and be unaffected by this.

Comment by Marko Mäkelä [ 2017-02-24 ]

The predicate Tablespace::is_undo_tablespace() is only invoked from buf_page_io_complete().
Due to the bug, we are disabling the change buffer merge for the system tablespace, corrupting the secondary indexes in the system tablespace.

There will be no crash if I run the test with

./mtr --mysqld=--innodb-change-buffering=none …

This bug only affects big workloads that modify secondary indexes in some tables that are located in the innodb_system tablespace.
A work-around to avoid this corruption is to set the configuration parameter

innodb_change_buffering=none

Comment by Marko Mäkelä [ 2017-02-24 ]

bb-10.2-marko

Comment by Jan Lindström (Inactive) [ 2017-02-24 ]

ok to push.

Generated at Thu Feb 08 07:53:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.