Marko Mäkelä added a comment - 2016-11-28 09:55 The reason of the deadlock might be a violation of the B-tree latching rules, which were changed in WL#6326 in MySQL 5.7 . It could be that something (probably not `btr_cur_search_to_nth_level()`) would incorrectly release the `index->lock` X-latch and all parent page latches leading from the index tree root to the leaf page. The defragmentation code will need to modify node pointer pages. So, we should acquire and hold page latches to all potentially affected non-leaf pages, or (as a quick and dirty fix) we should hold `index->lock` X-latch (which would allow us to reacquire any non-leaf page latches in arbitrary order).

Marko Mäkelä added a comment - 2017-01-16 13:56 I repeated this once locally in 10.2, for the same test. Stack trace: #7 0x0000000000f0792b in ib::fatal::~fatal (this=0x7ff853ffb4a8) at /home/marko/mariadb/server/storage/innobase/ut/ut0ut.cc:849 #8 0x0000000000e94617 in sync_array_wait_event (arr=0x44c26a0, cell=@0x7ff853ffb650: 0x7ff8a61f9090) at /home/marko/mariadb/server/storage/innobase/sync/sync0arr.cc:482 #9 0x0000000000e9a2a5 in rw_lock_x_lock_wait_func (lock=0x7ff8a54d7de0, pass=0, threshold=-268435456, file_name=0x152ea94 "/home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc", line=1923) at /home/marko/mariadb/server/storage/innobase/sync/sync0rw.cc:477 #10 0x0000000000e9890d in rw_lock_x_lock_low (lock=0x7ff8a54d7de0, pass=0, file_name=0x152ea94 "/home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc", line=1923) at /home/marko/mariadb/server/storage/innobase/sync/sync0rw.cc:566 #11 0x0000000000e984a0 in rw_lock_x_lock_func (lock=0x7ff8a54d7de0, pass=0, file_name=0x152ea94 "/home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc", line=1923) at /home/marko/mariadb/server/storage/innobase/sync/sync0rw.cc:705 #12 0x0000000000f7fc3f in pfs_rw_lock_x_lock_func (lock=0x7ff8a54d7de0, pass=0, file_name=0x152ea94 "/home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc", line=1923) at /home/marko/mariadb/server/storage/innobase/include/sync0rw.ic:562 #13 0x0000000000f84040 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=2, guess=0x0, mode=10, file=0x152ea94 "/home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc", line=1923, mtr=0x7ff853ffe9d0, err=0x7ff853ffbffc) at /home/marko/mariadb/server/storage/innobase/buf/buf0buf.cc:4823 #14 0x0000000000f37d91 in btr_block_get_func (page_id=..., page_size=..., mode=2, file=0x152ea94 "/home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc", line=1923, index=0x7ff83c145578, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/include/btr0btr.ic:61 #15 0x0000000000f3d072 in btr_cur_search_to_nth_level (index=0x7ff83c145578, level=2, tuple=0x7ff8240461e8, mode=PAGE_CUR_LE, latch_mode=34, cursor=0x7ff853ffddf0, has_search_latch=0, file=0x152c19b "/home/marko/mariadb/server/storage/innobase/btr/btr0btr.cc", line=1023, mtr=0x7ff853ffe9d0, autoinc=0) at /home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc:1921 #16 0x0000000000f29876 in btr_page_get_father_node_ptr_func (offsets=0x0, heap=0x7ff824046060, cursor=0x7ff853ffddf0, latch_mode=34, file=0x152c19b "/home/marko/mariadb/server/storage/innobase/btr/btr0btr.cc", line=1023, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/btr/btr0btr.cc:922 #17 0x0000000000f1e737 in btr_page_get_father (index=0x7ff83c145578, block=0x7ff8a54e8cd8, mtr=0x7ff853ffe9d0, cursor=0x7ff853ffddf0) at /home/marko/mariadb/server/storage/innobase/btr/btr0btr.cc:1023 #18 0x0000000000f1e5e6 in btr_node_ptr_delete (index=0x7ff83c145578, block=0x7ff8a54e8cd8, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/btr/btr0btr.cc:3391 #19 0x0000000000f4b65d in btr_cur_pessimistic_delete (err=0x7ff853ffe104, has_reserved_extents=1, cursor=0x7ff853ffe110, flags=16, rollback=false, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/btr/btr0cur.cc:5161 #20 0x0000000000f1e60e in btr_node_ptr_delete (index=0x7ff83c145578, block=0x7ff8a54fea70, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/btr/btr0btr.cc:3393 #21 0x0000000000f701d9 in btr_defragment_merge_pages (index=0x7ff83c145578, from_block=0x7ff8a54fea70, to_block=0x7ff8a54cb730, page_size=..., reserved_space=1625, max_data_size=0x7ff853ffe3b8, heap=0x7ff824005530, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/btr/btr0defragment.cc:542 #22 0x0000000000f70f9d in btr_defragment_n_pages (block=0x7ff8a5529b08, index=0x7ff83c145578, n_pages=20, mtr=0x7ff853ffe9d0) at /home/marko/mariadb/server/storage/innobase/btr/btr0defragment.cc:714 #23 0x0000000000f6e04a in btr_defragment_thread (arg=0x0) at /home/marko/mariadb/server/storage/innobase/btr/btr0defragment.cc:803

Jan Lindström (Inactive) added a comment - 2017-05-02 10:55 - edited Sorry, do not know how to fix this, above latching rules change causes this. marko can you help ?

Marko Mäkelä added a comment - 2017-05-08 14:23 jplindst , I believe that you temporarily worked around this by disabling the feature (but curiously, not disabling the tests). Did you already document this somewhere? It should make its way to the release notes.

Marko Mäkelä added a comment - 2017-05-08 18:54 - edited It looks like the tests innodb.innodb_defragment_small and innodb.defrag_mdl-9155 and possibly others should have been disabled when disabling the feature. The following tests outside sys_vars appear to have something to do with innodb_defragment: encryption.innodb-bad-key-change4 innodb.defrag_mdl-9155 innodb.innodb_defrag_concurrent innodb.innodb_defrag_stats innodb.innodb_defrag_stats_many_tables innodb.innodb_defragment innodb.innodb_defragment_fill_factor innodb.innodb_defragment_small innodb.innodb_defrag_binlog I disabled these tests .

Hartmut Holzgraefe added a comment - 2017-07-04 15:32 Shouldn't we prevent setting innodb_defragment to ON with a warning as long as the feature is disabled in 10.2? (and mention that it only works in 10.1 so far on the related knowledge base page? https://mariadb.com/kb/en/mariadb/defragmenting-innodb-tablespaces/

Jan Lindström (Inactive) added a comment - 2017-10-10 09:28 Do not know how to fix this. marko any better guidance, see the patch that does not work. Stack: ----------SERVER LOG START----------- ######################################## DEADLOCK of threads detected! rw-lock 0x7f863f3acf78 --Thread 140214561253120 has waited at btr0cur.cc line 6035 for 0.00 seconds the semaphore: S-lock on RW-latch at 0x7f863f3acf78 created in file buf0buf.cc line 1468 a writer (thread id 140214332794624) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: ffffffffdffffff0 Last time read locked in file btr0cur.cc line 6035 Last time write locked in file btr0cur.cc line 1917 Locked: thread 140214332794624 file btr0cur.cc line 1917 X-LOCK rw-lock 0x7f863f3ae8f8 --Thread 140214332794624 has waited at btr0cur.cc line 1917 for 0.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7f863f3ae8f8 created in file buf0buf.cc line 1468 a writer (thread id 140214332794624) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffefffffff Last time read locked in file btr0cur.cc line 6035 Last time write locked in file btr0btr.cc line 168 Locked: thread 140214561253120 file btr0cur.cc line 6035 S-LOCK Locked: thread 140214561253120 file btr0cur.cc line 6035 S-LOCK 2017-10-10 12:15:51 140214332794624 [ERROR] [FATAL] InnoDB: ######################################## Deadlock Detected! 171010 12:15:51 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.2.10-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=5 max_threads=153 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63097 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /home/jan/mysql/10.2-orig/sql/mysqld(my_print_stacktrace+0x38)[0x55e2c22075b0] /home/jan/mysql/10.2-orig/sql/mysqld(handle_fatal_signal+0x3a3)[0x55e2c1a9b168] /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f864bf3c0c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f864970efcf] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f86497103fa] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc3fd5d)[0x55e2c1ec1d5d] /home/jan/mysql/10.2-orig/sql/mysqld(+0xbe35af)[0x55e2c1e655af] /home/jan/mysql/10.2-orig/sql/mysqld(+0xbe6751)[0x55e2c1e68751] /home/jan/mysql/10.2-orig/sql/mysqld(+0xbe6926)[0x55e2c1e68926] /home/jan/mysql/10.2-orig/sql/mysqld(+0xbe6c6d)[0x55e2c1e68c6d] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc90f34)[0x55e2c1f12f34] /home/jan/mysql/10.2-orig/sql/mysqld(+0xca123b)[0x55e2c1f2323b] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc640dc)[0x55e2c1ee60dc] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc69e05)[0x55e2c1eebe05] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc4896e)[0x55e2c1eca96e] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc48fbc)[0x55e2c1ecafbc] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc4f1d4)[0x55e2c1ed11d4] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc72814)[0x55e2c1ef4814] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc4f204)[0x55e2c1ed1204] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc8e8aa)[0x55e2c1f108aa] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc8ef72)[0x55e2c1f10f72] /home/jan/mysql/10.2-orig/sql/mysqld(+0xc8f28f)[0x55e2c1f1128f] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7f864bf32494] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f86497c4aff] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file ----------SERVER LOG END-------------     - saving '/dev/shm/5/log/innodb.innodb_defrag_concurrent-innodb/' to '/dev/shm/log/innodb.innodb_defrag_concurrent-innodb/' - found 'core' (0/5)   Trying 'dbx' to get a backtrace   Trying 'gdb' to get a backtrace from coredump /dev/shm/log/innodb.innodb_defrag_concurrent-innodb/mysqld.1/data/core Core generated by '/home/jan/mysql/10.2-orig/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 3802] [New LWP 3669] [New LWP 3671] [New LWP 3674] [New LWP 3726] [New LWP 3778] [New LWP 3728] [New LWP 3672] [New LWP 3727] [New LWP 3779] [New LWP 3801] [New LWP 3673] [New LWP 3843] [New LWP 3803] [New LWP 3675] [New LWP 3804] [New LWP 3660] [New LWP 3676] [New LWP 3677] [New LWP 3854] [New LWP 3780] [New LWP 3855] [New LWP 3781] [New LWP 3856] [New LWP 3782] [New LWP 3857] [New LWP 3783] [New LWP 3784] [New LWP 3816] [New LWP 3818] [New LWP 3820] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/home/jan/mysql/10.2-orig/sql/mysqld --defaults-group-suffix=.1 --defaults-file'. Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 57 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory. [Current thread is 1 (Thread 0x7f86317fa700 (LWP 3802))] #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 #1 0x000055e2c22076a2 in my_write_core (sig=6) at /home/jan/mysql/10.2-orig/mysys/stacktrace.c:477 #2 0x000055e2c1a9b514 in handle_fatal_signal (sig=6) at /home/jan/mysql/10.2-orig/sql/signal_handler.cc:303 #3 <signal handler called> #4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #5 0x00007f86497103fa in __GI_abort () at abort.c:89 #6 0x000055e2c1ec1d5d in ib::fatal::~fatal (this=0x7f86317f76c0, __in_chrg=<optimized out>) at /home/jan/mysql/10.2-orig/storage/innobase/ut/ut0ut.cc:850 #7 0x000055e2c1e655af in sync_array_wait_event (arr=0x55e2c552f170, cell=@0x7f86317f7898: 0x7f864c346010) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0arr.cc:467 #8 0x000055e2c1e68751 in rw_lock_x_lock_wait_func (lock=0x7f863f3ae8f8, pass=0, threshold=-268435456, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:472 #9 0x000055e2c1e68926 in rw_lock_x_lock_low (lock=0x7f863f3ae8f8, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:557 #10 0x000055e2c1e68c6d in rw_lock_x_lock_func (lock=0x7f863f3ae8f8, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:687 #11 0x000055e2c1f12f34 in pfs_rw_lock_x_lock_func (lock=0x7f863f3ae8f8, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/include/sync0rw.ic:545 #12 0x000055e2c1f2323b in buf_page_get_gen (page_id=..., page_size=..., rw_latch=2, guess=0x0, mode=10, file=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917, mtr=0x7f86317f9860, err=0x7f86317f7c74) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0buf.cc:4770 #13 0x000055e2c1ee60dc in btr_block_get_func (page_id=..., page_size=..., mode=2, file=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917, index=0x7f85f804d278, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/include/btr0btr.ic:61 #14 0x000055e2c1eebe05 in btr_cur_search_to_nth_level (index=0x7f85f804d278, level=2, tuple=0x7f85cc0327c8, mode=PAGE_CUR_LE, latch_mode=34, cursor=0x7f86317f8ec0, has_search_latch=0, file=0x55e2c2468580 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc", line=1026, mtr=0x7f86317f9860, autoinc=0) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc:1917 #15 0x000055e2c1eca96e in btr_page_get_father_node_ptr_func (offsets=0x0, heap=0x7f85cc02c0d0, cursor=0x7f86317f8ec0, latch_mode=34, file=0x55e2c2468580 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc", line=1026, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:924 #16 0x000055e2c1ecafbc in btr_page_get_father (index=0x7f85f804d278, block=0x7f863f3c66a0, mtr=0x7f86317f9860, cursor=0x7f86317f8ec0) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:1026 #17 0x000055e2c1ed11d4 in btr_node_ptr_delete (index=0x7f85f804d278, block=0x7f863f3c66a0, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:3418 #18 0x000055e2c1ef4814 in btr_cur_pessimistic_delete (err=0x7f86317f9144, has_reserved_extents=1, cursor=0x7f86317f9150, flags=16, rollback=false, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc:5190 #19 0x000055e2c1ed1204 in btr_node_ptr_delete (index=0x7f85f804d278, block=0x7f863f3e9470, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:3421 #20 0x000055e2c1f108aa in btr_defragment_merge_pages (index=0x7f85f804d278, from_block=0x7f863f3e9470, to_block=0x7f863f3ae4f0, page_size=..., reserved_space=1625, max_data_size=0x7f86317f9330, heap=0x7f85cc000c20, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0defragment.cc:562 #21 0x000055e2c1f10f72 in btr_defragment_n_pages (block=0x7f863f3fc670, index=0x7f85f804d278, n_pages=20, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0defragment.cc:718 #22 0x000055e2c1f1128f in btr_defragment_thread () at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0defragment.cc:811 #23 0x00007f864bf32494 in start_thread (arg=0x7f86317fa700) at pthread_create.c:333 #24 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 31 (Thread 0x7f863f34c700 (LWP 3820)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c220cc20 in safe_cond_wait (cond=0x55e2c2d5ce40 <COND_slave_background>, mp=0x55e2c2d60040 <LOCK_slave_background>, file=0x55e2c227ed48 "/home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h", line=1138) at /home/jan/mysql/10.2-orig/mysys/thr_mutex.c:491 #2 0x000055e2c1778890 in inline_mysql_cond_wait (that=0x55e2c2d5ce40 <COND_slave_background>, mutex=0x55e2c2d60040 <LOCK_slave_background>, src_file=0x55e2c227f670 "/home/jan/mysql/10.2-orig/sql/slave.cc", src_line=334) at /home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h:1138 #3 0x000055e2c17794c2 in handle_slave_background (arg=0x0) at /home/jan/mysql/10.2-orig/sql/slave.cc:334 #4 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5a8e0e0) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #5 0x00007f864bf32494 in start_thread (arg=0x7f863f34c700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 30 (Thread 0x7f863f396700 (LWP 3818)): #0 do_sigwait (set=<optimized out>, set@entry=0x7f863f395cb0, sig=sig@entry=0x7f863f395c80) at ../sysdeps/unix/sysv/linux/sigwait.c:64 #1 0x00007f864bf3bced in __sigwait (set=0x7f863f395cb0, sig=0x7f863f395c80) at ../sysdeps/unix/sysv/linux/sigwait.c:96 #2 0x000055e2c1751e02 in signal_hand (arg=0x0) at /home/jan/mysql/10.2-orig/sql/mysqld.cc:3499 #3 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5a08940) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #4 0x00007f864bf32494 in start_thread (arg=0x7f863f396700) at pthread_create.c:333 #5 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 29 (Thread 0x7f86177fe700 (LWP 3816)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c220cf2a in safe_cond_timedwait (cond=0x55e2c35dfca0 <COND_checkpoint>, mp=0x55e2c35dfbe0 <LOCK_checkpoint>, abstime=0x7f86177fdc80, file=0x55e2c2508c70 "/home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h", line=1175) at /home/jan/mysql/10.2-orig/mysys/thr_mutex.c:545 #2 0x000055e2c208d509 in inline_mysql_cond_timedwait (that=0x55e2c35dfca0 <COND_checkpoint>, mutex=0x55e2c35dfbe0 <LOCK_checkpoint>, abstime=0x7f86177fdc80, src_file=0x55e2c2508cb0 "/home/jan/mysql/10.2-orig/storage/maria/ma_servicethread.c", src_line=116) at /home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h:1175 #3 0x000055e2c208da0b in my_service_thread_sleep (control=0x55e2c2b9eb80 <checkpoint_control>, sleep_time=30000000000) at /home/jan/mysql/10.2-orig/storage/maria/ma_servicethread.c:115 #4 0x000055e2c2082657 in ma_checkpoint_background (arg=0x1e) at /home/jan/mysql/10.2-orig/storage/maria/ma_checkpoint.c:709 #5 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c59e54c0) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #6 0x00007f864bf32494 in start_thread (arg=0x7f86177fe700) at pthread_create.c:333 #7 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 28 (Thread 0x7f86327fc700 (LWP 3784)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c5545d80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c5545d80, reset_sig_count=7442) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c5545d80, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1e55342 in srv_resume_thread (slot=0x55e2c2b9b900 <srv_sys+384>, sig_count=0, wait=true, timeout_usec=0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:940 #5 0x000055e2c1e58e43 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2647 #6 0x00007f864bf32494 in start_thread (arg=0x7f86327fc700) at pthread_create.c:333 #7 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 27 (Thread 0x7f8632ffd700 (LWP 3783)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c5545e00) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c5545e00, reset_sig_count=7016) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c5545e00, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1e55342 in srv_resume_thread (slot=0x55e2c2b9b938 <srv_sys+440>, sig_count=0, wait=true, timeout_usec=0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:940 #5 0x000055e2c1e58e43 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2647 #6 0x00007f864bf32494 in start_thread (arg=0x7f8632ffd700) at pthread_create.c:333 #7 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 26 (Thread 0x7f863f1da700 (LWP 3857)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c55da170) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c55da170, reset_sig_count=3) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c55da170, reset_sig_count=3) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1e655f2 in sync_array_wait_event (arr=0x55e2c552f170, cell=@0x7f863f1d5bb8: 0x7f864c346050) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0arr.cc:475 #5 0x000055e2c1e685b0 in rw_lock_s_lock_spin (lock=0x7f863f3acf78, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=6035) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:386 #6 0x000055e2c1f1275e in rw_lock_s_lock_func (lock=0x7f863f3acf78, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=6035) at /home/jan/mysql/10.2-orig/storage/innobase/include/sync0rw.ic:291 #7 0x000055e2c1f13104 in pfs_rw_lock_s_lock_func (lock=0x7f863f3acf78, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=6035) at /home/jan/mysql/10.2-orig/storage/innobase/include/sync0rw.ic:638 #8 0x000055e2c1f231a9 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=1, guess=0x0, mode=10, file=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=6035, mtr=0x7f863f1d6ad0, err=0x7f863f1d5f20) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0buf.cc:4757 #9 0x000055e2c1eed7b3 in btr_cur_open_at_rnd_pos_func (index=0x7f85f804d278, latch_mode=1, cursor=0x7f863f1d6a30, file=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=6035, mtr=0x7f863f1d6ad0) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc:2511 #10 0x000055e2c1ef60f7 in btr_estimate_number_of_different_key_vals (index=0x7f85f804d278) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc:6035 #11 0x000055e2c1f92a62 in dict_stats_update_transient_for_index (index=0x7f85f804d278) at /home/jan/mysql/10.2-orig/storage/innobase/dict/dict0stats.cc:922 #12 0x000055e2c1f92c89 in dict_stats_update_transient (table=0x7f85f8062118) at /home/jan/mysql/10.2-orig/storage/innobase/dict/dict0stats.cc:980 #13 0x000055e2c1f981d5 in dict_stats_update (table=0x7f85f8062118, stats_upd_option=DICT_STATS_RECALC_TRANSIENT) at /home/jan/mysql/10.2-orig/storage/innobase/dict/dict0stats.cc:3391 #14 0x000055e2c1f9b700 in dict_stats_update_if_needed (table=0x7f85f8062118) at /home/jan/mysql/10.2-orig/storage/innobase/dict/dict0stats_bg.cc:183 #15 0x000055e2c1deace7 in row_update_for_mysql (prebuilt=0x7f85f805dd18) at /home/jan/mysql/10.2-orig/storage/innobase/row/row0mysql.cc:2105 #16 0x000055e2c1caa65d in ha_innobase::update_row (this=0x7f85f804ca88, old_row=0x7f85f808c068 "\374\306:", new_row=0x7f85f808bf58 "\374\306:") at /home/jan/mysql/10.2-orig/storage/innobase/handler/ha_innodb.cc:9354 #17 0x000055e2c1aaaa00 in handler::ha_update_row (this=0x7f85f804ca88, old_data=0x7f85f808c068 "\374\306:", new_data=0x7f85f808bf58 "\374\306:") at /home/jan/mysql/10.2-orig/sql/handler.cc:6027 #18 0x000055e2c19194f1 in mysql_update (thd=0x7f85e8000b00, table_list=0x7f85e80112d8, fields=..., values=..., conds=0x7f85e8011eb8, order_num=0, order=0x0, limit=18446744073709549200, handle_duplicates=DUP_ERROR, ignore=false, found_return=0x7f863f1d86e0, updated_return=0x7f863f1d8790) at /home/jan/mysql/10.2-orig/sql/sql_update.cc:808 #19 0x000055e2c18317d4 in mysql_execute_command (thd=0x7f85e8000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:4268 #20 0x000055e2c183d013 in mysql_parse (thd=0x7f85e8000b00, rawbuf=0x7f85e80111c0 "UPDATE t1 SET c = c + 1 where c between 2000 and 8000", length=53, parser_state=0x7f863f1d9080, is_com_multi=false, is_next_command=false) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:7868 #21 0x000055e2c182ad27 in dispatch_command (command=COM_QUERY, thd=0x7f85e8000b00, packet=0x7f85e8008891 "UPDATE t1 SET c = c + 1 where c between 2000 and 8000;", packet_length=54, is_com_multi=false, is_next_command=false) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1812 #22 0x000055e2c18296d6 in do_command (thd=0x7f85e8000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1359 #23 0x000055e2c19756f5 in do_handle_one_connection (connect=0x55e2c5a2fdf0) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1354 #24 0x000055e2c1975475 in handle_one_connection (arg=0x55e2c5a2fdf0) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1260 #25 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5ad34b0) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #26 0x00007f864bf32494 in start_thread (arg=0x7f863f1da700) at pthread_create.c:333 #27 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 25 (Thread 0x7f86337fe700 (LWP 3782)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c5545e80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c5545e80, reset_sig_count=6587) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c5545e80, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1e55342 in srv_resume_thread (slot=0x55e2c2b9b970 <srv_sys+496>, sig_count=0, wait=true, timeout_usec=0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:940 #5 0x000055e2c1e58e43 in srv_worker_thread (arg=0x0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2647 #6 0x00007f864bf32494 in start_thread (arg=0x7f86337fe700) at pthread_create.c:333 #7 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 24 (Thread 0x7f863f224700 (LWP 3856)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c1d68802 in os_event::timed_wait (this=0x7f85e406e750, abstime=0x7f863f2218c0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:286 #2 0x000055e2c1d68b36 in os_event::wait_time_low (this=0x7f85e406e750, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:413 #3 0x000055e2c1d68cf0 in os_event_wait_time_low (event=0x7f85e406e750, time_in_usec=1000000, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:518 #4 0x000055e2c1cb5ff6 in ha_innobase::defragment_table (this=0x7f85e406f0d8, name=0x7f85f805cfd0 "test/t1", index_name=0x0, async=false) at /home/jan/mysql/10.2-orig/storage/innobase/handler/ha_innodb.cc:15017 #5 0x000055e2c1cb6153 in ha_innobase::optimize (this=0x7f85e406f0d8, thd=0x7f85e4000b00, check_opt=0x7f85e4005588) at /home/jan/mysql/10.2-orig/storage/innobase/handler/ha_innodb.cc:15072 #6 0x000055e2c1aa5f15 in handler::ha_optimize (this=0x7f85e406f0d8, thd=0x7f85e4000b00, check_opt=0x7f85e4005588) at /home/jan/mysql/10.2-orig/sql/handler.cc:4112 #7 0x000055e2c19879d9 in mysql_admin_table(THD *, TABLE_LIST *, HA_CHECK_OPT *, const char *, thr_lock_type, bool, bool, uint, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *), struct {...}, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *)) (thd=0x7f85e4000b00, tables=0x7f85e4011290, check_opt=0x7f85e4005588, operator_name=0x55e2c22cd8e0 "optimize", lock_type=TL_WRITE, open_for_modify=true, repair_table_use_frm=false, extra_open_options=0, prepare_func=0x0, operator_func=(int (handler::*)(handler * const, THD *, HA_CHECK_OPT *)) 0x55e2c1aa5e94 <handler::ha_optimize(THD*, st_ha_check_opt*)>, view_operator_func=0x0) at /home/jan/mysql/10.2-orig/sql/sql_admin.cc:798 #8 0x000055e2c1989cfa in Sql_cmd_optimize_table::execute (this=0x7f85e40118a8, thd=0x7f85e4000b00) at /home/jan/mysql/10.2-orig/sql/sql_admin.cc:1360 #9 0x000055e2c1838482 in mysql_execute_command (thd=0x7f85e4000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:6196 #10 0x000055e2c183d013 in mysql_parse (thd=0x7f85e4000b00, rawbuf=0x7f85e40111c0 "optimize table t1", length=17, parser_state=0x7f863f223080, is_com_multi=false, is_next_command=false) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:7868 #11 0x000055e2c182ad27 in dispatch_command (command=COM_QUERY, thd=0x7f85e4000b00, packet=0x7f85e4008891 "", packet_length=18, is_com_multi=false, is_next_command=false) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1812 #12 0x000055e2c18296d6 in do_command (thd=0x7f85e4000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1359 #13 0x000055e2c19756f5 in do_handle_one_connection (connect=0x55e2c5ab8750) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1354 #14 0x000055e2c1975475 in handle_one_connection (arg=0x55e2c5ab8750) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1260 #15 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5ad30f0) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #16 0x00007f864bf32494 in start_thread (arg=0x7f863f224700) at pthread_create.c:333 #17 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 23 (Thread 0x7f8633fff700 (LWP 3781)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c5545d00) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c5545d00, reset_sig_count=4112) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c5545d00, reset_sig_count=4112) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1e55342 in srv_resume_thread (slot=0x55e2c2b9b8c8 <srv_sys+328>, sig_count=4112, wait=true, timeout_usec=0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:940 #5 0x000055e2c1e59312 in srv_purge_coordinator_suspend (slot=0x55e2c2b9b8c8 <srv_sys+328>, rseg_history_len=12) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2800 #6 0x000055e2c1e596c7 in srv_purge_coordinator_thread (arg=0x0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2887 #7 0x00007f864bf32494 in start_thread (arg=0x7f8633fff700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 22 (Thread 0x7f863f26e700 (LWP 3855)): #0 0x00007f86497bb6ad in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x000055e2c226ce06 in vio_io_wait (vio=0x55e2c5ad25b0, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/jan/mysql/10.2-orig/vio/viosocket.c:945 #2 0x000055e2c226b615 in vio_socket_io_wait (vio=0x55e2c5ad25b0, event=VIO_IO_EVENT_READ) at /home/jan/mysql/10.2-orig/vio/viosocket.c:108 #3 0x000055e2c226b7cb in vio_read (vio=0x55e2c5ad25b0, buf=0x7f85f0008890 "\a", size=4) at /home/jan/mysql/10.2-orig/vio/viosocket.c:184 #4 0x000055e2c176095f in my_real_read (net=0x7f85f0000dc8, complen=0x7f863f26d7c0, header=1 '\001') at /home/jan/mysql/10.2-orig/sql/net_serv.cc:888 #5 0x000055e2c176115e in my_net_read_packet_reallen (net=0x7f85f0000dc8, read_from_server=1 '\001', reallen=0x7f863f26d828) at /home/jan/mysql/10.2-orig/sql/net_serv.cc:1158 #6 0x000055e2c1761112 in my_net_read_packet (net=0x7f85f0000dc8, read_from_server=1 '\001') at /home/jan/mysql/10.2-orig/sql/net_serv.cc:1142 #7 0x000055e2c18291a5 in do_command (thd=0x7f85f0000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1242 #8 0x000055e2c19756f5 in do_handle_one_connection (connect=0x55e2c5aaf200) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1354 #9 0x000055e2c1975475 in handle_one_connection (arg=0x55e2c5aaf200) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1260 #10 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5ad2d30) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #11 0x00007f864bf32494 in start_thread (arg=0x7f863f26e700) at pthread_create.c:333 #12 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 21 (Thread 0x7f8638f41700 (LWP 3780)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c1d68802 in os_event::timed_wait (this=0x55e2c56e6150, abstime=0x7f8638f40af0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:286 #2 0x000055e2c1d68b36 in os_event::wait_time_low (this=0x55e2c56e6150, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:413 #3 0x000055e2c1d68cf0 in os_event_wait_time_low (event=0x55e2c56e6150, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:518 #4 0x000055e2c1ec26c2 in ib_wqueue_timedwait (wq=0x55e2c56e5fe0, wait_in_usecs=5000000) at /home/jan/mysql/10.2-orig/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055e2c1ffd21c in fts_optimize_thread (arg=0x55e2c56e5fe0) at /home/jan/mysql/10.2-orig/storage/innobase/fts/fts0opt.cc:3041 #6 0x00007f864bf32494 in start_thread (arg=0x7f8638f41700) at pthread_create.c:333 #7 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 20 (Thread 0x7f863f2b8700 (LWP 3854)): #0 0x00007f86497bb6ad in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x000055e2c226ce06 in vio_io_wait (vio=0x55e2c5a8e0e0, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/jan/mysql/10.2-orig/vio/viosocket.c:945 #2 0x000055e2c226b615 in vio_socket_io_wait (vio=0x55e2c5a8e0e0, event=VIO_IO_EVENT_READ) at /home/jan/mysql/10.2-orig/vio/viosocket.c:108 #3 0x000055e2c226b7cb in vio_read (vio=0x55e2c5a8e0e0, buf=0x7f85ec008890 "\n", size=4) at /home/jan/mysql/10.2-orig/vio/viosocket.c:184 #4 0x000055e2c176095f in my_real_read (net=0x7f85ec000dc8, complen=0x7f863f2b77c0, header=1 '\001') at /home/jan/mysql/10.2-orig/sql/net_serv.cc:888 #5 0x000055e2c176115e in my_net_read_packet_reallen (net=0x7f85ec000dc8, read_from_server=1 '\001', reallen=0x7f863f2b7828) at /home/jan/mysql/10.2-orig/sql/net_serv.cc:1158 #6 0x000055e2c1761112 in my_net_read_packet (net=0x7f85ec000dc8, read_from_server=1 '\001') at /home/jan/mysql/10.2-orig/sql/net_serv.cc:1142 #7 0x000055e2c18291a5 in do_command (thd=0x7f85ec000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1242 #8 0x000055e2c19756f5 in do_handle_one_connection (connect=0x55e2c5a2fdf0) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1354 #9 0x000055e2c1975475 in handle_one_connection (arg=0x55e2c5a2fdf0) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1260 #10 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5ad2970) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #11 0x00007f864bf32494 in start_thread (arg=0x7f863f2b8700) at pthread_create.c:333 #12 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 19 (Thread 0x7f863c16d700 (LWP 3677)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f864bf34b95 in __GI___pthread_mutex_lock (mutex=0x55e2c2dda0d0 <rw_lock_debug_mutex+16>) at ../nptl/pthread_mutex_lock.c:80 #2 0x000055e2c1d14765 in OSMutex::enter (this=0x55e2c2dda0c8 <rw_lock_debug_mutex+8>) at /home/jan/mysql/10.2-orig/storage/innobase/include/sync0types.h:472 #3 0x000055e2c1d66363 in OSTrackMutex<GenericPolicy>::enter (this=0x55e2c2dda0c0 <rw_lock_debug_mutex>, max_spins=30, max_delay=6, filename=0x55e2c244ad18 "/home/jan/mysql/10.2-orig/storage/innobase/sync/sync0debug.cc", line=1268) at /home/jan/mysql/10.2-orig/storage/innobase/include/ib0mutex.h:109 #4 0x000055e2c1d65b3d in PolicyMutex<OSTrackMutex<GenericPolicy> >::enter (this=0x55e2c2dda0c0 <rw_lock_debug_mutex>, n_spins=30, n_delay=6, name=0x55e2c244ad18 "/home/jan/mysql/10.2-orig/storage/innobase/sync/sync0debug.cc", line=1268) at /home/jan/mysql/10.2-orig/storage/innobase/include/ib0mutex.h:635 #5 0x000055e2c1e6fd8a in rw_lock_debug_mutex_enter () at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0debug.cc:1268 #6 0x000055e2c1e694f9 in rw_lock_own (lock=0x55e2c561b900, lock_type=5) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:1024 #7 0x000055e2c1f3794d in buf_page_hash_get_locked (buf_pool=0x55e2c55c8550, page_id=..., lock=0x0, lock_mode=0, watch=false) at /home/jan/mysql/10.2-orig/storage/innobase/include/buf0buf.ic:1150 #8 0x000055e2c1f3acd7 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=162, n_to_flush=450) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:1462 #9 0x000055e2c1f3b193 in buf_flush_page_and_try_neighbors (bpage=0x7f863f3f0730, flush_type=BUF_FLUSH_LIST, n_to_flush=450, count=0x7f863c16c868) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:1562 #10 0x000055e2c1f3bc79 in buf_do_flush_list_batch (buf_pool=0x55e2c55c8550, min_n=450, lsn_limit=18446744073709551615) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:1823 #11 0x000055e2c1f3c115 in buf_flush_batch (buf_pool=0x55e2c55c8550, flush_type=BUF_FLUSH_LIST, min_n=450, lsn_limit=18446744073709551615, n=0x7f863c16cb10) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:1892 #12 0x000055e2c1f3c7c5 in buf_flush_do_batch (buf_pool=0x55e2c55c8550, type=BUF_FLUSH_LIST, min_n=450, lsn_limit=18446744073709551615, n=0x7f863c16cb10) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:2062 #13 0x000055e2c1f3f24e in pc_flush_slot () at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:2898 #14 0x000055e2c1f40209 in buf_flush_page_cleaner_coordinator () at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0flu.cc:3320 #15 0x00007f864bf32494 in start_thread (arg=0x7f863c16d700) at pthread_create.c:333 #16 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 18 (Thread 0x7f863c96e700 (LWP 3676)): #0 0x00007f864b2ae6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x000055e2c1d5c7e5 in LinuxAIOHandler::collect (this=0x7f863c96daa0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:1896 #2 0x000055e2c1d5cd1b in LinuxAIOHandler::poll (this=0x7f863c96daa0, m1=0x7f863c96db48, m2=0x7f863c96db50, request=0x7f863c96db80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2041 #3 0x000055e2c1d5ce95 in os_aio_linux_handler (global_segment=5, m1=0x7f863c96db48, m2=0x7f863c96db50, request=0x7f863c96db80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2095 #4 0x000055e2c1d61a11 in os_aio_handler (segment=5, m1=0x7f863c96db48, m2=0x7f863c96db50, request=0x7f863c96db80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:5678 #5 0x000055e2c1fb5535 in fil_aio_wait (segment=5) at /home/jan/mysql/10.2-orig/storage/innobase/fil/fil0fil.cc:5460 #6 0x000055e2c1e5bdf1 in io_handler_thread (arg=0x55e2c2dcf9c8 <n+40>) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0start.cc:343 #7 0x00007f864bf32494 in start_thread (arg=0x7f863c96e700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 17 (Thread 0x7f864c2add00 (LWP 3660)): #0 0x00007f86497bb6ad in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x000055e2c1757bf5 in handle_connections_sockets () at /home/jan/mysql/10.2-orig/sql/mysqld.cc:6598 #2 0x000055e2c1756eb2 in mysqld_main (argc=148, argv=0x55e2c5390160) at /home/jan/mysql/10.2-orig/sql/mysqld.cc:6064 #3 0x000055e2c174b590 in main (argc=26, argv=0x7ffe83e547a8) at /home/jan/mysql/10.2-orig/sql/main.cc:25   Thread 16 (Thread 0x7f8617fff700 (LWP 3804)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c220cc20 in safe_cond_wait (cond=0x7f8617ffec40, mp=0x7f8617ffec80, file=0x55e2c23a2168 "/home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h", line=1149) at /home/jan/mysql/10.2-orig/mysys/thr_mutex.c:491 #2 0x000055e2c1c9558a in inline_mysql_cond_wait (that=0x7f8617ffec40, mutex=0x7f8617ffec80, src_file=0x55e2c23a3be8 "/home/jan/mysql/10.2-orig/storage/innobase/handler/ha_innodb.cc", src_line=341) at /home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h:1149 #3 0x000055e2c1c99ec8 in thd_destructor_proxy () at /home/jan/mysql/10.2-orig/storage/innobase/handler/ha_innodb.cc:341 #4 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5705e80) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #5 0x00007f864bf32494 in start_thread (arg=0x7f8617fff700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 15 (Thread 0x7f863d16f700 (LWP 3675)): #0 0x00007f864b2ae6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x000055e2c1d5c7e5 in LinuxAIOHandler::collect (this=0x7f863d16eaa0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:1896 #2 0x000055e2c1d5cd1b in LinuxAIOHandler::poll (this=0x7f863d16eaa0, m1=0x7f863d16eb48, m2=0x7f863d16eb50, request=0x7f863d16eb80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2041 #3 0x000055e2c1d5ce95 in os_aio_linux_handler (global_segment=4, m1=0x7f863d16eb48, m2=0x7f863d16eb50, request=0x7f863d16eb80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2095 #4 0x000055e2c1d61a11 in os_aio_handler (segment=4, m1=0x7f863d16eb48, m2=0x7f863d16eb50, request=0x7f863d16eb80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:5678 #5 0x000055e2c1fb5535 in fil_aio_wait (segment=4) at /home/jan/mysql/10.2-orig/storage/innobase/fil/fil0fil.cc:5460 #6 0x000055e2c1e5bdf1 in io_handler_thread (arg=0x55e2c2dcf9c0 <n+32>) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0start.cc:343 #7 0x00007f864bf32494 in start_thread (arg=0x7f863d16f700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 14 (Thread 0x7f8630ff9700 (LWP 3803)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c5546100) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c5546100, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c5546100, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1f1e63b in buf_resize_thread () at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0buf.cc:3080 #5 0x00007f864bf32494 in start_thread (arg=0x7f8630ff9700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 13 (Thread 0x7f863f302700 (LWP 3843)): #0 0x00007f86497bb6ad in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x000055e2c226ce06 in vio_io_wait (vio=0x55e2c5ad34b0, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/jan/mysql/10.2-orig/vio/viosocket.c:945 #2 0x000055e2c226b615 in vio_socket_io_wait (vio=0x55e2c5ad34b0, event=VIO_IO_EVENT_READ) at /home/jan/mysql/10.2-orig/vio/viosocket.c:108 #3 0x000055e2c226b7cb in vio_read (vio=0x55e2c5ad34b0, buf=0x7f85f80849a0 "\n", size=4) at /home/jan/mysql/10.2-orig/vio/viosocket.c:184 #4 0x000055e2c176095f in my_real_read (net=0x7f85f8000dc8, complen=0x7f863f3017c0, header=1 '\001') at /home/jan/mysql/10.2-orig/sql/net_serv.cc:888 #5 0x000055e2c176115e in my_net_read_packet_reallen (net=0x7f85f8000dc8, read_from_server=1 '\001', reallen=0x7f863f301828) at /home/jan/mysql/10.2-orig/sql/net_serv.cc:1158 #6 0x000055e2c1761112 in my_net_read_packet (net=0x7f85f8000dc8, read_from_server=1 '\001') at /home/jan/mysql/10.2-orig/sql/net_serv.cc:1142 #7 0x000055e2c18291a5 in do_command (thd=0x7f85f8000b00) at /home/jan/mysql/10.2-orig/sql/sql_parse.cc:1242 #8 0x000055e2c19756f5 in do_handle_one_connection (connect=0x55e2c5aaf200) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1354 #9 0x000055e2c1975475 in handle_one_connection (arg=0x55e2c5aaf200) at /home/jan/mysql/10.2-orig/sql/sql_connect.cc:1260 #10 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c5ad25b0) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #11 0x00007f864bf32494 in start_thread (arg=0x7f863f302700) at pthread_create.c:333 #12 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 12 (Thread 0x7f863e171700 (LWP 3673)): #0 0x00007f864b2ae6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x000055e2c1d5c7e5 in LinuxAIOHandler::collect (this=0x7f863e170aa0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:1896 #2 0x000055e2c1d5cd1b in LinuxAIOHandler::poll (this=0x7f863e170aa0, m1=0x7f863e170b48, m2=0x7f863e170b50, request=0x7f863e170b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2041 #3 0x000055e2c1d5ce95 in os_aio_linux_handler (global_segment=2, m1=0x7f863e170b48, m2=0x7f863e170b50, request=0x7f863e170b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2095 #4 0x000055e2c1d61a11 in os_aio_handler (segment=2, m1=0x7f863e170b48, m2=0x7f863e170b50, request=0x7f863e170b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:5678 #5 0x000055e2c1fb5535 in fil_aio_wait (segment=2) at /home/jan/mysql/10.2-orig/storage/innobase/fil/fil0fil.cc:5460 #6 0x000055e2c1e5bdf1 in io_handler_thread (arg=0x55e2c2dcf9b0 <n+16>) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0start.cc:343 #7 0x00007f864bf32494 in start_thread (arg=0x7f863e171700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 11 (Thread 0x7f8631ffb700 (LWP 3801)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000055e2c1d68f6c in os_event::wait (this=0x55e2c5546000) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:166 #2 0x000055e2c1d68998 in os_event::wait_low (this=0x55e2c5546000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:336 #3 0x000055e2c1d68d15 in os_event_wait_low (event=0x55e2c5546000, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:535 #4 0x000055e2c1f35766 in buf_dump_thread () at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0dump.cc:824 #5 0x00007f864bf32494 in start_thread (arg=0x7f8631ffb700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 10 (Thread 0x7f8639742700 (LWP 3779)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c1d68802 in os_event::timed_wait (this=0x55e2c56966c0, abstime=0x7f8639741d20) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:286 #2 0x000055e2c1d68b36 in os_event::wait_time_low (this=0x55e2c56966c0, time_in_usec=10000000, reset_sig_count=33) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:413 #3 0x000055e2c1d68cf0 in os_event_wait_time_low (event=0x55e2c56966c0, time_in_usec=10000000, reset_sig_count=0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:518 #4 0x000055e2c1f9be21 in dict_stats_thread () at /home/jan/mysql/10.2-orig/storage/innobase/dict/dict0stats_bg.cc:451 #5 0x00007f864bf32494 in start_thread (arg=0x7f8639742700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 9 (Thread 0x7f863a744700 (LWP 3727)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c1d68802 in os_event::timed_wait (this=0x55e2c5545f00, abstime=0x7f863a743b40) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:286 #2 0x000055e2c1d68b36 in os_event::wait_time_low (this=0x55e2c5545f00, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:413 #3 0x000055e2c1d68cf0 in os_event_wait_time_low (event=0x55e2c5545f00, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:518 #4 0x000055e2c1e577cf in srv_error_monitor_thread () at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:1969 #5 0x00007f864bf32494 in start_thread (arg=0x7f863a744700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 8 (Thread 0x7f863e972700 (LWP 3672)): #0 0x00007f864b2ae6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x000055e2c1d5c7e5 in LinuxAIOHandler::collect (this=0x7f863e971aa0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:1896 #2 0x000055e2c1d5cd1b in LinuxAIOHandler::poll (this=0x7f863e971aa0, m1=0x7f863e971b48, m2=0x7f863e971b50, request=0x7f863e971b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2041 #3 0x000055e2c1d5ce95 in os_aio_linux_handler (global_segment=1, m1=0x7f863e971b48, m2=0x7f863e971b50, request=0x7f863e971b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2095 #4 0x000055e2c1d61a11 in os_aio_handler (segment=1, m1=0x7f863e971b48, m2=0x7f863e971b50, request=0x7f863e971b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:5678 #5 0x000055e2c1fb5535 in fil_aio_wait (segment=1) at /home/jan/mysql/10.2-orig/storage/innobase/fil/fil0fil.cc:5460 #6 0x000055e2c1e5bdf1 in io_handler_thread (arg=0x55e2c2dcf9a8 <n+8>) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0start.cc:343 #7 0x00007f864bf32494 in start_thread (arg=0x7f863e972700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 7 (Thread 0x7f8639f43700 (LWP 3728)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c1d68802 in os_event::timed_wait (this=0x55e2c5545f80, abstime=0x7f8639f42cd0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:286 #2 0x000055e2c1d68b36 in os_event::wait_time_low (this=0x55e2c5545f80, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:413 #3 0x000055e2c1d68cf0 in os_event_wait_time_low (event=0x55e2c5545f80, time_in_usec=5000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:518 #4 0x000055e2c1e570f6 in srv_monitor_thread () at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:1744 #5 0x00007f864bf32494 in start_thread (arg=0x7f8639f43700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 6 (Thread 0x7f863b746700 (LWP 3778)): #0 0x00007f864bf3b7dd in nanosleep () at ../sysdeps/unix/syscall-template.S:84 #1 0x000055e2c1d6928c in os_thread_sleep (tm=1000000) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0thread.cc:230 #2 0x000055e2c1e58870 in srv_master_sleep () at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2458 #3 0x000055e2c1e589ae in srv_master_thread (arg=0x0) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0srv.cc:2503 #4 0x00007f864bf32494 in start_thread (arg=0x7f863b746700) at pthread_create.c:333 #5 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 5 (Thread 0x7f863af45700 (LWP 3726)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c1d68802 in os_event::timed_wait (this=0x55e2c56833c0, abstime=0x7f863af44d00) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:286 #2 0x000055e2c1d68b36 in os_event::wait_time_low (this=0x55e2c56833c0, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:413 #3 0x000055e2c1d68cf0 in os_event_wait_time_low (event=0x55e2c56833c0, time_in_usec=1000000, reset_sig_count=1) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0event.cc:518 #4 0x000055e2c1d37629 in lock_wait_timeout_thread () at /home/jan/mysql/10.2-orig/storage/innobase/lock/lock0wait.cc:529 #5 0x00007f864bf32494 in start_thread (arg=0x7f863af45700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 4 (Thread 0x7f863d970700 (LWP 3674)): #0 0x00007f864b2ae6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x000055e2c1d5c7e5 in LinuxAIOHandler::collect (this=0x7f863d96faa0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:1896 #2 0x000055e2c1d5cd1b in LinuxAIOHandler::poll (this=0x7f863d96faa0, m1=0x7f863d96fb48, m2=0x7f863d96fb50, request=0x7f863d96fb80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2041 #3 0x000055e2c1d5ce95 in os_aio_linux_handler (global_segment=3, m1=0x7f863d96fb48, m2=0x7f863d96fb50, request=0x7f863d96fb80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2095 #4 0x000055e2c1d61a11 in os_aio_handler (segment=3, m1=0x7f863d96fb48, m2=0x7f863d96fb50, request=0x7f863d96fb80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:5678 #5 0x000055e2c1fb5535 in fil_aio_wait (segment=3) at /home/jan/mysql/10.2-orig/storage/innobase/fil/fil0fil.cc:5460 #6 0x000055e2c1e5bdf1 in io_handler_thread (arg=0x55e2c2dcf9b8 <n+24>) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0start.cc:343 #7 0x00007f864bf32494 in start_thread (arg=0x7f863d970700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 3 (Thread 0x7f863f173700 (LWP 3671)): #0 0x00007f864b2ae6ca in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x000055e2c1d5c7e5 in LinuxAIOHandler::collect (this=0x7f863f172aa0) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:1896 #2 0x000055e2c1d5cd1b in LinuxAIOHandler::poll (this=0x7f863f172aa0, m1=0x7f863f172b48, m2=0x7f863f172b50, request=0x7f863f172b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2041 #3 0x000055e2c1d5ce95 in os_aio_linux_handler (global_segment=0, m1=0x7f863f172b48, m2=0x7f863f172b50, request=0x7f863f172b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:2095 #4 0x000055e2c1d61a11 in os_aio_handler (segment=0, m1=0x7f863f172b48, m2=0x7f863f172b50, request=0x7f863f172b80) at /home/jan/mysql/10.2-orig/storage/innobase/os/os0file.cc:5678 #5 0x000055e2c1fb5535 in fil_aio_wait (segment=0) at /home/jan/mysql/10.2-orig/storage/innobase/fil/fil0fil.cc:5460 #6 0x000055e2c1e5bdf1 in io_handler_thread (arg=0x55e2c2dcf9a0 <n>) at /home/jan/mysql/10.2-orig/storage/innobase/srv/srv0start.cc:343 #7 0x00007f864bf32494 in start_thread (arg=0x7f863f173700) at pthread_create.c:333 #8 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 2 (Thread 0x7f8644bf3700 (LWP 3669)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055e2c220cf2a in safe_cond_timedwait (cond=0x55e2c35f13a0 <COND_timer>, mp=0x55e2c35f12e0 <LOCK_timer>, abstime=0x7f8644bf2d30, file=0x55e2c2545cd0 "/home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h", line=1175) at /home/jan/mysql/10.2-orig/mysys/thr_mutex.c:545 #2 0x000055e2c220de9f in inline_mysql_cond_timedwait (that=0x55e2c35f13a0 <COND_timer>, mutex=0x55e2c35f12e0 <LOCK_timer>, abstime=0x7f8644bf2d30, src_file=0x55e2c2545d10 "/home/jan/mysql/10.2-orig/mysys/thr_timer.c", src_line=292) at /home/jan/mysql/10.2-orig/include/mysql/psi/mysql_thread.h:1175 #3 0x000055e2c220e9ac in timer_handler (arg=0x0) at /home/jan/mysql/10.2-orig/mysys/thr_timer.c:292 #4 0x000055e2c21ad98f in pfs_spawn_thread (arg=0x55e2c54d55e0) at /home/jan/mysql/10.2-orig/storage/perfschema/pfs.cc:1862 #5 0x00007f864bf32494 in start_thread (arg=0x7f8644bf3700) at pthread_create.c:333 #6 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97   Thread 1 (Thread 0x7f86317fa700 (LWP 3802)): #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 #1 0x000055e2c22076a2 in my_write_core (sig=6) at /home/jan/mysql/10.2-orig/mysys/stacktrace.c:477 #2 0x000055e2c1a9b514 in handle_fatal_signal (sig=6) at /home/jan/mysql/10.2-orig/sql/signal_handler.cc:303 #3 <signal handler called> #4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #5 0x00007f86497103fa in __GI_abort () at abort.c:89 #6 0x000055e2c1ec1d5d in ib::fatal::~fatal (this=0x7f86317f76c0, __in_chrg=<optimized out>) at /home/jan/mysql/10.2-orig/storage/innobase/ut/ut0ut.cc:850 #7 0x000055e2c1e655af in sync_array_wait_event (arr=0x55e2c552f170, cell=@0x7f86317f7898: 0x7f864c346010) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0arr.cc:467 #8 0x000055e2c1e68751 in rw_lock_x_lock_wait_func (lock=0x7f863f3ae8f8, pass=0, threshold=-268435456, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:472 #9 0x000055e2c1e68926 in rw_lock_x_lock_low (lock=0x7f863f3ae8f8, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:557 #10 0x000055e2c1e68c6d in rw_lock_x_lock_func (lock=0x7f863f3ae8f8, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/sync/sync0rw.cc:687 #11 0x000055e2c1f12f34 in pfs_rw_lock_x_lock_func (lock=0x7f863f3ae8f8, pass=0, file_name=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917) at /home/jan/mysql/10.2-orig/storage/innobase/include/sync0rw.ic:545 #12 0x000055e2c1f2323b in buf_page_get_gen (page_id=..., page_size=..., rw_latch=2, guess=0x0, mode=10, file=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917, mtr=0x7f86317f9860, err=0x7f86317f7c74) at /home/jan/mysql/10.2-orig/storage/innobase/buf/buf0buf.cc:4770 #13 0x000055e2c1ee60dc in btr_block_get_func (page_id=..., page_size=..., mode=2, file=0x55e2c2475240 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc", line=1917, index=0x7f85f804d278, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/include/btr0btr.ic:61 #14 0x000055e2c1eebe05 in btr_cur_search_to_nth_level (index=0x7f85f804d278, level=2, tuple=0x7f85cc0327c8, mode=PAGE_CUR_LE, latch_mode=34, cursor=0x7f86317f8ec0, has_search_latch=0, file=0x55e2c2468580 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc", line=1026, mtr=0x7f86317f9860, autoinc=0) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc:1917 #15 0x000055e2c1eca96e in btr_page_get_father_node_ptr_func (offsets=0x0, heap=0x7f85cc02c0d0, cursor=0x7f86317f8ec0, latch_mode=34, file=0x55e2c2468580 "/home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc", line=1026, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:924 #16 0x000055e2c1ecafbc in btr_page_get_father (index=0x7f85f804d278, block=0x7f863f3c66a0, mtr=0x7f86317f9860, cursor=0x7f86317f8ec0) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:1026 #17 0x000055e2c1ed11d4 in btr_node_ptr_delete (index=0x7f85f804d278, block=0x7f863f3c66a0, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:3418 #18 0x000055e2c1ef4814 in btr_cur_pessimistic_delete (err=0x7f86317f9144, has_reserved_extents=1, cursor=0x7f86317f9150, flags=16, rollback=false, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0cur.cc:5190 #19 0x000055e2c1ed1204 in btr_node_ptr_delete (index=0x7f85f804d278, block=0x7f863f3e9470, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0btr.cc:3421 #20 0x000055e2c1f108aa in btr_defragment_merge_pages (index=0x7f85f804d278, from_block=0x7f863f3e9470, to_block=0x7f863f3ae4f0, page_size=..., reserved_space=1625, max_data_size=0x7f86317f9330, heap=0x7f85cc000c20, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0defragment.cc:562 #21 0x000055e2c1f10f72 in btr_defragment_n_pages (block=0x7f863f3fc670, index=0x7f85f804d278, n_pages=20, mtr=0x7f86317f9860) at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0defragment.cc:718 #22 0x000055e2c1f1128f in btr_defragment_thread () at /home/jan/mysql/10.2-orig/storage/innobase/btr/btr0defragment.cc:811 #23 0x00007f864bf32494 in start_thread (arg=0x7f86317fa700) at pthread_create.c:333 #24 0x00007f86497c4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Marko Mäkelä added a comment - 2017-10-10 10:08 jplindst , it could be an acceptable fix to acquire dict_index_t::lock in RW_LOCK_X mode at the start of the mini-transaction, and then let innodb_defragment_n_pages (up to 32 pages) be defragmented before committing the mini-transaction (which would release the latches). Note that btr_cur_search_to_nth_level() with latch_mode=BTR_MODIFY_TREE will only acquire an SX-latch on index->lock. You would probably want BTR_MODIFY_TREE|BTR_INTENTION_BOTH. Also, be careful to test the code on SPATIAL INDEX (R-tree index). Perhaps before re-enabling the code, we should import and run all the innodb_gis tests from MySQL 5.7. The current test coverage is poor, because it failed to find MDEV-13512 and MDEV-13942 . What is the exact algorithm that is used for defragmentation? If it does not look compatible with the access patterns of the R-tree indexes (especially the split sequence number and purge), perhaps it should remain disabled for spatial indexes.

Jan Lindström (Inactive) added a comment - 2017-10-10 10:38 BTR_INTENTION_BOTH is not available outside of btr0cur.cc but this seams to work fine with X-lock on dict_index_t::lock. I will see if gis is supported (do not very well know this code).

Jan Lindström (Inactive) added a comment - 2017-10-12 05:17 I do not want to add GIS-tests on this commit as they need some additional work and we do not defragment GIS-indexes (this fact is tested). https://github.com/MariaDB/server/commit/8610156a337184e6bb7bd64467a8f69ae7235985

Marko Mäkelä added a comment - 2017-10-12 06:22 The code change looks OK. But, please address my comments regarding tests and especially source code comments. I think that we must document the WL#6326 and WL#6363 rules in the source code. Currently the comment at the start of btr0btr.cc is conflicting with WL#6326.

Jan Lindström (Inactive) added a comment - 2017-10-12 10:04 commit a4fa940bad7f3a2c4afff776977ad5e46dfe95d3 Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Thu Oct 12 12:56:20 2017 +0300 MDEV-11336 : Enable defragmentation on 10.2 when tests pass Problem was that we could take page latches on different order than wat is entitled with SX-lock. To follow the latching order defined in WL#6326, acquire index->lock X-latch. This entitles us to acquire page latches in any order for the index. btr0btr.cc Document latch rules before and after MariaDB 10.2.2 sync0rw.cc Document latch compatibility rules better. btr_defragment_merge_pages Fix parameter value. btr_defragment_thread Acquire X-lock to dict_index_t::lock before restoring cursor position and continuing defragmentation. ha_innobase::optimize Restore defragment feature. Testing Add GIS-index and FT-index to table being defragmented. Defragmentation is not done to GIS-indexes and FT auxiliary tables.

Jan Lindström (Inactive) added a comment - 2017-10-12 10:05 greenman Can you document that from MariaDB 10.2.11 defragmentation feature is again available.