Details

    Description

      We recently upgraded from 10.1 to 10.2, but unfortunately MariaDB 10.2 becomes unresponsive multiple times every day. This is the log we see:

      10.2.9-MariaDB-10.2.9+maria~trusty-log
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200436983552 has waited at read0read.cc line 579 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140202383660800 has waited at lock0lock.cc line 7784 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fceccddc0, Mutex LOCK_SYS created lock0lock.cc:495, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140203085694720 has waited at read0read.cc line 687 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200439097088 has waited at read0read.cc line 579 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200431699712 has waited at lock0lock.cc line 5075 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fceccddc0, Mutex LOCK_SYS created lock0lock.cc:495, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200429586176 has waited at read0read.cc line 579 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200443324160 has waited at trx0trx.cc line 664 for 303.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140202384189184 has waited at read0read.cc line 579 for 301.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200442267392 has waited at read0read.cc line 579 for 299.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140201363072768 has waited at read0read.cc line 579 for 299.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140200430114560 has waited at read0read.cc line 579 for 299.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140201362544384 has waited at trx0trx.cc line 3091 for 286.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140203127658240 has waited at lock0wait.cc line 484 for 282.00 seconds the semaphore:
      Mutex at 0x7f8fceccddc0, Mutex LOCK_SYS created lock0lock.cc:495, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140203034810112 has waited at trx0trx.cc line 1275 for 242.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
      2017-10-03 10:21:11 140203119265536 [Note] InnoDB: A semaphore wait:
      --Thread 140203110872832 has waited at read0read.cc line 714 for 60.00 seconds the semaphore:
      Mutex at 0x7f8fced4d2b0, Mutex TRX_SYS created trx0sys.cc:558, lock var 2
       
       
      ....
      

      Attachments

        Issue Links

          Activity

            btginsb Brandon Ginsberg added a comment - - edited

            We had another occurrence of this last night. Log reports it occurred around 23:55 and a thread had been waiting for roughly 900s before the semaphore wait limit crashed the server. Not sure how it reached 900s if the limit is 600s?

            Version: 10.2.21 on AWS RDS

            Log is attached for reference:
            mysql-error-running.log.1

            btginsb Brandon Ginsberg added a comment - - edited We had another occurrence of this last night. Log reports it occurred around 23:55 and a thread had been waiting for roughly 900s before the semaphore wait limit crashed the server. Not sure how it reached 900s if the limit is 600s? Version: 10.2.21 on AWS RDS Log is attached for reference: mysql-error-running.log.1

            The 600-second limit is some kind of a ‘soft limit’ that avoids false positive alerts from the InnoDB watchdog. Before I fixed it years ago in MySQL, you could get bogus watchdog kills on a heavily loaded system, possibly because the same mutex happened to be occupied (by varying threads) every time a sample was taken. The hard limit has always been 900 seconds (15 minutes) as long as I remember.

            marko Marko Mäkelä added a comment - The 600-second limit is some kind of a ‘soft limit’ that avoids false positive alerts from the InnoDB watchdog. Before I fixed it years ago in MySQL, you could get bogus watchdog kills on a heavily loaded system, possibly because the same mutex happened to be occupied (by varying threads) every time a sample was taken. The hard limit has always been 900 seconds (15 minutes) as long as I remember.

            It looks like this bug was introduced in an attempt to fix MDEV-11896 and fixed for 10.3.3 in a cleanup that was in MDEV-16467 partially backported to 10.1.41 and 10.2.25.

            The history of this goes all the way back to 2004, when the LOCK_thread_count acquisition was introduced in InnoDB. Back then, the logic was clear and correct: Mutexes outside InnoDB are conceptually above any InnoDB mutex or rw-lock in the latching order, and thus they should not be acquired while holding any InnoDB mutex. For displaying the details of other sessions that participate in a deadlock between InnoDB transactions, InnoDB would first acquire LOCK_thread_count upfront, and only after that acquire its own mutexes (originally kernel_mutex, later trx_sys_t::mutex, lock_sys_t::mutex, trx_t::mutex). At some point, the LOCK_thread_count acquisition was removed from the ‘upper level’ along with the function innobase_mysql_prepare_print_arbitrary_thd(). Finally, the attempted MDEV-11896 fix broke the assumptions by introducing the offending mutex acquisition at the low level.

            We did some analysis of stack trace dumps of a support customer using MariaDB Server 10.2.21. The deadlocks that they were experiencing involve 3 threads:

            • Thread A (user transaction deadlock reporter in InnoDB) is holding lock_sys_t::mutex and waiting for LOCK_thread_count.
            • Thread B (handle_slave_background() but it could also be something else handling a KILL statement) acquired THD::LOCK_ha_data on the victim’s handle before entering THD::awake(), and is waiting for lock_sys_t::mutex in lock_trx_handle_wait(), invoked via innobase_hton->kill_query = innobase_kill_query.
            • Thread C (SELECT…FROM INFORMATION_SCHEMA.PROCESSLIST or SHOW PROCESSLIST) acquired LOCK_thread_count in fill_schema_processlist() or mysqld_list_processes and is waiting for THD::lock_ha_data later in the same function, on the KILL victim’s handle.

            This deadlock was fixed by removing the offending wait in Thread A, in the code that InnoDB calls while holding its own mutexes.

            Note: In MySQL 5.7 and MariaDB 10.2.2, the InnoDB deadlock checker was refactored. Before that, the mutex functionality was basically the same: lock_deadlock_trx_print() is holding lock_sys->mutex and trx_sys->mutex before invoking innobase_mysql_print_thd() in trx_print_low(). Starting with 10.2.2, the diagnostic output is initiated by DeadlockChecker::notify().

            marko Marko Mäkelä added a comment - It looks like this bug was introduced in an attempt to fix MDEV-11896 and fixed for 10.3.3 in a cleanup that was in MDEV-16467 partially backported to 10.1.41 and 10.2.25 . The history of this goes all the way back to 2004 , when the LOCK_thread_count acquisition was introduced in InnoDB. Back then, the logic was clear and correct: Mutexes outside InnoDB are conceptually above any InnoDB mutex or rw-lock in the latching order, and thus they should not be acquired while holding any InnoDB mutex. For displaying the details of other sessions that participate in a deadlock between InnoDB transactions, InnoDB would first acquire LOCK_thread_count upfront, and only after that acquire its own mutexes (originally kernel_mutex , later trx_sys_t::mutex , lock_sys_t::mutex , trx_t::mutex ). At some point, the LOCK_thread_count acquisition was removed from the ‘upper level’ along with the function innobase_mysql_prepare_print_arbitrary_thd() . Finally, the attempted MDEV-11896 fix broke the assumptions by introducing the offending mutex acquisition at the low level. We did some analysis of stack trace dumps of a support customer using MariaDB Server 10.2.21. The deadlocks that they were experiencing involve 3 threads: Thread A (user transaction deadlock reporter in InnoDB) is holding lock_sys_t::mutex and waiting for LOCK_thread_count . Thread B ( handle_slave_background() but it could also be something else handling a KILL statement) acquired THD::LOCK_ha_data on the victim’s handle before entering THD::awake() , and is waiting for lock_sys_t::mutex in lock_trx_handle_wait() , invoked via innobase_hton->kill_query = innobase_kill_query . Thread C ( SELECT…FROM INFORMATION_SCHEMA.PROCESSLIST or SHOW PROCESSLIST ) acquired LOCK_thread_count in fill_schema_processlist() or mysqld_list_processes and is waiting for THD::lock_ha_data later in the same function, on the KILL victim’s handle. This deadlock was fixed by removing the offending wait in Thread A, in the code that InnoDB calls while holding its own mutexes. Note: In MySQL 5.7 and MariaDB 10.2.2, the InnoDB deadlock checker was refactored. Before that, the mutex functionality was basically the same: lock_deadlock_trx_print() is holding lock_sys->mutex and trx_sys->mutex before invoking innobase_mysql_print_thd() in trx_print_low() . Starting with 10.2.2, the diagnostic output is initiated by DeadlockChecker::notify() .
            CrewOne Sander Pilon added a comment -

            Wow. Thanks for fixing this. We upgraded to Percona early 2019 because of this, but good to know it's fixed.

            CrewOne Sander Pilon added a comment - Wow. Thanks for fixing this. We upgraded to Percona early 2019 because of this, but good to know it's fixed.

            I repeated this hang locally. My test ran for a couple of seconds from the server startup until the deadlock occurred. After reverting the attempted fix of MDEV-11896, the server would not hang. I interrupted the test after 3 minutes.

            I will not disclose my scripts (involving 4 clients, 2 of them causing user transaction deadlocks when accessing an InnoDB table), because it might make the life too easy for script kiddies.

            Here is the server error log output:

            mariadb-10.2.21

            2020-01-31  8:28:22 140120539419456 [Note] ../sql/mysqld: ready for connections.
            Version: '10.2.21-MariaDB'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
            2020-01-31  8:32:27 140116577302272 [Warning] InnoDB: A long semaphore wait:
            --Thread 140120359331584 has waited at lock0lock.cc line 6956 for 241.00 seconds the semaphore:
            Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
             
            2020-01-31  8:32:27 140116577302272 [Note] InnoDB: A semaphore wait:
            --Thread 140120359331584 has waited at lock0lock.cc line 6956 for 241.00 seconds the semaphore:
            Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
             
            2020-01-31  8:32:27 140116577302272 [Note] InnoDB: A semaphore wait:
            --Thread 140116746655488 has waited at lock0wait.cc line 501 for 190.00 seconds the semaphore:
            Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2
             
            InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
            InnoDB: Pending reads 0, writes 0
            

            Here are the stack traces of the hung server process:

            mariadb-10.2.21

            Thread 34 (Thread 0x7f705039b700 (LWP 26484)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6f10006644) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6f100065f0, cond=0x7f6f10006618) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x7f6f10006618, mutex=0x7f6f100065f0) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x7f6f100065e0) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x7f6f100065e0, reset_sig_count=406) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba12bcbfa in lock_wait_suspend_thread (thr=0x7f6f2402c458) at /mariadb/server/storage/innobase/lock/lock0wait.cc:355
            #6  0x0000557ba131cf74 in row_mysql_handle_errors (new_err=0x7f7050397cdc, trx=0x7f7051ebc118, thr=0x7f6f2402c458, savept=0x0) at /mariadb/server/storage/innobase/row/row0mysql.cc:741
            #7  0x0000557ba1336dab in row_search_mvcc (buf=<optimized out>, mode=PAGE_CUR_GE, prebuilt=0x7f6f2402bd90, match_mode=<optimized out>, direction=<optimized out>) at /mariadb/server/storage/innobase/row/row0sel.cc:5693
            #8  0x0000557ba126c3ed in ha_innobase::index_read (this=0x7f6f2402b620, buf=0x7f6f2402b238 "\377", key_ptr=0x7f6f10011a58 "\001", key_len=4, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:9431
            #9  0x0000557ba10eb70a in handler::index_read_idx_map (this=0x7f6f2402b620, buf=0x7f6f2402b238 "\377", index=0, key=0x7f6f10011a58 "\001", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:5487
            #10 0x0000557ba10e5f99 in handler::ha_index_read_idx_map (this=0x7f6f2402b620, buf=0x80 <error: Cannot access memory at address 0x80>, index=<optimized out>, key=0x7f6f10011a58 "\001", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:2657
            #11 0x0000557ba0fa8606 in join_read_const (tab=0x7f6f10010928) at /mariadb/server/sql/sql_select.cc:19336
            #12 0x0000557ba0fa81a2 in join_read_const_table (thd=0x7f6f10000c48, tab=0x7f6f10010928, pos=0x7f6f10010ee0) at /mariadb/server/sql/sql_select.cc:19216
            #13 0x0000557ba0f7dd05 in make_join_statistics (join=0x7f6f1000fea8, tables_list=..., keyuse_array=0x7f6f10010198) at /mariadb/server/sql/sql_select.cc:4310
            #14 JOIN::optimize_inner (this=<optimized out>) at /mariadb/server/sql/sql_select.cc:1582
            #15 0x0000557ba0f7bbda in JOIN::optimize (this=0x7f6f1000fea8) at /mariadb/server/sql/sql_select.cc:1115
            #16 0x0000557ba0f7980e in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /mariadb/server/sql/sql_select.cc:3804
            #17 0x0000557ba0f7964f in handle_select (thd=0x7f6f10000c48, lex=0x7f6f100045a0, result=0x7f6f1000fe88, setup_tables_done_option=<optimized out>) at /mariadb/server/sql/sql_select.cc:364
            #18 0x0000557ba0f5b38d in execute_sqlcom_select (thd=0x7f6f10000c48, all_tables=<optimized out>) at /mariadb/server/sql/sql_parse.cc:6481
            #19 0x0000557ba0f53b5d in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:3487
            #20 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f10000c48, rawbuf=0x7f6f1000f2a0 "select * from t1 where id=1 for update", length=<optimized out>, parser_state=0x7f705039a350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
            #21 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f10000c48, packet=0x7f6f10006f89 "select * from t1 where id=1 for update", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
            #22 0x0000557ba0f50220 in do_command (thd=0x7f6f10000c48) at /mariadb/server/sql/sql_parse.cc:1378
            #23 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
            #24 0x0000557ba1023973 in handle_one_connection (arg=0x557ba85fa0d8) at /mariadb/server/sql/sql_connect.cc:1241
            #25 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #26 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 33 (Thread 0x7f70503e6700 (LWP 26472)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82a6f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82a6f10, cond=0x557ba82a6f38) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba82a6f38, mutex=0x557ba82a6f10) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82a6f00) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba82a6f00, reset_sig_count=3) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f70503e4118: 0x7f70522bb010) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469
            #6  0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x7f705003d050, max_spins=30, max_delay=6, filename=0x557ba169c700 "/mariadb/server/storage/innobase/lock/lock0lock.cc", line=6956) at /mariadb/server/storage/innobase/include/ib0mutex.h:516
            #7  0x0000557ba12b9808 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x7f705003d050, n_spins=30, n_delay=6, name=<optimized out>, line=6956) at /mariadb/server/storage/innobase/include/ib0mutex.h:637
            #8  lock_trx_handle_wait (trx=0x7f7051ebb098) at /mariadb/server/storage/innobase/lock/lock0lock.cc:6956
            #9  0x0000557ba10e32b4 in kill_handlerton (thd=0x7f6f24000c48, plugin=<optimized out>, level=0x7f70503e42f4) at /mariadb/server/sql/handler.cc:809
            #10 0x0000557ba0f6431f in plugin_foreach_with_mask (thd=0x7f6f24000c48, func=0x557ba10e3280 <kill_handlerton(THD*, st_plugin_int*, void*)>, type=<optimized out>, state_mask=8, arg=0x7f70503e42f4) at /mariadb/server/sql/sql_plugin.cc:2396
            #11 0x0000557ba10e3270 in ha_kill_query (thd=0x557ba82a6f60, level=<optimized out>) at /mariadb/server/sql/handler.cc:816
            #12 0x0000557ba0f204ad in THD::awake (this=0x7f6f24000c48, state_to_set=KILL_QUERY_HARD) at /mariadb/server/sql/sql_class.cc:1715
            #13 0x0000557ba0f5ebf2 in kill_one_thread (thd=0x7f6f1c000c48, id=<optimized out>, kill_signal=KILL_QUERY_HARD, type=KILL_TYPE_ID) at /mariadb/server/sql/sql_parse.cc:8903
            #14 0x0000557ba0f584e8 in sql_kill (thd=0x7f6f1c000c48, id=8, state=NOT_KILLED, type=(KILL_TYPE_USER | unknown: 1538883012)) at /mariadb/server/sql/sql_parse.cc:9009
            #15 mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:5475
            #16 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f1c000c48, rawbuf=0x7f6f1c00f2a0 "kill query 8", length=<optimized out>, parser_state=0x7f70503e5350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
            #17 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f1c000c48, packet=0x7f6f1c006f89 "kill query 8", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
            #18 0x0000557ba0f50220 in do_command (thd=0x7f6f1c000c48) at /mariadb/server/sql/sql_parse.cc:1378
            #19 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
            #20 0x0000557ba1023973 in handle_one_connection (arg=0x557ba85fa698) at /mariadb/server/sql/sql_connect.cc:1241
            #21 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #22 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 32 (Thread 0x7f7050431700 (LWP 26471)):
            #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
            #1  0x00007f705bb948d9 in __GI___pthread_mutex_lock (mutex=0x7f6f240024d8) at ../nptl/pthread_mutex_lock.c:135
            #2  0x0000557ba0fb3e61 in inline_mysql_mutex_lock (that=0x7f6f240024d8, src_file=<optimized out>, src_line=2623) at /mariadb/server/include/mysql/psi/mysql_thread.h:683
            #3  mysqld_list_processes (thd=0x7f6f18000c48, user=0x0, verbose=<optimized out>) at /mariadb/server/sql/sql_show.cc:2623
            #4  0x0000557ba0f57776 in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:4776
            #5  0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f18000c48, rawbuf=0x7f6f1800f2a0 "show processlist", length=<optimized out>, parser_state=0x7f7050430350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
            #6  0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f18000c48, packet=0x7f6f18006f89 "", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
            #7  0x0000557ba0f50220 in do_command (thd=0x7f6f18000c48) at /mariadb/server/sql/sql_parse.cc:1378
            #8  0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
            #9  0x0000557ba1023973 in handle_one_connection (arg=0x557ba82f2a98) at /mariadb/server/sql/sql_connect.cc:1241
            #10 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #11 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 31 (Thread 0x7f705047c700 (LWP 26466)):
            #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
            #1  0x00007f705bb948d9 in __GI___pthread_mutex_lock (mutex=0x557ba1c25530 <LOCK_thread_count>) at ../nptl/pthread_mutex_lock.c:135
            #2  0x0000557ba0fc4fbb in inline_mysql_mutex_lock (that=0xfffffffffffffe00, src_file=<optimized out>, src_line=9923) at /mariadb/server/include/mysql/psi/mysql_thread.h:683
            #3  thd_get_error_context_description (thd=0x7f6f10000c48, buffer=0x7f7050477fe0 "H\264*\250{U", length=1024, max_query_len=3000) at /mariadb/server/sql/sql_show.cc:9923
            #4  0x0000557ba1263a71 in innobase_mysql_print_thd (f=0x557ba82a7230, thd=0x557ba1c25530 <LOCK_thread_count>, max_query_len=1538895916) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:2153
            #5  0x0000557ba12b9f21 in DeadlockChecker::print (trx=0x7f7051ebc118, max_query_len=3) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7224
            #6  0x0000557ba12ba28d in DeadlockChecker::notify (this=0x7f70504784e8, lock=0x7f7051ebb140) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7371
            #7  0x0000557ba12ba8d6 in DeadlockChecker::search (this=0x7f70504784e8) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7487
            #8  0x0000557ba12af6ac in DeadlockChecker::check_and_resolve (lock=0x7f7051ebb298, trx=0x7f7051ebb098) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7622
            #9  0x0000557ba12af0a1 in lock_rec_enqueue_waiting (c_lock=<optimized out>, type_mode=<optimized out>, block=<optimized out>, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>, prdt=0x0) at /mariadb/server/storage/innobase/lock/lock0lock.cc:1796
            #10 0x0000557ba12b7a21 in lock_rec_lock_slow (impl=<optimized out>, mode=<optimized out>, block=0x7f7043cfe140, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>) at /mariadb/server/storage/innobase/lock/lock0lock.cc:2113
            #11 lock_rec_lock (impl=<optimized out>, mode=<optimized out>, block=0x7f7043cfe140, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>) at /mariadb/server/storage/innobase/lock/lock0lock.cc:2178
            #12 0x0000557ba12b8150 in lock_clust_rec_read_check_and_lock (flags=<optimized out>, block=0x7f7043cfe140, rec=<optimized out>, index=0x7f6f240221b0, offsets=<optimized out>, mode=<optimized out>, gap_mode=<optimized out>, thr=0x7f6f24019898) at /mariadb/server/storage/innobase/lock/lock0lock.cc:6393
            #13 0x0000557ba133a1fa in sel_set_rec_lock (pcur=<optimized out>, rec=0x7f70444d0093 "\200", index=0x7f6f240221b0, offsets=<optimized out>, mode=<optimized out>, type=<optimized out>, thr=0x7f6f24019898, mtr=0x7f70504790f8) at /mariadb/server/storage/innobase/row/row0sel.cc:1261
            #14 0x0000557ba1337dde in row_search_mvcc (buf=<optimized out>, mode=PAGE_CUR_GE, prebuilt=0x7f6f240191d0, match_mode=<optimized out>, direction=<optimized out>) at /mariadb/server/storage/innobase/row/row0sel.cc:5090
            #15 0x0000557ba126c3ed in ha_innobase::index_read (this=0x7f6f24018a60, buf=0x7f6f24018678 "\377", key_ptr=0x7f6f24011a58 "\002", key_len=4, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:9431
            #16 0x0000557ba10eb70a in handler::index_read_idx_map (this=0x7f6f24018a60, buf=0x7f6f24018678 "\377", index=0, key=0x7f6f24011a58 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:5487
            #17 0x0000557ba10e5f99 in handler::ha_index_read_idx_map (this=0x7f6f24018a60, buf=0x80 <error: Cannot access memory at address 0x80>, index=<optimized out>, key=0x7f6f24011a58 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:2657
            #18 0x0000557ba0fa8606 in join_read_const (tab=0x7f6f24010928) at /mariadb/server/sql/sql_select.cc:19336
            #19 0x0000557ba0fa81a2 in join_read_const_table (thd=0x7f6f24000c48, tab=0x7f6f24010928, pos=0x7f6f24010ee0) at /mariadb/server/sql/sql_select.cc:19216
            #20 0x0000557ba0f7dd05 in make_join_statistics (join=0x7f6f2400fea8, tables_list=..., keyuse_array=0x7f6f24010198) at /mariadb/server/sql/sql_select.cc:4310
            #21 JOIN::optimize_inner (this=<optimized out>) at /mariadb/server/sql/sql_select.cc:1582
            #22 0x0000557ba0f7bbda in JOIN::optimize (this=0x7f6f2400fea8) at /mariadb/server/sql/sql_select.cc:1115
            #23 0x0000557ba0f7980e in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /mariadb/server/sql/sql_select.cc:3804
            #24 0x0000557ba0f7964f in handle_select (thd=0x7f6f24000c48, lex=0x7f6f240045a0, result=0x7f6f2400fe88, setup_tables_done_option=<optimized out>) at /mariadb/server/sql/sql_select.cc:364
            #25 0x0000557ba0f5b38d in execute_sqlcom_select (thd=0x7f6f24000c48, all_tables=<optimized out>) at /mariadb/server/sql/sql_parse.cc:6481
            #26 0x0000557ba0f53b5d in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:3487
            #27 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f24000c48, rawbuf=0x7f6f2400f2a0 "select * from t1 where id=2 for update", length=<optimized out>, parser_state=0x7f705047b350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015
            #28 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f24000c48, packet=0x7f6f24006f89 "select * from t1 where id=2 for update", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825
            #29 0x0000557ba0f50220 in do_command (thd=0x7f6f24000c48) at /mariadb/server/sql/sql_parse.cc:1378
            #30 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335
            #31 0x0000557ba1023973 in handle_one_connection (arg=0x557ba8608058) at /mariadb/server/sql/sql_connect.cc:1241
            #32 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #33 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 30 (Thread 0x7f70504c7700 (LWP 26463)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba1c2306c <COND_slave_background+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba1c25830 <LOCK_slave_background>, cond=0x557ba1c23040 <COND_slave_background>) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba1c23040 <COND_slave_background>, mutex=0x557ba1c25830 <LOCK_slave_background>) at pthread_cond_wait.c:655
            #3  0x0000557ba0edca0b in inline_mysql_cond_wait (that=0x557ba1c23040 <COND_slave_background>, mutex=<optimized out>, src_file=<optimized out>, src_line=334) at /mariadb/server/include/mysql/psi/mysql_thread.h:1149
            #4  handle_slave_background (arg=<optimized out>) at /mariadb/server/sql/slave.cc:334
            #5  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #6  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 29 (Thread 0x7f7050512700 (LWP 26462)):
            #0  0x00007f705b12ddd2 in __GI___sigtimedwait (set=set@entry=0x7f7050511dc8, info=info@entry=0x7f7050511cb0, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
            #1  0x00007f705bb9c07c in __sigwait (set=0x7f7050511dc8, sig=0x7f7050511d60) at ../sysdeps/unix/sysv/linux/sigwait.c:28
            #2  0x0000557ba0ec317b in signal_hand (arg=<optimized out>) at /mariadb/server/sql/mysqld.cc:3513
            #3  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #4  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 28 (Thread 0x7f6f4daaf700 (LWP 26461)):
            #0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f4daaed98, expected=0, futex_word=0x557ba249f178 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7f6f4daaed98, mutex=0x557ba249f118 <LOCK_checkpoint>, cond=0x557ba249f150 <COND_checkpoint>) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x557ba249f150 <COND_checkpoint>, mutex=0x557ba249f118 <LOCK_checkpoint>, abstime=0x7f6f4daaed98) at pthread_cond_wait.c:667
            #3  0x0000557ba14bbb5f in inline_mysql_cond_timedwait (that=0x557ba249f150 <COND_checkpoint>, mutex=0x557ba249f118 <LOCK_checkpoint>, abstime=0x7f6f4daaed98, src_file=<optimized out>, src_line=116) at /mariadb/server/include/mysql/psi/mysql_thread.h:1186
            #4  my_service_thread_sleep (control=0x557ba1ba82b0 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/server/storage/maria/ma_servicethread.c:115
            #5  0x0000557ba14b4c4c in ma_checkpoint_background (arg=<optimized out>) at /mariadb/server/storage/maria/ma_checkpoint.c:709
            #6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 27 (Thread 0x7f6f4e7fc700 (LWP 26460)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6f4e7fbe00) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6f4e7fbda8, cond=0x7f6f4e7fbdd8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x7f6f4e7fbdd8, mutex=0x7f6f4e7fbda8) at pthread_cond_wait.c:655
            #3  0x0000557ba126327b in inline_mysql_cond_wait (that=0x7f6f4e7fbdd8, mutex=0x7f6f4e7fbda8, src_file=<optimized out>, src_line=332) at /mariadb/server/include/mysql/psi/mysql_thread.h:1149
            #4  thd_destructor_proxy () at /mariadb/server/storage/innobase/handler/ha_innodb.cc:332
            #5  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #6  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 26 (Thread 0x7f6f4effd700 (LWP 26459)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327b30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327ae0, cond=0x557ba4327b08) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba4327b08, mutex=0x557ba4327ae0) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327ad0) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba4327ad0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba13c9ee2 in buf_resize_thread () at /mariadb/server/storage/innobase/buf/buf0buf.cc:3102
            #6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 25 (Thread 0x7f6f4f7fe700 (LWP 26458)):
            #0  0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f4f7fd7a8, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
            #1  0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225
            #2  0x0000557ba13c675a in btr_defragment_thread () at /mariadb/server/storage/innobase/btr/btr0defragment.cc:709
            #3  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #4  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 24 (Thread 0x7f6f4ffff700 (LWP 26457)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327a50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327a00, cond=0x557ba4327a28) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba4327a28, mutex=0x557ba4327a00) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43279f0) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba43279f0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba13deb60 in buf_dump_thread () at /mariadb/server/storage/innobase/buf/buf0dump.cc:807
            #6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 23 (Thread 0x7f6f6c8e6700 (LWP 26456)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327900) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba43278b0, cond=0x557ba43278d8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba43278d8, mutex=0x557ba43278b0) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43278a0) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba43278a0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba6968 <srv_sys+360>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
            #6  srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580
            #7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 22 (Thread 0x7f6f6d0e7700 (LWP 26455)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327890) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327840, cond=0x557ba4327868) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba4327868, mutex=0x557ba4327840) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327830) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba4327830, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba6930 <srv_sys+304>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
            #6  srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580
            #7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 21 (Thread 0x7f6f6d8e8700 (LWP 26454)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327820) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba43277d0, cond=0x557ba43277f8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba43277f8, mutex=0x557ba43277d0) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43277c0) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba43277c0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba68f8 <srv_sys+248>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
            #6  srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580
            #7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 20 (Thread 0x7f6f76690700 (LWP 26453)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba43277b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327760, cond=0x557ba4327788) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba4327788, mutex=0x557ba4327760) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327750) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba4327750, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba135b821 in srv_resume_thread (slot=<optimized out>, sig_count=<optimized out>, wait=<optimized out>, timeout_usec=10000) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930
            #6  srv_purge_coordinator_suspend (slot=<optimized out>, rseg_history_len=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2730
            #7  srv_purge_coordinator_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2819
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 19 (Thread 0x7f6f76e91700 (LWP 26452)):
            #0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f76e90bc0, expected=0, futex_word=0x557ba8306b50) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7f6f76e90bc0, mutex=0x557ba8306b00, cond=0x557ba8306b28) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x557ba8306b28, mutex=0x557ba8306b00, abstime=0x7f6f76e90bc0) at pthread_cond_wait.c:667
            #3  0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f76e90bc0) at /mariadb/server/storage/innobase/os/os0event.cc:279
            #4  0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba8306af0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:406
            #5  0x0000557ba13995bf in ib_wqueue_timedwait (wq=0x557ba42f6f20, wait_in_usecs=5000000) at /mariadb/server/storage/innobase/ut/ut0wqueue.cc:161
            #6  0x0000557ba14686c4 in fts_optimize_thread (arg=0x557ba42f6f20) at /mariadb/server/storage/innobase/fts/fts0opt.cc:2897
            #7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 18 (Thread 0x7f6f77692700 (LWP 26451)):
            #0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f77691dc0, expected=0, futex_word=0x557ba82aa6f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7f6f77691dc0, mutex=0x557ba82aa6a0, cond=0x557ba82aa6c8) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x557ba82aa6c8, mutex=0x557ba82aa6a0, abstime=0x7f6f77691dc0) at pthread_cond_wait.c:667
            #3  0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f77691dc0) at /mariadb/server/storage/innobase/os/os0event.cc:279
            #4  0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba82aa690, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:406
            #5  0x0000557ba1424089 in dict_stats_thread () at /mariadb/server/storage/innobase/dict/dict0stats_bg.cc:458
            #6  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #7  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 17 (Thread 0x7f6f77e93700 (LWP 26450)):
            #0  0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f77e92c68, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
            #1  0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225
            #2  0x0000557ba1359e4f in srv_master_sleep () at /mariadb/server/storage/innobase/srv/srv0srv.cc:2385
            #3  srv_master_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2426
            #4  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #5  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 16 (Thread 0x7f6f78694700 (LWP 26449)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82ab4f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82ab4a0, cond=0x557ba82ab4c8) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba82ab4c8, mutex=0x557ba82ab4a0) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82ab490) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba82ab490, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f6f78693ca8: 0x7f70522bb090) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469
            #6  0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x557ba82aaa80, max_spins=30, max_delay=6, filename=0x557ba16a2e36 "/mariadb/server/storage/innobase/read/read0read.cc", line=714) at /mariadb/server/storage/innobase/include/ib0mutex.h:516
            #7  0x0000557ba12f79f0 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x557ba82aaa80, n_spins=30, n_delay=6, name=<optimized out>, line=714) at /mariadb/server/storage/innobase/include/ib0mutex.h:637
            #8  MVCC::size (this=0x557ba42e8940) at /mariadb/server/storage/innobase/read/read0read.cc:714
            #9  0x0000557ba1356f86 in srv_printf_innodb_monitor (file=0x7f705b2ad680 <_IO_2_1_stderr_>, nowait=<optimized out>, trx_start_pos=<optimized out>, trx_end=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:1393
            #10 0x0000557ba135901e in srv_monitor_thread () at /mariadb/server/storage/innobase/srv/srv0srv.cc:1752
            #11 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #12 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 15 (Thread 0x7f6f6ed13700 (LWP 26448)):
            #0  0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f6ed12a28, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
            #1  0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225
            #2  0x0000557ba1365d04 in sync_array_print_long_waits (waiter=<optimized out>, sema=<optimized out>) at /mariadb/server/storage/innobase/sync/sync0arr.cc:1104
            #3  0x0000557ba13598c5 in srv_error_monitor_thread () at /mariadb/server/storage/innobase/srv/srv0srv.cc:1861
            #4  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #5  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 14 (Thread 0x7f6f78e95700 (LWP 26447)):
            #0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82a6f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
            #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82a6f10, cond=0x557ba82a6f38) at pthread_cond_wait.c:502
            #2  __pthread_cond_wait (cond=0x557ba82a6f38, mutex=0x557ba82a6f10) at pthread_cond_wait.c:655
            #3  0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82a6f00) at /mariadb/server/storage/innobase/os/os0event.cc:162
            #4  os_event::wait_low (this=0x557ba82a6f00, reset_sig_count=3) at /mariadb/server/storage/innobase/os/os0event.cc:329
            #5  0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f6f78e94da8: 0x7f70522bb050) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469
            #6  0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x7f705003d050, max_spins=30, max_delay=6, filename=0x557ba169d35a "/mariadb/server/storage/innobase/lock/lock0wait.cc", line=501) at /mariadb/server/storage/innobase/include/ib0mutex.h:516
            #7  0x0000557ba12bd649 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x7f705003d050, n_spins=30, n_delay=6, name=<optimized out>, line=501) at /mariadb/server/storage/innobase/include/ib0mutex.h:637
            #8  lock_wait_check_and_cancel (slot=0x7f705003d138) at /mariadb/server/storage/innobase/lock/lock0wait.cc:501
            #9  lock_wait_timeout_thread () at /mariadb/server/storage/innobase/lock/lock0wait.cc:569
            #10 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #11 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 13 (Thread 0x7f6f6f719700 (LWP 26444)):
            #0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f6f718b70, expected=0, futex_word=0x557ba4327ac4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7f6f6f718b70, mutex=0x557ba4327a70, cond=0x557ba4327a98) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x557ba4327a98, mutex=0x557ba4327a70, abstime=0x7f6f6f718b70) at pthread_cond_wait.c:667
            #3  0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f6f718b70) at /mariadb/server/storage/innobase/os/os0event.cc:279
            #4  0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba4327a60, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/server/storage/innobase/os/os0event.cc:406
            #5  0x0000557ba13e4047 in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/server/storage/innobase/buf/buf0flu.cc:2702
            #6  buf_flush_page_cleaner_coordinator () at /mariadb/server/storage/innobase/buf/buf0flu.cc:3168
            #7  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #8  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 12 (Thread 0x7f6f6ff1a700 (LWP 26443)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f6ff19b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f6ff19b60, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=9) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 11 (Thread 0x7f6f7071b700 (LWP 26442)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7071ab60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7071ab60, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=8) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 10 (Thread 0x7f6f70f1c700 (LWP 26441)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f70f1bb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f70f1bb60, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=7) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 9 (Thread 0x7f6f7171d700 (LWP 26440)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7171cb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7171cb60, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=6) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 8 (Thread 0x7f6f71f1e700 (LWP 26439)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f71f1db60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f71f1db60, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=5) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 7 (Thread 0x7f6f7271f700 (LWP 26438)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7271eb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7271eb60, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=4) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 6 (Thread 0x7f6f72f20700 (LWP 26437)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f72f1fb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f72f1fb60, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=3) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 5 (Thread 0x7f6f73721700 (LWP 26436)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f73720b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f73720b60, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=2) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 4 (Thread 0x7f6f73f22700 (LWP 26435)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f73f21b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f73f21b60, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=1) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 3 (Thread 0x7f6f74723700 (LWP 26434)):
            #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
            #1  0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
            #2  0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f74722b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924
            #3  0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f74722b60, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069
            #4  0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123
            #5  os_aio_handler (segment=<optimized out>, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728
            #6  0x0000557ba1437d2a in fil_aio_wait (segment=0) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211
            #7  0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333
            #8  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #9  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 2 (Thread 0x7f705adff700 (LWP 26433)):
            #0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f705adfee00, expected=0, futex_word=0x557ba24ae818 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            #1  __pthread_cond_wait_common (abstime=0x7f705adfee00, mutex=0x557ba24ae7c0 <LOCK_timer>, cond=0x557ba24ae7f0 <COND_timer>) at pthread_cond_wait.c:539
            #2  __pthread_cond_timedwait (cond=0x557ba24ae7f0 <COND_timer>, mutex=0x557ba24ae7c0 <LOCK_timer>, abstime=0x7f705adfee00) at pthread_cond_wait.c:667
            #3  0x0000557ba15a5ed6 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=0x7f705adfee00, src_file=<optimized out>, src_line=292) at /mariadb/server/include/mysql/psi/mysql_thread.h:1186
            #4  timer_handler (arg=<optimized out>) at /mariadb/server/mysys/thr_timer.c:292
            #5  0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
            #6  0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
             
            Thread 1 (Thread 0x7f705afa5340 (LWP 26432)):
            #0  0x00007f705b1e2d0f in __GI___poll (fds=0x7ffc43d161a0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
            #1  0x0000557ba0ec8e39 in handle_connections_sockets () at /mariadb/server/sql/mysqld.cc:6619
            #2  0x0000557ba0ec4797 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/server/sql/mysqld.cc:6085
            #3  0x00007f705b119bbb in __libc_start_main (main=0x557ba0ebf760 <main(int, char**)>, argc=5, argv=0x7ffc43d168a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc43d16898) at ../csu/libc-start.c:308
            #4  0x0000557ba0ebf69a in _start ()
            

            The threads that participate in the deadlock are as follows, using the identifiers introduced in my previous comment:

            • Thread A = Thread 31
            • Thread B = Thread 33
            • Thread C = Thread 32
            marko Marko Mäkelä added a comment - I repeated this hang locally. My test ran for a couple of seconds from the server startup until the deadlock occurred. After reverting the attempted fix of MDEV-11896 , the server would not hang. I interrupted the test after 3 minutes. I will not disclose my scripts (involving 4 clients, 2 of them causing user transaction deadlocks when accessing an InnoDB table), because it might make the life too easy for script kiddies. Here is the server error log output: mariadb-10.2.21 2020-01-31 8:28:22 140120539419456 [Note] ../sql/mysqld: ready for connections. Version: '10.2.21-MariaDB' socket: '/tmp/mysql.sock' port: 3306 Source distribution 2020-01-31 8:32:27 140116577302272 [Warning] InnoDB: A long semaphore wait: --Thread 140120359331584 has waited at lock0lock.cc line 6956 for 241.00 seconds the semaphore: Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2   2020-01-31 8:32:27 140116577302272 [Note] InnoDB: A semaphore wait: --Thread 140120359331584 has waited at lock0lock.cc line 6956 for 241.00 seconds the semaphore: Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2   2020-01-31 8:32:27 140116577302272 [Note] InnoDB: A semaphore wait: --Thread 140116746655488 has waited at lock0wait.cc line 501 for 190.00 seconds the semaphore: Mutex at 0x7f705003d050, Mutex LOCK_SYS created lock0lock.cc:473, lock var 2   InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending reads 0, writes 0 Here are the stack traces of the hung server process: mariadb-10.2.21 Thread 34 (Thread 0x7f705039b700 (LWP 26484)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6f10006644) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6f100065f0, cond=0x7f6f10006618) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x7f6f10006618, mutex=0x7f6f100065f0) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x7f6f100065e0) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x7f6f100065e0, reset_sig_count=406) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba12bcbfa in lock_wait_suspend_thread (thr=0x7f6f2402c458) at /mariadb/server/storage/innobase/lock/lock0wait.cc:355 #6 0x0000557ba131cf74 in row_mysql_handle_errors (new_err=0x7f7050397cdc, trx=0x7f7051ebc118, thr=0x7f6f2402c458, savept=0x0) at /mariadb/server/storage/innobase/row/row0mysql.cc:741 #7 0x0000557ba1336dab in row_search_mvcc (buf=<optimized out>, mode=PAGE_CUR_GE, prebuilt=0x7f6f2402bd90, match_mode=<optimized out>, direction=<optimized out>) at /mariadb/server/storage/innobase/row/row0sel.cc:5693 #8 0x0000557ba126c3ed in ha_innobase::index_read (this=0x7f6f2402b620, buf=0x7f6f2402b238 "\377", key_ptr=0x7f6f10011a58 "\001", key_len=4, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:9431 #9 0x0000557ba10eb70a in handler::index_read_idx_map (this=0x7f6f2402b620, buf=0x7f6f2402b238 "\377", index=0, key=0x7f6f10011a58 "\001", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:5487 #10 0x0000557ba10e5f99 in handler::ha_index_read_idx_map (this=0x7f6f2402b620, buf=0x80 <error: Cannot access memory at address 0x80>, index=<optimized out>, key=0x7f6f10011a58 "\001", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:2657 #11 0x0000557ba0fa8606 in join_read_const (tab=0x7f6f10010928) at /mariadb/server/sql/sql_select.cc:19336 #12 0x0000557ba0fa81a2 in join_read_const_table (thd=0x7f6f10000c48, tab=0x7f6f10010928, pos=0x7f6f10010ee0) at /mariadb/server/sql/sql_select.cc:19216 #13 0x0000557ba0f7dd05 in make_join_statistics (join=0x7f6f1000fea8, tables_list=..., keyuse_array=0x7f6f10010198) at /mariadb/server/sql/sql_select.cc:4310 #14 JOIN::optimize_inner (this=<optimized out>) at /mariadb/server/sql/sql_select.cc:1582 #15 0x0000557ba0f7bbda in JOIN::optimize (this=0x7f6f1000fea8) at /mariadb/server/sql/sql_select.cc:1115 #16 0x0000557ba0f7980e in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /mariadb/server/sql/sql_select.cc:3804 #17 0x0000557ba0f7964f in handle_select (thd=0x7f6f10000c48, lex=0x7f6f100045a0, result=0x7f6f1000fe88, setup_tables_done_option=<optimized out>) at /mariadb/server/sql/sql_select.cc:364 #18 0x0000557ba0f5b38d in execute_sqlcom_select (thd=0x7f6f10000c48, all_tables=<optimized out>) at /mariadb/server/sql/sql_parse.cc:6481 #19 0x0000557ba0f53b5d in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:3487 #20 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f10000c48, rawbuf=0x7f6f1000f2a0 "select * from t1 where id=1 for update", length=<optimized out>, parser_state=0x7f705039a350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015 #21 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f10000c48, packet=0x7f6f10006f89 "select * from t1 where id=1 for update", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825 #22 0x0000557ba0f50220 in do_command (thd=0x7f6f10000c48) at /mariadb/server/sql/sql_parse.cc:1378 #23 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335 #24 0x0000557ba1023973 in handle_one_connection (arg=0x557ba85fa0d8) at /mariadb/server/sql/sql_connect.cc:1241 #25 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #26 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 33 (Thread 0x7f70503e6700 (LWP 26472)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82a6f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82a6f10, cond=0x557ba82a6f38) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba82a6f38, mutex=0x557ba82a6f10) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82a6f00) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba82a6f00, reset_sig_count=3) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f70503e4118: 0x7f70522bb010) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469 #6 0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x7f705003d050, max_spins=30, max_delay=6, filename=0x557ba169c700 "/mariadb/server/storage/innobase/lock/lock0lock.cc", line=6956) at /mariadb/server/storage/innobase/include/ib0mutex.h:516 #7 0x0000557ba12b9808 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x7f705003d050, n_spins=30, n_delay=6, name=<optimized out>, line=6956) at /mariadb/server/storage/innobase/include/ib0mutex.h:637 #8 lock_trx_handle_wait (trx=0x7f7051ebb098) at /mariadb/server/storage/innobase/lock/lock0lock.cc:6956 #9 0x0000557ba10e32b4 in kill_handlerton (thd=0x7f6f24000c48, plugin=<optimized out>, level=0x7f70503e42f4) at /mariadb/server/sql/handler.cc:809 #10 0x0000557ba0f6431f in plugin_foreach_with_mask (thd=0x7f6f24000c48, func=0x557ba10e3280 <kill_handlerton(THD*, st_plugin_int*, void*)>, type=<optimized out>, state_mask=8, arg=0x7f70503e42f4) at /mariadb/server/sql/sql_plugin.cc:2396 #11 0x0000557ba10e3270 in ha_kill_query (thd=0x557ba82a6f60, level=<optimized out>) at /mariadb/server/sql/handler.cc:816 #12 0x0000557ba0f204ad in THD::awake (this=0x7f6f24000c48, state_to_set=KILL_QUERY_HARD) at /mariadb/server/sql/sql_class.cc:1715 #13 0x0000557ba0f5ebf2 in kill_one_thread (thd=0x7f6f1c000c48, id=<optimized out>, kill_signal=KILL_QUERY_HARD, type=KILL_TYPE_ID) at /mariadb/server/sql/sql_parse.cc:8903 #14 0x0000557ba0f584e8 in sql_kill (thd=0x7f6f1c000c48, id=8, state=NOT_KILLED, type=(KILL_TYPE_USER | unknown: 1538883012)) at /mariadb/server/sql/sql_parse.cc:9009 #15 mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:5475 #16 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f1c000c48, rawbuf=0x7f6f1c00f2a0 "kill query 8", length=<optimized out>, parser_state=0x7f70503e5350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015 #17 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f1c000c48, packet=0x7f6f1c006f89 "kill query 8", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825 #18 0x0000557ba0f50220 in do_command (thd=0x7f6f1c000c48) at /mariadb/server/sql/sql_parse.cc:1378 #19 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335 #20 0x0000557ba1023973 in handle_one_connection (arg=0x557ba85fa698) at /mariadb/server/sql/sql_connect.cc:1241 #21 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #22 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 32 (Thread 0x7f7050431700 (LWP 26471)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103 #1 0x00007f705bb948d9 in __GI___pthread_mutex_lock (mutex=0x7f6f240024d8) at ../nptl/pthread_mutex_lock.c:135 #2 0x0000557ba0fb3e61 in inline_mysql_mutex_lock (that=0x7f6f240024d8, src_file=<optimized out>, src_line=2623) at /mariadb/server/include/mysql/psi/mysql_thread.h:683 #3 mysqld_list_processes (thd=0x7f6f18000c48, user=0x0, verbose=<optimized out>) at /mariadb/server/sql/sql_show.cc:2623 #4 0x0000557ba0f57776 in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:4776 #5 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f18000c48, rawbuf=0x7f6f1800f2a0 "show processlist", length=<optimized out>, parser_state=0x7f7050430350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015 #6 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f18000c48, packet=0x7f6f18006f89 "", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825 #7 0x0000557ba0f50220 in do_command (thd=0x7f6f18000c48) at /mariadb/server/sql/sql_parse.cc:1378 #8 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335 #9 0x0000557ba1023973 in handle_one_connection (arg=0x557ba82f2a98) at /mariadb/server/sql/sql_connect.cc:1241 #10 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #11 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 31 (Thread 0x7f705047c700 (LWP 26466)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103 #1 0x00007f705bb948d9 in __GI___pthread_mutex_lock (mutex=0x557ba1c25530 <LOCK_thread_count>) at ../nptl/pthread_mutex_lock.c:135 #2 0x0000557ba0fc4fbb in inline_mysql_mutex_lock (that=0xfffffffffffffe00, src_file=<optimized out>, src_line=9923) at /mariadb/server/include/mysql/psi/mysql_thread.h:683 #3 thd_get_error_context_description (thd=0x7f6f10000c48, buffer=0x7f7050477fe0 "H\264*\250{U", length=1024, max_query_len=3000) at /mariadb/server/sql/sql_show.cc:9923 #4 0x0000557ba1263a71 in innobase_mysql_print_thd (f=0x557ba82a7230, thd=0x557ba1c25530 <LOCK_thread_count>, max_query_len=1538895916) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:2153 #5 0x0000557ba12b9f21 in DeadlockChecker::print (trx=0x7f7051ebc118, max_query_len=3) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7224 #6 0x0000557ba12ba28d in DeadlockChecker::notify (this=0x7f70504784e8, lock=0x7f7051ebb140) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7371 #7 0x0000557ba12ba8d6 in DeadlockChecker::search (this=0x7f70504784e8) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7487 #8 0x0000557ba12af6ac in DeadlockChecker::check_and_resolve (lock=0x7f7051ebb298, trx=0x7f7051ebb098) at /mariadb/server/storage/innobase/lock/lock0lock.cc:7622 #9 0x0000557ba12af0a1 in lock_rec_enqueue_waiting (c_lock=<optimized out>, type_mode=<optimized out>, block=<optimized out>, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>, prdt=0x0) at /mariadb/server/storage/innobase/lock/lock0lock.cc:1796 #10 0x0000557ba12b7a21 in lock_rec_lock_slow (impl=<optimized out>, mode=<optimized out>, block=0x7f7043cfe140, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>) at /mariadb/server/storage/innobase/lock/lock0lock.cc:2113 #11 lock_rec_lock (impl=<optimized out>, mode=<optimized out>, block=0x7f7043cfe140, heap_no=<optimized out>, index=<optimized out>, thr=<optimized out>) at /mariadb/server/storage/innobase/lock/lock0lock.cc:2178 #12 0x0000557ba12b8150 in lock_clust_rec_read_check_and_lock (flags=<optimized out>, block=0x7f7043cfe140, rec=<optimized out>, index=0x7f6f240221b0, offsets=<optimized out>, mode=<optimized out>, gap_mode=<optimized out>, thr=0x7f6f24019898) at /mariadb/server/storage/innobase/lock/lock0lock.cc:6393 #13 0x0000557ba133a1fa in sel_set_rec_lock (pcur=<optimized out>, rec=0x7f70444d0093 "\200", index=0x7f6f240221b0, offsets=<optimized out>, mode=<optimized out>, type=<optimized out>, thr=0x7f6f24019898, mtr=0x7f70504790f8) at /mariadb/server/storage/innobase/row/row0sel.cc:1261 #14 0x0000557ba1337dde in row_search_mvcc (buf=<optimized out>, mode=PAGE_CUR_GE, prebuilt=0x7f6f240191d0, match_mode=<optimized out>, direction=<optimized out>) at /mariadb/server/storage/innobase/row/row0sel.cc:5090 #15 0x0000557ba126c3ed in ha_innobase::index_read (this=0x7f6f24018a60, buf=0x7f6f24018678 "\377", key_ptr=0x7f6f24011a58 "\002", key_len=4, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/storage/innobase/handler/ha_innodb.cc:9431 #16 0x0000557ba10eb70a in handler::index_read_idx_map (this=0x7f6f24018a60, buf=0x7f6f24018678 "\377", index=0, key=0x7f6f24011a58 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:5487 #17 0x0000557ba10e5f99 in handler::ha_index_read_idx_map (this=0x7f6f24018a60, buf=0x80 <error: Cannot access memory at address 0x80>, index=<optimized out>, key=0x7f6f24011a58 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /mariadb/server/sql/handler.cc:2657 #18 0x0000557ba0fa8606 in join_read_const (tab=0x7f6f24010928) at /mariadb/server/sql/sql_select.cc:19336 #19 0x0000557ba0fa81a2 in join_read_const_table (thd=0x7f6f24000c48, tab=0x7f6f24010928, pos=0x7f6f24010ee0) at /mariadb/server/sql/sql_select.cc:19216 #20 0x0000557ba0f7dd05 in make_join_statistics (join=0x7f6f2400fea8, tables_list=..., keyuse_array=0x7f6f24010198) at /mariadb/server/sql/sql_select.cc:4310 #21 JOIN::optimize_inner (this=<optimized out>) at /mariadb/server/sql/sql_select.cc:1582 #22 0x0000557ba0f7bbda in JOIN::optimize (this=0x7f6f2400fea8) at /mariadb/server/sql/sql_select.cc:1115 #23 0x0000557ba0f7980e in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /mariadb/server/sql/sql_select.cc:3804 #24 0x0000557ba0f7964f in handle_select (thd=0x7f6f24000c48, lex=0x7f6f240045a0, result=0x7f6f2400fe88, setup_tables_done_option=<optimized out>) at /mariadb/server/sql/sql_select.cc:364 #25 0x0000557ba0f5b38d in execute_sqlcom_select (thd=0x7f6f24000c48, all_tables=<optimized out>) at /mariadb/server/sql/sql_parse.cc:6481 #26 0x0000557ba0f53b5d in mysql_execute_command (thd=<optimized out>) at /mariadb/server/sql/sql_parse.cc:3487 #27 0x0000557ba0f515c0 in mysql_parse (thd=0x7f6f24000c48, rawbuf=0x7f6f2400f2a0 "select * from t1 where id=2 for update", length=<optimized out>, parser_state=0x7f705047b350, is_com_multi=<optimized out>, is_next_command=false) at /mariadb/server/sql/sql_parse.cc:8015 #28 0x0000557ba0f4f02c in dispatch_command (command=<optimized out>, thd=0x7f6f24000c48, packet=0x7f6f24006f89 "select * from t1 where id=2 for update", packet_length=3, is_com_multi=false, is_next_command=<optimized out>) at /mariadb/server/sql/sql_parse.cc:1825 #29 0x0000557ba0f50220 in do_command (thd=0x7f6f24000c48) at /mariadb/server/sql/sql_parse.cc:1378 #30 0x0000557ba1023be0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/server/sql/sql_connect.cc:1335 #31 0x0000557ba1023973 in handle_one_connection (arg=0x557ba8608058) at /mariadb/server/sql/sql_connect.cc:1241 #32 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #33 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 30 (Thread 0x7f70504c7700 (LWP 26463)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba1c2306c <COND_slave_background+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba1c25830 <LOCK_slave_background>, cond=0x557ba1c23040 <COND_slave_background>) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba1c23040 <COND_slave_background>, mutex=0x557ba1c25830 <LOCK_slave_background>) at pthread_cond_wait.c:655 #3 0x0000557ba0edca0b in inline_mysql_cond_wait (that=0x557ba1c23040 <COND_slave_background>, mutex=<optimized out>, src_file=<optimized out>, src_line=334) at /mariadb/server/include/mysql/psi/mysql_thread.h:1149 #4 handle_slave_background (arg=<optimized out>) at /mariadb/server/sql/slave.cc:334 #5 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #6 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 29 (Thread 0x7f7050512700 (LWP 26462)): #0 0x00007f705b12ddd2 in __GI___sigtimedwait (set=set@entry=0x7f7050511dc8, info=info@entry=0x7f7050511cb0, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29 #1 0x00007f705bb9c07c in __sigwait (set=0x7f7050511dc8, sig=0x7f7050511d60) at ../sysdeps/unix/sysv/linux/sigwait.c:28 #2 0x0000557ba0ec317b in signal_hand (arg=<optimized out>) at /mariadb/server/sql/mysqld.cc:3513 #3 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #4 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 28 (Thread 0x7f6f4daaf700 (LWP 26461)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f4daaed98, expected=0, futex_word=0x557ba249f178 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6f4daaed98, mutex=0x557ba249f118 <LOCK_checkpoint>, cond=0x557ba249f150 <COND_checkpoint>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x557ba249f150 <COND_checkpoint>, mutex=0x557ba249f118 <LOCK_checkpoint>, abstime=0x7f6f4daaed98) at pthread_cond_wait.c:667 #3 0x0000557ba14bbb5f in inline_mysql_cond_timedwait (that=0x557ba249f150 <COND_checkpoint>, mutex=0x557ba249f118 <LOCK_checkpoint>, abstime=0x7f6f4daaed98, src_file=<optimized out>, src_line=116) at /mariadb/server/include/mysql/psi/mysql_thread.h:1186 #4 my_service_thread_sleep (control=0x557ba1ba82b0 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/server/storage/maria/ma_servicethread.c:115 #5 0x0000557ba14b4c4c in ma_checkpoint_background (arg=<optimized out>) at /mariadb/server/storage/maria/ma_checkpoint.c:709 #6 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 27 (Thread 0x7f6f4e7fc700 (LWP 26460)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6f4e7fbe00) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6f4e7fbda8, cond=0x7f6f4e7fbdd8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x7f6f4e7fbdd8, mutex=0x7f6f4e7fbda8) at pthread_cond_wait.c:655 #3 0x0000557ba126327b in inline_mysql_cond_wait (that=0x7f6f4e7fbdd8, mutex=0x7f6f4e7fbda8, src_file=<optimized out>, src_line=332) at /mariadb/server/include/mysql/psi/mysql_thread.h:1149 #4 thd_destructor_proxy () at /mariadb/server/storage/innobase/handler/ha_innodb.cc:332 #5 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #6 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 26 (Thread 0x7f6f4effd700 (LWP 26459)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327b30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327ae0, cond=0x557ba4327b08) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba4327b08, mutex=0x557ba4327ae0) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327ad0) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba4327ad0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba13c9ee2 in buf_resize_thread () at /mariadb/server/storage/innobase/buf/buf0buf.cc:3102 #6 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 25 (Thread 0x7f6f4f7fe700 (LWP 26458)): #0 0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f4f7fd7a8, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557ba13c675a in btr_defragment_thread () at /mariadb/server/storage/innobase/btr/btr0defragment.cc:709 #3 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #4 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 24 (Thread 0x7f6f4ffff700 (LWP 26457)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327a50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327a00, cond=0x557ba4327a28) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba4327a28, mutex=0x557ba4327a00) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43279f0) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba43279f0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba13deb60 in buf_dump_thread () at /mariadb/server/storage/innobase/buf/buf0dump.cc:807 #6 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 23 (Thread 0x7f6f6c8e6700 (LWP 26456)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327900) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba43278b0, cond=0x557ba43278d8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba43278d8, mutex=0x557ba43278b0) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43278a0) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba43278a0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba6968 <srv_sys+360>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930 #6 srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580 #7 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 22 (Thread 0x7f6f6d0e7700 (LWP 26455)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327890) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327840, cond=0x557ba4327868) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba4327868, mutex=0x557ba4327840) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327830) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba4327830, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba6930 <srv_sys+304>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930 #6 srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580 #7 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 21 (Thread 0x7f6f6d8e8700 (LWP 26454)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba4327820) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba43277d0, cond=0x557ba43277f8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba43277f8, mutex=0x557ba43277d0) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba43277c0) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba43277c0, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba135af3c in srv_resume_thread (slot=0x557ba1ba68f8 <srv_sys+248>, sig_count=0, wait=255, timeout_usec=0) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930 #6 srv_worker_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2580 #7 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 20 (Thread 0x7f6f76690700 (LWP 26453)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba43277b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba4327760, cond=0x557ba4327788) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba4327788, mutex=0x557ba4327760) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba4327750) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba4327750, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba135b821 in srv_resume_thread (slot=<optimized out>, sig_count=<optimized out>, wait=<optimized out>, timeout_usec=10000) at /mariadb/server/storage/innobase/srv/srv0srv.cc:930 #6 srv_purge_coordinator_suspend (slot=<optimized out>, rseg_history_len=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2730 #7 srv_purge_coordinator_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2819 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 19 (Thread 0x7f6f76e91700 (LWP 26452)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f76e90bc0, expected=0, futex_word=0x557ba8306b50) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6f76e90bc0, mutex=0x557ba8306b00, cond=0x557ba8306b28) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x557ba8306b28, mutex=0x557ba8306b00, abstime=0x7f6f76e90bc0) at pthread_cond_wait.c:667 #3 0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f76e90bc0) at /mariadb/server/storage/innobase/os/os0event.cc:279 #4 0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba8306af0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:406 #5 0x0000557ba13995bf in ib_wqueue_timedwait (wq=0x557ba42f6f20, wait_in_usecs=5000000) at /mariadb/server/storage/innobase/ut/ut0wqueue.cc:161 #6 0x0000557ba14686c4 in fts_optimize_thread (arg=0x557ba42f6f20) at /mariadb/server/storage/innobase/fts/fts0opt.cc:2897 #7 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 18 (Thread 0x7f6f77692700 (LWP 26451)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f77691dc0, expected=0, futex_word=0x557ba82aa6f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6f77691dc0, mutex=0x557ba82aa6a0, cond=0x557ba82aa6c8) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x557ba82aa6c8, mutex=0x557ba82aa6a0, abstime=0x7f6f77691dc0) at pthread_cond_wait.c:667 #3 0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f77691dc0) at /mariadb/server/storage/innobase/os/os0event.cc:279 #4 0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba82aa690, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:406 #5 0x0000557ba1424089 in dict_stats_thread () at /mariadb/server/storage/innobase/dict/dict0stats_bg.cc:458 #6 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 17 (Thread 0x7f6f77e93700 (LWP 26450)): #0 0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f77e92c68, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557ba1359e4f in srv_master_sleep () at /mariadb/server/storage/innobase/srv/srv0srv.cc:2385 #3 srv_master_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:2426 #4 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #5 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 16 (Thread 0x7f6f78694700 (LWP 26449)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82ab4f0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82ab4a0, cond=0x557ba82ab4c8) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba82ab4c8, mutex=0x557ba82ab4a0) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82ab490) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba82ab490, reset_sig_count=1) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f6f78693ca8: 0x7f70522bb090) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469 #6 0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x557ba82aaa80, max_spins=30, max_delay=6, filename=0x557ba16a2e36 "/mariadb/server/storage/innobase/read/read0read.cc", line=714) at /mariadb/server/storage/innobase/include/ib0mutex.h:516 #7 0x0000557ba12f79f0 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x557ba82aaa80, n_spins=30, n_delay=6, name=<optimized out>, line=714) at /mariadb/server/storage/innobase/include/ib0mutex.h:637 #8 MVCC::size (this=0x557ba42e8940) at /mariadb/server/storage/innobase/read/read0read.cc:714 #9 0x0000557ba1356f86 in srv_printf_innodb_monitor (file=0x7f705b2ad680 <_IO_2_1_stderr_>, nowait=<optimized out>, trx_start_pos=<optimized out>, trx_end=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0srv.cc:1393 #10 0x0000557ba135901e in srv_monitor_thread () at /mariadb/server/storage/innobase/srv/srv0srv.cc:1752 #11 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #12 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 15 (Thread 0x7f6f6ed13700 (LWP 26448)): #0 0x00007f705bb9b9a5 in __GI___nanosleep (requested_time=0x7f6f6ed12a28, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x0000557ba12dd29f in os_thread_sleep (tm=<optimized out>) at /mariadb/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557ba1365d04 in sync_array_print_long_waits (waiter=<optimized out>, sema=<optimized out>) at /mariadb/server/storage/innobase/sync/sync0arr.cc:1104 #3 0x0000557ba13598c5 in srv_error_monitor_thread () at /mariadb/server/storage/innobase/srv/srv0srv.cc:1861 #4 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #5 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 14 (Thread 0x7f6f78e95700 (LWP 26447)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x557ba82a6f60) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x557ba82a6f10, cond=0x557ba82a6f38) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x557ba82a6f38, mutex=0x557ba82a6f10) at pthread_cond_wait.c:655 #3 0x0000557ba12dcbf6 in os_event::wait (this=0x557ba82a6f00) at /mariadb/server/storage/innobase/os/os0event.cc:162 #4 os_event::wait_low (this=0x557ba82a6f00, reset_sig_count=3) at /mariadb/server/storage/innobase/os/os0event.cc:329 #5 0x0000557ba13657f5 in sync_array_wait_event (arr=0x557ba431f250, cell=@0x7f6f78e94da8: 0x7f70522bb050) at /mariadb/server/storage/innobase/sync/sync0arr.cc:469 #6 0x0000557ba1283334 in TTASEventMutex<GenericPolicy>::enter (this=0x7f705003d050, max_spins=30, max_delay=6, filename=0x557ba169d35a "/mariadb/server/storage/innobase/lock/lock0wait.cc", line=501) at /mariadb/server/storage/innobase/include/ib0mutex.h:516 #7 0x0000557ba12bd649 in PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0x7f705003d050, n_spins=30, n_delay=6, name=<optimized out>, line=501) at /mariadb/server/storage/innobase/include/ib0mutex.h:637 #8 lock_wait_check_and_cancel (slot=0x7f705003d138) at /mariadb/server/storage/innobase/lock/lock0wait.cc:501 #9 lock_wait_timeout_thread () at /mariadb/server/storage/innobase/lock/lock0wait.cc:569 #10 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #11 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 13 (Thread 0x7f6f6f719700 (LWP 26444)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6f6f718b70, expected=0, futex_word=0x557ba4327ac4) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6f6f718b70, mutex=0x557ba4327a70, cond=0x557ba4327a98) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x557ba4327a98, mutex=0x557ba4327a70, abstime=0x7f6f6f718b70) at pthread_cond_wait.c:667 #3 0x0000557ba12dca5b in os_event::timed_wait (this=<optimized out>, abstime=0x7f6f6f718b70) at /mariadb/server/storage/innobase/os/os0event.cc:279 #4 0x0000557ba12dcd46 in os_event::wait_time_low (this=0x557ba4327a60, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/server/storage/innobase/os/os0event.cc:406 #5 0x0000557ba13e4047 in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/server/storage/innobase/buf/buf0flu.cc:2702 #6 buf_flush_page_cleaner_coordinator () at /mariadb/server/storage/innobase/buf/buf0flu.cc:3168 #7 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 12 (Thread 0x7f6f6ff1a700 (LWP 26443)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f6ff19b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f6ff19b60, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f6ff19c50, m2=0x7f6f6ff19c58, request=0x7f6f6ff19de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=9) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 11 (Thread 0x7f6f7071b700 (LWP 26442)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7071ab60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7071ab60, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f7071ac50, m2=0x7f6f7071ac58, request=0x7f6f7071ade0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=8) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 10 (Thread 0x7f6f70f1c700 (LWP 26441)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f70f1bb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f70f1bb60, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f70f1bc50, m2=0x7f6f70f1bc58, request=0x7f6f70f1bde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=7) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 9 (Thread 0x7f6f7171d700 (LWP 26440)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7171cb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7171cb60, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f7171cc50, m2=0x7f6f7171cc58, request=0x7f6f7171cde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=6) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 8 (Thread 0x7f6f71f1e700 (LWP 26439)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f71f1db60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f71f1db60, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f71f1dc50, m2=0x7f6f71f1dc58, request=0x7f6f71f1dde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=5) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 7 (Thread 0x7f6f7271f700 (LWP 26438)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f7271eb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f7271eb60, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f7271ec50, m2=0x7f6f7271ec58, request=0x7f6f7271ede0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=4) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 6 (Thread 0x7f6f72f20700 (LWP 26437)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f72f1fb60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f72f1fb60, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f72f1fc50, m2=0x7f6f72f1fc58, request=0x7f6f72f1fde0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=3) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 5 (Thread 0x7f6f73721700 (LWP 26436)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f73720b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f73720b60, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f73720c50, m2=0x7f6f73720c58, request=0x7f6f73720de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=2) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 4 (Thread 0x7f6f73f22700 (LWP 26435)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f73f21b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f73f21b60, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f73f21c50, m2=0x7f6f73f21c58, request=0x7f6f73f21de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=1) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 3 (Thread 0x7f6f74723700 (LWP 26434)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f705baf527a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000557ba12d3880 in LinuxAIOHandler::collect (this=0x7f6f74722b60) at /mariadb/server/storage/innobase/os/os0file.cc:1924 #3 0x0000557ba12d3dfc in LinuxAIOHandler::poll (this=0x7f6f74722b60, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:2069 #4 0x0000557ba12d82b9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:2123 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6f74722c50, m2=0x7f6f74722c58, request=0x7f6f74722de0) at /mariadb/server/storage/innobase/os/os0file.cc:5728 #6 0x0000557ba1437d2a in fil_aio_wait (segment=0) at /mariadb/server/storage/innobase/fil/fil0fil.cc:5211 #7 0x0000557ba135c988 in io_handler_thread (arg=<optimized out>) at /mariadb/server/storage/innobase/srv/srv0start.cc:333 #8 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 2 (Thread 0x7f705adff700 (LWP 26433)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f705adfee00, expected=0, futex_word=0x557ba24ae818 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f705adfee00, mutex=0x557ba24ae7c0 <LOCK_timer>, cond=0x557ba24ae7f0 <COND_timer>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x557ba24ae7f0 <COND_timer>, mutex=0x557ba24ae7c0 <LOCK_timer>, abstime=0x7f705adfee00) at pthread_cond_wait.c:667 #3 0x0000557ba15a5ed6 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=0x7f705adfee00, src_file=<optimized out>, src_line=292) at /mariadb/server/include/mysql/psi/mysql_thread.h:1186 #4 timer_handler (arg=<optimized out>) at /mariadb/server/mysys/thr_timer.c:292 #5 0x00007f705bb91fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486 #6 0x00007f705b1ed2cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95   Thread 1 (Thread 0x7f705afa5340 (LWP 26432)): #0 0x00007f705b1e2d0f in __GI___poll (fds=0x7ffc43d161a0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x0000557ba0ec8e39 in handle_connections_sockets () at /mariadb/server/sql/mysqld.cc:6619 #2 0x0000557ba0ec4797 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/server/sql/mysqld.cc:6085 #3 0x00007f705b119bbb in __libc_start_main (main=0x557ba0ebf760 <main(int, char**)>, argc=5, argv=0x7ffc43d168a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc43d16898) at ../csu/libc-start.c:308 #4 0x0000557ba0ebf69a in _start () The threads that participate in the deadlock are as follows, using the identifiers introduced in my previous comment: Thread A = Thread 31 Thread B = Thread 33 Thread C = Thread 32

            People

              marko Marko Mäkelä
              CrewOne Sander Pilon
              Votes:
              6 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.