Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34135

Replication slave is stuck without any error

Details

    Description

      I have an issue very similar to MDEV-8311. As it was never solved due to lack of activity, and it is almost 10 years old, I took the liberty to open a new issue.

      My replica stopped replicating, and it is stuck.
      One slave worker is stuck with the status "Waiting for prior transaction to start commit".
      The slave itself is stuck with the status "Waiting for room in worker thread event queue".

      I cannot "stop slave;" as it is also stuck.
      I can manually kill the slave PIDs. But then if I "start slave", it is stuck directly.
      After manually killing the slave PIDs, I can do:

      SET GLOBAL sql_slave_skip_counter = 1;
      start slave;
      

      Then it starts properly ("Write_rows_log_event::write_row(-1) on table `my_table`"), and get stuck again a few minutes later.

      Please note that I have 2 other replicas running on the same master with the exact same config.
      Please also note that those 3 replicas have been running like that for months/years.
      Potentially interesting context: most of my tables are using RocksDB as an engine. The rest use InnoDB.


      mariadb --version
      mariadb  Ver 15.1 Distrib 10.6.16-MariaDB, for debian-linux-gnu (x86_64) using  EditLine wrapper
      

      Client output when started:

      Server version: 10.6.16-MariaDB-0ubuntu0.22.04.1 Ubuntu 22.04
      

      MariaDB [my_db]> show slave status\G
      *************************** 1. row ***************************
                      Slave_IO_State: Waiting for master to send event
                         Master_Host: 192.168.1.154
                         Master_User: my_user
                         Master_Port: 3306
                       Connect_Retry: 60
                     Master_Log_File: master10-bin.000291
                 Read_Master_Log_Pos: 107402253
                      Relay_Log_File: mysqld-relay-bin.000552
                       Relay_Log_Pos: 12124351
               Relay_Master_Log_File: master10-bin.000265
                    Slave_IO_Running: Yes
                   Slave_SQL_Running: Yes
                     Replicate_Do_DB: 
                 Replicate_Ignore_DB: 
                  Replicate_Do_Table: 
              Replicate_Ignore_Table: 
             Replicate_Wild_Do_Table: 
         Replicate_Wild_Ignore_Table: 
                          Last_Errno: 0
                          Last_Error: 
                        Skip_Counter: 0
                 Exec_Master_Log_Pos: 417796162
                     Relay_Log_Space: 10110622341
                     Until_Condition: None
                      Until_Log_File: 
                       Until_Log_Pos: 0
                  Master_SSL_Allowed: No
                  Master_SSL_CA_File: 
                  Master_SSL_CA_Path: 
                     Master_SSL_Cert: 
                   Master_SSL_Cipher: 
                      Master_SSL_Key: 
               Seconds_Behind_Master: 38815
       Master_SSL_Verify_Server_Cert: No
                       Last_IO_Errno: 0
                       Last_IO_Error: 
                      Last_SQL_Errno: 0
                      Last_SQL_Error: 
         Replicate_Ignore_Server_Ids: 
                    Master_Server_Id: 10
                      Master_SSL_Crl: 
                  Master_SSL_Crlpath: 
                          Using_Gtid: No
                         Gtid_IO_Pos: 
             Replicate_Do_Domain_Ids: 
         Replicate_Ignore_Domain_Ids: 
                       Parallel_Mode: conservative
                           SQL_Delay: 0
                 SQL_Remaining_Delay: NULL
             Slave_SQL_Running_State: Waiting for room in worker thread event queue
                    Slave_DDL_Groups: 0
      Slave_Non_Transactional_Groups: 0
          Slave_Transactional_Groups: 464
      1 row in set (0.025 sec)
      

      MariaDB [my_db]> show processlist;
      +------+-------------+---------------------------------------------------+--------------+--------------+------+-----------------------------------------------+------------------------------------------------------------------------------------------------------+----------+
      | Id   | User        | Host                                              | db           | Command      | Time | State                                         | Info                                                                                                 | Progress |
      +------+-------------+---------------------------------------------------+--------------+--------------+------+-----------------------------------------------+------------------------------------------------------------------------------------------------------+----------+
      | 1544 | system user |                                                   | NULL         | Slave_IO     | 1263 | Waiting for master to send event              | NULL                                                                                                 |    0.000 |
      | 1548 | system user |                                                   | NULL         | Slave_worker | 1187 | Waiting for work from SQL thread              | NULL                                                                                                 |    0.000 |
      | 1546 | system user |                                                   | NULL         | Slave_worker | 1187 | Waiting for prior transaction to start commit | NULL                                                                                                 |    0.000 |
      | 1549 | system user |                                                   | NULL         | Slave_worker | 1187 | Waiting for work from SQL thread              | NULL                                                                                                 |    0.000 |
      | 1547 | system user |                                                   | NULL         | Slave_worker | 1187 | Waiting for work from SQL thread              | NULL                                                                                                 |    0.000 |
      | 1550 | system user |                                                   | NULL         | Slave_worker | 1187 | Waiting for work from SQL thread              | NULL                                                                                                 |    0.000 |
      | 1545 | system user |                                                   | NULL         | Slave_SQL    | 1187 | Waiting for room in worker thread event queue | NULL                                                                                                 |    0.000 |
      [...] Normal queries [...]
      +------+-------------+---------------------------------------------------+--------------+--------------+------+-----------------------------------------------+------------------------------------------------------------------------------------------------------+----------+
      10 rows in set (0.024 sec)
      

      Attachments

        Issue Links

          Activity

            Hi @Kristian,

            It's back. I have left it untouched, haven't tried to restart mariadb or anything.
            I can leave it like this for a couple of days, but at some point, I'll need to make it work somehow.

            Here is the full processlist:

            MariaDB [redacted]> show full processlist;
            +-------+-------------+---------------------------------------------------+-----------------+--------------+-------+-----------------------------------------------+-----------------------+----------+
            | Id    | User        | Host                                              | db              | Command      | Time  | State                                         | Info                  | Progress |
            +-------+-------------+---------------------------------------------------+-----------------+--------------+-------+-----------------------------------------------+-----------------------+----------+
            |     4 | system user |                                                   | NULL            | Slave_IO     | 72097 | Waiting for master to send event              | NULL                  |    0.000 |
            |     7 | system user |                                                   | NULL            | Slave_worker | 72023 | Waiting for work from SQL thread              | NULL                  |    0.000 |
            |     6 | system user |                                                   | NULL            | Slave_worker | 72023 | Waiting for work from SQL thread              | NULL                  |    0.000 |
            |     8 | system user |                                                   | NULL            | Slave_worker | 72023 | Waiting for prior transaction to start commit | NULL                  |    0.000 |
            |     9 | system user |                                                   | NULL            | Slave_worker | 72023 | Waiting for work from SQL thread              | NULL                  |    0.000 |
            |    10 | system user |                                                   | NULL            | Slave_worker | 72023 | Waiting for work from SQL thread              | NULL                  |    0.000 |
            |     5 | system user |                                                   | NULL            | Slave_SQL    | 72023 | Waiting for room in worker thread event queue | NULL                  |    0.000 |
            | 17806 | redacted    | redacted_host                                     | redacted_db     | Sleep        |   551 |                                               | NULL                  |    0.000 | 
            | 17981 | root        | localhost                                         | redacted_db     | Query        |     0 | starting                                      | show full processlist |    0.000 |
            +-------+-------------+---------------------------------------------------+-----------------+--------------+-------+-----------------------------------------------+-----------------------+----------+
            9 rows in set (0.000 sec)
            

            The slave status:

            MariaDB [redacted]> show slave status\G
            *************************** 1. row ***************************
                            Slave_IO_State: Waiting for master to send event
                               Master_Host: 192.168.1.154
                               Master_User: repl
                               Master_Port: 3306
                             Connect_Retry: 60
                           Master_Log_File: master10-bin.006016
                       Read_Master_Log_Pos: 28632
                            Relay_Log_File: mysqld-relay-bin.010391
                             Relay_Log_Pos: 294543565
                     Relay_Master_Log_File: master10-bin.005935
                          Slave_IO_Running: Yes
                         Slave_SQL_Running: Yes
                           Replicate_Do_DB: 
                       Replicate_Ignore_DB: 
                        Replicate_Do_Table: 
                    Replicate_Ignore_Table: 
                   Replicate_Wild_Do_Table: 
               Replicate_Wild_Ignore_Table: 
                                Last_Errno: 0
                                Last_Error: 
                              Skip_Counter: 0
                       Exec_Master_Log_Pos: 750962706
                           Relay_Log_Space: 34052988805
                           Until_Condition: None
                            Until_Log_File: 
                             Until_Log_Pos: 0
                        Master_SSL_Allowed: No
                        Master_SSL_CA_File: 
                        Master_SSL_CA_Path: 
                           Master_SSL_Cert: 
                         Master_SSL_Cipher: 
                            Master_SSL_Key: 
                     Seconds_Behind_Master: 71965
             Master_SSL_Verify_Server_Cert: No
                             Last_IO_Errno: 0
                             Last_IO_Error: 
                            Last_SQL_Errno: 0
                            Last_SQL_Error: 
               Replicate_Ignore_Server_Ids: 
                          Master_Server_Id: 10
                            Master_SSL_Crl: 
                        Master_SSL_Crlpath: 
                                Using_Gtid: No
                               Gtid_IO_Pos: 
                   Replicate_Do_Domain_Ids: 
               Replicate_Ignore_Domain_Ids: 
                             Parallel_Mode: conservative
                                 SQL_Delay: 0
                       SQL_Remaining_Delay: NULL
                   Slave_SQL_Running_State: Waiting for room in worker thread event queue
                          Slave_DDL_Groups: 0
            Slave_Non_Transactional_Groups: 0
                Slave_Transactional_Groups: 78
            1 row in set (0.000 sec)
            

            And the ```gdb``` ```thread apply all bt``` :

            (gdb) thread apply all bt
             
            Thread 46 (Thread 0x7f1fc67fc640 (LWP 3190450) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=537376365, cancel=true, abstime=0x7f1fc67fbc30, op=137, expected=0, futex_word=0x5586034980ec) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=537376365, abstime=0x7f1fc67fbc30, clockid=0, expected=0, futex_word=0x5586034980ec) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586034980ec, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f1fc67fbc30, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f202002a2dd in __pthread_cond_wait_common (abstime=0x7f1fc67fbc30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034980c0) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_clockwait64 (abstime=0x7f1fc67fbc30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034980c0) at ./nptl/pthread_cond_wait.c:691
            #5  ___pthread_cond_clockwait64 (cond=0x5586034980c0, mutex=0x5586011e0bf0, clockid=1, abstime=0x7f1fc67fbc30) at ./nptl/pthread_cond_wait.c:679
            #6  0x00005585ffb76535 in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) ()
            #7  0x00005585ffb76de8 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) ()
            #8  0x00005585ffb76f5d in tpool::thread_pool_generic::worker_main(tpool::worker_data*) ()
            #9  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 45 (Thread 0x7f200171d640 (LWP 3189008) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=832010992, cancel=true, abstime=0x7f200171cc80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=832010992, abstime=0x7f200171cc80, clockid=832003512, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200171cc80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200171cc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200171cc80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 44 (Thread 0x7f20010f6640 (LWP 3189007) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-654408672, cancel=true, abstime=0x7f20010f5c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-654408672, abstime=0x7f20010f5c80, clockid=-654416152, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20010f5c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20010f5c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20010f5c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 43 (Thread 0x7f20103d2640 (LWP 3189005) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-128790480, cancel=true, abstime=0x7f20103d1c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-128790480, abstime=0x7f20103d1c80, clockid=-128797960, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20103d1c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20103d1c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20103d1c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 42 (Thread 0x7f201017a640 (LWP 3189004) "mariadbd"):
            #0  0x00007f20200aebcf in __GI___poll (fds=0x7f2010179918, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
            #1  0x00005585ff8b0e7f in vio_io_wait ()
            #2  0x00005585ff8b0f54 in vio_socket_io_wait ()
            #3  0x00005585ff8b1013 in vio_read ()
            #4  0x00005585ff840f6e in ?? ()
            #5  0x00005585ff8421f0 in my_net_read_packet_reallen ()
            #6  0x00005585ff842301 in my_net_read_packet ()
            #7  0x00005585ff4d7812 in do_command(THD*, bool) ()
            #8  0x00005585ff5e210f in do_handle_one_connection(CONNECT*, bool) ()
            #9  0x00005585ff5e248d in handle_one_connection ()
            #10 0x00005585ff9443c6 in ?? ()
            #11 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #12 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 41 (Thread 0x7f200155b640 (LWP 3189003) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f200155ac80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f200155ac80, clockid=-592399872, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200155ac80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200155ac80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200155ac80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 40 (Thread 0x7f20102f1640 (LWP 3189002) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=402662960, cancel=true, abstime=0x7f20102f0c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=402662960, abstime=0x7f20102f0c80, clockid=402655480, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20102f0c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20102f0c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20102f0c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 39 (Thread 0x7f20013e4640 (LWP 3189001) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f20013e3c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f20013e3c80, clockid=-592399872, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20013e3c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20013e3c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20013e3c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 38 (Thread 0x7f20015a6640 (LWP 3189000) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-1472854160, cancel=true, abstime=0x7f20015a5c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-1472854160, abstime=0x7f20015a5c80, clockid=-1472861640, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20015a5c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20015a5c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20015a5c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 37 (Thread 0x7f20104b3640 (LWP 3188999) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f20104b2c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f20104b2c80, clockid=-592399872, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20104b2c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20104b2c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20104b2c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 36 (Thread 0x7f20101c5640 (LWP 3188981) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-1159479664, cancel=true, abstime=0x7f20101c4c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-1159479664, abstime=0x7f20101c4c80, clockid=-1159487144, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20101c4c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20101c4c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20101c4c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 35 (Thread 0x7f20011d7640 (LWP 3188980) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=1298423536, cancel=true, abstime=0x7f20011d6c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=1298423536, abstime=0x7f20011d6c80, clockid=1298416056, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20011d6c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20011d6c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20011d6c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 34 (Thread 0x7f200142f640 (LWP 3188977) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f200142ec80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f200142ec80, clockid=-592399872, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200142ec80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200142ec80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200142ec80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            --Type <RET> for more, q to quit, c to continue without paging-
            

            Vongo Adrian Delmarre added a comment - Hi @Kristian, It's back. I have left it untouched, haven't tried to restart mariadb or anything. I can leave it like this for a couple of days, but at some point, I'll need to make it work somehow. Here is the full processlist: MariaDB [redacted]> show full processlist; +-------+-------------+---------------------------------------------------+-----------------+--------------+-------+-----------------------------------------------+-----------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +-------+-------------+---------------------------------------------------+-----------------+--------------+-------+-----------------------------------------------+-----------------------+----------+ | 4 | system user | | NULL | Slave_IO | 72097 | Waiting for master to send event | NULL | 0.000 | | 7 | system user | | NULL | Slave_worker | 72023 | Waiting for work from SQL thread | NULL | 0.000 | | 6 | system user | | NULL | Slave_worker | 72023 | Waiting for work from SQL thread | NULL | 0.000 | | 8 | system user | | NULL | Slave_worker | 72023 | Waiting for prior transaction to start commit | NULL | 0.000 | | 9 | system user | | NULL | Slave_worker | 72023 | Waiting for work from SQL thread | NULL | 0.000 | | 10 | system user | | NULL | Slave_worker | 72023 | Waiting for work from SQL thread | NULL | 0.000 | | 5 | system user | | NULL | Slave_SQL | 72023 | Waiting for room in worker thread event queue | NULL | 0.000 | | 17806 | redacted | redacted_host | redacted_db | Sleep | 551 | | NULL | 0.000 | | 17981 | root | localhost | redacted_db | Query | 0 | starting | show full processlist | 0.000 | +-------+-------------+---------------------------------------------------+-----------------+--------------+-------+-----------------------------------------------+-----------------------+----------+ 9 rows in set (0.000 sec) The slave status: MariaDB [redacted]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.1.154 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master10-bin.006016 Read_Master_Log_Pos: 28632 Relay_Log_File: mysqld-relay-bin.010391 Relay_Log_Pos: 294543565 Relay_Master_Log_File: master10-bin.005935 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 750962706 Relay_Log_Space: 34052988805 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 71965 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 10 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: No Gtid_IO_Pos: Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Waiting for room in worker thread event queue Slave_DDL_Groups: 0 Slave_Non_Transactional_Groups: 0 Slave_Transactional_Groups: 78 1 row in set (0.000 sec) And the ```gdb``` ```thread apply all bt``` : (gdb) thread apply all bt   Thread 46 (Thread 0x7f1fc67fc640 (LWP 3190450) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=537376365, cancel=true, abstime=0x7f1fc67fbc30, op=137, expected=0, futex_word=0x5586034980ec) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=537376365, abstime=0x7f1fc67fbc30, clockid=0, expected=0, futex_word=0x5586034980ec) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586034980ec, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f1fc67fbc30, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f202002a2dd in __pthread_cond_wait_common (abstime=0x7f1fc67fbc30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034980c0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_clockwait64 (abstime=0x7f1fc67fbc30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034980c0) at ./nptl/pthread_cond_wait.c:691 #5 ___pthread_cond_clockwait64 (cond=0x5586034980c0, mutex=0x5586011e0bf0, clockid=1, abstime=0x7f1fc67fbc30) at ./nptl/pthread_cond_wait.c:679 #6 0x00005585ffb76535 in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) () #7 0x00005585ffb76de8 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) () #8 0x00005585ffb76f5d in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () #9 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 45 (Thread 0x7f200171d640 (LWP 3189008) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=832010992, cancel=true, abstime=0x7f200171cc80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=832010992, abstime=0x7f200171cc80, clockid=832003512, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200171cc80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200171cc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200171cc80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 44 (Thread 0x7f20010f6640 (LWP 3189007) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-654408672, cancel=true, abstime=0x7f20010f5c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-654408672, abstime=0x7f20010f5c80, clockid=-654416152, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20010f5c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20010f5c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20010f5c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 43 (Thread 0x7f20103d2640 (LWP 3189005) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-128790480, cancel=true, abstime=0x7f20103d1c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-128790480, abstime=0x7f20103d1c80, clockid=-128797960, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20103d1c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20103d1c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20103d1c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 42 (Thread 0x7f201017a640 (LWP 3189004) "mariadbd"): #0 0x00007f20200aebcf in __GI___poll (fds=0x7f2010179918, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00005585ff8b0e7f in vio_io_wait () #2 0x00005585ff8b0f54 in vio_socket_io_wait () #3 0x00005585ff8b1013 in vio_read () #4 0x00005585ff840f6e in ?? () #5 0x00005585ff8421f0 in my_net_read_packet_reallen () #6 0x00005585ff842301 in my_net_read_packet () #7 0x00005585ff4d7812 in do_command(THD*, bool) () #8 0x00005585ff5e210f in do_handle_one_connection(CONNECT*, bool) () #9 0x00005585ff5e248d in handle_one_connection () #10 0x00005585ff9443c6 in ?? () #11 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #12 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 41 (Thread 0x7f200155b640 (LWP 3189003) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f200155ac80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f200155ac80, clockid=-592399872, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200155ac80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200155ac80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200155ac80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 40 (Thread 0x7f20102f1640 (LWP 3189002) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=402662960, cancel=true, abstime=0x7f20102f0c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=402662960, abstime=0x7f20102f0c80, clockid=402655480, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20102f0c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20102f0c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20102f0c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 39 (Thread 0x7f20013e4640 (LWP 3189001) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f20013e3c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f20013e3c80, clockid=-592399872, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20013e3c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20013e3c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20013e3c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 38 (Thread 0x7f20015a6640 (LWP 3189000) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-1472854160, cancel=true, abstime=0x7f20015a5c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1472854160, abstime=0x7f20015a5c80, clockid=-1472861640, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20015a5c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20015a5c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20015a5c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 37 (Thread 0x7f20104b3640 (LWP 3188999) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f20104b2c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f20104b2c80, clockid=-592399872, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20104b2c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20104b2c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20104b2c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 36 (Thread 0x7f20101c5640 (LWP 3188981) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-1159479664, cancel=true, abstime=0x7f20101c4c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1159479664, abstime=0x7f20101c4c80, clockid=-1159487144, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20101c4c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20101c4c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20101c4c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 35 (Thread 0x7f20011d7640 (LWP 3188980) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=1298423536, cancel=true, abstime=0x7f20011d6c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=1298423536, abstime=0x7f20011d6c80, clockid=1298416056, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20011d6c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20011d6c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20011d6c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 34 (Thread 0x7f200142f640 (LWP 3188977) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f200142ec80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f200142ec80, clockid=-592399872, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200142ec80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200142ec80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200142ec80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () --Type <RET> for more, q to quit, c to continue without paging-

            Vongo, thanks a lot for catching the problem again and supplying the additional information!

            Unfortunately, the output from gdb is not complete, by default gdb shows only one page of output and is waiting for the user to press enter to continue. So threads 1-33 are missing, including the 5 interesting threads that contain the function handle_rpl_parallel_thread() in the stack trace. Can you obtain the full output from gdb containing all 46 threads?

            We see in the processlist that one thread is stuck in "Waiting for prior transaction to start commit". This is definitely a bug, since the other 4 threads are idle. There have unfortunately been a number of bugs in this particular area, hopefully I will be able to track down and fix this one.

            It you have problems in your production due to this bug, one work-around may be to run in optimistic parallel replication mode instead of conservative; this particular situation should be much less likely to occur in optimistic mode. Just STOP SLAVE (might need to kill the waiting thread that is stuck, not sure) and START SLAVE should break the hang and allow replication to continue, when you experience it in conservative mode. Of course, the bug still needs to be fixed, just suggested work-arounds in case it helps.

            - Kristian.

            knielsen Kristian Nielsen added a comment - Vongo , thanks a lot for catching the problem again and supplying the additional information! Unfortunately, the output from gdb is not complete, by default gdb shows only one page of output and is waiting for the user to press enter to continue. So threads 1-33 are missing, including the 5 interesting threads that contain the function handle_rpl_parallel_thread() in the stack trace. Can you obtain the full output from gdb containing all 46 threads? We see in the processlist that one thread is stuck in "Waiting for prior transaction to start commit". This is definitely a bug, since the other 4 threads are idle. There have unfortunately been a number of bugs in this particular area, hopefully I will be able to track down and fix this one. It you have problems in your production due to this bug, one work-around may be to run in optimistic parallel replication mode instead of conservative; this particular situation should be much less likely to occur in optimistic mode. Just STOP SLAVE (might need to kill the waiting thread that is stuck, not sure) and START SLAVE should break the hang and allow replication to continue, when you experience it in conservative mode. Of course, the bug still needs to be fixed, just suggested work-arounds in case it helps. - Kristian.

            Hi @Kristian Nielsen, thanks for your answer, and sorry for the incomplete output.

            Here is the complete output :

            Thread 31 (Thread 0x7f2001303640 (LWP 3300512) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-1811929552, cancel=true, abstime=0x7f2001302c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-1811929552, abstime=0x7f2001302c80, clockid=-1811937032, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2001302c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2001302c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2001302c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 30 (Thread 0x7f2001060640 (LWP 3300511) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=536880688, cancel=true, abstime=0x7f200105fc80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=536880688, abstime=0x7f200105fc80, clockid=536873208, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200105fc80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200105fc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200105fc80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 29 (Thread 0x7f2001399640 (LWP 3300505) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-2132796320, cancel=true, abstime=0x7f2001398c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-2132796320, abstime=0x7f2001398c80, clockid=-2132803800, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2001398c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2001398c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2001398c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 28 (Thread 0x7f20104fe640 (LWP 3300503) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-1924900784, cancel=true, abstime=0x7f20104fdc80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-1924900784, abstime=0x7f20104fdc80, clockid=-1924908264, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20104fdc80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20104fdc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20104fdc80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 27 (Thread 0x7f2010594640 (LWP 3300498) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=2013275696, cancel=true, abstime=0x7f2010593c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=2013275696, abstime=0x7f2010593c80, clockid=2013268216, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2010593c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2010593c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2010593c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 26 (Thread 0x7f20102f1640 (LWP 3300494) "mariadbd"):
            #0  0x00007f20200aebcf in __GI___poll (fds=0x7f20102f0918, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
            #1  0x00005585ff8b0e7f in vio_io_wait ()
            #2  0x00005585ff8b0f54 in vio_socket_io_wait ()
            #3  0x00005585ff8b1013 in vio_read ()
            #4  0x00005585ff840f6e in ?? ()
            #5  0x00005585ff8421f0 in my_net_read_packet_reallen ()
            #6  0x00005585ff842301 in my_net_read_packet ()
            #7  0x00005585ff4d7812 in do_command(THD*, bool) ()
            #8  0x00005585ff5e210f in do_handle_one_connection(CONNECT*, bool) ()
            #9  0x00005585ff5e248d in handle_one_connection ()
            #10 0x00005585ff9443c6 in ?? ()
            #11 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #12 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 25 (Thread 0x7f200126d640 (LWP 3299572) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=1207969328, cancel=true, abstime=0x7f200126cc80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=1207969328, abstime=0x7f200126cc80, clockid=1207961848, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200126cc80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200126cc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200126cc80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 24 (Thread 0x7f2001687640 (LWP 3299571) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=1422390464, cancel=true, abstime=0x7f2001686c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=1422390464, abstime=0x7f2001686c80, clockid=1422382984, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2001686c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2001686c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2001686c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 23 (Thread 0x7f20014c5640 (LWP 3291154) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=149265072, cancel=true, abstime=0x7f20014c4c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=149265072, abstime=0x7f20014c4c80, clockid=149257592, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20014c4c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20014c4c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20014c4c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) ()
            #6  0x00005585ff5e248d in handle_one_connection ()
            #7  0x00005585ff9443c6 in ?? ()
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 22 (Thread 0x7f1ff0ff9640 (LWP 3252392) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=537376365, cancel=true, abstime=0x7f1ff0ff8c30, op=137, expected=0, futex_word=0x5586034985e8) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=537376365, abstime=0x7f1ff0ff8c30, clockid=0, expected=0, futex_word=0x5586034985e8) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586034985e8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f1ff0ff8c30, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f202002a2dd in __pthread_cond_wait_common (abstime=0x7f1ff0ff8c30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034985c0) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_clockwait64 (abstime=0x7f1ff0ff8c30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034985c0) at ./nptl/pthread_cond_wait.c:691
            #5  ___pthread_cond_clockwait64 (cond=0x5586034985c0, mutex=0x5586011e0bf0, clockid=1, abstime=0x7f1ff0ff8c30) at ./nptl/pthread_cond_wait.c:679
            #6  0x00005585ffb76535 in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) ()
            #7  0x00005585ffb76de8 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) ()
            #8  0x00005585ffb76f5d in tpool::thread_pool_generic::worker_main(tpool::worker_data*) ()
            #9  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 21 (Thread 0x7f20105df640 (LWP 3067940) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac00a7b0) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac00a7b0) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac00a7b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac00a758, cond=0x7f1fac00a788) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x7f1fac00a788, mutex=0x7f1fac00a758) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff646c96 in handle_rpl_parallel_thread ()
            #6  0x00005585ff9443c6 in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 20 (Thread 0x7f201062a640 (LWP 3067939) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac00a12c) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac00a12c) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac00a12c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac00a0d0, cond=0x7f1fac00a100) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x7f1fac00a100, mutex=0x7f1fac00a0d0) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff646c96 in handle_rpl_parallel_thread ()
            #6  0x00005585ff9443c6 in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 19 (Thread 0x7f2010675640 (LWP 3067938) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fad07e4e0) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fad07e4e0) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fad07e4e0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac01e618, cond=0x7f1fad07e4b8) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x7f1fad07e4b8, mutex=0x7f1fac01e618) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff6477be in handle_rpl_parallel_thread ()
            #6  0x00005585ff9443c6 in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 18 (Thread 0x7f20106c0640 (LWP 3067937) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac00941c) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac00941c) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac00941c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac0093c0, cond=0x7f1fac0093f0) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x7f1fac0093f0, mutex=0x7f1fac0093c0) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff646c96 in handle_rpl_parallel_thread ()
            #6  0x00005585ff9443c6 in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 17 (Thread 0x7f201070b640 (LWP 3067936) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac008d94) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac008d94) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac008d94, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac008d38, cond=0x7f1fac008d68) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x7f1fac008d68, mutex=0x7f1fac008d38) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff646c96 in handle_rpl_parallel_thread ()
            #6  0x00005585ff9443c6 in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 16 (Thread 0x7f2010756640 (LWP 3067935) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac009adc) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac009adc) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac009adc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac009a48, cond=0x7f1fac009ab0) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x7f1fac009ab0, mutex=0x7f1fac009a48) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff64903f in rpl_parallel_entry::choose_thread(rpl_group_info*, bool*, PSI_stage_info_v1*, Gtid_log_event*) ()
            #6  0x00005585ff64a1ae in rpl_parallel::do_event(rpl_group_info*, Log_event*, unsigned long long) ()
            #7  0x00005585ff4237e5 in handle_slave_sql ()
            #8  0x00005585ff9443c6 in ?? ()
            #9  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #10 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 15 (Thread 0x7f201c07f640 (LWP 3067934) "mariadbd"):
            #0  0x00007f20200aebcf in __GI___poll (fds=0x7f201c07e698, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29
            #1  0x00005585ff8b0e7f in vio_io_wait ()
            #2  0x00005585ff8b0f54 in vio_socket_io_wait ()
            #3  0x00005585ff8b1013 in vio_read ()
            #4  0x00005585ff8b113a in vio_read_buff ()
            #5  0x00005585ff840f6e in ?? ()
            #6  0x00005585ff8421f0 in my_net_read_packet_reallen ()
            #7  0x00005585ff6d7ae1 in cli_safe_read_reallen ()
            #8  0x00005585ff4287f2 in handle_slave_io ()
            #9  0x00005585ff9443c6 in ?? ()
            #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 14 (Thread 0x7f201c0ca640 (LWP 3067933) "mariadbd"):
            #0  0x00007f201ffd921a in __GI___sigtimedwait (set=0x7f201c0c9af0, info=0x7f201c0c9bf0, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61
            #1  0x00005585ff3ff213 in signal_hand ()
            #2  0x00005585ff9443c6 in ?? ()
            #3  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #4  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 13 (Thread 0x7f201c115640 (LWP 3067932) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558600495dac <COND_manager+44>) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x558600495dac <COND_manager+44>) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600495dac <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558600495dc0 <LOCK_manager>, cond=0x558600495d80 <COND_manager>) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x558600495d80 <COND_manager>, mutex=0x558600495dc0 <LOCK_manager>) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ff4c3d00 in handle_manager ()
            #6  0x00005585ff9443c6 in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 12 (Thread 0x7f2001fff640 (LWP 3067925) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558600410524) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x558600410524) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600410524, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558600410480, cond=0x5586004104f8) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x5586004104f8, mutex=0x558600410480) at ./nptl/pthread_cond_wait.c:627
            #5  0x00005585ffb04dde in ?? ()
            #6  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 11 (Thread 0x7f2010f57640 (LWP 3067924) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=-1, cancel=true, abstime=0x7f2010f56c60, op=393, expected=0, futex_word=0x558600cc8448) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-1, abstime=0x7f2010f56c60, clockid=4, expected=0, futex_word=0x558600cc8448) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600cc8448, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2010f56c60, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2010f56c60, clockid=0, mutex=0x558600cc8460, cond=0x558600cc8420) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600cc8420, mutex=0x558600cc8460, abstime=0x7f2010f56c60) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ff8e19c8 in ?? ()
            #6  0x00005585ff8da37e in ?? ()
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 10 (Thread 0x7f2011758640 (LWP 3067923) "myrocks-mc"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f2011757c80, op=393, expected=0, futex_word=0x7f201e3f4ab0) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f2011757c80, clockid=0, expected=0, futex_word=0x7f201e3f4ab0) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f201e3f4ab0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2011757c80, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2011757c80, clockid=0, mutex=0x7f201e3f4a58, cond=0x7f201e3f4a88) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x7f201e3f4a88, mutex=0x7f201e3f4a58, abstime=0x7f2011757c80) at ./nptl/pthread_cond_wait.c:652
            #5  0x00007f201df94279 in myrocks::Rdb_manual_compaction_thread::run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201dfebb60 in myrocks::Rdb_thread::thread_func(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 9 (Thread 0x7f1ff2ffd640 (LWP 3067922) "myrocks-index"):
            #0  __futex_abstimed_wait_common64 (private=-592399872, cancel=true, abstime=0x7f1ff2ffcb50, op=393, expected=0, futex_word=0x7f201e3f47b4) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-592399872, abstime=0x7f1ff2ffcb50, clockid=-592399872, expected=0, futex_word=0x7f201e3f47b4) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f201e3f47b4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff2ffcb50, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff2ffcb50, clockid=0, mutex=0x7f201e3f4758, cond=0x7f201e3f4788) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x7f201e3f4788, mutex=0x7f201e3f4758, abstime=0x7f1ff2ffcb50) at ./nptl/pthread_cond_wait.c:652
            #5  0x00007f201dfa8ca3 in myrocks::Rdb_drop_index_thread::run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201dfebb60 in myrocks::Rdb_thread::thread_func(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 8 (Thread 0x7f1ff27fc640 (LWP 3067921) "myrocks-bg"):
            #0  __futex_abstimed_wait_common64 (private=537113683, cancel=true, abstime=0x7f1ff27fbc40, op=393, expected=0, futex_word=0x7f201e3f4bd0) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=537113683, abstime=0x7f1ff27fbc40, clockid=503428599, expected=0, futex_word=0x7f201e3f4bd0) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f201e3f4bd0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff27fbc40, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff27fbc40, clockid=0, mutex=0x7f201e3f4b78, cond=0x7f201e3f4ba8) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x7f201e3f4ba8, mutex=0x7f201e3f4b78, abstime=0x7f1ff27fbc40) at ./nptl/pthread_cond_wait.c:652
            #5  0x00007f201dfaa764 in myrocks::Rdb_background_thread::run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201dfebb60 in myrocks::Rdb_thread::thread_func(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #8  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 7 (Thread 0x7f1ff1ffb640 (LWP 3067920) "rocksdb:pst_st"):
            #0  __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f1ff1ffab90, op=393, expected=0, futex_word=0x55860136db38) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f1ff1ffab90, clockid=537113683, expected=0, futex_word=0x55860136db38) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860136db38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff1ffab90, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff1ffab90, clockid=0, mutex=0x55860136dac8, cond=0x55860136db10) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x55860136db10, mutex=0x55860136dac8, abstime=0x7f1ff1ffab90) at ./nptl/pthread_cond_wait.c:652
            #5  0x00007f201e245092 in rocksdb::port::CondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201e14db19 in rocksdb::InstrumentedCondVar::TimedWaitInternal(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f201e14dc06 in rocksdb::InstrumentedCondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #8  0x00007f201e0224db in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #9  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 6 (Thread 0x7f1ff17fa640 (LWP 3067919) "rocksdb:dump_st"):
            #0  __futex_abstimed_wait_common64 (private=1952672118, cancel=true, abstime=0x7f1ff17f9b90, op=393, expected=0, futex_word=0x558601147b08) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=1952672118, abstime=0x7f1ff17f9b90, clockid=-243295576, expected=0, futex_word=0x558601147b08) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558601147b08, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff17f9b90, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff17f9b90, clockid=0, mutex=0x558601147a98, cond=0x558601147ae0) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558601147ae0, mutex=0x558601147a98, abstime=0x7f1ff17f9b90) at ./nptl/pthread_cond_wait.c:652
            #5  0x00007f201e245092 in rocksdb::port::CondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201e14db19 in rocksdb::InstrumentedCondVar::TimedWaitInternal(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f201e14dc06 in rocksdb::InstrumentedCondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #8  0x00007f201e0224db in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #9  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 5 (Thread 0x7f201cafc640 (LWP 3067903) "rocksdb:high0"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5586010cccc4) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5586010cccc4) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586010cccc4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5586010ccc70, cond=0x5586010ccc98) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x5586010ccc98, mutex=0x5586010ccc70) at ./nptl/pthread_cond_wait.c:627
            #5  0x00007f201e1f5730 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201e1f5ace in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 4 (Thread 0x7f201d2fd640 (LWP 3067902) "rocksdb:low0"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5586010c9030) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5586010c9030) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586010c9030, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5586010c8fe0, cond=0x5586010c9008) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x5586010c9008, mutex=0x5586010c8fe0) at ./nptl/pthread_cond_wait.c:627
            #5  0x00007f201e1f5730 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201e1f5ace in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #8  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #9  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 3 (Thread 0x7f201dafe640 (LWP 3067901) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558601127290) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x558601127290) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558601127290, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558601127188, cond=0x558601127268) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_wait (cond=0x558601127268, mutex=0x558601127188) at ./nptl/pthread_cond_wait.c:627
            #5  0x00007f201e245011 in rocksdb::port::CondVar::Wait() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #6  0x00007f201e14d92d in rocksdb::InstrumentedCondVar::Wait() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #7  0x00007f201e134398 in rocksdb::DeleteScheduler::BackgroundEmptyTrash() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so
            #8  0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
            #9  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #10 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 2 (Thread 0x7f201fb89640 (LWP 3067900) "mariadbd"):
            #0  __futex_abstimed_wait_common64 (private=17563920, cancel=true, abstime=0x7f201fb88cd0, op=393, expected=0, futex_word=0x558600d62428) at ./nptl/futex-internal.c:57
            #1  __futex_abstimed_wait_common (cancel=true, private=17563920, abstime=0x7f201fb88cd0, clockid=17563488, expected=0, futex_word=0x558600d62428) at ./nptl/futex-internal.c:87
            #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600d62428, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f201fb88cd0, private=private@entry=0) at ./nptl/futex-internal.c:139
            #3  0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f201fb88cd0, clockid=0, mutex=0x558600d62440, cond=0x558600d62400) at ./nptl/pthread_cond_wait.c:503
            #4  ___pthread_cond_timedwait64 (cond=0x558600d62400, mutex=0x558600d62440, abstime=0x7f201fb88cd0) at ./nptl/pthread_cond_wait.c:652
            #5  0x00005585ffbd9137 in ?? ()
            #6  0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
            #7  0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            Thread 1 (Thread 0x7f201fd06880 (LWP 3067898) "mariadbd"):
            #0  0x00007f20200aebcf in __GI___poll (fds=0x5586023f8818, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
            #1  0x00005585ff40741a in handle_connections_sockets() ()
            #2  0x00005585ff40844f in mysqld_main(int, char**) ()
            #3  0x00007f201ffbfd90 in __libc_start_call_main (main=main@entry=0x5585ff3b6360 <main>, argc=argc@entry=1, argv=argv@entry=0x7ffd0f99d628) at ../sysdeps/nptl/libc_start_call_main.h:58
            #4  0x00007f201ffbfe40 in __libc_start_main_impl (main=0x5585ff3b6360 <main>, argc=1, argv=0x7ffd0f99d628, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd0f99d618) at ../csu/libc-start.c:392
            #5  0x00005585ff3fccb5 in _start ()
            
            

            As for optimistic mode, I used to use it, and at least once per month I would have the slave stuck because it would replicate things in the wrong order.
            My remark about production was just that I will gladly keep the server blocked for a few days if it helps with tracking the bug. I use MariaDB everyday, it's beautiful, and I am grateful that this kind of robust software exists for free. So, if I can help in any way, that's the least I can do.
            Still, at some point, I'll have to fix it (first I'll try to restart it, if it's not enough, I'll probably try the optimistic mode, and last resort, I'll restore it from a master dump). But don't worry, I'm not expecting you to fix my production.

            Vongo Adrian Delmarre added a comment - Hi @Kristian Nielsen, thanks for your answer, and sorry for the incomplete output. Here is the complete output : Thread 31 (Thread 0x7f2001303640 (LWP 3300512) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-1811929552, cancel=true, abstime=0x7f2001302c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1811929552, abstime=0x7f2001302c80, clockid=-1811937032, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2001302c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2001302c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2001302c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 30 (Thread 0x7f2001060640 (LWP 3300511) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=536880688, cancel=true, abstime=0x7f200105fc80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=536880688, abstime=0x7f200105fc80, clockid=536873208, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200105fc80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200105fc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200105fc80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 29 (Thread 0x7f2001399640 (LWP 3300505) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-2132796320, cancel=true, abstime=0x7f2001398c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-2132796320, abstime=0x7f2001398c80, clockid=-2132803800, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2001398c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2001398c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2001398c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 28 (Thread 0x7f20104fe640 (LWP 3300503) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-1924900784, cancel=true, abstime=0x7f20104fdc80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1924900784, abstime=0x7f20104fdc80, clockid=-1924908264, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20104fdc80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20104fdc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20104fdc80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 27 (Thread 0x7f2010594640 (LWP 3300498) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=2013275696, cancel=true, abstime=0x7f2010593c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=2013275696, abstime=0x7f2010593c80, clockid=2013268216, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2010593c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2010593c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2010593c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 26 (Thread 0x7f20102f1640 (LWP 3300494) "mariadbd"): #0 0x00007f20200aebcf in __GI___poll (fds=0x7f20102f0918, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00005585ff8b0e7f in vio_io_wait () #2 0x00005585ff8b0f54 in vio_socket_io_wait () #3 0x00005585ff8b1013 in vio_read () #4 0x00005585ff840f6e in ?? () #5 0x00005585ff8421f0 in my_net_read_packet_reallen () #6 0x00005585ff842301 in my_net_read_packet () #7 0x00005585ff4d7812 in do_command(THD*, bool) () #8 0x00005585ff5e210f in do_handle_one_connection(CONNECT*, bool) () #9 0x00005585ff5e248d in handle_one_connection () #10 0x00005585ff9443c6 in ?? () #11 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #12 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 25 (Thread 0x7f200126d640 (LWP 3299572) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=1207969328, cancel=true, abstime=0x7f200126cc80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=1207969328, abstime=0x7f200126cc80, clockid=1207961848, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f200126cc80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f200126cc80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f200126cc80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 24 (Thread 0x7f2001687640 (LWP 3299571) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=1422390464, cancel=true, abstime=0x7f2001686c80, op=393, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=1422390464, abstime=0x7f2001686c80, clockid=1422382984, expected=0, futex_word=0x55860049338c <thread_cache+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860049338c <thread_cache+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2001686c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2001686c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f2001686c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 23 (Thread 0x7f20014c5640 (LWP 3291154) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=149265072, cancel=true, abstime=0x7f20014c4c80, op=393, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=149265072, abstime=0x7f20014c4c80, clockid=149257592, expected=0, futex_word=0x558600493388 <thread_cache+40>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600493388 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f20014c4c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f20014c4c80, clockid=0, mutex=0x5586004933d0 <thread_cache+112>, cond=0x558600493360 <thread_cache>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600493360 <thread_cache>, mutex=0x5586004933d0 <thread_cache+112>, abstime=0x7f20014c4c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff5e219a in do_handle_one_connection(CONNECT*, bool) () #6 0x00005585ff5e248d in handle_one_connection () #7 0x00005585ff9443c6 in ?? () #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 22 (Thread 0x7f1ff0ff9640 (LWP 3252392) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=537376365, cancel=true, abstime=0x7f1ff0ff8c30, op=137, expected=0, futex_word=0x5586034985e8) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=537376365, abstime=0x7f1ff0ff8c30, clockid=0, expected=0, futex_word=0x5586034985e8) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586034985e8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f1ff0ff8c30, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f202002a2dd in __pthread_cond_wait_common (abstime=0x7f1ff0ff8c30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034985c0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_clockwait64 (abstime=0x7f1ff0ff8c30, clockid=1, mutex=0x5586011e0bf0, cond=0x5586034985c0) at ./nptl/pthread_cond_wait.c:691 #5 ___pthread_cond_clockwait64 (cond=0x5586034985c0, mutex=0x5586011e0bf0, clockid=1, abstime=0x7f1ff0ff8c30) at ./nptl/pthread_cond_wait.c:679 #6 0x00005585ffb76535 in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) () #7 0x00005585ffb76de8 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) () #8 0x00005585ffb76f5d in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () #9 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 21 (Thread 0x7f20105df640 (LWP 3067940) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac00a7b0) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac00a7b0) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac00a7b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac00a758, cond=0x7f1fac00a788) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x7f1fac00a788, mutex=0x7f1fac00a758) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff646c96 in handle_rpl_parallel_thread () #6 0x00005585ff9443c6 in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 20 (Thread 0x7f201062a640 (LWP 3067939) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac00a12c) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac00a12c) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac00a12c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac00a0d0, cond=0x7f1fac00a100) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x7f1fac00a100, mutex=0x7f1fac00a0d0) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff646c96 in handle_rpl_parallel_thread () #6 0x00005585ff9443c6 in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 19 (Thread 0x7f2010675640 (LWP 3067938) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fad07e4e0) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fad07e4e0) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fad07e4e0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac01e618, cond=0x7f1fad07e4b8) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x7f1fad07e4b8, mutex=0x7f1fac01e618) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff6477be in handle_rpl_parallel_thread () #6 0x00005585ff9443c6 in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 18 (Thread 0x7f20106c0640 (LWP 3067937) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac00941c) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac00941c) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac00941c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac0093c0, cond=0x7f1fac0093f0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x7f1fac0093f0, mutex=0x7f1fac0093c0) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff646c96 in handle_rpl_parallel_thread () #6 0x00005585ff9443c6 in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 17 (Thread 0x7f201070b640 (LWP 3067936) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac008d94) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac008d94) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac008d94, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac008d38, cond=0x7f1fac008d68) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x7f1fac008d68, mutex=0x7f1fac008d38) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff646c96 in handle_rpl_parallel_thread () #6 0x00005585ff9443c6 in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 16 (Thread 0x7f2010756640 (LWP 3067935) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f1fac009adc) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f1fac009adc) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1fac009adc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f1fac009a48, cond=0x7f1fac009ab0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x7f1fac009ab0, mutex=0x7f1fac009a48) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff64903f in rpl_parallel_entry::choose_thread(rpl_group_info*, bool*, PSI_stage_info_v1*, Gtid_log_event*) () #6 0x00005585ff64a1ae in rpl_parallel::do_event(rpl_group_info*, Log_event*, unsigned long long) () #7 0x00005585ff4237e5 in handle_slave_sql () #8 0x00005585ff9443c6 in ?? () #9 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #10 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 15 (Thread 0x7f201c07f640 (LWP 3067934) "mariadbd"): #0 0x00007f20200aebcf in __GI___poll (fds=0x7f201c07e698, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00005585ff8b0e7f in vio_io_wait () #2 0x00005585ff8b0f54 in vio_socket_io_wait () #3 0x00005585ff8b1013 in vio_read () #4 0x00005585ff8b113a in vio_read_buff () #5 0x00005585ff840f6e in ?? () #6 0x00005585ff8421f0 in my_net_read_packet_reallen () #7 0x00005585ff6d7ae1 in cli_safe_read_reallen () #8 0x00005585ff4287f2 in handle_slave_io () #9 0x00005585ff9443c6 in ?? () #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 14 (Thread 0x7f201c0ca640 (LWP 3067933) "mariadbd"): #0 0x00007f201ffd921a in __GI___sigtimedwait (set=0x7f201c0c9af0, info=0x7f201c0c9bf0, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61 #1 0x00005585ff3ff213 in signal_hand () #2 0x00005585ff9443c6 in ?? () #3 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #4 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 13 (Thread 0x7f201c115640 (LWP 3067932) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558600495dac <COND_manager+44>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x558600495dac <COND_manager+44>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600495dac <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558600495dc0 <LOCK_manager>, cond=0x558600495d80 <COND_manager>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x558600495d80 <COND_manager>, mutex=0x558600495dc0 <LOCK_manager>) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ff4c3d00 in handle_manager () #6 0x00005585ff9443c6 in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 12 (Thread 0x7f2001fff640 (LWP 3067925) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558600410524) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x558600410524) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600410524, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558600410480, cond=0x5586004104f8) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x5586004104f8, mutex=0x558600410480) at ./nptl/pthread_cond_wait.c:627 #5 0x00005585ffb04dde in ?? () #6 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 11 (Thread 0x7f2010f57640 (LWP 3067924) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=-1, cancel=true, abstime=0x7f2010f56c60, op=393, expected=0, futex_word=0x558600cc8448) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1, abstime=0x7f2010f56c60, clockid=4, expected=0, futex_word=0x558600cc8448) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600cc8448, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2010f56c60, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2010f56c60, clockid=0, mutex=0x558600cc8460, cond=0x558600cc8420) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600cc8420, mutex=0x558600cc8460, abstime=0x7f2010f56c60) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ff8e19c8 in ?? () #6 0x00005585ff8da37e in ?? () #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 10 (Thread 0x7f2011758640 (LWP 3067923) "myrocks-mc"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f2011757c80, op=393, expected=0, futex_word=0x7f201e3f4ab0) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f2011757c80, clockid=0, expected=0, futex_word=0x7f201e3f4ab0) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f201e3f4ab0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f2011757c80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f2011757c80, clockid=0, mutex=0x7f201e3f4a58, cond=0x7f201e3f4a88) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f201e3f4a88, mutex=0x7f201e3f4a58, abstime=0x7f2011757c80) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f201df94279 in myrocks::Rdb_manual_compaction_thread::run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201dfebb60 in myrocks::Rdb_thread::thread_func(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 9 (Thread 0x7f1ff2ffd640 (LWP 3067922) "myrocks-index"): #0 __futex_abstimed_wait_common64 (private=-592399872, cancel=true, abstime=0x7f1ff2ffcb50, op=393, expected=0, futex_word=0x7f201e3f47b4) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-592399872, abstime=0x7f1ff2ffcb50, clockid=-592399872, expected=0, futex_word=0x7f201e3f47b4) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f201e3f47b4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff2ffcb50, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff2ffcb50, clockid=0, mutex=0x7f201e3f4758, cond=0x7f201e3f4788) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f201e3f4788, mutex=0x7f201e3f4758, abstime=0x7f1ff2ffcb50) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f201dfa8ca3 in myrocks::Rdb_drop_index_thread::run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201dfebb60 in myrocks::Rdb_thread::thread_func(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 8 (Thread 0x7f1ff27fc640 (LWP 3067921) "myrocks-bg"): #0 __futex_abstimed_wait_common64 (private=537113683, cancel=true, abstime=0x7f1ff27fbc40, op=393, expected=0, futex_word=0x7f201e3f4bd0) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=537113683, abstime=0x7f1ff27fbc40, clockid=503428599, expected=0, futex_word=0x7f201e3f4bd0) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f201e3f4bd0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff27fbc40, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff27fbc40, clockid=0, mutex=0x7f201e3f4b78, cond=0x7f201e3f4ba8) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f201e3f4ba8, mutex=0x7f201e3f4b78, abstime=0x7f1ff27fbc40) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f201dfaa764 in myrocks::Rdb_background_thread::run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201dfebb60 in myrocks::Rdb_thread::thread_func(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #8 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 7 (Thread 0x7f1ff1ffb640 (LWP 3067920) "rocksdb:pst_st"): #0 __futex_abstimed_wait_common64 (private=-456, cancel=true, abstime=0x7f1ff1ffab90, op=393, expected=0, futex_word=0x55860136db38) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-456, abstime=0x7f1ff1ffab90, clockid=537113683, expected=0, futex_word=0x55860136db38) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55860136db38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff1ffab90, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff1ffab90, clockid=0, mutex=0x55860136dac8, cond=0x55860136db10) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x55860136db10, mutex=0x55860136dac8, abstime=0x7f1ff1ffab90) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f201e245092 in rocksdb::port::CondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201e14db19 in rocksdb::InstrumentedCondVar::TimedWaitInternal(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f201e14dc06 in rocksdb::InstrumentedCondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #8 0x00007f201e0224db in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #9 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 6 (Thread 0x7f1ff17fa640 (LWP 3067919) "rocksdb:dump_st"): #0 __futex_abstimed_wait_common64 (private=1952672118, cancel=true, abstime=0x7f1ff17f9b90, op=393, expected=0, futex_word=0x558601147b08) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=1952672118, abstime=0x7f1ff17f9b90, clockid=-243295576, expected=0, futex_word=0x558601147b08) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558601147b08, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ff17f9b90, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f1ff17f9b90, clockid=0, mutex=0x558601147a98, cond=0x558601147ae0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558601147ae0, mutex=0x558601147a98, abstime=0x7f1ff17f9b90) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f201e245092 in rocksdb::port::CondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201e14db19 in rocksdb::InstrumentedCondVar::TimedWaitInternal(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f201e14dc06 in rocksdb::InstrumentedCondVar::TimedWait(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #8 0x00007f201e0224db in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rocksdb::RepeatableThread::RepeatableThread(std::function<void ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, unsigned long, unsigned long)::{lambda()#1}> > >::_M_run() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #9 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #11 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 5 (Thread 0x7f201cafc640 (LWP 3067903) "rocksdb:high0"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5586010cccc4) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5586010cccc4) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586010cccc4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5586010ccc70, cond=0x5586010ccc98) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x5586010ccc98, mutex=0x5586010ccc70) at ./nptl/pthread_cond_wait.c:627 #5 0x00007f201e1f5730 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201e1f5ace in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 4 (Thread 0x7f201d2fd640 (LWP 3067902) "rocksdb:low0"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5586010c9030) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5586010c9030) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5586010c9030, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5586010c8fe0, cond=0x5586010c9008) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x5586010c9008, mutex=0x5586010c8fe0) at ./nptl/pthread_cond_wait.c:627 #5 0x00007f201e1f5730 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201e1f5ace in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 3 (Thread 0x7f201dafe640 (LWP 3067901) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558601127290) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x558601127290) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558601127290, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558601127188, cond=0x558601127268) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x558601127268, mutex=0x558601127188) at ./nptl/pthread_cond_wait.c:627 #5 0x00007f201e245011 in rocksdb::port::CondVar::Wait() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #6 0x00007f201e14d92d in rocksdb::InstrumentedCondVar::Wait() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #7 0x00007f201e134398 in rocksdb::DeleteScheduler::BackgroundEmptyTrash() () from target:/usr/lib/mysql/plugin/ha_rocksdb.so #8 0x00007f2020384253 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #10 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 2 (Thread 0x7f201fb89640 (LWP 3067900) "mariadbd"): #0 __futex_abstimed_wait_common64 (private=17563920, cancel=true, abstime=0x7f201fb88cd0, op=393, expected=0, futex_word=0x558600d62428) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=17563920, abstime=0x7f201fb88cd0, clockid=17563488, expected=0, futex_word=0x558600d62428) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558600d62428, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f201fb88cd0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f2020029e9b in __pthread_cond_wait_common (abstime=0x7f201fb88cd0, clockid=0, mutex=0x558600d62440, cond=0x558600d62400) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x558600d62400, mutex=0x558600d62440, abstime=0x7f201fb88cd0) at ./nptl/pthread_cond_wait.c:652 #5 0x00005585ffbd9137 in ?? () #6 0x00007f202002aac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #7 0x00007f20200bc850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   Thread 1 (Thread 0x7f201fd06880 (LWP 3067898) "mariadbd"): #0 0x00007f20200aebcf in __GI___poll (fds=0x5586023f8818, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00005585ff40741a in handle_connections_sockets() () #2 0x00005585ff40844f in mysqld_main(int, char**) () #3 0x00007f201ffbfd90 in __libc_start_call_main (main=main@entry=0x5585ff3b6360 <main>, argc=argc@entry=1, argv=argv@entry=0x7ffd0f99d628) at ../sysdeps/nptl/libc_start_call_main.h:58 #4 0x00007f201ffbfe40 in __libc_start_main_impl (main=0x5585ff3b6360 <main>, argc=1, argv=0x7ffd0f99d628, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd0f99d618) at ../csu/libc-start.c:392 #5 0x00005585ff3fccb5 in _start () As for optimistic mode, I used to use it, and at least once per month I would have the slave stuck because it would replicate things in the wrong order. My remark about production was just that I will gladly keep the server blocked for a few days if it helps with tracking the bug. I use MariaDB everyday, it's beautiful, and I am grateful that this kind of robust software exists for free. So, if I can help in any way, that's the least I can do. Still, at some point, I'll have to fix it (first I'll try to restart it, if it's not enough, I'll probably try the optimistic mode, and last resort, I'll restore it from a master dump). But don't worry, I'm not expecting you to fix my production.

            Understood Vongo, appreciate the effort.

            Another piece of information that would be useful is the binlog events
            around the point where the slave gets stuck, especially the first event that
            has not been applied on the slave, as this is the event that gets stuck
            somehow. You can obtain this either from the master's binlog or the slave's
            binlog, whichever is easiest and has not yet been deleted by the system.

            Once you have that (if possible), I would request:

            1. Stop the slave. First try STOP SLAVE; if that hangs (which I expect it
            will), try KILL CONNECTION 8 the stuck worker and observe if that still
            hangs. Stop the server with kill -9 or similar if required.

            2. Restart the slave without skip_slave_counter set, see if it proceeds or
            if it hangs again. If it hangs, I would like to see the SHOW PROCESSLIST and
            SHOW SLAVE STATUS of this second hang, to see if it hangs the same way and
            in the same place.

            After that feel free to fix the server in whatever way you need (if it still
            is stuck).

            I would also like to know if you think a FLUSH TABLES WITH READ LOCK could
            have happened around when the hangs occured, just to rule out one corner
            case I found in the code.

            Some more technical points from my investigation so far:

            From the SHOW PROCESSLIST output, I would expect the stuch transaction to be
            larger than @@slave_parallel_max_queued, as the SQL driver thread is
            waiting for room to queue a single transaction while all other workers are
            idle with no events queued; this is the case in both occurrences.

            On the other hand, the processlist combined with the stack trace looks like
            the one worker thread is stuck in the function do_gco_wait() (inlined so not
            directly visible in the stack trace). This does not seem to relate to having
            a single large transaction, and it is especially surprising that this
            problem would persist after slave restart, requiring slave_skip_counter=1.
            So I'm trying with the request for more information to understand more about
            how the server is stuck and which of these seemingly conflicting
            circumstances is the relevant one.

            Thanks for the additional information about experiencing hangs in optimistic
            mode earlier. Such hangs would also be very good to get debugged and fixed.
            Though using RocksDB with optimistic parallel replication might be
            challenging (as you found), and conservative might be a good compromise for
            you (hopefully we can get this bug fixed).

            There were several bugs that could cause parallel replication hangs fixed
            since 10.6.16, including one (MDEV-34696) that will only be in 10.6.20. But
            I am not sure if any of them applies to your specific problem.

            knielsen Kristian Nielsen added a comment - Understood Vongo , appreciate the effort. Another piece of information that would be useful is the binlog events around the point where the slave gets stuck, especially the first event that has not been applied on the slave, as this is the event that gets stuck somehow. You can obtain this either from the master's binlog or the slave's binlog, whichever is easiest and has not yet been deleted by the system. Once you have that (if possible), I would request: 1. Stop the slave. First try STOP SLAVE; if that hangs (which I expect it will), try KILL CONNECTION 8 the stuck worker and observe if that still hangs. Stop the server with kill -9 or similar if required. 2. Restart the slave without skip_slave_counter set, see if it proceeds or if it hangs again. If it hangs, I would like to see the SHOW PROCESSLIST and SHOW SLAVE STATUS of this second hang, to see if it hangs the same way and in the same place. After that feel free to fix the server in whatever way you need (if it still is stuck). I would also like to know if you think a FLUSH TABLES WITH READ LOCK could have happened around when the hangs occured, just to rule out one corner case I found in the code. Some more technical points from my investigation so far: From the SHOW PROCESSLIST output, I would expect the stuch transaction to be larger than @@slave_parallel_max_queued, as the SQL driver thread is waiting for room to queue a single transaction while all other workers are idle with no events queued; this is the case in both occurrences. On the other hand, the processlist combined with the stack trace looks like the one worker thread is stuck in the function do_gco_wait() (inlined so not directly visible in the stack trace). This does not seem to relate to having a single large transaction, and it is especially surprising that this problem would persist after slave restart, requiring slave_skip_counter=1. So I'm trying with the request for more information to understand more about how the server is stuck and which of these seemingly conflicting circumstances is the relevant one. Thanks for the additional information about experiencing hangs in optimistic mode earlier. Such hangs would also be very good to get debugged and fixed. Though using RocksDB with optimistic parallel replication might be challenging (as you found), and conservative might be a good compromise for you (hopefully we can get this bug fixed). There were several bugs that could cause parallel replication hangs fixed since 10.6.16, including one ( MDEV-34696 ) that will only be in 10.6.20. But I am not sure if any of them applies to your specific problem.

            HI Kristian, thank you very much for investigating.
            Unfortunately, and believe me when I say that I am very sorry about that, I had to fix the server yesterday morning, because some users were starting to complain that their data was not updating any more.
            You might find it interesting, though, that it didn't get stuck after restarting, and I didn't have to skip any transaction.
            To fix it, I just restarted the mariadb.service (with optimistic mode as you suggested). As it got stuck, I killed maria processes 8 and 5. And then, it was good as new.
            Next time it happens, I'll get the binlog events and follow your procedure.

            Vongo Adrian Delmarre added a comment - HI Kristian, thank you very much for investigating. Unfortunately, and believe me when I say that I am very sorry about that, I had to fix the server yesterday morning, because some users were starting to complain that their data was not updating any more. You might find it interesting, though, that it didn't get stuck after restarting, and I didn't have to skip any transaction. To fix it, I just restarted the mariadb.service (with optimistic mode as you suggested). As it got stuck, I killed maria processes 8 and 5. And then, it was good as new. Next time it happens, I'll get the binlog events and follow your procedure.

            People

              knielsen Kristian Nielsen
              Vongo Adrian Delmarre
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.