Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29388

Galera cluster hanging

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.6.9
    • 10.6.10
    • Galera, Server
    • None

    Description

      On a 3-nodes cluster, 2 nodes got completely stuck. Anything trying to connect would also get stuck. The third node had 200+ (or 500?) queries running, but stuck as well.

      Stopping those 2 servers got stuck to, had to kill the processes manually. Logs shows the server being disconnected from the cluster. Cluster size went from 3 to 1 as it should, but mariadb processes where stuck on those 2 servers until manually killed.

      Starting first node back after a minute got it to sync using an IST, the other node restarted after 5 minutes synced using an SST. As if nothing happened!?

      Digging into logs of the 3 servers shows nothing, no output at all during the whole period they where stuck.

      Anything I could do to track this down if it happens again?

      Server 1 log (stuck):

      2022-08-25  0:29:16 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000191
      		 
      2022-08-25  0:29:32 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000192 of size 134217696 bytes
      		 
      2022-08-25  0:29:39 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000192
      		 
      2022-08-25  0:30:38 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000193 of size 134217696 bytes
      		 
      2022-08-25  0:30:47 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000194 of size 134217696 bytes
      		 
      2022-08-25  0:30:56 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000193
      		 
      2022-08-25  0:30:57 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000194
      		 
      2022-08-25  0:32:23 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000195 of size 134217696 bytes
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: declaring 19edde19-9435 at tcp://xxxx:4567 stable
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Node 19edde19-9435 state prim
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,63) memb {
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: save pc into disk
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 0 (ovh7.1check.com)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 1 (ovh1.1check.com)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Quorum results:
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Flow-control interval: [23, 23]
      		 
      2022-08-25  1:03:26 0 [Note] WSREP:  cleaning up e26a09b5-b02e (tcp://xxxx:4567)
      		 
      2022-08-25  1:06:00 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: Shutdown replication
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: Server status change synced -> disconnecting
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: Closing send monitor...
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: Closed send monitor.
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: gcomm: terminating thread
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: gcomm: joining thread
      		 
      2022-08-25  1:06:00 0 [Note] WSREP: gcomm: closing backend
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: view(view_id(NON_PRIM,19edde19-9435,63) memb {
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: PC protocol downgrade 1 -> 0
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: view((empty))
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: gcomm: closed
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Flow-control interval: [16, 16]
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Received NON-PRIMARY.
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 70213440)
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: New SELF-LEAVE.
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Flow-control interval: [0, 0]
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 70213440)
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: RECV thread exiting 0: Success
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: recv_thread() joined.
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Closing replication queue.
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Closing slave action queue.
      		 
      2022-08-25  1:07:18 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 2c96698a-0fdf-11ed-90d6-7ecc8fa70984:70213386
      		 
      2022-08-25  1:07:18 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'

      Server 2 log (stuck):

      2022-08-25  0:29:32 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000192 of size 134217696 bytes
      		 
      2022-08-25  0:29:39 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000192
      		 
      2022-08-25  0:30:38 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000193 of size 134217696 bytes
      		 
      2022-08-25  0:30:47 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000194 of size 134217696 bytes
      		 
      2022-08-25  0:30:56 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000193
      		 
      2022-08-25  0:30:57 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000194
      		 
      2022-08-25  0:32:23 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000195 of size 134217696 bytes
      		 
      2022-08-25  1:03:21 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Shutdown replication
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Server status change synced -> disconnecting
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Closing send monitor...
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Closed send monitor.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: gcomm: terminating thread
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: gcomm: joining thread
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: gcomm: closing backend
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: view(view_id(NON_PRIM,19edde19-9435,62) memb {
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: PC protocol downgrade 1 -> 0
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: view((empty))
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: gcomm: closed
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Flow-control interval: [16, 16]
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Received NON-PRIMARY.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 70213433)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: New SELF-LEAVE.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Flow-control interval: [0, 0]
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 70213433)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: RECV thread exiting 0: Success
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: recv_thread() joined.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Closing replication queue.
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Closing slave action queue.
      		 
      2022-08-25  1:08:05 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 2c96698a-0fdf-11ed-90d6-7ecc8fa70984:70213385
      		 
      2022-08-25  1:08:05 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
      		 
      2022-08-25  1:08:05 0 [Note] WSREP: wsrep_load(): Galera 26.4.12(r1eac5b64) by Codership Oy <info@codership.com> loaded successfully.
      		 
      2022-08-25  1:08:05 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.

      Server 3 log (still alive):

      2022-08-25  0:30:56 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000389
      		 
      2022-08-25  0:30:57 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000390
      		 
      2022-08-25  0:32:23 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000391 of size 134217696 bytes
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: declaring 3e14de76-94d4 at tcp://xxxx:4567 stable
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://xxxx:44208
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7fd768845210
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Deferred close timer destruct
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Node 19edde19-9435 state prim
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,63) memb {
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: save pc into disk
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: f2ffea9d-2400-11ed-9136-ef240a6d8bee
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 0 (ovh7.1check.com)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 1 (ovh1.1check.com)
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Quorum results:
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Flow-control interval: [23, 23]
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: ####### processing CC 70213434, local, ordered
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: ####### My UUID: 19edde19-2139-11ed-9435-e69b7d8e37e9
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: Skipping cert index reset
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: REPL Protocols: 10 (5)
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: ####### Adjusting cert position: 70213433 -> 70213434
      		 
      2022-08-25  1:03:21 0 [Note] WSREP: Service thread queue flushed.
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: ================================================
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: Lowest cert index boundary for CC from group: 70213347
      		 
      2022-08-25  1:03:21 17 [Note] WSREP: Min available from gcache for CC from group: 70213347
      		 
      2022-08-25  1:03:26 0 [Note] WSREP:  cleaning up e26a09b5-b02e (tcp://xxxx:4567)
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://xxxx:36660
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: forgetting 3e14de76-94d4 (tcp://xxxx:4567)
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Node 19edde19-9435 state prim
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,64) memb {
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: save pc into disk
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: forgetting 3e14de76-94d4 (tcp://xxxx:4567)
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7fd7344f9160
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Deferred close timer destruct
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 523fb653-2401-11ed-8253-82473e2187ea
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 523fb653-2401-11ed-8253-82473e2187ea
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: STATE EXCHANGE: got state msg: 523fb653-2401-11ed-8253-82473e2187ea from 0 (ovh7.1check.com)
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Quorum results:
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Flow-control interval: [16, 16]
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: ####### processing CC 70213441, local, ordered
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: ####### My UUID: 19edde19-2139-11ed-9435-e69b7d8e37e9
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: Skipping cert index reset
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: REPL Protocols: 10 (5)
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: ####### Adjusting cert position: 70213440 -> 70213441
      		 
      2022-08-25  1:06:01 0 [Note] WSREP: Service thread queue flushed.
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: ================================================
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: Lowest cert index boundary for CC from group: 70213347
      		 
      2022-08-25  1:06:01 17 [Note] WSREP: Min available from gcache for CC from group: 70213347
      		 
      2022-08-25  1:06:06 0 [Note] WSREP:  cleaning up 3e14de76-94d4 (tcp://xxxx:4567)
      		 
      2022-08-25  1:06:17 16935047 [Warning] Aborted connection 16935047 to db: '_1check_mail_export' user: 'bo_1check' host: '127.0.0.1' (Got an error reading communication packets)
      		 
      2022-08-25  1:06:17 16935044 [Warning] Aborted connection 16935044 to db: '_1check_mail_export' user: 'bo_1check' host: '127.0.0.1' (Got an error reading communication packets)
      		 
      2022-08-25  1:07:12 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000392 of size 134217696 bytes
      		 
      2022-08-25  1:07:18 0 [Note] WSREP: (19edde19-9435, 'tcp://0.0.0.0:4567') connection established to 807a59e3-896d tcp://xxxx:4567
      		 
      2022-08-25  1:07:18 0 [Note] WSREP: (19edde19-9435, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: declaring 807a59e3-896d at tcp://xxxx:4567 stable
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: Node 19edde19-9435 state prim
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,65) memb {
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: save pc into disk
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 80c6eee7-2401-11ed-a085-a2c1fb1fa522
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 80c6eee7-2401-11ed-a085-a2c1fb1fa522
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: 80c6eee7-2401-11ed-a085-a2c1fb1fa522 from 0 (ovh7.1check.com)
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: 80c6eee7-2401-11ed-a085-a2c1fb1fa522 from 1 (ovh1.1check.com)
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: Quorum results:
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: Flow-control interval: [23, 23]
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: ####### processing CC 70224411, local, ordered
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: ####### My UUID: 19edde19-2139-11ed-9435-e69b7d8e37e9
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: Skipping cert index reset
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: REPL Protocols: 10 (5)
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: ####### Adjusting cert position: 70224410 -> 70224411
      		 
      2022-08-25  1:07:19 0 [Note] WSREP: Service thread queue flushed.
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: ================================================
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: Lowest cert index boundary for CC from group: 70213347
      		 
      2022-08-25  1:07:19 17 [Note] WSREP: Min available from gcache for CC from group: 70213347

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          View 7 older comments
          ccounotte COUNOTTE CEDRIC added a comment -

          @Jan Lindström, as requested, output of thread apply all bt:

          Thread 24 (Thread 0x7efefde1d640 (LWP 2545245) "mariadbd"):
          		 
          #0  0x00007eff10676d7f in __GI___poll (fds=0x7efefde1c8c8, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
          		 
          #1  0x00005603e32edc9f in vio_io_wait ()
          		 
          #2  0x00005603e32edd74 in vio_socket_io_wait ()
          		 
          #3  0x00005603e32ede33 in vio_read ()
          		 
          #4  0x00005603e327a535 in ?? ()
          		 
          #5  0x00005603e327b9d0 in my_net_read_packet_reallen ()
          		 
          #6  0x00005603e327bac1 in my_net_read_packet ()
          		 
          #7  0x00005603e2f172e2 in do_command(THD*, bool) ()
          		 
          #8  0x00005603e3024f6f in do_handle_one_connection(CONNECT*, bool) ()
          		 
          #9  0x00005603e30252bd in handle_one_connection ()
          		 
          #10 0x00005603e3390db6 in ?? ()
          		 
          #11 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #12 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 23 (Thread 0x7efea56b0640 (LWP 2495611) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=-477129360, cancel=true, abstime=0x7efea56afc30, op=137, expected=0, futex_word=0x5603e5dddbdc) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=-477129360, abstime=0x7efea56afc30, clockid=0, expected=0, futex_word=0x5603e5dddbdc) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5dddbdc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7efea56afc30, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f235d in __pthread_cond_wait_common (abstime=0x7efea56afc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5dddbb0) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_clockwait64 (abstime=0x7efea56afc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5dddbb0) at ./nptl/pthread_cond_wait.c:691
          		 
          #5  ___pthread_cond_clockwait64 (cond=0x5603e5dddbb0, mutex=0x5603e5cc4a00, clockid=1, abstime=0x7efea56afc30) at ./nptl/pthread_cond_wait.c:679
          		 
          #6  0x00005603e3619aaf in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) ()
          		 
          #7  0x00005603e361a608 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) ()
          		 
          #8  0x00005603e361b5a5 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) ()
          		 
          #9  0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
          		 
          #10 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #11 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 22 (Thread 0x7ee836ffd640 (LWP 2487980) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=-477129360, cancel=true, abstime=0x7ee836ffcc30, op=137, expected=0, futex_word=0x5603e5ddd5d8) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=-477129360, abstime=0x7ee836ffcc30, clockid=0, expected=0, futex_word=0x5603e5ddd5d8) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5ddd5d8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7ee836ffcc30, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f235d in __pthread_cond_wait_common (abstime=0x7ee836ffcc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5ddd5b0) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_clockwait64 (abstime=0x7ee836ffcc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5ddd5b0) at ./nptl/pthread_cond_wait.c:691
          		 
          #5  ___pthread_cond_clockwait64 (cond=0x5603e5ddd5b0, mutex=0x5603e5cc4a00, clockid=1, abstime=0x7ee836ffcc30) at ./nptl/pthread_cond_wait.c:679
          		 
          #6  0x00005603e3619aaf in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) ()
          		 
          #7  0x00005603e361a608 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) ()
          		 
          #8  0x00005603e361b5a5 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) ()
          		 
          #9  0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
          		 
          #10 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #11 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 21 (Thread 0x7efeec063640 (LWP 2352786) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ee83832c508) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ee83832c508) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ee83832c508, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c94f88, cond=0x7ee83832c4e0) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x7ee83832c4e0, mutex=0x5603e5c94f88) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00007efeff0e1343 in gu_cond_wait_SYS (mutex=0x5603e5c94f88, cond=0x7ee83832c4e0) at ./galerautils/src/gu_threads.h:251
          		 
          #6  gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
          		 
          #7  galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x5603e5c94f88, obj=...) at ./galera/src/monitor.hpp:194
          		 
          #8  0x00007efeff0ba425 in galera::ReplicatorSMM::commit_order_enter_remote (trx=..., this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:1315
          		 
          #9  galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at ./galera/src/wsrep_provider.cpp:762
          		 
          #10 0x00005603e37214aa in wsrep::wsrep_provider_v26::commit_order_enter(wsrep::ws_handle const&, wsrep::ws_meta const&) ()
          		 
          #11 0x00005603e371c91d in wsrep::transaction::before_commit() ()
          		 
          #12 0x00005603e3155d01 in ha_commit_trans(THD*, bool) ()
          		 
          #13 0x00005603e3034f17 in trans_commit(THD*) ()
          		 
          #14 0x00005603e33ff178 in Wsrep_high_priority_service::commit(wsrep::ws_handle const&, wsrep::ws_meta const&) ()
          		 
          #15 0x00005603e370dde3 in ?? ()
          		 
          #16 0x00005603e3720c55 in ?? ()
          		 
          #17 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee83832c290, recv_ctx=0x7efeec062be0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp:396
          		 
          #18 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec062be0, ts=...) at ./galera/src/replicator_smm.cpp:516
          		 
          #19 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec062be0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136
          		 
          #20 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efeec06280f: false, act=..., recv_ctx=0x7efeec062be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:62
          		 
          #21 galera::GcsActionSource::dispatch (exit_loop=@0x7efeec06280f: false, act=..., recv_ctx=0x7efeec062be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110
          		 
          #22 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efeec062be0, exit_loop=@0x7efeec06280f: false) at ./galera/src/gcs_action_source.cpp:186
          		 
          --Type <RET> for more, q to quit, c to continue without paging--c
          		 
          #23 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec062be0) at ./galera/src/replicator_smm.cpp:402
          		 
          #24 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #25 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #26 0x00005603e341a267 in ?? ()
          		 
          #27 0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #28 0x00005603e3390db6 in ?? ()
          		 
          #29 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #30 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 20 (Thread 0x7efeec0ae640 (LWP 2352785) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efeec0ad7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efeec0ad7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec0adbe0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 19 (Thread 0x7efeec0f9640 (LWP 2352784) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efeec0f87c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efeec0f87c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec0f8be0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 18 (Thread 0x7efeec144640 (LWP 2352783) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ee83010b6b8) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ee83010b6b8) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ee83010b6b8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c94f88, cond=0x7ee83010b690) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x7ee83010b690, mutex=0x5603e5c94f88) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00007efeff0e1343 in gu_cond_wait_SYS (mutex=0x5603e5c94f88, cond=0x7ee83010b690) at ./galerautils/src/gu_threads.h:251
          		 
          #6  gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
          		 
          #7  galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x5603e5c94f88, obj=...) at ./galera/src/monitor.hpp:194
          		 
          #8  0x00007efeff0ba425 in galera::ReplicatorSMM::commit_order_enter_remote (trx=..., this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:1315
          		 
          #9  galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at ./galera/src/wsrep_provider.cpp:762
          		 
          #10 0x00005603e37214aa in wsrep::wsrep_provider_v26::commit_order_enter(wsrep::ws_handle const&, wsrep::ws_meta const&) ()
          		 
          #11 0x00005603e371c91d in wsrep::transaction::before_commit() ()
          		 
          #12 0x00005603e3155d01 in ha_commit_trans(THD*, bool) ()
          		 
          #13 0x00005603e3034f17 in trans_commit(THD*) ()
          		 
          #14 0x00005603e33ff178 in Wsrep_high_priority_service::commit(wsrep::ws_handle const&, wsrep::ws_meta const&) ()
          		 
          #15 0x00005603e370dde3 in ?? ()
          		 
          #16 0x00005603e3720c55 in ?? ()
          		 
          #17 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee83010b440, recv_ctx=0x7efeec143be0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp:396
          		 
          #18 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec143be0, ts=...) at ./galera/src/replicator_smm.cpp:516
          		 
          #19 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec143be0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136
          		 
          #20 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efeec14380f: false, act=..., recv_ctx=0x7efeec143be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:62
          		 
          #21 galera::GcsActionSource::dispatch (exit_loop=@0x7efeec14380f: false, act=..., recv_ctx=0x7efeec143be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110
          		 
          #22 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efeec143be0, exit_loop=@0x7efeec14380f: false) at ./galera/src/gcs_action_source.cpp:186
          		 
          #23 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec143be0) at ./galera/src/replicator_smm.cpp:402
          		 
          #24 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #25 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #26 0x00005603e341a267 in ?? ()
          		 
          #27 0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #28 0x00005603e3390db6 in ?? ()
          		 
          #29 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #30 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 17 (Thread 0x7efefc053640 (LWP 2352782) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc0527c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc0527c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc052be0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 16 (Thread 0x7efefc09e640 (LWP 2352781) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc09d7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc09d7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc09dbe0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 15 (Thread 0x7efefc0e9640 (LWP 2352780) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94f08) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c94f08) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c94f08, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c94e98, cond=0x5603e5c94ee0) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e5c94ee0, mutex=0x5603e5c94e98) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00007efeff0dea1d in gu_cond_wait_SYS (mutex=<optimized out>, cond=0x5603e5c94ee0) at ./galerautils/src/gu_threads.h:251
          		 
          #6  gu::Lock::wait (this=0x7efefc0e7ac0, this=0x7efefc0e7ac0, cond=...) at ./galerautils/src/gu_lock.hpp:46
          		 
          #7  galera::Monitor<galera::ReplicatorSMM::ApplyOrder>::drain_common (this=0x5603e5c94e98, seqno=<optimized out>, lock=...) at ./galera/src/monitor.hpp:564
          		 
          #8  0x00007efeff0e40b0 in galera::Monitor<galera::ReplicatorSMM::ApplyOrder>::drain (this=0x5603e5c94e98, seqno=129147258) at ./galera/src/monitor.hpp:368
          		 
          #9  0x00007efeff0d897b in galera::ReplicatorSMM::drain_monitors (upto=129147258, this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:2208
          		 
          #10 galera::ReplicatorSMM::drain_monitors_for_local_conf_change (this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:2489
          		 
          #11 0x00007efeff0dbe4e in galera::ReplicatorSMM::process_non_prim_conf_change (my_index=<optimized out>, conf=..., recv_ctx=0x7efefc0e8be0, this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:2521
          		 
          #12 galera::ReplicatorSMM::process_conf_change (this=0x5603e5c93bf0, recv_ctx=0x7efefc0e8be0, conf=..., cc=...) at ./galera/src/replicator_smm.cpp:2459
          		 
          #13 0x00007efeff0f85f1 in galera::GcsActionSource::dispatch (exit_loop=<optimized out>, act=..., recv_ctx=0x7efefc0e8be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:127
          		 
          #14 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efefc0e8be0, exit_loop=<optimized out>) at ./galera/src/gcs_action_source.cpp:186
          		 
          #15 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc0e8be0) at ./galera/src/replicator_smm.cpp:402
          		 
          #16 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #17 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #18 0x00005603e341a267 in ?? ()
          		 
          #19 0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #20 0x00005603e3390db6 in ?? ()
          		 
          #21 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #22 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 14 (Thread 0x7efefc134640 (LWP 2352779) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc1337c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc1337c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc133be0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 13 (Thread 0x7efefc17f640 (LWP 2352778) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc17e7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc17e7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc17ebe0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 12 (Thread 0x7efefc1ca640 (LWP 2352777) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc1c97c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc1c97c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc1c9be0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 11 (Thread 0x7efefc215640 (LWP 2352776) "mariadbd"):
          		 
          #0  0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc2147c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
          		 
          #1  0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc2147c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
          		 
          #2  0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
          		 
          #3  0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc214be0) at ./galera/src/replicator_smm.cpp:408
          		 
          #4  0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #5  0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #6  0x00005603e341a267 in ?? ()
          		 
          #7  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #8  0x00005603e3390db6 in ?? ()
          		 
          #9  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 10 (Thread 0x7efefc260640 (LWP 2352773) "mariadbd"):
          		 
          #0  0x00007eff105a121a in __GI___sigtimedwait (set=0x7efefc25fb90, info=0x7efefc25fc10, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61
          		 
          #1  0x00005603e2e3169b in signal_hand ()
          		 
          #2  0x00005603e3390db6 in ?? ()
          		 
          #3  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #4  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 9 (Thread 0x7efefc2ab640 (LWP 2352772) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3f6efac <mysql_bin_log+3276>) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e3f6efac <mysql_bin_log+3276>) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e3f6efac <mysql_bin_log+3276>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e3f6ef50 <mysql_bin_log+3184>, cond=0x5603e3f6ef80 <mysql_bin_log+3232>) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e3f6ef80 <mysql_bin_log+3232>, mutex=0x5603e3f6ef50 <mysql_bin_log+3184>) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00005603e324a57e in binlog_background_thread ()
          		 
          #6  0x00005603e3390db6 in ?? ()
          		 
          #7  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #8  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 8 (Thread 0x7efefc2f6640 (LWP 2352771) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3f40dac <COND_manager+44>) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e3f40dac <COND_manager+44>) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e3f40dac <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e3f40dc0 <LOCK_manager>, cond=0x5603e3f40d80 <COND_manager>) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e3f40d80 <COND_manager>, mutex=0x5603e3f40dc0 <LOCK_manager>) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00005603e2f01480 in handle_manager ()
          		 
          #6  0x00005603e3390db6 in ?? ()
          		 
          #7  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #8  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 7 (Thread 0x7efe8e0c7640 (LWP 2352709) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3ebb89c) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e3ebb89c) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e3ebb89c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e3ebb800, cond=0x5603e3ebb870) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e3ebb870, mutex=0x5603e3ebb800) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00005603e358aaa3 in ?? ()
          		 
          #6  0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
          		 
          #7  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #8  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 6 (Thread 0x7efe8e8c8640 (LWP 2352708) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7efe8e8c7c60, op=393, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7efe8e8c7c60, clockid=1602883208, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e47728a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7efe8e8c7c60, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1f1b in __pthread_cond_wait_common (abstime=0x7efe8e8c7c60, clockid=0, mutex=0x5603e47728c0, cond=0x5603e4772880) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_timedwait64 (cond=0x5603e4772880, mutex=0x5603e47728c0, abstime=0x7efe8e8c7c60) at ./nptl/pthread_cond_wait.c:652
          		 
          #5  0x00005603e3325402 in ?? ()
          		 
          #6  0x00005603e331d7be in ?? ()
          		 
          #7  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #8  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 5 (Thread 0x7efefcefc640 (LWP 2280234) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c93dc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c93d60, cond=0x5603e5c93da0) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e5c93da0, mutex=0x5603e5c93d60) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00007efeff0cc92c in gu_cond_wait_SYS (mutex=0x5603e5c93d60, cond=0x5603e5c93da0) at ./galerautils/src/gu_threads.h:251
          		 
          #6  gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
          		 
          #7  galera::ReplicatorSMM::wait_for_CLOSED (lock=..., this=<optimized out>) at ./galera/src/replicator_smm.cpp:285
          		 
          #8  galera::ReplicatorSMM::close (this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:376
          		 
          #9  0x00007efeff0b77a1 in galera_disconnect (gh=<optimized out>) at ./galera/src/wsrep_provider.cpp:235
          		 
          #10 0x00005603e3720506 in wsrep::wsrep_provider_v26::disconnect() ()
          		 
          #11 0x00005603e2e3383a in unireg_abort ()
          		 
          #12 0x00005603e3408298 in wsrep_handle_mdl_conflict(MDL_context*, MDL_ticket const*, MDL_key const*) ()
          		 
          #13 0x00005603e302e2cb in MDL_lock::can_grant_lock(enum_mdl_type, MDL_context*, bool) const ()
          		 
          #14 0x00005603e302e4ea in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) ()
          		 
          #15 0x00005603e302e942 in MDL_context::acquire_lock(MDL_request*, double) ()
          		 
          #16 0x00005603e302f5ae in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) ()
          		 
          #17 0x00005603e2e9f80a in wait_while_table_is_used(THD*, TABLE*, ha_extra_function) ()
          		 
          #18 0x00005603e2fbe618 in mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool) ()
          		 
          #19 0x00005603e2fbf454 in mysql_recreate_table(THD*, TABLE_LIST*, bool) ()
          		 
          #20 0x00005603e30301cd in ?? ()
          		 
          #21 0x00005603e30327e8 in ?? ()
          		 
          #22 0x00005603e30342a5 in Sql_cmd_optimize_table::execute(THD*) ()
          		 
          #23 0x00005603e2f11d9c in mysql_execute_command(THD*, bool) ()
          		 
          #24 0x00005603e2f12f57 in mysql_parse(THD*, char*, unsigned int, Parser_state*) ()
          		 
          #25 0x00005603e3269071 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) ()
          		 
          #26 0x00005603e34183c0 in wsrep_apply_events(THD*, Relay_log_info*, void const*, unsigned long) ()
          		 
          #27 0x00005603e33fee0b in Wsrep_high_priority_service::apply_toi(wsrep::ws_meta const&, wsrep::const_buffer const&, wsrep::mutable_buffer&) ()
          		 
          #28 0x00005603e370ed69 in wsrep::server_state::on_apply(wsrep::high_priority_service&, wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&) ()
          		 
          #29 0x00005603e3720c55 in ?? ()
          		 
          #30 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee73016b410, recv_ctx=0x7efefcefbbe0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp:396
          		 
          #31 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0, ts=...) at ./galera/src/replicator_smm.cpp:516
          		 
          #32 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136
          		 
          #33 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efefcefb80f: false, act=..., recv_ctx=0x7efefcefbbe0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:62
          		 
          #34 galera::GcsActionSource::dispatch (exit_loop=@0x7efefcefb80f: false, act=..., recv_ctx=0x7efefcefbbe0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110
          		 
          #35 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efefcefbbe0, exit_loop=@0x7efefcefb80f: false) at ./galera/src/gcs_action_source.cpp:186
          		 
          #36 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0) at ./galera/src/replicator_smm.cpp:402
          		 
          #37 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263
          		 
          #38 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) ()
          		 
          #39 0x00005603e341a267 in ?? ()
          		 
          #40 0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #41 0x00005603e3390db6 in ?? ()
          		 
          #42 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #43 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 4 (Thread 0x7efefcf47640 (LWP 2280233) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7efee40134a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7efee4013448, cond=0x7efee4013478) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x7efee4013478, mutex=0x7efee4013448) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00005603e3419415 in ?? ()
          		 
          #6  0x00005603e340a033 in start_wsrep_THD(void*) ()
          		 
          #7  0x00005603e3390db6 in ?? ()
          		 
          #8  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #9  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 3 (Thread 0x7efefeecd640 (LWP 2280228) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c94544, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c944d8, cond=0x5603e5c94518) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e5c94518, mutex=0x5603e5c944d8) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00007efeff0f55f3 in gu_cond_wait_SYS (mutex=0x5603e5c944d8, cond=0x5603e5c94518) at ./galerautils/src/gu_threads.h:251
          		 
          #6  gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
          		 
          #7  galera::ServiceThd::thd_func (arg=0x5603e5c944b8) at ./galera/src/galera_service_thd.cpp:31
          		 
          #8  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #9  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 2 (Thread 0x7efeff24a640 (LWP 2280227) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=40, cancel=true, abstime=0x7efeff249cd0, op=393, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=40, abstime=0x7efeff249cd0, clockid=3, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e480cacc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7efeff249cd0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1f1b in __pthread_cond_wait_common (abstime=0x7efeff249cd0, clockid=0, mutex=0x5603e480cae0, cond=0x5603e480caa0) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_timedwait64 (cond=0x5603e480caa0, mutex=0x5603e480cae0, abstime=0x7efeff249cd0) at ./nptl/pthread_cond_wait.c:652
          		 
          #5  0x00005603e3689d07 in ?? ()
          		 
          #6  0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
          		 
          #7  0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
          		 
           
          		 
          Thread 1 (Thread 0x7eff1024ad00 (LWP 2280226) "mariadbd"):
          		 
          #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) at ./nptl/futex-internal.c:57
          		 
          #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) at ./nptl/futex-internal.c:87
          		 
          #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e47bf348 <COND_wsrep_server_state+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
          		 
          #3  0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e47bf360 <LOCK_wsrep_server_state>, cond=0x5603e47bf320 <COND_wsrep_server_state>) at ./nptl/pthread_cond_wait.c:503
          		 
          #4  ___pthread_cond_wait (cond=0x5603e47bf320 <COND_wsrep_server_state>, mutex=0x5603e47bf360 <LOCK_wsrep_server_state>) at ./nptl/pthread_cond_wait.c:627
          		 
          #5  0x00005603e3708e0b in wsrep::server_state::wait_until_state(std::unique_lock<wsrep::mutex>&, wsrep::server_state::state) const ()
          		 
          #6  0x00005603e340c0ad in wsrep_shutdown_replication() ()
          		 
          #7  0x00005603e2e3b5d7 in mysqld_main(int, char**) ()
          		 
          #8  0x00007eff10587d90 in __libc_start_call_main (main=main@entry=0x5603e2dead10 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffdab8ea7c8) at ../sysdeps/nptl/libc_start_call_main.h:58
          		 
          #9  0x00007eff10587e40 in __libc_start_main_impl (main=0x5603e2dead10 <main>, argc=2, argv=0x7ffdab8ea7c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdab8ea7b8) at ../csu/libc-start.c:392
          		 
          #10 0x00005603e2e2ef45 in _start ()

          ccounotte COUNOTTE CEDRIC added a comment - @Jan Lindström, as requested, output of thread apply all bt: Thread 24 (Thread 0x7efefde1d640 (LWP 2545245 ) "mariadbd" ): # 0 0x00007eff10676d7f in __GI___poll (fds= 0x7efefde1c8c8 , nfds= 1 , timeout= 28800000 ) at ../sysdeps/unix/sysv/linux/poll.c: 29 # 1 0x00005603e32edc9f in vio_io_wait () # 2 0x00005603e32edd74 in vio_socket_io_wait () # 3 0x00005603e32ede33 in vio_read () # 4 0x00005603e327a535 in ?? () # 5 0x00005603e327b9d0 in my_net_read_packet_reallen () # 6 0x00005603e327bac1 in my_net_read_packet () # 7 0x00005603e2f172e2 in do_command(THD*, bool) () # 8 0x00005603e3024f6f in do_handle_one_connection(CONNECT*, bool) () # 9 0x00005603e30252bd in handle_one_connection () # 10 0x00005603e3390db6 in ?? () # 11 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 12 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 23 (Thread 0x7efea56b0640 (LWP 2495611 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private =- 477129360 , cancel= true , abstime= 0x7efea56afc30 , op= 137 , expected= 0 , futex_word= 0x5603e5dddbdc ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private =- 477129360 , abstime= 0x7efea56afc30 , clockid= 0 , expected= 0 , futex_word= 0x5603e5dddbdc ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e5dddbdc , expected=expected @entry = 0 , clockid=clockid @entry = 1 , abstime=abstime @entry = 0x7efea56afc30 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f235d in __pthread_cond_wait_common (abstime= 0x7efea56afc30 , clockid= 1 , mutex= 0x5603e5cc4a00 , cond= 0x5603e5dddbb0 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_clockwait64 (abstime= 0x7efea56afc30 , clockid= 1 , mutex= 0x5603e5cc4a00 , cond= 0x5603e5dddbb0 ) at ./nptl/pthread_cond_wait.c: 691 # 5 ___pthread_cond_clockwait64 (cond= 0x5603e5dddbb0 , mutex= 0x5603e5cc4a00 , clockid= 1 , abstime= 0x7efea56afc30 ) at ./nptl/pthread_cond_wait.c: 679 # 6 0x00005603e3619aaf in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) () # 7 0x00005603e361a608 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) () # 8 0x00005603e361b5a5 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () # 9 0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so. 6 # 10 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 11 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 22 (Thread 0x7ee836ffd640 (LWP 2487980 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private =- 477129360 , cancel= true , abstime= 0x7ee836ffcc30 , op= 137 , expected= 0 , futex_word= 0x5603e5ddd5d8 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private =- 477129360 , abstime= 0x7ee836ffcc30 , clockid= 0 , expected= 0 , futex_word= 0x5603e5ddd5d8 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e5ddd5d8 , expected=expected @entry = 0 , clockid=clockid @entry = 1 , abstime=abstime @entry = 0x7ee836ffcc30 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f235d in __pthread_cond_wait_common (abstime= 0x7ee836ffcc30 , clockid= 1 , mutex= 0x5603e5cc4a00 , cond= 0x5603e5ddd5b0 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_clockwait64 (abstime= 0x7ee836ffcc30 , clockid= 1 , mutex= 0x5603e5cc4a00 , cond= 0x5603e5ddd5b0 ) at ./nptl/pthread_cond_wait.c: 691 # 5 ___pthread_cond_clockwait64 (cond= 0x5603e5ddd5b0 , mutex= 0x5603e5cc4a00 , clockid= 1 , abstime= 0x7ee836ffcc30 ) at ./nptl/pthread_cond_wait.c: 679 # 6 0x00005603e3619aaf in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) () # 7 0x00005603e361a608 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) () # 8 0x00005603e361b5a5 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () # 9 0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so. 6 # 10 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 11 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 21 (Thread 0x7efeec063640 (LWP 2352786 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x7ee83832c508 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x7ee83832c508 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x7ee83832c508 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e5c94f88 , cond= 0x7ee83832c4e0 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x7ee83832c4e0 , mutex= 0x5603e5c94f88 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00007efeff0e1343 in gu_cond_wait_SYS (mutex= 0x5603e5c94f88 , cond= 0x7ee83832c4e0 ) at ./galerautils/src/gu_threads.h: 251 # 6 gu::Lock::wait ( this =<synthetic pointer>, this =<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp: 46 # 7 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter ( this = 0x5603e5c94f88 , obj=...) at ./galera/src/monitor.hpp: 194 # 8 0x00007efeff0ba425 in galera::ReplicatorSMM::commit_order_enter_remote (trx=..., this = 0x5603e5c93bf0 ) at ./galera/src/replicator_smm.cpp: 1315 # 9 galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at ./galera/src/wsrep_provider.cpp: 762 # 10 0x00005603e37214aa in wsrep::wsrep_provider_v26::commit_order_enter(wsrep::ws_handle const &, wsrep::ws_meta const &) () # 11 0x00005603e371c91d in wsrep::transaction::before_commit() () # 12 0x00005603e3155d01 in ha_commit_trans(THD*, bool) () # 13 0x00005603e3034f17 in trans_commit(THD*) () # 14 0x00005603e33ff178 in Wsrep_high_priority_service::commit(wsrep::ws_handle const &, wsrep::ws_meta const &) () # 15 0x00005603e370dde3 in ?? () # 16 0x00005603e3720c55 in ?? () # 17 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply ( this = 0x7ee83832c290 , recv_ctx= 0x7efeec062be0 , apply_cb= 0x5603e3720b70 , meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp: 396 # 18 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec062be0 , ts=...) at ./galera/src/replicator_smm.cpp: 516 # 19 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec062be0 , ts_ptr=...) at ./galera/src/replicator_smm.cpp: 2136 # 20 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop= @0x7efeec06280f : false , act=..., recv_ctx= 0x7efeec062be0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 62 # 21 galera::GcsActionSource::dispatch (exit_loop= @0x7efeec06280f : false , act=..., recv_ctx= 0x7efeec062be0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 110 # 22 galera::GcsActionSource::process ( this = 0x5603e5ca1050 , recv_ctx= 0x7efeec062be0 , exit_loop= @0x7efeec06280f : false ) at ./galera/src/gcs_action_source.cpp: 186 --Type <RET> for more, q to quit, c to continue without paging--c # 23 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec062be0 ) at ./galera/src/replicator_smm.cpp: 402 # 24 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 25 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 26 0x00005603e341a267 in ?? () # 27 0x00005603e340a033 in start_wsrep_THD( void *) () # 28 0x00005603e3390db6 in ?? () # 29 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 30 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 20 (Thread 0x7efeec0ae640 (LWP 2352785 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efeec0ad7c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efeec0ad7c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec0adbe0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 19 (Thread 0x7efeec0f9640 (LWP 2352784 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efeec0f87c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efeec0f87c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec0f8be0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 18 (Thread 0x7efeec144640 (LWP 2352783 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x7ee83010b6b8 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x7ee83010b6b8 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x7ee83010b6b8 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e5c94f88 , cond= 0x7ee83010b690 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x7ee83010b690 , mutex= 0x5603e5c94f88 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00007efeff0e1343 in gu_cond_wait_SYS (mutex= 0x5603e5c94f88 , cond= 0x7ee83010b690 ) at ./galerautils/src/gu_threads.h: 251 # 6 gu::Lock::wait ( this =<synthetic pointer>, this =<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp: 46 # 7 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter ( this = 0x5603e5c94f88 , obj=...) at ./galera/src/monitor.hpp: 194 # 8 0x00007efeff0ba425 in galera::ReplicatorSMM::commit_order_enter_remote (trx=..., this = 0x5603e5c93bf0 ) at ./galera/src/replicator_smm.cpp: 1315 # 9 galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at ./galera/src/wsrep_provider.cpp: 762 # 10 0x00005603e37214aa in wsrep::wsrep_provider_v26::commit_order_enter(wsrep::ws_handle const &, wsrep::ws_meta const &) () # 11 0x00005603e371c91d in wsrep::transaction::before_commit() () # 12 0x00005603e3155d01 in ha_commit_trans(THD*, bool) () # 13 0x00005603e3034f17 in trans_commit(THD*) () # 14 0x00005603e33ff178 in Wsrep_high_priority_service::commit(wsrep::ws_handle const &, wsrep::ws_meta const &) () # 15 0x00005603e370dde3 in ?? () # 16 0x00005603e3720c55 in ?? () # 17 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply ( this = 0x7ee83010b440 , recv_ctx= 0x7efeec143be0 , apply_cb= 0x5603e3720b70 , meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp: 396 # 18 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec143be0 , ts=...) at ./galera/src/replicator_smm.cpp: 516 # 19 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec143be0 , ts_ptr=...) at ./galera/src/replicator_smm.cpp: 2136 # 20 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop= @0x7efeec14380f : false , act=..., recv_ctx= 0x7efeec143be0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 62 # 21 galera::GcsActionSource::dispatch (exit_loop= @0x7efeec14380f : false , act=..., recv_ctx= 0x7efeec143be0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 110 # 22 galera::GcsActionSource::process ( this = 0x5603e5ca1050 , recv_ctx= 0x7efeec143be0 , exit_loop= @0x7efeec14380f : false ) at ./galera/src/gcs_action_source.cpp: 186 # 23 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efeec143be0 ) at ./galera/src/replicator_smm.cpp: 402 # 24 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 25 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 26 0x00005603e341a267 in ?? () # 27 0x00005603e340a033 in start_wsrep_THD( void *) () # 28 0x00005603e3390db6 in ?? () # 29 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 30 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 17 (Thread 0x7efefc053640 (LWP 2352782 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efefc0527c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efefc0527c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc052be0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 16 (Thread 0x7efefc09e640 (LWP 2352781 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efefc09d7c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efefc09d7c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc09dbe0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 15 (Thread 0x7efefc0e9640 (LWP 2352780 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e5c94f08 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e5c94f08 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e5c94f08 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e5c94e98 , cond= 0x5603e5c94ee0 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e5c94ee0 , mutex= 0x5603e5c94e98 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00007efeff0dea1d in gu_cond_wait_SYS (mutex=<optimized out>, cond= 0x5603e5c94ee0 ) at ./galerautils/src/gu_threads.h: 251 # 6 gu::Lock::wait ( this = 0x7efefc0e7ac0 , this = 0x7efefc0e7ac0 , cond=...) at ./galerautils/src/gu_lock.hpp: 46 # 7 galera::Monitor<galera::ReplicatorSMM::ApplyOrder>::drain_common ( this = 0x5603e5c94e98 , seqno=<optimized out>, lock=...) at ./galera/src/monitor.hpp: 564 # 8 0x00007efeff0e40b0 in galera::Monitor<galera::ReplicatorSMM::ApplyOrder>::drain ( this = 0x5603e5c94e98 , seqno= 129147258 ) at ./galera/src/monitor.hpp: 368 # 9 0x00007efeff0d897b in galera::ReplicatorSMM::drain_monitors (upto= 129147258 , this = 0x5603e5c93bf0 ) at ./galera/src/replicator_smm.cpp: 2208 # 10 galera::ReplicatorSMM::drain_monitors_for_local_conf_change ( this = 0x5603e5c93bf0 ) at ./galera/src/replicator_smm.cpp: 2489 # 11 0x00007efeff0dbe4e in galera::ReplicatorSMM::process_non_prim_conf_change (my_index=<optimized out>, conf=..., recv_ctx= 0x7efefc0e8be0 , this = 0x5603e5c93bf0 ) at ./galera/src/replicator_smm.cpp: 2521 # 12 galera::ReplicatorSMM::process_conf_change ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc0e8be0 , conf=..., cc=...) at ./galera/src/replicator_smm.cpp: 2459 # 13 0x00007efeff0f85f1 in galera::GcsActionSource::dispatch (exit_loop=<optimized out>, act=..., recv_ctx= 0x7efefc0e8be0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 127 # 14 galera::GcsActionSource::process ( this = 0x5603e5ca1050 , recv_ctx= 0x7efefc0e8be0 , exit_loop=<optimized out>) at ./galera/src/gcs_action_source.cpp: 186 # 15 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc0e8be0 ) at ./galera/src/replicator_smm.cpp: 402 # 16 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 17 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 18 0x00005603e341a267 in ?? () # 19 0x00005603e340a033 in start_wsrep_THD( void *) () # 20 0x00005603e3390db6 in ?? () # 21 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 22 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 14 (Thread 0x7efefc134640 (LWP 2352779 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efefc1337c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efefc1337c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc133be0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 13 (Thread 0x7efefc17f640 (LWP 2352778 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efefc17e7c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efefc17e7c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc17ebe0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 12 (Thread 0x7efefc1ca640 (LWP 2352777 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efefc1c97c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efefc1c97c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc1c9be0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 11 (Thread 0x7efefc215640 (LWP 2352776 ) "mariadbd" ): # 0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id @entry = 0 , flags=flags @entry = 0 , req=req @entry = 0x7efefc2147c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c: 78 # 1 0x00007eff106486e7 in __GI___nanosleep (req=req @entry = 0x7efefc2147c0 , rem=rem @entry = 0x0 ) at ../sysdeps/unix/sysv/linux/nanosleep.c: 25 # 2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c: 31 # 3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefc214be0 ) at ./galera/src/replicator_smm.cpp: 408 # 4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 6 0x00005603e341a267 in ?? () # 7 0x00005603e340a033 in start_wsrep_THD( void *) () # 8 0x00005603e3390db6 in ?? () # 9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 10 (Thread 0x7efefc260640 (LWP 2352773 ) "mariadbd" ): # 0 0x00007eff105a121a in __GI___sigtimedwait (set= 0x7efefc25fb90 , info= 0x7efefc25fc10 , timeout= 0x0 ) at ../sysdeps/unix/sysv/linux/sigtimedwait.c: 61 # 1 0x00005603e2e3169b in signal_hand () # 2 0x00005603e3390db6 in ?? () # 3 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 4 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 9 (Thread 0x7efefc2ab640 (LWP 2352772 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e3f6efac <mysql_bin_log+ 3276 >) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e3f6efac <mysql_bin_log+ 3276 >) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e3f6efac <mysql_bin_log+ 3276 >, expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e3f6ef50 <mysql_bin_log+ 3184 >, cond= 0x5603e3f6ef80 <mysql_bin_log+ 3232 >) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e3f6ef80 <mysql_bin_log+ 3232 >, mutex= 0x5603e3f6ef50 <mysql_bin_log+ 3184 >) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00005603e324a57e in binlog_background_thread () # 6 0x00005603e3390db6 in ?? () # 7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 8 (Thread 0x7efefc2f6640 (LWP 2352771 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e3f40dac <COND_manager+ 44 >) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e3f40dac <COND_manager+ 44 >) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e3f40dac <COND_manager+ 44 >, expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e3f40dc0 <LOCK_manager>, cond= 0x5603e3f40d80 <COND_manager>) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e3f40d80 <COND_manager>, mutex= 0x5603e3f40dc0 <LOCK_manager>) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00005603e2f01480 in handle_manager () # 6 0x00005603e3390db6 in ?? () # 7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 7 (Thread 0x7efe8e0c7640 (LWP 2352709 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e3ebb89c ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e3ebb89c ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e3ebb89c , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e3ebb800 , cond= 0x5603e3ebb870 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e3ebb870 , mutex= 0x5603e3ebb800 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00005603e358aaa3 in ?? () # 6 0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so. 6 # 7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 6 (Thread 0x7efe8e8c8640 (LWP 2352708 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x7efe8e8c7c60 , op= 393 , expected= 0 , futex_word= 0x5603e47728a8 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x7efe8e8c7c60 , clockid= 1602883208 , expected= 0 , futex_word= 0x5603e47728a8 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e47728a8 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x7efe8e8c7c60 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1f1b in __pthread_cond_wait_common (abstime= 0x7efe8e8c7c60 , clockid= 0 , mutex= 0x5603e47728c0 , cond= 0x5603e4772880 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_timedwait64 (cond= 0x5603e4772880 , mutex= 0x5603e47728c0 , abstime= 0x7efe8e8c7c60 ) at ./nptl/pthread_cond_wait.c: 652 # 5 0x00005603e3325402 in ?? () # 6 0x00005603e331d7be in ?? () # 7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 5 (Thread 0x7efefcefc640 (LWP 2280234 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e5c93dc8 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e5c93dc8 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e5c93dc8 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e5c93d60 , cond= 0x5603e5c93da0 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e5c93da0 , mutex= 0x5603e5c93d60 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00007efeff0cc92c in gu_cond_wait_SYS (mutex= 0x5603e5c93d60 , cond= 0x5603e5c93da0 ) at ./galerautils/src/gu_threads.h: 251 # 6 gu::Lock::wait ( this =<synthetic pointer>, this =<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp: 46 # 7 galera::ReplicatorSMM::wait_for_CLOSED (lock=..., this =<optimized out>) at ./galera/src/replicator_smm.cpp: 285 # 8 galera::ReplicatorSMM::close ( this = 0x5603e5c93bf0 ) at ./galera/src/replicator_smm.cpp: 376 # 9 0x00007efeff0b77a1 in galera_disconnect (gh=<optimized out>) at ./galera/src/wsrep_provider.cpp: 235 # 10 0x00005603e3720506 in wsrep::wsrep_provider_v26::disconnect() () # 11 0x00005603e2e3383a in unireg_abort () # 12 0x00005603e3408298 in wsrep_handle_mdl_conflict(MDL_context*, MDL_ticket const *, MDL_key const *) () # 13 0x00005603e302e2cb in MDL_lock::can_grant_lock(enum_mdl_type, MDL_context*, bool) const () # 14 0x00005603e302e4ea in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) () # 15 0x00005603e302e942 in MDL_context::acquire_lock(MDL_request*, double ) () # 16 0x00005603e302f5ae in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double ) () # 17 0x00005603e2e9f80a in wait_while_table_is_used(THD*, TABLE*, ha_extra_function) () # 18 0x00005603e2fbe618 in mysql_alter_table(THD*, st_mysql_const_lex_string const *, st_mysql_const_lex_string const *, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int , st_order*, bool, bool) () # 19 0x00005603e2fbf454 in mysql_recreate_table(THD*, TABLE_LIST*, bool) () # 20 0x00005603e30301cd in ?? () # 21 0x00005603e30327e8 in ?? () # 22 0x00005603e30342a5 in Sql_cmd_optimize_table::execute(THD*) () # 23 0x00005603e2f11d9c in mysql_execute_command(THD*, bool) () # 24 0x00005603e2f12f57 in mysql_parse(THD*, char *, unsigned int , Parser_state*) () # 25 0x00005603e3269071 in Query_log_event::do_apply_event(rpl_group_info*, char const *, unsigned int ) () # 26 0x00005603e34183c0 in wsrep_apply_events(THD*, Relay_log_info*, void const *, unsigned long ) () # 27 0x00005603e33fee0b in Wsrep_high_priority_service::apply_toi(wsrep::ws_meta const &, wsrep::const_buffer const &, wsrep::mutable_buffer&) () # 28 0x00005603e370ed69 in wsrep::server_state::on_apply(wsrep::high_priority_service&, wsrep::ws_handle const &, wsrep::ws_meta const &, wsrep::const_buffer const &) () # 29 0x00005603e3720c55 in ?? () # 30 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply ( this = 0x7ee73016b410 , recv_ctx= 0x7efefcefbbe0 , apply_cb= 0x5603e3720b70 , meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp: 396 # 31 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefcefbbe0 , ts=...) at ./galera/src/replicator_smm.cpp: 516 # 32 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefcefbbe0 , ts_ptr=...) at ./galera/src/replicator_smm.cpp: 2136 # 33 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop= @0x7efefcefb80f : false , act=..., recv_ctx= 0x7efefcefbbe0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 62 # 34 galera::GcsActionSource::dispatch (exit_loop= @0x7efefcefb80f : false , act=..., recv_ctx= 0x7efefcefbbe0 , this = 0x5603e5ca1050 ) at ./galera/src/gcs_action_source.cpp: 110 # 35 galera::GcsActionSource::process ( this = 0x5603e5ca1050 , recv_ctx= 0x7efefcefbbe0 , exit_loop= @0x7efefcefb80f : false ) at ./galera/src/gcs_action_source.cpp: 186 # 36 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv ( this = 0x5603e5c93bf0 , recv_ctx= 0x7efefcefbbe0 ) at ./galera/src/replicator_smm.cpp: 402 # 37 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp: 263 # 38 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () # 39 0x00005603e341a267 in ?? () # 40 0x00005603e340a033 in start_wsrep_THD( void *) () # 41 0x00005603e3390db6 in ?? () # 42 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 43 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 4 (Thread 0x7efefcf47640 (LWP 2280233 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x7efee40134a0 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x7efee40134a0 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x7efee40134a0 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x7efee4013448 , cond= 0x7efee4013478 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x7efee4013478 , mutex= 0x7efee4013448 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00005603e3419415 in ?? () # 6 0x00005603e340a033 in start_wsrep_THD( void *) () # 7 0x00005603e3390db6 in ?? () # 8 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 9 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 3 (Thread 0x7efefeecd640 (LWP 2280228 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e5c94544 ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e5c94544 ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e5c94544 , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e5c944d8 , cond= 0x5603e5c94518 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e5c94518 , mutex= 0x5603e5c944d8 ) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00007efeff0f55f3 in gu_cond_wait_SYS (mutex= 0x5603e5c944d8 , cond= 0x5603e5c94518 ) at ./galerautils/src/gu_threads.h: 251 # 6 gu::Lock::wait ( this =<synthetic pointer>, this =<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp: 46 # 7 galera::ServiceThd::thd_func (arg= 0x5603e5c944b8 ) at ./galera/src/galera_service_thd.cpp: 31 # 8 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 9 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 2 (Thread 0x7efeff24a640 (LWP 2280227 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 40 , cancel= true , abstime= 0x7efeff249cd0 , op= 393 , expected= 0 , futex_word= 0x5603e480cacc ) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 40 , abstime= 0x7efeff249cd0 , clockid= 3 , expected= 0 , futex_word= 0x5603e480cacc ) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e480cacc , expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x7efeff249cd0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1f1b in __pthread_cond_wait_common (abstime= 0x7efeff249cd0 , clockid= 0 , mutex= 0x5603e480cae0 , cond= 0x5603e480caa0 ) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_timedwait64 (cond= 0x5603e480caa0 , mutex= 0x5603e480cae0 , abstime= 0x7efeff249cd0 ) at ./nptl/pthread_cond_wait.c: 652 # 5 0x00005603e3689d07 in ?? () # 6 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c: 442 # 7 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S: 81   Thread 1 (Thread 0x7eff1024ad00 (LWP 2280226 ) "mariadbd" ): # 0 __futex_abstimed_wait_common64 ( private = 0 , cancel= true , abstime= 0x0 , op= 393 , expected= 0 , futex_word= 0x5603e47bf348 <COND_wsrep_server_state+ 40 >) at ./nptl/futex-internal.c: 57 # 1 __futex_abstimed_wait_common (cancel= true , private = 0 , abstime= 0x0 , clockid= 0 , expected= 0 , futex_word= 0x5603e47bf348 <COND_wsrep_server_state+ 40 >) at ./nptl/futex-internal.c: 87 # 2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word @entry = 0x5603e47bf348 <COND_wsrep_server_state+ 40 >, expected=expected @entry = 0 , clockid=clockid @entry = 0 , abstime=abstime @entry = 0x0 , private = private @entry = 0 ) at ./nptl/futex-internal.c: 139 # 3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime= 0x0 , clockid= 0 , mutex= 0x5603e47bf360 <LOCK_wsrep_server_state>, cond= 0x5603e47bf320 <COND_wsrep_server_state>) at ./nptl/pthread_cond_wait.c: 503 # 4 ___pthread_cond_wait (cond= 0x5603e47bf320 <COND_wsrep_server_state>, mutex= 0x5603e47bf360 <LOCK_wsrep_server_state>) at ./nptl/pthread_cond_wait.c: 627 # 5 0x00005603e3708e0b in wsrep::server_state::wait_until_state(std::unique_lock<wsrep::mutex>&, wsrep::server_state::state) const () # 6 0x00005603e340c0ad in wsrep_shutdown_replication() () # 7 0x00005603e2e3b5d7 in mysqld_main( int , char **) () # 8 0x00007eff10587d90 in __libc_start_call_main (main=main @entry = 0x5603e2dead10 <main>, argc=argc @entry = 2 , argv=argv @entry = 0x7ffdab8ea7c8 ) at ../sysdeps/nptl/libc_start_call_main.h: 58 # 9 0x00007eff10587e40 in __libc_start_main_impl (main= 0x5603e2dead10 <main>, argc= 2 , argv= 0x7ffdab8ea7c8 , init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end= 0x7ffdab8ea7b8 ) at ../csu/libc-start.c: 392 # 10 0x00005603e2e2ef45 in _start ()
          ccounotte COUNOTTE CEDRIC added a comment - - edited

          jplindst
          Does the thread output gives any hint into what's going on ?
          I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue?

          I saw this change: MDEV-29433
          Could it help fix this issue?

          ccounotte COUNOTTE CEDRIC added a comment - - edited jplindst Does the thread output gives any hint into what's going on ? I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue? I saw this change: MDEV-29433 Could it help fix this issue?
          jplindst Jan Lindström (Inactive) added a comment -

          ccounotte I do not find anything clear from information provided. Do you have show processlist from moment when cluster is hung? This could be related to MDEV-29684.

          jplindst Jan Lindström (Inactive) added a comment - ccounotte I do not find anything clear from information provided. Do you have show processlist from moment when cluster is hung? This could be related to MDEV-29684 .
          ccounotte COUNOTTE CEDRIC added a comment -

          Upgrading MariaDB finally solved the issue, I had other issues with 10.6.10/11, less dramatic, and now upgraded to 10.6.12, in hope to resolve the last issues.

          So far didn't experience the cluster hanging any further.

          ----Message d'origine----
          De : Sergei Golubchik (Jira) <jira@mariadb.org>
          Envoyé : lundi 6 mars 2023 09:21
          À : Cédric Counotte <cedric.counotte@1check.com>
          Objet : [JIRA] (MDEV-29388) Galera cluster hanging

          [ https://jira.mariadb.org/browse/MDEV-29388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=235422#comment-235422 ]

          Sergei Golubchik edited comment on MDEV-29388 at 3/6/23 8:20 AM:
          -----------------------------------------------------------------

          jplindst
          Does the thread output gives any hint into what's going on ?
          I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue?

          I saw this change: MDEV-29433
          Could it help fix this issue?

          was (Author: JIRAUSER51971):
          @Jan Lindström
          Does the thread output gives any hint into what's going on ?
          I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue?

          I saw this change: https://jira.mariadb.org/browse/MDEV-29433
          Could it help fix this issue?

          –
          This message was sent by Atlassian Jira
          (v8.20.16#820016)

          ccounotte COUNOTTE CEDRIC added a comment - Upgrading MariaDB finally solved the issue, I had other issues with 10.6.10/11, less dramatic, and now upgraded to 10.6.12, in hope to resolve the last issues. So far didn't experience the cluster hanging any further. ---- Message d'origine ---- De : Sergei Golubchik (Jira) <jira@mariadb.org> Envoyé : lundi 6 mars 2023 09:21 À : Cédric Counotte <cedric.counotte@1check.com> Objet : [JIRA] ( MDEV-29388 ) Galera cluster hanging [ https://jira.mariadb.org/browse/MDEV-29388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=235422#comment-235422 ] Sergei Golubchik edited comment on MDEV-29388 at 3/6/23 8:20 AM: ----------------------------------------------------------------- jplindst Does the thread output gives any hint into what's going on ? I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue? I saw this change: MDEV-29433 Could it help fix this issue? was (Author: JIRAUSER51971): @Jan Lindström Does the thread output gives any hint into what's going on ? I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue? I saw this change: https://jira.mariadb.org/browse/MDEV-29433 Could it help fix this issue? – This message was sent by Atlassian Jira (v8.20.16#820016)
          psumner Phil Sumner added a comment -

          I know this ticket is closed but I've recently experienced something VERY similar and thought it was worth adding to the ticket for future web searchers.

          Environment: 3 node cluster, 10.6.15, CentOS 7.9.2009, 48GB RAM & 16x CPUs each. MaxScale readwritesplitter fronting.

          Symptoms: Sessions pile up on writer Node in commit state. Sessions on one writer node show all wsrep applier threads in "commit" state except one which is executing some SQL statement. `shutdown;` then `kill <pid>` on the MariaDB process got the writer working again, and restarting the affected node saw it do an IST and start working again. For a while. Then one or the other of the readers would have the same issue.

          Cause: a MASSIVE build-up of History List Length. While we were gathering the relevant data (Percona Monitoring) we weren't actively monitoring or alerting on this value. PMM graphs showed that HLL was in the 10s of millions (10,000,000) and higher on both reader nodes. During shutdown, this was being purged (before I killed the process!) and then building up again.

          Why this happened? I'm not entirely sure. Looks like we had a long-running session which caused it to grow, and it never managed to recover. I've since put a lot of effort into monitoring HLL and optimising long-running queries to prevent this from being as much of an issue in the future. I've also set `idle_readonly_transaction_timeout` and `idle_write_transaction_timeout`.

          psumner Phil Sumner added a comment - I know this ticket is closed but I've recently experienced something VERY similar and thought it was worth adding to the ticket for future web searchers. Environment: 3 node cluster, 10.6.15, CentOS 7.9.2009, 48GB RAM & 16x CPUs each. MaxScale readwritesplitter fronting. Symptoms: Sessions pile up on writer Node in commit state. Sessions on one writer node show all wsrep applier threads in "commit" state except one which is executing some SQL statement. `shutdown;` then `kill <pid>` on the MariaDB process got the writer working again, and restarting the affected node saw it do an IST and start working again. For a while. Then one or the other of the readers would have the same issue. Cause: a MASSIVE build-up of History List Length. While we were gathering the relevant data (Percona Monitoring) we weren't actively monitoring or alerting on this value. PMM graphs showed that HLL was in the 10s of millions (10,000,000) and higher on both reader nodes. During shutdown, this was being purged (before I killed the process!) and then building up again. Why this happened? I'm not entirely sure. Looks like we had a long-running session which caused it to grow, and it never managed to recover. I've since put a lot of effort into monitoring HLL and optimising long-running queries to prevent this from being as much of an issue in the future. I've also set `idle_readonly_transaction_timeout` and `idle_write_transaction_timeout`.

          People

            sysprg Julius Goryavsky
            ccounotte COUNOTTE CEDRIC
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.