Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.6.9
-
None
Description
On a 3-nodes cluster, 2 nodes got completely stuck. Anything trying to connect would also get stuck. The third node had 200+ (or 500?) queries running, but stuck as well.
Stopping those 2 servers got stuck to, had to kill the processes manually. Logs shows the server being disconnected from the cluster. Cluster size went from 3 to 1 as it should, but mariadb processes where stuck on those 2 servers until manually killed.
Starting first node back after a minute got it to sync using an IST, the other node restarted after 5 minutes synced using an SST. As if nothing happened!?
Digging into logs of the 3 servers shows nothing, no output at all during the whole period they where stuck.
Anything I could do to track this down if it happens again?
Server 1 log (stuck):
2022-08-25 0:29:16 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000191 |
2022-08-25 0:29:32 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000192 of size 134217696 bytes |
2022-08-25 0:29:39 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000192 |
2022-08-25 0:30:38 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000193 of size 134217696 bytes |
2022-08-25 0:30:47 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000194 of size 134217696 bytes |
2022-08-25 0:30:56 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000193 |
2022-08-25 0:30:57 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000194 |
2022-08-25 0:32:23 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000195 of size 134217696 bytes |
2022-08-25 1:03:21 0 [Note] WSREP: declaring 19edde19-9435 at tcp://xxxx:4567 stable |
2022-08-25 1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567) |
2022-08-25 1:03:21 0 [Note] WSREP: Node 19edde19-9435 state prim |
2022-08-25 1:03:21 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,63) memb { |
2022-08-25 1:03:21 0 [Note] WSREP: save pc into disk |
2022-08-25 1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567) |
2022-08-25 1:03:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 0 (ovh7.1check.com) |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 1 (ovh1.1check.com) |
2022-08-25 1:03:21 0 [Note] WSREP: Quorum results: |
2022-08-25 1:03:21 0 [Note] WSREP: Flow-control interval: [23, 23] |
2022-08-25 1:03:26 0 [Note] WSREP: cleaning up e26a09b5-b02e (tcp://xxxx:4567) |
2022-08-25 1:06:00 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown |
2022-08-25 1:06:00 0 [Note] WSREP: Shutdown replication |
2022-08-25 1:06:00 0 [Note] WSREP: Server status change synced -> disconnecting |
2022-08-25 1:06:00 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
2022-08-25 1:06:00 0 [Note] WSREP: Closing send monitor... |
2022-08-25 1:06:00 0 [Note] WSREP: Closed send monitor. |
2022-08-25 1:06:00 0 [Note] WSREP: gcomm: terminating thread |
2022-08-25 1:06:00 0 [Note] WSREP: gcomm: joining thread |
2022-08-25 1:06:00 0 [Note] WSREP: gcomm: closing backend |
2022-08-25 1:06:01 0 [Note] WSREP: view(view_id(NON_PRIM,19edde19-9435,63) memb { |
2022-08-25 1:06:01 0 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2022-08-25 1:06:01 0 [Note] WSREP: view((empty)) |
2022-08-25 1:06:01 0 [Note] WSREP: gcomm: closed |
2022-08-25 1:06:01 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2022-08-25 1:06:01 0 [Note] WSREP: Flow-control interval: [16, 16] |
2022-08-25 1:06:01 0 [Note] WSREP: Received NON-PRIMARY. |
2022-08-25 1:06:01 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 70213440) |
2022-08-25 1:06:01 0 [Note] WSREP: New SELF-LEAVE. |
2022-08-25 1:06:01 0 [Note] WSREP: Flow-control interval: [0, 0] |
2022-08-25 1:06:01 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. |
2022-08-25 1:06:01 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 70213440) |
2022-08-25 1:06:01 0 [Note] WSREP: RECV thread exiting 0: Success |
2022-08-25 1:06:01 0 [Note] WSREP: recv_thread() joined. |
2022-08-25 1:06:01 0 [Note] WSREP: Closing replication queue. |
2022-08-25 1:06:01 0 [Note] WSREP: Closing slave action queue. |
2022-08-25 1:07:18 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 2c96698a-0fdf-11ed-90d6-7ecc8fa70984:70213386 |
2022-08-25 1:07:18 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' |
Server 2 log (stuck):
2022-08-25 0:29:32 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000192 of size 134217696 bytes |
2022-08-25 0:29:39 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000192 |
2022-08-25 0:30:38 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000193 of size 134217696 bytes |
2022-08-25 0:30:47 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000194 of size 134217696 bytes |
2022-08-25 0:30:56 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000193 |
2022-08-25 0:30:57 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000194 |
2022-08-25 0:32:23 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000195 of size 134217696 bytes |
2022-08-25 1:03:21 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown |
2022-08-25 1:03:21 0 [Note] WSREP: Shutdown replication |
2022-08-25 1:03:21 0 [Note] WSREP: Server status change synced -> disconnecting |
2022-08-25 1:03:21 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
2022-08-25 1:03:21 0 [Note] WSREP: Closing send monitor... |
2022-08-25 1:03:21 0 [Note] WSREP: Closed send monitor. |
2022-08-25 1:03:21 0 [Note] WSREP: gcomm: terminating thread |
2022-08-25 1:03:21 0 [Note] WSREP: gcomm: joining thread |
2022-08-25 1:03:21 0 [Note] WSREP: gcomm: closing backend |
2022-08-25 1:03:21 0 [Note] WSREP: view(view_id(NON_PRIM,19edde19-9435,62) memb { |
2022-08-25 1:03:21 0 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2022-08-25 1:03:21 0 [Note] WSREP: view((empty)) |
2022-08-25 1:03:21 0 [Note] WSREP: gcomm: closed |
2022-08-25 1:03:21 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2022-08-25 1:03:21 0 [Note] WSREP: Flow-control interval: [16, 16] |
2022-08-25 1:03:21 0 [Note] WSREP: Received NON-PRIMARY. |
2022-08-25 1:03:21 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 70213433) |
2022-08-25 1:03:21 0 [Note] WSREP: New SELF-LEAVE. |
2022-08-25 1:03:21 0 [Note] WSREP: Flow-control interval: [0, 0] |
2022-08-25 1:03:21 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. |
2022-08-25 1:03:21 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 70213433) |
2022-08-25 1:03:21 0 [Note] WSREP: RECV thread exiting 0: Success |
2022-08-25 1:03:21 0 [Note] WSREP: recv_thread() joined. |
2022-08-25 1:03:21 0 [Note] WSREP: Closing replication queue. |
2022-08-25 1:03:21 0 [Note] WSREP: Closing slave action queue. |
2022-08-25 1:08:05 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 2c96698a-0fdf-11ed-90d6-7ecc8fa70984:70213385 |
2022-08-25 1:08:05 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' |
2022-08-25 1:08:05 0 [Note] WSREP: wsrep_load(): Galera 26.4.12(r1eac5b64) by Codership Oy <info@codership.com> loaded successfully. |
2022-08-25 1:08:05 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. |
Server 3 log (still alive):
2022-08-25 0:30:56 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000389 |
2022-08-25 0:30:57 0 [Note] WSREP: Deleted page /var/lib/mysql/galera.page.000390 |
2022-08-25 0:32:23 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000391 of size 134217696 bytes |
2022-08-25 1:03:21 0 [Note] WSREP: declaring 3e14de76-94d4 at tcp://xxxx:4567 stable |
2022-08-25 1:03:21 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://xxxx:44208 |
2022-08-25 1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567) |
2022-08-25 1:03:21 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7fd768845210 |
2022-08-25 1:03:21 0 [Note] WSREP: Deferred close timer destruct |
2022-08-25 1:03:21 0 [Note] WSREP: Node 19edde19-9435 state prim |
2022-08-25 1:03:21 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,63) memb { |
2022-08-25 1:03:21 0 [Note] WSREP: save pc into disk |
2022-08-25 1:03:21 0 [Note] WSREP: forgetting e26a09b5-b02e (tcp://xxxx:4567) |
2022-08-25 1:03:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 |
2022-08-25 1:03:21 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: f2ffea9d-2400-11ed-9136-ef240a6d8bee |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 0 (ovh7.1check.com) |
2022-08-25 1:03:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: f2ffea9d-2400-11ed-9136-ef240a6d8bee from 1 (ovh1.1check.com) |
2022-08-25 1:03:21 0 [Note] WSREP: Quorum results: |
2022-08-25 1:03:21 0 [Note] WSREP: Flow-control interval: [23, 23] |
2022-08-25 1:03:21 17 [Note] WSREP: ####### processing CC 70213434, local, ordered |
2022-08-25 1:03:21 17 [Note] WSREP: ####### My UUID: 19edde19-2139-11ed-9435-e69b7d8e37e9 |
2022-08-25 1:03:21 17 [Note] WSREP: Skipping cert index reset |
2022-08-25 1:03:21 17 [Note] WSREP: REPL Protocols: 10 (5) |
2022-08-25 1:03:21 17 [Note] WSREP: ####### Adjusting cert position: 70213433 -> 70213434 |
2022-08-25 1:03:21 0 [Note] WSREP: Service thread queue flushed. |
2022-08-25 1:03:21 17 [Note] WSREP: ================================================ |
2022-08-25 1:03:21 17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
2022-08-25 1:03:21 17 [Note] WSREP: Lowest cert index boundary for CC from group: 70213347 |
2022-08-25 1:03:21 17 [Note] WSREP: Min available from gcache for CC from group: 70213347 |
2022-08-25 1:03:26 0 [Note] WSREP: cleaning up e26a09b5-b02e (tcp://xxxx:4567) |
2022-08-25 1:06:01 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://xxxx:36660 |
2022-08-25 1:06:01 0 [Note] WSREP: forgetting 3e14de76-94d4 (tcp://xxxx:4567) |
2022-08-25 1:06:01 0 [Note] WSREP: Node 19edde19-9435 state prim |
2022-08-25 1:06:01 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,64) memb { |
2022-08-25 1:06:01 0 [Note] WSREP: save pc into disk |
2022-08-25 1:06:01 0 [Note] WSREP: forgetting 3e14de76-94d4 (tcp://xxxx:4567) |
2022-08-25 1:06:01 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 |
2022-08-25 1:06:01 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7fd7344f9160 |
2022-08-25 1:06:01 0 [Note] WSREP: Deferred close timer destruct |
2022-08-25 1:06:01 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 523fb653-2401-11ed-8253-82473e2187ea |
2022-08-25 1:06:01 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 523fb653-2401-11ed-8253-82473e2187ea |
2022-08-25 1:06:01 0 [Note] WSREP: STATE EXCHANGE: got state msg: 523fb653-2401-11ed-8253-82473e2187ea from 0 (ovh7.1check.com) |
2022-08-25 1:06:01 0 [Note] WSREP: Quorum results: |
2022-08-25 1:06:01 0 [Note] WSREP: Flow-control interval: [16, 16] |
2022-08-25 1:06:01 17 [Note] WSREP: ####### processing CC 70213441, local, ordered |
2022-08-25 1:06:01 17 [Note] WSREP: ####### My UUID: 19edde19-2139-11ed-9435-e69b7d8e37e9 |
2022-08-25 1:06:01 17 [Note] WSREP: Skipping cert index reset |
2022-08-25 1:06:01 17 [Note] WSREP: REPL Protocols: 10 (5) |
2022-08-25 1:06:01 17 [Note] WSREP: ####### Adjusting cert position: 70213440 -> 70213441 |
2022-08-25 1:06:01 0 [Note] WSREP: Service thread queue flushed. |
2022-08-25 1:06:01 17 [Note] WSREP: ================================================ |
2022-08-25 1:06:01 17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
2022-08-25 1:06:01 17 [Note] WSREP: Lowest cert index boundary for CC from group: 70213347 |
2022-08-25 1:06:01 17 [Note] WSREP: Min available from gcache for CC from group: 70213347 |
2022-08-25 1:06:06 0 [Note] WSREP: cleaning up 3e14de76-94d4 (tcp://xxxx:4567) |
2022-08-25 1:06:17 16935047 [Warning] Aborted connection 16935047 to db: '_1check_mail_export' user: 'bo_1check' host: '127.0.0.1' (Got an error reading communication packets) |
2022-08-25 1:06:17 16935044 [Warning] Aborted connection 16935044 to db: '_1check_mail_export' user: 'bo_1check' host: '127.0.0.1' (Got an error reading communication packets) |
2022-08-25 1:07:12 0 [Note] WSREP: Created page /var/lib/mysql/galera.page.000392 of size 134217696 bytes |
2022-08-25 1:07:18 0 [Note] WSREP: (19edde19-9435, 'tcp://0.0.0.0:4567') connection established to 807a59e3-896d tcp://xxxx:4567 |
2022-08-25 1:07:18 0 [Note] WSREP: (19edde19-9435, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: |
2022-08-25 1:07:19 0 [Note] WSREP: declaring 807a59e3-896d at tcp://xxxx:4567 stable |
2022-08-25 1:07:19 0 [Note] WSREP: Node 19edde19-9435 state prim |
2022-08-25 1:07:19 0 [Note] WSREP: view(view_id(PRIM,19edde19-9435,65) memb { |
2022-08-25 1:07:19 0 [Note] WSREP: save pc into disk |
2022-08-25 1:07:19 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 |
2022-08-25 1:07:19 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 80c6eee7-2401-11ed-a085-a2c1fb1fa522 |
2022-08-25 1:07:19 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 80c6eee7-2401-11ed-a085-a2c1fb1fa522 |
2022-08-25 1:07:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: 80c6eee7-2401-11ed-a085-a2c1fb1fa522 from 0 (ovh7.1check.com) |
2022-08-25 1:07:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: 80c6eee7-2401-11ed-a085-a2c1fb1fa522 from 1 (ovh1.1check.com) |
2022-08-25 1:07:19 0 [Note] WSREP: Quorum results: |
2022-08-25 1:07:19 0 [Note] WSREP: Flow-control interval: [23, 23] |
2022-08-25 1:07:19 17 [Note] WSREP: ####### processing CC 70224411, local, ordered |
2022-08-25 1:07:19 17 [Note] WSREP: ####### My UUID: 19edde19-2139-11ed-9435-e69b7d8e37e9 |
2022-08-25 1:07:19 17 [Note] WSREP: Skipping cert index reset |
2022-08-25 1:07:19 17 [Note] WSREP: REPL Protocols: 10 (5) |
2022-08-25 1:07:19 17 [Note] WSREP: ####### Adjusting cert position: 70224410 -> 70224411 |
2022-08-25 1:07:19 0 [Note] WSREP: Service thread queue flushed. |
2022-08-25 1:07:19 17 [Note] WSREP: ================================================ |
2022-08-25 1:07:19 17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
2022-08-25 1:07:19 17 [Note] WSREP: Lowest cert index boundary for CC from group: 70213347 |
2022-08-25 1:07:19 17 [Note] WSREP: Min available from gcache for CC from group: 70213347 |
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Fix Version/s | 10.6 [ 24028 ] | |
| Assignee | Jan Lindström [ jplindst ] |
| Status | Open [ 1 ] | Needs Feedback [ 10501 ] |
The issue occurred again last night at 00:16 on a single server, out of a 3 nodes cluster.
Running that command (or any other) on that server simply hangs permanently: sudo mysql --user=root -e "show status like 'wsrep%';"
I had to stop the service, kill the process manually and restart the service quickly so that the server did an IST to resync.
Other servers responded to the above command, however no queries (be it a simple select or anything else) were completing, except the simple show status.
Logs are as attached in description, plain empty, until I run 'systemctl stop mariadb'.
Cannot use debugger on production servers while our service is entirely down, even though one server is stuck.
The issue also occured a few days ago on our spare cluster, so I'll be sure to attach debugger to the stuck node when that happens again.
@cedric, do you see something similar in the jira that i reported over here MDEV-29346.
you will see a system user thread stuck in one of the processlist. I suspect the flow control kick in and stall the whole cluster.
our gcs.fc_limit is 16.
@Ping, your issue looks exactly like the one I'm seeing. One server hangs and the other cannot write anything to the DB, with only option to stop/kill hanging server and start it again.
I never looked at the process list, mainly because the hanging server cannot be accessed any more (connection attempts hang permanently) when that happens, but I will surely take a look at it from the remaining servers to see if I can get more info.
Where or how did you see those galera logs though? Never seen them and when server hangs, nothing is logged at all. I assume MariaDB logging level?
@cedric, those are our internal monitoring that spew out some of the galera wsrep status every 5mins.
I believe you should be able to still connect to it just it takes a little bit longer.
I suspect it has to do with the flow control kicking in, our setting is 16. When one of the node's receive queue exceed 16 , the whole replication will stop. Wondering anyone from MariaDB can help comfirm this is working as design or it is a bug?
Referencing this galera google forum : https://groups.google.com/g/codership-team/c/KJI1JyE9tuI
Flow control concept : https://www.percona.com/blog/2013/05/02/galera-flow-control-in-percona-xtradb-cluster-for-mysql/
As the hanging occurs always at the same time (at night around 0:10 when we clean-up tables), I've tested the clean-up script further and reproduce the hanging by running it during the day!
It always happens when large DELETE occurs and when the server is busy enough. When scrip run at night it'll hang the server after a few days, but during the day I'm sure to get it stuck. Sadly I have to kill the server to avoid hanging the whole cluster in a matter of seconds.
We have 3 tables containing large historical data, which are cleaned-up nightly, one is cleaned of 6 months old data, the other is cleaned of 3+ days old data. The later table may contain gigabyte of data per day. And we have such model for 300+ customers (in different DBs).
On a 3 server node, ran the script on node 1, nodes 1 and 2 got stuck, restarted node 1 and got into debugger of node 2 which was "stuck" differently than before. This node is hanging everything, thinking it's still part of a 3-node cluster.
Here are what I could gather from it using gdb:
Threads:
* 1 Thread 0x7eff1024ad00 (LWP 2280226) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, |
futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) at ./nptl/futex-internal.c:57 |
2 Thread 0x7efeff24a640 (LWP 2280227) "mariadbd" __futex_abstimed_wait_common64 (private=40, cancel=true, abstime=0x7efeff249cd0, op=393, expected=0, |
futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:57 |
3 Thread 0x7efefeecd640 (LWP 2280228) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94544) |
at ./nptl/futex-internal.c:57 |
4 Thread 0x7efefcf47640 (LWP 2280233) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7efee40134a0) |
at ./nptl/futex-internal.c:57 |
5 Thread 0x7efefcefc640 (LWP 2280234) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c93dc8) |
at ./nptl/futex-internal.c:57 |
6 Thread 0x7efe8e8c8640 (LWP 2352708) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7efe8e8c7c60, op=393, expected=0, |
futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:57 |
7 Thread 0x7efe8e0c7640 (LWP 2352709) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3ebb89c) |
at ./nptl/futex-internal.c:57 |
8 Thread 0x7efefc2f6640 (LWP 2352771) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, |
futex_word=0x5603e3f40dac <COND_manager+44>) at ./nptl/futex-internal.c:57 |
9 Thread 0x7efefc2ab640 (LWP 2352772) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, |
futex_word=0x5603e3f6efac <mysql_bin_log+3276>) at ./nptl/futex-internal.c:57 |
10 Thread 0x7efefc260640 (LWP 2352773) "mariadbd" 0x00007eff105a121a in __GI___sigtimedwait (set=0x7efefc25fb90, info=0x7efefc25fc10, timeout=0x0) |
at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61 |
11 Thread 0x7efefc215640 (LWP 2352776) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, |
req=req@entry=0x7efefc2147c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
12 Thread 0x7efefc1ca640 (LWP 2352777) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, |
req=req@entry=0x7efefc1c97c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
13 Thread 0x7efefc17f640 (LWP 2352778) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, |
req=req@entry=0x7efefc17e7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
14 Thread 0x7efefc134640 (LWP 2352779) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, |
req=req@entry=0x7efefc1337c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
15 Thread 0x7efefc0e9640 (LWP 2352780) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94f08) |
at ./nptl/futex-internal.c:57 |
16 Thread 0x7efefc09e640 (LWP 2352781) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, |
req=req@entry=0x7efefc09d7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
17 Thread 0x7efefc053640 (LWP 2352782) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, |
req=req@entry=0x7efefc0527c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
18 Thread 0x7efeec144640 (LWP 2352783) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ee83010b6b8) |
at ./nptl/futex-internal.c:57 |
--Type <RET> for more, q to quit, c to continue without paging--c |
19 Thread 0x7efeec0f9640 (LWP 2352784) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efeec0f87c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
20 Thread 0x7efeec0ae640 (LWP 2352785) "mariadbd" 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efeec0ad7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
21 Thread 0x7efeec063640 (LWP 2352786) "mariadbd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ee83832c508) at ./nptl/futex-internal.c:57 |
22 Thread 0x7ee836ffd640 (LWP 2487980) "mariadbd" __futex_abstimed_wait_common64 (private=-477129360, cancel=true, abstime=0x7ee836ffcc30, op=137, expected=0, futex_word=0x5603e5ddd5d8) at ./nptl/futex-internal.c:57 |
23 Thread 0x7efea56b0640 (LWP 2495611) "mariadbd" __futex_abstimed_wait_common64 (private=-477129360, cancel=true, abstime=0x7efea56afc30, op=137, expected=0, futex_word=0x5603e5dddbdc) at ./nptl/futex-internal.c:57 |
24 Thread 0x7efefde1d640 (LWP 2545245) "mariadbd" 0x00007eff10676d7f in __GI___poll (fds=0x7efefde1c8c8, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29 |
Here are 3 threads stack trace:
(gdb) thread 1 |
[Switching to thread 1 (Thread 0x7eff1024ad00 (LWP 2280226))] |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) |
at ./nptl/futex-internal.c:57 |
57 in ./nptl/futex-internal.c |
(gdb) bt
|
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) |
at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) |
at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e47bf348 <COND_wsrep_server_state+40>, expected=expected@entry=0, clockid=clockid@entry=0, |
abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e47bf360 <LOCK_wsrep_server_state>, cond=0x5603e47bf320 <COND_wsrep_server_state>) |
at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e47bf320 <COND_wsrep_server_state>, mutex=0x5603e47bf360 <LOCK_wsrep_server_state>) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e3708e0b in wsrep::server_state::wait_until_state(std::unique_lock<wsrep::mutex>&, wsrep::server_state::state) const () |
#6 0x00005603e340c0ad in wsrep_shutdown_replication() () |
#7 0x00005603e2e3b5d7 in mysqld_main(int, char**) () |
#8 0x00007eff10587d90 in __libc_start_call_main (main=main@entry=0x5603e2dead10 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffdab8ea7c8) |
at ../sysdeps/nptl/libc_start_call_main.h:58 |
#9 0x00007eff10587e40 in __libc_start_main_impl (main=0x5603e2dead10 <main>, argc=2, argv=0x7ffdab8ea7c8, init=<optimized out>, fini=<optimized out>, |
rtld_fini=<optimized out>, stack_end=0x7ffdab8ea7b8) at ../csu/libc-start.c:392 |
#10 0x00005603e2e2ef45 in _start () |
(gdb) thread 2 |
[Switching to thread 2 (Thread 0x7efeff24a640 (LWP 2280227))] |
#0 __futex_abstimed_wait_common64 (private=40, cancel=true, abstime=0x7efeff249cd0, op=393, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:57 |
57 in ./nptl/futex-internal.c |
(gdb) bt
|
#0 __futex_abstimed_wait_common64 (private=40, cancel=true, abstime=0x7efeff249cd0, op=393, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=40, abstime=0x7efeff249cd0, clockid=3, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e480cacc, expected=expected@entry=0, clockid=clockid@entry=0, |
abstime=abstime@entry=0x7efeff249cd0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1f1b in __pthread_cond_wait_common (abstime=0x7efeff249cd0, clockid=0, mutex=0x5603e480cae0, cond=0x5603e480caa0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_timedwait64 (cond=0x5603e480caa0, mutex=0x5603e480cae0, abstime=0x7efeff249cd0) at ./nptl/pthread_cond_wait.c:652 |
#5 0x00005603e3689d07 in ?? () |
#6 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#7 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
(gdb) thread 3 |
[Switching to thread 3 (Thread 0x7efefeecd640 (LWP 2280228))] |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:57 |
57 in ./nptl/futex-internal.c |
(gdb) bt
|
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c94544, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, |
private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c944d8, cond=0x5603e5c94518) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e5c94518, mutex=0x5603e5c944d8) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0f55f3 in gu_cond_wait_SYS (mutex=0x5603e5c944d8, cond=0x5603e5c94518) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::ServiceThd::thd_func (arg=0x5603e5c944b8) at ./galera/src/galera_service_thd.cpp:31 |
#8 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#9 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
and threads 4 to 6:
(gdb) thread 4 |
[Switching to thread 4 (Thread 0x7efefcf47640 (LWP 2280233))] |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:57 |
57 in ./nptl/futex-internal.c |
(gdb) bt
|
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7efee40134a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, |
private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7efee4013448, cond=0x7efee4013478) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x7efee4013478, mutex=0x7efee4013448) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e3419415 in ?? () |
#6 0x00005603e340a033 in start_wsrep_THD(void*) () |
#7 0x00005603e3390db6 in ?? () |
#8 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#9 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
(gdb) thread 5 |
[Switching to thread 5 (Thread 0x7efefcefc640 (LWP 2280234))] |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:57 |
57 in ./nptl/futex-internal.c |
(gdb) bt
|
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c93dc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, |
private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c93d60, cond=0x5603e5c93da0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e5c93da0, mutex=0x5603e5c93d60) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0cc92c in gu_cond_wait_SYS (mutex=0x5603e5c93d60, cond=0x5603e5c93da0) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::ReplicatorSMM::wait_for_CLOSED (lock=..., this=<optimized out>) at ./galera/src/replicator_smm.cpp:285 |
#8 galera::ReplicatorSMM::close (this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:376 |
#9 0x00007efeff0b77a1 in galera_disconnect (gh=<optimized out>) at ./galera/src/wsrep_provider.cpp:235 |
#10 0x00005603e3720506 in wsrep::wsrep_provider_v26::disconnect() () |
#11 0x00005603e2e3383a in unireg_abort () |
#12 0x00005603e3408298 in wsrep_handle_mdl_conflict(MDL_context*, MDL_ticket const*, MDL_key const*) () |
#13 0x00005603e302e2cb in MDL_lock::can_grant_lock(enum_mdl_type, MDL_context*, bool) const () |
#14 0x00005603e302e4ea in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) () |
#15 0x00005603e302e942 in MDL_context::acquire_lock(MDL_request*, double) () |
#16 0x00005603e302f5ae in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) () |
#17 0x00005603e2e9f80a in wait_while_table_is_used(THD*, TABLE*, ha_extra_function) () |
#18 0x00005603e2fbe618 in mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool) () |
#19 0x00005603e2fbf454 in mysql_recreate_table(THD*, TABLE_LIST*, bool) () |
#20 0x00005603e30301cd in ?? () |
#21 0x00005603e30327e8 in ?? () |
#22 0x00005603e30342a5 in Sql_cmd_optimize_table::execute(THD*) () |
#23 0x00005603e2f11d9c in mysql_execute_command(THD*, bool) () |
#24 0x00005603e2f12f57 in mysql_parse(THD*, char*, unsigned int, Parser_state*) () |
#25 0x00005603e3269071 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) () |
#26 0x00005603e34183c0 in wsrep_apply_events(THD*, Relay_log_info*, void const*, unsigned long) () |
#27 0x00005603e33fee0b in Wsrep_high_priority_service::apply_toi(wsrep::ws_meta const&, wsrep::const_buffer const&, wsrep::mutable_buffer&) () |
#28 0x00005603e370ed69 in wsrep::server_state::on_apply(wsrep::high_priority_service&, wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&) () |
#29 0x00005603e3720c55 in ?? () |
#30 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee73016b410, recv_ctx=0x7efefcefbbe0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) |
at ./galera/src/trx_handle.cpp:396 |
#31 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0, ts=...) at ./galera/src/replicator_smm.cpp:516 |
#32 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136 |
#33 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efefcefb80f: false, act=..., recv_ctx=0x7efefcefbbe0, this=0x5603e5ca1050) |
at ./galera/src/gcs_action_source.cpp:62 |
#34 galera::GcsActionSource::dispatch (exit_loop=@0x7efefcefb80f: false, act=..., recv_ctx=0x7efefcefbbe0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110 |
#35 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efefcefbbe0, exit_loop=@0x7efefcefb80f: false) at ./galera/src/gcs_action_source.cpp:186 |
#36 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0) at ./galera/src/replicator_smm.cpp:402 |
#37 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#38 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#39 0x00005603e341a267 in ?? () |
#40 0x00005603e340a033 in start_wsrep_THD(void*) () |
#41 0x00005603e3390db6 in ?? () |
#42 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#43 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
(gdb) thread 6 |
[Switching to thread 6 (Thread 0x7efe8e8c8640 (LWP 2352708))] |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7efe8e8c7c60, op=393, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:57 |
57 in ./nptl/futex-internal.c |
(gdb) bt
|
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7efe8e8c7c60, op=393, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7efe8e8c7c60, clockid=1602883208, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e47728a8, expected=expected@entry=0, clockid=clockid@entry=0, |
abstime=abstime@entry=0x7efe8e8c7c60, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1f1b in __pthread_cond_wait_common (abstime=0x7efe8e8c7c60, clockid=0, mutex=0x5603e47728c0, cond=0x5603e4772880) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_timedwait64 (cond=0x5603e4772880, mutex=0x5603e47728c0, abstime=0x7efe8e8c7c60) at ./nptl/pthread_cond_wait.c:652 |
#5 0x00005603e3325402 in ?? () |
#6 0x00005603e331d7be in ?? () |
#7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
@Jan Lindström, as requested, output of thread apply all bt:
Thread 24 (Thread 0x7efefde1d640 (LWP 2545245) "mariadbd"): |
#0 0x00007eff10676d7f in __GI___poll (fds=0x7efefde1c8c8, nfds=1, timeout=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29 |
#1 0x00005603e32edc9f in vio_io_wait () |
#2 0x00005603e32edd74 in vio_socket_io_wait () |
#3 0x00005603e32ede33 in vio_read () |
#4 0x00005603e327a535 in ?? () |
#5 0x00005603e327b9d0 in my_net_read_packet_reallen () |
#6 0x00005603e327bac1 in my_net_read_packet () |
#7 0x00005603e2f172e2 in do_command(THD*, bool) () |
#8 0x00005603e3024f6f in do_handle_one_connection(CONNECT*, bool) () |
#9 0x00005603e30252bd in handle_one_connection () |
#10 0x00005603e3390db6 in ?? () |
#11 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#12 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 23 (Thread 0x7efea56b0640 (LWP 2495611) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=-477129360, cancel=true, abstime=0x7efea56afc30, op=137, expected=0, futex_word=0x5603e5dddbdc) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=-477129360, abstime=0x7efea56afc30, clockid=0, expected=0, futex_word=0x5603e5dddbdc) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5dddbdc, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7efea56afc30, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f235d in __pthread_cond_wait_common (abstime=0x7efea56afc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5dddbb0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_clockwait64 (abstime=0x7efea56afc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5dddbb0) at ./nptl/pthread_cond_wait.c:691 |
#5 ___pthread_cond_clockwait64 (cond=0x5603e5dddbb0, mutex=0x5603e5cc4a00, clockid=1, abstime=0x7efea56afc30) at ./nptl/pthread_cond_wait.c:679 |
#6 0x00005603e3619aaf in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) () |
#7 0x00005603e361a608 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) () |
#8 0x00005603e361b5a5 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () |
#9 0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 |
#10 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#11 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 22 (Thread 0x7ee836ffd640 (LWP 2487980) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=-477129360, cancel=true, abstime=0x7ee836ffcc30, op=137, expected=0, futex_word=0x5603e5ddd5d8) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=-477129360, abstime=0x7ee836ffcc30, clockid=0, expected=0, futex_word=0x5603e5ddd5d8) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5ddd5d8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7ee836ffcc30, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f235d in __pthread_cond_wait_common (abstime=0x7ee836ffcc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5ddd5b0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_clockwait64 (abstime=0x7ee836ffcc30, clockid=1, mutex=0x5603e5cc4a00, cond=0x5603e5ddd5b0) at ./nptl/pthread_cond_wait.c:691 |
#5 ___pthread_cond_clockwait64 (cond=0x5603e5ddd5b0, mutex=0x5603e5cc4a00, clockid=1, abstime=0x7ee836ffcc30) at ./nptl/pthread_cond_wait.c:679 |
#6 0x00005603e3619aaf in tpool::thread_pool_generic::wait_for_tasks(std::unique_lock<std::mutex>&, tpool::worker_data*) () |
#7 0x00005603e361a608 in tpool::thread_pool_generic::get_task(tpool::worker_data*, tpool::task**) () |
#8 0x00005603e361b5a5 in tpool::thread_pool_generic::worker_main(tpool::worker_data*) () |
#9 0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 |
#10 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#11 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 21 (Thread 0x7efeec063640 (LWP 2352786) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ee83832c508) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ee83832c508) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ee83832c508, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c94f88, cond=0x7ee83832c4e0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x7ee83832c4e0, mutex=0x5603e5c94f88) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0e1343 in gu_cond_wait_SYS (mutex=0x5603e5c94f88, cond=0x7ee83832c4e0) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x5603e5c94f88, obj=...) at ./galera/src/monitor.hpp:194 |
#8 0x00007efeff0ba425 in galera::ReplicatorSMM::commit_order_enter_remote (trx=..., this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:1315 |
#9 galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at ./galera/src/wsrep_provider.cpp:762 |
#10 0x00005603e37214aa in wsrep::wsrep_provider_v26::commit_order_enter(wsrep::ws_handle const&, wsrep::ws_meta const&) () |
#11 0x00005603e371c91d in wsrep::transaction::before_commit() () |
#12 0x00005603e3155d01 in ha_commit_trans(THD*, bool) () |
#13 0x00005603e3034f17 in trans_commit(THD*) () |
#14 0x00005603e33ff178 in Wsrep_high_priority_service::commit(wsrep::ws_handle const&, wsrep::ws_meta const&) () |
#15 0x00005603e370dde3 in ?? () |
#16 0x00005603e3720c55 in ?? () |
#17 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee83832c290, recv_ctx=0x7efeec062be0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp:396 |
#18 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec062be0, ts=...) at ./galera/src/replicator_smm.cpp:516 |
#19 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec062be0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136 |
#20 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efeec06280f: false, act=..., recv_ctx=0x7efeec062be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:62 |
#21 galera::GcsActionSource::dispatch (exit_loop=@0x7efeec06280f: false, act=..., recv_ctx=0x7efeec062be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110 |
#22 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efeec062be0, exit_loop=@0x7efeec06280f: false) at ./galera/src/gcs_action_source.cpp:186 |
--Type <RET> for more, q to quit, c to continue without paging--c |
#23 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec062be0) at ./galera/src/replicator_smm.cpp:402 |
#24 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#25 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#26 0x00005603e341a267 in ?? () |
#27 0x00005603e340a033 in start_wsrep_THD(void*) () |
#28 0x00005603e3390db6 in ?? () |
#29 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#30 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 20 (Thread 0x7efeec0ae640 (LWP 2352785) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efeec0ad7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efeec0ad7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec0adbe0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 19 (Thread 0x7efeec0f9640 (LWP 2352784) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efeec0f87c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efeec0f87c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec0f8be0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 18 (Thread 0x7efeec144640 (LWP 2352783) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ee83010b6b8) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ee83010b6b8) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ee83010b6b8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c94f88, cond=0x7ee83010b690) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x7ee83010b690, mutex=0x5603e5c94f88) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0e1343 in gu_cond_wait_SYS (mutex=0x5603e5c94f88, cond=0x7ee83010b690) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x5603e5c94f88, obj=...) at ./galera/src/monitor.hpp:194 |
#8 0x00007efeff0ba425 in galera::ReplicatorSMM::commit_order_enter_remote (trx=..., this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:1315 |
#9 galera_commit_order_enter (gh=<optimized out>, ws_handle=<optimized out>, meta=<optimized out>) at ./galera/src/wsrep_provider.cpp:762 |
#10 0x00005603e37214aa in wsrep::wsrep_provider_v26::commit_order_enter(wsrep::ws_handle const&, wsrep::ws_meta const&) () |
#11 0x00005603e371c91d in wsrep::transaction::before_commit() () |
#12 0x00005603e3155d01 in ha_commit_trans(THD*, bool) () |
#13 0x00005603e3034f17 in trans_commit(THD*) () |
#14 0x00005603e33ff178 in Wsrep_high_priority_service::commit(wsrep::ws_handle const&, wsrep::ws_meta const&) () |
#15 0x00005603e370dde3 in ?? () |
#16 0x00005603e3720c55 in ?? () |
#17 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee83010b440, recv_ctx=0x7efeec143be0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp:396 |
#18 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec143be0, ts=...) at ./galera/src/replicator_smm.cpp:516 |
#19 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efeec143be0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136 |
#20 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efeec14380f: false, act=..., recv_ctx=0x7efeec143be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:62 |
#21 galera::GcsActionSource::dispatch (exit_loop=@0x7efeec14380f: false, act=..., recv_ctx=0x7efeec143be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110 |
#22 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efeec143be0, exit_loop=@0x7efeec14380f: false) at ./galera/src/gcs_action_source.cpp:186 |
#23 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efeec143be0) at ./galera/src/replicator_smm.cpp:402 |
#24 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#25 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#26 0x00005603e341a267 in ?? () |
#27 0x00005603e340a033 in start_wsrep_THD(void*) () |
#28 0x00005603e3390db6 in ?? () |
#29 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#30 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 17 (Thread 0x7efefc053640 (LWP 2352782) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc0527c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc0527c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc052be0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 16 (Thread 0x7efefc09e640 (LWP 2352781) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc09d7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc09d7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc09dbe0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 15 (Thread 0x7efefc0e9640 (LWP 2352780) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94f08) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c94f08) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c94f08, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c94e98, cond=0x5603e5c94ee0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e5c94ee0, mutex=0x5603e5c94e98) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0dea1d in gu_cond_wait_SYS (mutex=<optimized out>, cond=0x5603e5c94ee0) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=0x7efefc0e7ac0, this=0x7efefc0e7ac0, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::Monitor<galera::ReplicatorSMM::ApplyOrder>::drain_common (this=0x5603e5c94e98, seqno=<optimized out>, lock=...) at ./galera/src/monitor.hpp:564 |
#8 0x00007efeff0e40b0 in galera::Monitor<galera::ReplicatorSMM::ApplyOrder>::drain (this=0x5603e5c94e98, seqno=129147258) at ./galera/src/monitor.hpp:368 |
#9 0x00007efeff0d897b in galera::ReplicatorSMM::drain_monitors (upto=129147258, this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:2208 |
#10 galera::ReplicatorSMM::drain_monitors_for_local_conf_change (this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:2489 |
#11 0x00007efeff0dbe4e in galera::ReplicatorSMM::process_non_prim_conf_change (my_index=<optimized out>, conf=..., recv_ctx=0x7efefc0e8be0, this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:2521 |
#12 galera::ReplicatorSMM::process_conf_change (this=0x5603e5c93bf0, recv_ctx=0x7efefc0e8be0, conf=..., cc=...) at ./galera/src/replicator_smm.cpp:2459 |
#13 0x00007efeff0f85f1 in galera::GcsActionSource::dispatch (exit_loop=<optimized out>, act=..., recv_ctx=0x7efefc0e8be0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:127 |
#14 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efefc0e8be0, exit_loop=<optimized out>) at ./galera/src/gcs_action_source.cpp:186 |
#15 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc0e8be0) at ./galera/src/replicator_smm.cpp:402 |
#16 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#17 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#18 0x00005603e341a267 in ?? () |
#19 0x00005603e340a033 in start_wsrep_THD(void*) () |
#20 0x00005603e3390db6 in ?? () |
#21 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#22 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 14 (Thread 0x7efefc134640 (LWP 2352779) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc1337c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc1337c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc133be0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 13 (Thread 0x7efefc17f640 (LWP 2352778) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc17e7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc17e7c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc17ebe0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 12 (Thread 0x7efefc1ca640 (LWP 2352777) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc1c97c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc1c97c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc1c9be0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 11 (Thread 0x7efefc215640 (LWP 2352776) "mariadbd"): |
#0 0x00007eff10643868 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7efefc2147c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 |
#1 0x00007eff106486e7 in __GI___nanosleep (req=req@entry=0x7efefc2147c0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25 |
#2 0x00007eff1067a0df in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31 |
#3 0x00007efeff0d1095 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefc214be0) at ./galera/src/replicator_smm.cpp:408 |
#4 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#5 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#6 0x00005603e341a267 in ?? () |
#7 0x00005603e340a033 in start_wsrep_THD(void*) () |
#8 0x00005603e3390db6 in ?? () |
#9 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#10 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 10 (Thread 0x7efefc260640 (LWP 2352773) "mariadbd"): |
#0 0x00007eff105a121a in __GI___sigtimedwait (set=0x7efefc25fb90, info=0x7efefc25fc10, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61 |
#1 0x00005603e2e3169b in signal_hand () |
#2 0x00005603e3390db6 in ?? () |
#3 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#4 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 9 (Thread 0x7efefc2ab640 (LWP 2352772) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3f6efac <mysql_bin_log+3276>) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e3f6efac <mysql_bin_log+3276>) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e3f6efac <mysql_bin_log+3276>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e3f6ef50 <mysql_bin_log+3184>, cond=0x5603e3f6ef80 <mysql_bin_log+3232>) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e3f6ef80 <mysql_bin_log+3232>, mutex=0x5603e3f6ef50 <mysql_bin_log+3184>) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e324a57e in binlog_background_thread () |
#6 0x00005603e3390db6 in ?? () |
#7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 8 (Thread 0x7efefc2f6640 (LWP 2352771) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3f40dac <COND_manager+44>) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e3f40dac <COND_manager+44>) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e3f40dac <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e3f40dc0 <LOCK_manager>, cond=0x5603e3f40d80 <COND_manager>) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e3f40d80 <COND_manager>, mutex=0x5603e3f40dc0 <LOCK_manager>) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e2f01480 in handle_manager () |
#6 0x00005603e3390db6 in ?? () |
#7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 7 (Thread 0x7efe8e0c7640 (LWP 2352709) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e3ebb89c) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e3ebb89c) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e3ebb89c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e3ebb800, cond=0x5603e3ebb870) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e3ebb870, mutex=0x5603e3ebb800) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e358aaa3 in ?? () |
#6 0x00007eff1094b2b3 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6 |
#7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 6 (Thread 0x7efe8e8c8640 (LWP 2352708) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7efe8e8c7c60, op=393, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7efe8e8c7c60, clockid=1602883208, expected=0, futex_word=0x5603e47728a8) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e47728a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7efe8e8c7c60, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1f1b in __pthread_cond_wait_common (abstime=0x7efe8e8c7c60, clockid=0, mutex=0x5603e47728c0, cond=0x5603e4772880) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_timedwait64 (cond=0x5603e4772880, mutex=0x5603e47728c0, abstime=0x7efe8e8c7c60) at ./nptl/pthread_cond_wait.c:652 |
#5 0x00005603e3325402 in ?? () |
#6 0x00005603e331d7be in ?? () |
#7 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#8 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 5 (Thread 0x7efefcefc640 (LWP 2280234) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c93dc8) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c93dc8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c93d60, cond=0x5603e5c93da0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e5c93da0, mutex=0x5603e5c93d60) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0cc92c in gu_cond_wait_SYS (mutex=0x5603e5c93d60, cond=0x5603e5c93da0) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::ReplicatorSMM::wait_for_CLOSED (lock=..., this=<optimized out>) at ./galera/src/replicator_smm.cpp:285 |
#8 galera::ReplicatorSMM::close (this=0x5603e5c93bf0) at ./galera/src/replicator_smm.cpp:376 |
#9 0x00007efeff0b77a1 in galera_disconnect (gh=<optimized out>) at ./galera/src/wsrep_provider.cpp:235 |
#10 0x00005603e3720506 in wsrep::wsrep_provider_v26::disconnect() () |
#11 0x00005603e2e3383a in unireg_abort () |
#12 0x00005603e3408298 in wsrep_handle_mdl_conflict(MDL_context*, MDL_ticket const*, MDL_key const*) () |
#13 0x00005603e302e2cb in MDL_lock::can_grant_lock(enum_mdl_type, MDL_context*, bool) const () |
#14 0x00005603e302e4ea in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) () |
#15 0x00005603e302e942 in MDL_context::acquire_lock(MDL_request*, double) () |
#16 0x00005603e302f5ae in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) () |
#17 0x00005603e2e9f80a in wait_while_table_is_used(THD*, TABLE*, ha_extra_function) () |
#18 0x00005603e2fbe618 in mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool) () |
#19 0x00005603e2fbf454 in mysql_recreate_table(THD*, TABLE_LIST*, bool) () |
#20 0x00005603e30301cd in ?? () |
#21 0x00005603e30327e8 in ?? () |
#22 0x00005603e30342a5 in Sql_cmd_optimize_table::execute(THD*) () |
#23 0x00005603e2f11d9c in mysql_execute_command(THD*, bool) () |
#24 0x00005603e2f12f57 in mysql_parse(THD*, char*, unsigned int, Parser_state*) () |
#25 0x00005603e3269071 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) () |
#26 0x00005603e34183c0 in wsrep_apply_events(THD*, Relay_log_info*, void const*, unsigned long) () |
#27 0x00005603e33fee0b in Wsrep_high_priority_service::apply_toi(wsrep::ws_meta const&, wsrep::const_buffer const&, wsrep::mutable_buffer&) () |
#28 0x00005603e370ed69 in wsrep::server_state::on_apply(wsrep::high_priority_service&, wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&) () |
#29 0x00005603e3720c55 in ?? () |
#30 0x00007efeff0c4e74 in galera::TrxHandleSlave::apply (this=0x7ee73016b410, recv_ctx=0x7efefcefbbe0, apply_cb=0x5603e3720b70, meta=..., exit_loop=<optimized out>) at ./galera/src/trx_handle.cpp:396 |
#31 0x00007efeff0cce75 in galera::ReplicatorSMM::apply_trx (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0, ts=...) at ./galera/src/replicator_smm.cpp:516 |
#32 0x00007efeff0d8ec8 in galera::ReplicatorSMM::process_trx (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0, ts_ptr=...) at ./galera/src/replicator_smm.cpp:2136 |
#33 0x00007efeff0f8965 in galera::GcsActionSource::process_writeset (exit_loop=@0x7efefcefb80f: false, act=..., recv_ctx=0x7efefcefbbe0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:62 |
#34 galera::GcsActionSource::dispatch (exit_loop=@0x7efefcefb80f: false, act=..., recv_ctx=0x7efefcefbbe0, this=0x5603e5ca1050) at ./galera/src/gcs_action_source.cpp:110 |
#35 galera::GcsActionSource::process (this=0x5603e5ca1050, recv_ctx=0x7efefcefbbe0, exit_loop=@0x7efefcefb80f: false) at ./galera/src/gcs_action_source.cpp:186 |
#36 0x00007efeff0d0ff0 in galera::ReplicatorSMM::async_recv (this=0x5603e5c93bf0, recv_ctx=0x7efefcefbbe0) at ./galera/src/replicator_smm.cpp:402 |
#37 0x00007efeff0b7811 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:263 |
#38 0x00005603e37212a2 in wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*) () |
#39 0x00005603e341a267 in ?? () |
#40 0x00005603e340a033 in start_wsrep_THD(void*) () |
#41 0x00005603e3390db6 in ?? () |
#42 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#43 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 4 (Thread 0x7efefcf47640 (LWP 2280233) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7efee40134a0) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7efee40134a0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7efee4013448, cond=0x7efee4013478) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x7efee4013478, mutex=0x7efee4013448) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e3419415 in ?? () |
#6 0x00005603e340a033 in start_wsrep_THD(void*) () |
#7 0x00005603e3390db6 in ?? () |
#8 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#9 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 3 (Thread 0x7efefeecd640 (LWP 2280228) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e5c94544) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e5c94544, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e5c944d8, cond=0x5603e5c94518) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e5c94518, mutex=0x5603e5c944d8) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00007efeff0f55f3 in gu_cond_wait_SYS (mutex=0x5603e5c944d8, cond=0x5603e5c94518) at ./galerautils/src/gu_threads.h:251 |
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46 |
#7 galera::ServiceThd::thd_func (arg=0x5603e5c944b8) at ./galera/src/galera_service_thd.cpp:31 |
#8 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#9 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 2 (Thread 0x7efeff24a640 (LWP 2280227) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=40, cancel=true, abstime=0x7efeff249cd0, op=393, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=40, abstime=0x7efeff249cd0, clockid=3, expected=0, futex_word=0x5603e480cacc) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e480cacc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7efeff249cd0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1f1b in __pthread_cond_wait_common (abstime=0x7efeff249cd0, clockid=0, mutex=0x5603e480cae0, cond=0x5603e480caa0) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_timedwait64 (cond=0x5603e480caa0, mutex=0x5603e480cae0, abstime=0x7efeff249cd0) at ./nptl/pthread_cond_wait.c:652 |
#5 0x00005603e3689d07 in ?? () |
#6 0x00007eff105f2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 |
#7 0x00007eff10684a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 |
|
|
Thread 1 (Thread 0x7eff1024ad00 (LWP 2280226) "mariadbd"): |
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) at ./nptl/futex-internal.c:57 |
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5603e47bf348 <COND_wsrep_server_state+40>) at ./nptl/futex-internal.c:87 |
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5603e47bf348 <COND_wsrep_server_state+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 |
#3 0x00007eff105f1ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5603e47bf360 <LOCK_wsrep_server_state>, cond=0x5603e47bf320 <COND_wsrep_server_state>) at ./nptl/pthread_cond_wait.c:503 |
#4 ___pthread_cond_wait (cond=0x5603e47bf320 <COND_wsrep_server_state>, mutex=0x5603e47bf360 <LOCK_wsrep_server_state>) at ./nptl/pthread_cond_wait.c:627 |
#5 0x00005603e3708e0b in wsrep::server_state::wait_until_state(std::unique_lock<wsrep::mutex>&, wsrep::server_state::state) const () |
#6 0x00005603e340c0ad in wsrep_shutdown_replication() () |
#7 0x00005603e2e3b5d7 in mysqld_main(int, char**) () |
#8 0x00007eff10587d90 in __libc_start_call_main (main=main@entry=0x5603e2dead10 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffdab8ea7c8) at ../sysdeps/nptl/libc_start_call_main.h:58 |
#9 0x00007eff10587e40 in __libc_start_main_impl (main=0x5603e2dead10 <main>, argc=2, argv=0x7ffdab8ea7c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdab8ea7b8) at ../csu/libc-start.c:392 |
#10 0x00005603e2e2ef45 in _start () |
|
jplindst
Does the thread output gives any hint into what's going on ?
I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue?
I saw this change: MDEV-29433
Could it help fix this issue?
| Status | Needs Feedback [ 10501 ] | Open [ 1 ] |
ccounotte I do not find anything clear from information provided. Do you have show processlist from moment when cluster is hung? This could be related to MDEV-29684.
| Status | Open [ 1 ] | Needs Feedback [ 10501 ] |
| Assignee | Jan Lindström [ jplindst ] | Julius Goryavsky [ sysprg ] |
Upgrading MariaDB finally solved the issue, I had other issues with 10.6.10/11, less dramatic, and now upgraded to 10.6.12, in hope to resolve the last issues.
So far didn't experience the cluster hanging any further.
----Message d'origine----
De : Sergei Golubchik (Jira) <jira@mariadb.org>
Envoyé : lundi 6 mars 2023 09:21
À : Cédric Counotte <cedric.counotte@1check.com>
Objet : [JIRA] (MDEV-29388) Galera cluster hanging
Sergei Golubchik edited comment on MDEV-29388 at 3/6/23 8:20 AM:
-----------------------------------------------------------------
jplindst
Does the thread output gives any hint into what's going on ?
I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue?
I saw this change: MDEV-29433
Could it help fix this issue?
was (Author: JIRAUSER51971):
@Jan Lindström
Does the thread output gives any hint into what's going on ?
I'm currently upgrading the servers to 10.6.10, any hope it'll solve this issue?
I saw this change: https://jira.mariadb.org/browse/MDEV-29433
Could it help fix this issue?
–
This message was sent by Atlassian Jira
(v8.20.16#820016)
| Status | Needs Feedback [ 10501 ] | Open [ 1 ] |
| Fix Version/s | 10.6.11 [ 28441 ] | |
| Fix Version/s | 10.6 [ 24028 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Closed [ 6 ] |
| Fix Version/s | 10.6.10 [ 28407 ] | |
| Fix Version/s | 10.6.11 [ 28441 ] |
I know this ticket is closed but I've recently experienced something VERY similar and thought it was worth adding to the ticket for future web searchers.
Environment: 3 node cluster, 10.6.15, CentOS 7.9.2009, 48GB RAM & 16x CPUs each. MaxScale readwritesplitter fronting.
Symptoms: Sessions pile up on writer Node in commit state. Sessions on one writer node show all wsrep applier threads in "commit" state except one which is executing some SQL statement. `shutdown;` then `kill <pid>` on the MariaDB process got the writer working again, and restarting the affected node saw it do an IST and start working again. For a while. Then one or the other of the readers would have the same issue.
Cause: a MASSIVE build-up of History List Length. While we were gathering the relevant data (Percona Monitoring) we weren't actively monitoring or alerting on this value. PMM graphs showed that HLL was in the 10s of millions (10,000,000) and higher on both reader nodes. During shutdown, this was being purged (before I killed the process!) and then building up again.
Why this happened? I'm not entirely sure. Looks like we had a long-running session which caused it to grow, and it never managed to recover. I've since put a lot of effort into monitoring HLL and optimising long-running queries to prevent this from being as much of an issue in the future. I've also set `idle_readonly_transaction_timeout` and `idle_write_transaction_timeout`.
ccounotte To fully analyze your situation I would need unedited error logs from all nodes, your node configuration files and output from show processlist and show engine innodb status. If your node becomes stuck you could also attach debugger on that node and using debug packages get output of : thread apply all bt