[MDEV-22342] Galera nodes hang during shutdown, until killed manually Created: 2020-04-22  Updated: 2021-04-19  Resolved: 2021-02-15

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.8
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Hartmut Holzgraefe Assignee: Hartmut Holzgraefe
Resolution: Incomplete Votes: 1
Labels: need_feedback

Attachments: Text File backup_backtrace_1.txt    
Issue Links:
Relates
relates to MDEV-18665 Server hangs on shutdown after settin... Closed

 Description   

Nodes hang on shutdown, until finally killed manually.

There are other similar bug reports about shutdown issues, but the error log entries and gdb backtrace here don't really seem to match up with the other reported cases.

Error log from shutdown request up to first restart messages after kill:

2020-03-04 20:16:11 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
 
2020-03-04 20:16:11 0 [Note] WSREP: Shutdown replication
 
2020-03-04 20:16:11 0 [Note] WSREP: Server status change synced -> disconnecting
 
2020-03-04 20:16:11 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-03-04 20:16:11 0 [Note] WSREP: Closing send monitor...
 
2020-03-04 20:16:11 0 [Note] WSREP: Closed send monitor.
 
2020-03-04 20:16:11 0 [Note] WSREP: gcomm: terminating thread
 
2020-03-04 20:16:11 0 [Note] WSREP: gcomm: joining thread
 
2020-03-04 20:16:11 0 [Note] WSREP: gcomm: closing backend
 
2020-03-04 20:16:14 0 [Note] WSREP: (2b1056a5, 'tcp://0.0.0.0:4567') connection to peer ef32f7e1 with addr tcp://10.240.240.157:4567 timed out, no messages seen in PT3S
 
2020-03-04 20:16:14 0 [Note] WSREP: (2b1056a5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.240.240.157:4567
 
2020-03-04 20:16:16 0 [Note] WSREP: (2b1056a5, 'tcp://0.0.0.0:4567') reconnecting to ef32f7e1 (tcp://10.240.240.157:4567), attempt 0
 
2020-03-04 20:16:16 0 [Note] WSREP: evs::proto(2b1056a5, LEAVING, view_id(REG,2b1056a5,130)) suspecting node: ef32f7e1
 
2020-03-04 20:16:16 0 [Note] WSREP: evs::proto(2b1056a5, LEAVING, view_id(REG,2b1056a5,130)) suspected node without join message, declaring inactive
 
2020-03-04 20:16:16 0 [Note] WSREP: view(view_id(NON_PRIM,2b1056a5,130) memb {
 
        2b1056a5,0
 
} joined {
 
} left {
 
} partitioned {
 
        ef32f7e1,0
 
})
 
2020-03-04 20:16:16 0 [Note] WSREP: PC protocol downgrade 1 -> 0
 
2020-03-04 20:16:16 0 [Note] WSREP: view((empty))
 
2020-03-04 20:16:16 0 [Note] WSREP: gcomm: closed
 
2020-03-04 20:16:16 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
2020-03-04 20:16:16 0 [Note] WSREP: Writing down CC checksum: 9ad8d59a 5509f1a7 0dafec6c 7d5b0627 at offset 128
 
2020-03-04 20:16:16 0 [Note] WSREP: Flow-control interval: [48, 60]
 
2020-03-04 20:16:16 0 [Note] WSREP: Trying to continue unpaused monitor
 
2020-03-04 20:16:16 0 [Note] WSREP: Received NON-PRIMARY.
 
2020-03-04 20:16:16 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 814658)
 
2020-03-04 20:16:16 0 [Note] WSREP: Received self-leave message.
 
2020-03-04 20:16:16 0 [Note] WSREP: Writing down CC checksum: e9c379a5 3a8874b7 e8cc08bb 81bf91a0 at offset 64
 
2020-03-04 20:16:16 0 [Note] WSREP: Flow-control interval: [0, 0]
 
2020-03-04 20:16:16 0 [Note] WSREP: Trying to continue unpaused monitor
 
2020-03-04 20:16:16 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
 
2020-03-04 20:16:16 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
 
2020-03-04 20:16:16 0 [Note] WSREP: RECV thread exiting 0: Success
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### processing CC -1, local, ordered
 
2020-03-04 20:16:16 336618 [Warning] Aborted connection 336618 to db: 'unconnected' user: 'monitoruser' host: '10.240.240.154' (Got an error reading communication packets)
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### drain monitors upto 814658
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### My UUID: 2b1056a5-32b2-11ea-90f7-57fab51ae3b5
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### ST not required
 
2020-03-04 20:16:16 0 [Note] WSREP: recv_thread() joined.
 
2020-03-04 20:16:16 0 [Note] WSREP: Closing replication queue.
 
2020-03-04 20:16:16 0 [Note] WSREP: Closing slave action queue.
 
2020-03-04 20:16:16 2 [Note] WSREP: ================================================
 
View:
 
  id: 6fd050b9-ff17-11e9-bfb0-b2894f03dddb:-1
 
  status: non-primary
 
  protocol_version: 4
 
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
  final: no
 
  own_index: 0
 
  members(1):
 
        0: 2b1056a5-32b2-11ea-90f7-57fab51ae3b5, dessanmdb04
 
=================================================
 
2020-03-04 20:16:16 2 [Note] WSREP: Non-primary view
 
2020-03-04 20:16:16 336628 [Warning] Aborted connection 336628 to db: 'unconnected' user: 'monitoruser' host: '10.240.240.155' (Got an error reading communication packets)
 
2020-03-04 20:16:16 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### processing CC -1, local, ordered
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### drain monitors upto 814658
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### My UUID: 2b1056a5-32b2-11ea-90f7-57fab51ae3b5
 
2020-03-04 20:16:16 2 [Note] WSREP: ####### ST not required
 
2020-03-04 20:16:16 2 [Note] WSREP: ================================================
 
View:
 
  id: 6fd050b9-ff17-11e9-bfb0-b2894f03dddb:-1
 
  status: non-primary
 
  protocol_version: 4
 
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
  final: yes
 
  own_index: -1
 
  members(0):
 
=================================================
 
2020-03-04 20:16:16 2 [Note] WSREP: Non-primary view
 
2020-03-04 20:16:16 2 [Note] WSREP: Server status change disconnecting -> disconnected
 
2020-03-04 20:16:16 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-03-04 20:16:16 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-03-04 20:16:16 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
 
2020-03-04 20:16:16 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
 
2020-03-04 20:16:16 0 [Note] WSREP: killing local connection: 336645
 
2020-03-04 20:16:16 0 [Note] WSREP: killing local connection: 336642
 
 
 
[... node killed hard here ...]
 
 
 
2020-03-04 20:35:26 0 [Note] WSREP: Loading provider /usr/lib64/galera-4/libgalera_smm.so initial position: 6fd050b9-ff17-11e9-bfb0-b2894f03dddb:814658
 
wsrep loader: [INFO] wsrep_load(): loading provider library '/usr/lib64/galera-4/libgalera_smm.so'
 
wsrep loader: [INFO] wsrep_load(): Galera 26.4.2(r4498) by Codership Oy <info@codership.com> loaded successfully.

GDB backtrace snapshot from hanging node:

Thread 64 (Thread 0x7f7cdd90c700 (LWP 120701)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c5d5e47 in timer_handler ()
 
#2  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#3  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 63 (Thread 0x7f7cdc987700 (LWP 120702)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x00007f7cdcc52935 in wait (this=<synthetic pointer>, cond=...) at galerautils/src/gu_lock.hpp:45
 
#2  galera::ServiceThd::thd_func (arg=0x55902f6c1e98) at galera/src/galera_service_thd.cpp:30
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 62 (Thread 0x7f7cdc186700 (LWP 120705)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902bfdac2b in wsrep_rollback_process(THD*, void*) ()
 
#2  0x000055902bfcd294 in start_wsrep_THD(void*) ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 61 (Thread 0x7f7ca7ffe700 (LWP 121331)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c4b15e6 in my_service_thread_sleep ()
 
#2  0x000055902c4a8f6e in ma_checkpoint_background ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 60 (Thread 0x7f7a867ba700 (LWP 121333)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 59 (Thread 0x7f7a85fb9700 (LWP 121334)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 58 (Thread 0x7f7a857b8700 (LWP 121335)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 57 (Thread 0x7f7a84fb7700 (LWP 121336)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 56 (Thread 0x7f7a847b6700 (LWP 121337)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 55 (Thread 0x7f7a83fb5700 (LWP 121338)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 54 (Thread 0x7f7a837b4700 (LWP 121339)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 53 (Thread 0x7f7a82fb3700 (LWP 121340)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 52 (Thread 0x7f7a827b2700 (LWP 121341)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 51 (Thread 0x7f7a81fb1700 (LWP 121342)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 50 (Thread 0x7f7a817b0700 (LWP 121343)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 49 (Thread 0x7f7a80faf700 (LWP 121344)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 48 (Thread 0x7f7a807ae700 (LWP 121345)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 47 (Thread 0x7f7a7ffad700 (LWP 121346)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 46 (Thread 0x7f7a7f7ac700 (LWP 121347)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 45 (Thread 0x7f7a7efab700 (LWP 121348)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 44 (Thread 0x7f7a7e7aa700 (LWP 121349)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 43 (Thread 0x7f7a7dfa9700 (LWP 121350)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 42 (Thread 0x7f7a7d7a8700 (LWP 121351)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 41 (Thread 0x7f7a7cfa7700 (LWP 121352)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 40 (Thread 0x7f7a7c7a6700 (LWP 121353)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 39 (Thread 0x7f7a7bfa5700 (LWP 121354)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 38 (Thread 0x7f7a7b7a4700 (LWP 121355)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 37 (Thread 0x7f7a7afa3700 (LWP 121356)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 36 (Thread 0x7f7a7a7a2700 (LWP 121357)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 35 (Thread 0x7f7a79fa1700 (LWP 121358)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 34 (Thread 0x7f7a797a0700 (LWP 121359)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 33 (Thread 0x7f7a78f9f700 (LWP 121360)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 32 (Thread 0x7f7a7879e700 (LWP 121361)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 31 (Thread 0x7f7a77f9d700 (LWP 121362)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 30 (Thread 0x7f7a7779c700 (LWP 121363)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 29 (Thread 0x7f7a76f9b700 (LWP 121364)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 28 (Thread 0x7f7a7679a700 (LWP 121365)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 27 (Thread 0x7f7a75f99700 (LWP 121366)):
 
#0  0x00007f7ce1d8c644 in __io_getevents_0_4 () from /lib64/libaio.so.1
 
#1  0x000055902c28024a in LinuxAIOHandler::collect() ()
 
#2  0x000055902c280852 in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) ()
 
#3  0x000055902c288479 in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) ()
 
#4  0x000055902c403306 in fil_aio_wait(unsigned long) ()
 
#5  0x000055902c314918 in io_handler_thread ()
 
#6  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 26 (Thread 0x7f7a75798700 (LWP 121367)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289353 in os_event::timed_wait(timespec const*) ()
 
#2  0x000055902c289561 in os_event::wait_time_low(unsigned long, long) ()
 
#3  0x000055902c3b071d in buf_flush_page_cleaner_coordinator ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 25 (Thread 0x7f7a74f97700 (LWP 121368)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c3aff13 in buf_flush_page_cleaner_worker ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 24 (Thread 0x7f7a6ffff700 (LWP 121369)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c3aff13 in buf_flush_page_cleaner_worker ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 23 (Thread 0x7f7a6f7fe700 (LWP 121370)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c3aff13 in buf_flush_page_cleaner_worker ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 22 (Thread 0x7f7a6effd700 (LWP 121375)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289353 in os_event::timed_wait(timespec const*) ()
 
#2  0x000055902c289561 in os_event::wait_time_low(unsigned long, long) ()
 
#3  0x000055902c25dd7f in lock_wait_timeout_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 21 (Thread 0x7f7a5effd700 (LWP 121376)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289353 in os_event::timed_wait(timespec const*) ()
 
#2  0x000055902c289561 in os_event::wait_time_low(unsigned long, long) ()
 
#3  0x000055902c31189c in srv_error_monitor_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 20 (Thread 0x7f7a92b95700 (LWP 121377)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289353 in os_event::timed_wait(timespec const*) ()
 
#2  0x000055902c289561 in os_event::wait_time_low(unsigned long, long) ()
 
#3  0x000055902c311190 in srv_monitor_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 19 (Thread 0x7f7a92394700 (LWP 121378)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289353 in os_event::timed_wait(timespec const*) ()
 
#2  0x000055902c289561 in os_event::wait_time_low(unsigned long, long) ()
 
#3  0x000055902c3f7587 in dict_stats_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 18 (Thread 0x7f7a91b93700 (LWP 121379)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289353 in os_event::timed_wait(timespec const*) ()
 
#2  0x000055902c289561 in os_event::wait_time_low(unsigned long, long) ()
 
#3  0x000055902c35a7de in ib_wqueue_timedwait(ib_wqueue_t*, unsigned long) ()
 
#4  0x000055902c440528 in fts_optimize_thread(void*) ()
 
#5  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 17 (Thread 0x7f7a91392700 (LWP 121380)):
 
#0  0x00007f7ce21c2eed in nanosleep () from /lib64/libpthread.so.0
 
#1  0x000055902c289a70 in os_thread_sleep(unsigned long) ()
 
#2  0x000055902c3122c9 in srv_master_thread ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 16 (Thread 0x7f7a90b91700 (LWP 121381)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c31395a in srv_purge_coordinator_thread ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 15 (Thread 0x7f7a90390700 (LWP 121382)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c30d1fb in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) [clone .constprop.70] ()
 
#3  0x000055902c312c77 in srv_worker_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 14 (Thread 0x7f7a8fb8f700 (LWP 121383)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c30d1fb in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) [clone .constprop.70] ()
 
#3  0x000055902c312c77 in srv_worker_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 13 (Thread 0x7f7a8f38e700 (LWP 121384)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c30d1fb in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) [clone .constprop.70] ()
 
#3  0x000055902c312c77 in srv_worker_thread ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 12 (Thread 0x7f7a8eb8d700 (LWP 121385)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c3a9d37 in buf_dump_thread ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 11 (Thread 0x7f7a8e38c700 (LWP 121386)):
 
#0  0x00007f7ce21c2eed in nanosleep () from /lib64/libpthread.so.0
 
#1  0x000055902c289a70 in os_thread_sleep(unsigned long) ()
 
#2  0x000055902c392f7a in btr_defragment_thread ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 10 (Thread 0x7f7a8db8b700 (LWP 121387)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c289861 in os_event_wait_low(os_event*, long) ()
 
#2  0x000055902c3a2532 in buf_resize_thread ()
 
#3  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 9 (Thread 0x7f7a8d38a700 (LWP 121388)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c2074f3 in thd_destructor_proxy ()
 
#2  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#3  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 8 (Thread 0x7f7a8cb89700 (LWP 121389)):
 
#0  0x00007f7ce21bfcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902c1f2fe6 in timer_thread(void*) ()
 
#2  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#3  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 7 (Thread 0x7f7ccc23a700 (LWP 121390)):
 
#0  0x00007f7ce049940a in sigwaitinfo () from /lib64/libc.so.6
 
#1  0x000055902bd943c3 in signal_hand ()
 
#2  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#3  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 6 (Thread 0x7f7ccc1ef700 (LWP 121391)):
 
#0  0x00007f7ce21bf945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 
#1  0x000055902bdb0a4b in handle_slave_background ()
 
#2  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#3  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 5 (Thread 0x7f7a8c04f700 (LWP 85470)):
 
#0  0x00007f7ce0561113 in epoll_wait () from /lib64/libc.so.6
 
#1  0x000055902c1f353b in io_poll_wait(int, epoll_event*, int, int) ()
 
#2  0x000055902c1f3bfe in get_event(worker_thread_t*, thread_group_t*, timespec*) ()
 
#3  0x000055902c1f4160 in worker_main(void*) ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 4 (Thread 0x7f7a8be42700 (LWP 85472)):
 
#0  0x00007f7ce0561113 in epoll_wait () from /lib64/libc.so.6
 
#1  0x000055902c1f353b in io_poll_wait(int, epoll_event*, int, int) ()
 
#2  0x000055902c1f3bfe in get_event(worker_thread_t*, thread_group_t*, timespec*) ()
 
#3  0x000055902c1f4160 in worker_main(void*) ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 3 (Thread 0x7f7a8c130700 (LWP 104980)):
 
#0  0x00007f7ce0561113 in epoll_wait () from /lib64/libc.so.6
 
#1  0x000055902c1f353b in io_poll_wait(int, epoll_event*, int, int) ()
 
#2  0x000055902c1f3bfe in get_event(worker_thread_t*, thread_group_t*, timespec*) ()
 
#3  0x000055902c1f4160 in worker_main(void*) ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 2 (Thread 0x7f7a8c0e5700 (LWP 104993)):
 
#0  0x00007f7ce0561113 in epoll_wait () from /lib64/libc.so.6
 
#1  0x000055902c1f353b in io_poll_wait(int, epoll_event*, int, int) ()
 
#2  0x000055902c1f3bfe in get_event(worker_thread_t*, thread_group_t*, timespec*) ()
 
#3  0x000055902c1f4160 in worker_main(void*) ()
 
#4  0x00007f7ce21bbdd5 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007f7ce0560b3d in clone () from /lib64/libc.so.6
 
 
 
Thread 1 (Thread 0x7f7ce25da8c0 (LWP 120699)):
 
#0  0x00007f7ce05274fd in nanosleep () from /lib64/libc.so.6
 
#1  0x00007f7ce0527394 in sleep () from /lib64/libc.so.6
 
#2  0x000055902bfcba45 in wsrep_close_client_connections(char, THD*) ()
 
#3  0x000055902bfcbff0 in wsrep_shutdown_replication() ()
 
#4  0x000055902bd9ee75 in mysqld_main(int, char**) ()
 
#5  0x00007f7ce04843d5 in __libc_start_main () from /lib64/libc.so.6
 
#6  0x000055902bd91bd4 in _start ()



 Comments   
Comment by Jan Lindström (Inactive) [ 2020-04-28 ]

Does problem repeat if you use Galera library 26.4.4 ?

Comment by Hartmut Holzgraefe [ 2020-04-28 ]

26.4.4 will only be part of upcoming releases, latest 10.4.12 still comes with 26.4.3?

Comment by Claudio Nanni [ 2020-05-15 ]

Also observed on 10.4.10, 10.4.11, 10.4.12 o Ubuntu 18.0.4 and CentOS 7

Comment by Jan Lindström (Inactive) [ 2020-12-18 ]

I could not find anything clear from stack traces or error log. I recommend using more recent version of MariaDB server and Galera library and if problem is reproducible please provide error logs from shutdown and then from kill.

Generated at Thu Feb 08 09:14:00 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.