[MDEV-20929] Galera node startup fails with " free(): invalid pointer" Created: 2019-10-30  Updated: 2021-04-19  Resolved: 2020-10-30

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.8
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Hartmut Holzgraefe Assignee: Alexey
Resolution: Cannot Reproduce Votes: 1
Labels: need_feedback


 Description   

After an upgrade from MariaDB 10.3.16 to 10.4.8 nodes fail with the following backtrace:

> *** Error in `mysqld': free(): invalid pointer: 0x00007f9703fff500 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x81609)[0x7f99d6186609]
> /usr/lib64/galera-4/libgalera_smm.so(gcache::MemStore::discard(gcache::BufferHeader*)+0x1b)[0x7f998b65418b]
> /usr/lib64/galera-4/libgalera_smm.so(gcache::GCache::free_common(gcache::BufferHeader*)+0x5e)[0x7f998b651c0e]
> /usr/lib64/galera-4/libgalera_smm.so(gcache::GCache::seqno_release(long)+0x122)[0x7f998b63b852]
> /usr/lib64/galera-4/libgalera_smm.so(galera::ServiceThd::thd_func(void*)+0x14b)[0x7f998b7509db]
> /lib64/libpthread.so.0(+0x7dd5)[0x7f99d7e61dd5]
> /lib64/libc.so.6(clone+0x6d)[0x7f99d620302d]

The only similar report that I could find so far is:

https://github.com/codership/galera/issues/419

in which the problem was caused by two different node instances running on the same machine where accidentally configured to use the same datadir. This can most certainly be ruled out as a possible cause in this case though.



 Comments   
Comment by Hartmut Holzgraefe [ 2019-10-30 ]

Full log output before running the backtrace part through c++filt

2019-10-29 23:11:33 0 [Note] mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2019-10-29 23:11:33 0 [Note] WSREP: Shutdown replication
2019-10-29 23:11:33 0 [Note] WSREP: Server status change synced -> disconnecting
2019-10-29 23:11:33 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:33 0 [Note] WSREP: Closing send monitor...
2019-10-29 23:11:33 0 [Note] WSREP: Closed send monitor.
2019-10-29 23:11:33 0 [Note] WSREP: gcomm: terminating thread
2019-10-29 23:11:33 0 [Note] WSREP: gcomm: joining thread
2019-10-29 23:11:33 0 [Note] WSREP: gcomm: closing backend
2019-10-29 23:11:36 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') connection to peer f6e2057e with addr tcp://10.142.8.142:4333 timed out, no messages seen in PT3S
2019-10-29 23:11:36 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') turning message relay requesting on, nonlive peers: tcp://10.142.8.142:4333
2019-10-29 23:11:38 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') reconnecting to f6e2057e (tcp://10.142.8.142:4333), attempt 0
2019-10-29 23:11:41 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') connection to peer f6e2057e with addr tcp://10.142.8.142:4333 timed out, no messages seen in PT3S
2019-10-29 23:11:43 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') connection established to f6e2057e tcp://10.142.8.142:4333
2019-10-29 23:11:45 0 [Note] WSREP: evs::proto(eb0b5481, LEAVING, view_id(REG,eb0b5481,6)) suspecting node: f6e2057e
2019-10-29 23:11:45 0 [Note] WSREP: evs::proto(eb0b5481, LEAVING, view_id(REG,eb0b5481,6)) suspected node without join message, declaring inactive
2019-10-29 23:11:45 0 [Note] WSREP: view(view_id(NON_PRIM,eb0b5481,6) memb {
eb0b5481,0
} joined {
} left {
} partitioned {
f6e2057e,0
})
2019-10-29 23:11:45 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2019-10-29 23:11:45 0 [Note] WSREP: view((empty))
2019-10-29 23:11:45 0 [Note] WSREP: gcomm: closed
2019-10-29 23:11:45 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2019-10-29 23:11:45 0 [Note] WSREP: Writing down CC checksum: 39b2e180 6e10a4de 74927600 a6dff24f at offset 120
2019-10-29 23:11:45 0 [Note] WSREP: Flow-control interval: [64, 64]
2019-10-29 23:11:45 0 [Note] WSREP: Trying to continue unpaused monitor
2019-10-29 23:11:45 0 [Note] WSREP: Received NON-PRIMARY.
2019-10-29 23:11:45 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 9)
2019-10-29 23:11:45 0 [Note] WSREP: Received self-leave message.
2019-10-29 23:11:45 0 [Note] WSREP: Writing down CC checksum: 94d1c656 7a51b027 67573338 31d1e2be at offset 64
2019-10-29 23:11:45 0 [Note] WSREP: Flow-control interval: [0, 0]
2019-10-29 23:11:45 0 [Note] WSREP: Trying to continue unpaused monitor
2019-10-29 23:11:45 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2019-10-29 23:11:45 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
2019-10-29 23:11:45 2 [Note] WSREP: ####### processing CC -1, local, ordered
2019-10-29 23:11:45 0 [Note] WSREP: RECV thread exiting 0: Success
2019-10-29 23:11:45 0 [Note] WSREP: recv_thread() joined.
2019-10-29 23:11:45 0 [Note] WSREP: Closing replication queue.
2019-10-29 23:11:45 0 [Note] WSREP: Closing slave action queue.
2019-10-29 23:11:45 2 [Note] WSREP: ####### drain monitors upto 9
2019-10-29 23:11:45 2 [Note] WSREP: ####### My UUID: eb0b5481-fa31-11e9-b396-3e828533d8bb
2019-10-29 23:11:45 2 [Note] WSREP: ####### ST not required
2019-10-29 23:11:45 2 [Note] WSREP: ================================================
View:
id: eb0c6714-fa31-11e9-8759-afff06cd3fd6:-1
status: non-primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: eb0b5481-fa31-11e9-b396-3e828533d8bb, rndmdc401
=================================================
2019-10-29 23:11:45 2 [Note] WSREP: Non-primary view
2019-10-29 23:11:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:45 2 [Note] WSREP: ####### processing CC -1, local, ordered
2019-10-29 23:11:45 2 [Note] WSREP: ####### drain monitors upto 9
2019-10-29 23:11:45 2 [Note] WSREP: ####### My UUID: eb0b5481-fa31-11e9-b396-3e828533d8bb
2019-10-29 23:11:45 2 [Note] WSREP: ####### ST not required
2019-10-29 23:11:45 2 [Note] WSREP: ================================================
View:
id: eb0c6714-fa31-11e9-8759-afff06cd3fd6:-1
status: non-primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: yes
own_index: -1
members(0):
=================================================
2019-10-29 23:11:45 2 [Note] WSREP: Non-primary view
2019-10-29 23:11:45 2 [Note] WSREP: Server status change disconnecting -> disconnected
2019-10-29 23:11:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:45 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
2019-10-29 23:11:45 1 [Note] WSREP: rollbacker thread exiting 1
2019-10-29 23:11:45 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
2019-10-29 23:11:45 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
2019-10-29 23:11:45 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 9
2019-10-29 23:11:45 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply
2019-10-29 23:11:45 0 [Note] Event Scheduler: Stopped
2019-10-29 23:11:45 0 [Note] Event Scheduler: Purging the queue. 0 events
2019-10-29 23:11:45 0 [Note] InnoDB: FTS optimize thread exiting.
2019-10-29 23:11:45 0 [Note] WSREP: dtor state: CLOSED
2019-10-29 23:11:45 0 [Note] WSREP: MemPool(TrxHandleSlave): hit ratio: 0, misses: 3, in use: 3, in pool: 0
2019-10-29 23:11:45 0 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
2019-10-29 23:11:45 0 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
2019-10-29 23:11:45 0 [Note] WSREP: mon: entered 21 oooe fraction 0 oool fraction 0
2019-10-29 23:11:45 0 [Note] WSREP: cert index usage at exit 1
2019-10-29 23:11:45 0 [Note] WSREP: cert trx map usage at exit 3
2019-10-29 23:11:45 0 [Note] WSREP: deps set usage at exit 0
2019-10-29 23:11:45 0 [Note] WSREP: avg deps dist 1
2019-10-29 23:11:45 0 [Note] WSREP: avg cert interval 0
2019-10-29 23:11:45 0 [Note] WSREP: cert index size 1
*** Error in `mysqld': free(): invalid pointer: 0x00007f9703fff500 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81609)[0x7f99d6186609]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6gcache8MemStore7discardEPNS_12BufferHeaderE+0x1b)[0x7f998b65418b]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6gcache6GCache11free_commonEPNS_12BufferHeaderE+0x5e)[0x7f998b651c0e]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6gcache6GCache13seqno_releaseEl+0x122)[0x7f998b63b852]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6galera10ServiceThd8thd_funcEPv+0x14b)[0x7f998b7509db]
/lib64/libpthread.so.0(+0x7dd5)[0x7f99d7e61dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f99d620302d]
======= Memory map: ========
55ada4d60000-55ada61a8000 r-xp 00000000 fd:01 33598536 /usr/sbin/mysqld
55ada63a8000-55ada64d3000 r--p 01448000 fd:01 33598536 /usr/sbin/mysqld
55ada64d3000-55ada658d000 rw-p 01573000 fd:01 33598536 /usr/sbin/mysqld
55ada658d000-55ada6e3c000 rw-p 00000000 00:00 0
55ada853c000-55adb2f27000 rw-p 00000000 00:00 0 [heap]
7f9450000000-7f945043e000 rw-p 00000000 00:00 0
7f945043e000-7f9454000000 ---p 00000000 00:00 0
7f94561fd000-7f94561fe000 ---p 00000000 00:00 0
7f94561fe000-7f9456bfe000 rw-p 00000000 00:00 0
7f9456bfe000-7f9456bff000 ---p 00000000 00:00 0
7f9456bff000-7f94575ff000 rw-p 00000000 00:00 0
7f94575ff000-7f9457600000 ---p 00000000 00:00 0
7f9457600000-7f9458000000 rw-p 00000000 00:00 0
7f9458000000-7f9458457000 rw-p 00000000 00:00 0
7f9458457000-7f945c000000 ---p 00000000 00:00 0
7f945c000000-7f945c021000 rw-p 00000000 00:00 0
7f945c021000-7f9460000000 ---p 00000000 00:00 0
7f9460000000-7f9460021000 rw-p 00000000 00:00 0
7f9460021000-7f9464000000 ---p 00000000 00:00 0
7f9464000000-7f9464021000 rw-p 00000000 00:00 0
7f9464021000-7f9468000000 ---p 00000000 00:00 0
7f9468000000-7f9468021000 rw-p 00000000 00:00 0
7f9468021000-7f946c000000 ---p 00000000 00:00 0
7f946c000000-7f946c021000 rw-p 00000000 00:00 0
7f946c021000-7f9470000000 ---p 00000000 00:00 0
7f9470000000-7f9470021000 rw-p 00000000 00:00 0
7f9470021000-7f9474000000 ---p 00000000 00:00 0
7f9474000000-7f9474021000 rw-p 00000000 00:00 0
7f9474021000-7f9478000000 ---p 00000000 00:00 0
7f9478000000-7f9478021000 rw-p 00000000 00:00 0
7f9478021000-7f947c000000 ---p 00000000 00:00 0
7f947c000000-7f947c021000 rw-p 00000000 00:00 0
7f947c021000-7f9480000000 ---p 00000000 00:00 0
7f9480000000-7f9480021000 rw-p 00000000 00:00 0
7f9480021000-7f9484000000 ---p 00000000 00:00 0
7f9484000000-7f9484021000 rw-p 00000000 00:00 0
7f9484021000-7f9488000000 ---p 00000000 00:00 0
7f9488000000-7f9488021000 rw-p 00000000 00:00 0
7f9488021000-7f948c000000 ---p 00000000 00:00 0
7f948c000000-7f948c021000 rw-p 00000000 00:00 0
7f948c021000-7f9490000000 ---p 00000000 00:00 0
7f9490000000-7f9490021000 rw-p 00000000 00:00 0
7f9490021000-7f9494000000 ---p 00000000 00:00 0
7f9494000000-7f9494021000 rw-p 00000000 00:00 0
7f9494021000-7f9498000000 ---p 00000000 00:00 0
7f9498000000-7f9498021000 rw-p 00000000 00:00 0
7f9498021000-7f949c000000 ---p 00000000 00:00 0
7f949c000000-7f949c021000 rw-p 00000000 00:00 0
7f949c021000-7f94a0000000 ---p 00000000 00:00 0
7f94a0000000-7f94a0021000 rw-p 00000000 00:00 0
7f94a0021000-7f94a4000000 ---p 00000000 00:00 0
7f94a4359000-7f94a435a000 ---p 00000000 00:00 0
7f94a435a000-7f94a8000000 rw-p 00000000 00:00 0
7f94a8000000-7f94a8021000 rw-p 00000000 00:00 0
7f94a8021000-7f94ac000000 ---p 00000000 00:00 0
7f94ac000000-7f94ac021000 rw-p 00000000 00:00 0
7f94ac021000-7f94b0000000 ---p 00000000 00:00 0
7f94b0000000-7f94b0021000 rw-p 00000000 00:00 0
7f94b0021000-7f94b4000000 ---p 00000000 00:00 0
7f94b4000000-7f94b4021000 rw-p 00000000 00:00 0
7f94b4021000-7f94b8000000 ---p 00000000 00:00 0
7f94b8702000-7f94b8907000 rw-p 00000000 00:00 0
7f94b9308000-7f94b9309000 ---p 00000000 00:00 0
7f94b9309000-7f94b9d09000 rw-p 00000000 00:00 0
7f94b9d09000-7f94b9d0a000 ---p 00000000 00:00 0
7f94b9d0a000-7f94ba70a000 rw-p 00000000 00:00 0
7f94ba70a000-7f94ba70b000 ---p 00000000 00:00 0
7f94ba70b000-7f94bb10b000 rw-p 00000000 00:00 0
7f94bb10b000-7f94bb10c000 ---p 00000000 00:00 0
7f94bb10c000-7f94bbb0c000 rw-p 00000000 00:00 0
7f94bbb0c000-7f94bbb0d000 ---p 00000000 00:00 0
7f94bbb0d000-7f94bc50d000 rw-p 00000000 00:00 0
7f94bc50d000-7f94bc50e000 ---p 00000000 00:00 0
7f94bc50e000-7f94bcf0e000 rw-p 00000000 00:00 0
7f94bcf0e000-7f94bcf0f000 ---p 00000000 00:00 0
7f94bcf0f000-7f94bd90f000 rw-p 00000000 00:00 0
7f94bd90f000-7f94bd910000 ---p 00000000 00:00 0
7f94bd910000-7f94be310000 rw-p 00000000 00:00 0
7f94be310000-7f94be311000 ---p 00000000 00:00 0
7f94be311000-7f94bed11000 rw-p 00000000 00:00 0
7f94bed11000-7f94bed12000 ---p 00000000 00:00 0
7f94bed12000-7f94bf712000 rw-p 00000000 00:00 0
7f94bf712000-7f94bf713000 ---p 00000000 00:00 0
7f94bf713000-7f94c0113000 rw-p 00000000 00:00 0
7f94c0113000-7f94c0114000 ---p 00000000 00:00 0
7f94c0114000-7f94c0b14000 rw-p 00000000 00:00 0
7f94c0b14000-7f94c0b15000 ---p 00000000 00:00 0
7f94c0b15000-7f94c51e4000 rw-p 00000000 00:00 0
7f94c54dc000-7f94c54e1000 r-xp 00000000 fd:01 33665962 /usr/lib64/libnss_dns-2.17.so
7f94c54e1000-7f94c56e1000 ---p 00005000 fd:01 33665962 /usr/lib64/libnss_dns-2.17.so
7f94c56e1000-7f94c56e2000 r--p 00005000 fd:01 33665962 /usr/lib64/libnss_dns-2.17.so
7f94c56e2000-7f94c56e3000 rw-p 00006000 fd:01 33665962 /usr/lib64/libnss_dns-2.17.so
7f94c56e3000-7f94c56e4000 ---p 00000000 00:00 0
7f94c56e4000-7f94c60e4000 rw-p 00000000 00:00 0
7f94c60e4000-7f94c60e5000 ---p 00000000 00:00 0
7f94c60e5000-7f94c6ae5000 rw-p 00000000 00:00 0
7f94c6ae5000-7f94c6ae6000 ---p 00000000 00:00 0
7f94c6ae6000-7f94c74e6000 rw-p 00000000 00:00 0
7f94c74e6000-7f94c74e7000 ---p 00000000 00:00 0
7f94c74e7000-7f94c7ee7000 rw-p 00000000 00:00 0
7f94c88e8000-7f94c88e9000 ---p 00000000 00:00 0
7f94c88e9000-7f94c92e9000 rw-p 00000000 00:00 0
7f94c92e9000-7f94c92ea000 ---p 00000000 00:00 0
7f94c92ea000-7f94c9cea000 rw-p 00000000 00:00 0
7f94c9cea000-7f94c9ceb000 ---p 00000000 00:00 0
7f94c9ceb000-7f94ca6eb000 rw-p 00000000 00:00 0
7f94cb0ec000-7f94cb0ed000 ---p 00000000 00:00 0
7f94cb0ed000-7f94cbaed000 rw-p 00000000 00:00 0
7f94cbaed000-7f94cbaee000 ---p 00000000 00:00 0
7f94cbaee000-7f94cc4ee000 rw-p 00000000 00:00 0
7f94cc4ee000-7f94cc4ef000 ---p 00000000 00:00 0
7f94cc4ef000-7f94cceef000 rw-p 00000000 00:00 0
7f94cd8f0000-7f94cf8f0000 rw-p 00000000 00:00 0
7f94cf8f0000-7f94d8000000 rw-p 00000000 00:00 0
7f94d8000000-7f96d8000000 rw-p 00000000 00:00 0
7f96d8000000-7f96d8021000 rw-p 00000000 00:00 0
7f96d8021000-7f96dc000000 ---p 00000000 00:00 0
7f96dc147000-7f96dc148000 ---p 00000000 00:00 0
7f96dc148000-7f96dcd60000 rw-p 00000000 00:00 0
7f96dcf60000-7f96ec000000 rw-p 00000000 00:00 0
7f96ec000000-7f96ec038000 rw-p 00000000 00:00 0
7f96ec038000-7f96f0000000 ---p 00000000 00:00 0
7f96f0000000-7f96f0021000 rw-p 00000000 00:00 0
7f96f0021000-7f96f4000000 ---p 00000000 00:00 0
7f96f4000000-7f96f4021000 rw-p 00000000 00:00 0
7f96f4021000-7f96f8000000 ---p 00000000 00:00 0
7f96f8000000-7f96f8021000 rw-p 00000000 00:00 0
7f96f8021000-7f96fc000000 ---p 00000000 00:00 0
7f96fc000000-7f96fc021000 rw-p 00000000 00:00 0
7f96fc021000-7f9700000000 ---p 00000000 00:00 0
7f970004c000-7f970004d000 ---p 00000000 00:00 0
7f970004d000-7f9700b68000 rw-p 00000000 00:00 0
7f9700b68000-7f9700b69000 ---p 00000000 00:00 0
7f9700b69000-7f9701c9e000 rw-p 00000000 00:00 0
7f9701c9e000-7f9701ca2000 r-xp 00000000 fd:01 33697530 /usr/lib64/libcap-ng.so.0.0.0
7f9701ca2000-7f9701ea2000 ---p 00004000 fd:01 33697530 /usr/lib64/libcap-ng.so.0.0.0
7f9701ea2000-7f9701ea3000 r--p 00004000 fd:01 33697530 /usr/lib64/libcap-ng.so.0.0.0
7f9701ea3000-7f9701ea4000 rw-p 00005000 fd:01 33697530 /usr/lib64/libcap-ng.so.0.0.0
7f9701ea4000-7f97035fe000 rw-p 00000000 00:00 0
7f9703fff000-7f9984000000 rw-s 00000000 fd:03 19 /db/data3/galera.cache
7f9984000000-7f9984021000 rw-p 00000000 00:00 0
7f9984021000-7f9988000000 ---p 00000000 00:00 0
7f998805c000-7f99880dd000 rw-p 00000000 00:00 0
7f99880dd000-7f99880fb000 r-xp 00000000 fd:01 34044702 /usr/lib64/libaudit.so.1.0.0
7f99880fb000-7f99882fa000 ---p 0001e000 fd:01 34044702 /usr/lib64/libaudit.so.1.0.0
7f99882fa000-7f99882fb000 r--p 0001d000 fd:01 34044702 /usr/lib64/libaudit.so.1.0.0
7f99882fb000-7f99882fc000 rw-p 0001e000 fd:01 34044702 /usr/lib64/libaudit.so.1.0.0
7f99882fc000-7f9988306000 rw-p 00000000 00:00 0
7f9988306000-7f9988313000 r-xp 00000000 fd:01 33764806 /usr/lib64/libpam.so.0.83.1
7f9988313000-7f9988513000 ---p 0000d000 fd:01 33764806 /usr/lib64/libpam.so.0.83.1
7f9988513000-7f9988514000 r--p 0000d000 fd:01 33764806 /usr/lib64/libpam.so.0.83.1
7f9988514000-7f9988515000 rw-p 0000e000 fd:01 33764806 /usr/lib64/libpam.so.0.83.1
7f9988515000-7f9988517000 r-xp 00000000 fd:01 83916812 /usr/lib64/mysql/plugin/auth_pam.so
7f9988517000-7f9988716000 ---p 00002000 fd:01 83916812 /usr/lib64/mysql/plugin/auth_pam.so
7f9988716000-7f9988717000 r--p 00001000 fd:01 83916812 /usr/lib64/mysql/plugin/auth_pam.so
7f9988717000-7f9988718000 rw-p 00002000 fd:01 83916812 /usr/lib64/mysql/plugin/auth_pam.so
7f9988718000-7f9988789000 rw-p 00000000 00:00 0
7f998aa23000-7f998aa24000 ---p 00000000 00:00 0
7f998aa24000-7f998b587000 rw-p 00000000 00:00 0
7f998b587000-7f998b7ec000 r-xp 00000000 fd:01 134217799 /usr/lib64/galera-4/libgalera_smm.so
7f998b7ec000-7f998b9eb000 ---p 00265000 fd:01 134217799 /usr/lib64/galera-4/libgalera_smm.so
7f998b9eb000-7f998b9f3000 r--p 00264000 fd:01 134217799 /usr/lib64/galera-4/libgalera_smm.so
7f998b9f3000-7f998b9f5000 rw-p 0026c000 fd:01 134217799 /usr/lib64/galera-4/libgalera_smm.so
7f998b9f5000-7f998b9f7000 rw-p 00000000 00:00 0
7f998b9f7000-7f998b9f8000 ---p 00000000 00:00 0
7f998b9f8000-7f998c4f4000 rw-p 00000000 00:00 0
7f998c4f4000-7f998c4f6000 r-xp 00000000 fd:01 118138806 /usr/lib64/samba/libreplace-samba4.so
7f998c4f6000-7f998c6f5000 ---p 00002000 fd:01 118138806 /usr/lib64/samba/libreplace-samba4.so
7f998c6f5000-7f998c6f6000 r--p 00001000 fd:01 118138806 /usr/lib64/samba/libreplace-samba4.so
7f998c6f6000-7f998c6f7000 rw-p 00000000 00:00 0
7f998c6f7000-7f998c6fa000 r-xp 00000000 fd:01 118279933 /usr/lib64/samba/libwinbind-client-samba4.so
7f998c6fa000-7f998c8f9000 ---p 00003000 fd:01 118279933 /usr/lib64/samba/libwinbind-client-samba4.so
7f998c8f9000-7f998c8fa000 r--p 00002000 fd:01 118279933 /usr/lib64/samba/libwinbind-client-samba4.so
7f998c8fa000-7f998c8fb000 rw-p 00003000 fd:01 118279933 /usr/lib64/samba/libwinbind-client-samba4.so
7f998c8fb000-7f998c8fe000 r-xp 00000000 fd:01 33863279 /usr/lib64/libnss_winbind.so.2
7f998c8fe000-7f998cafd000 ---p 00003000 fd:01 33863279 /usr/lib64/libnss_winbind.so.2
7f998cafd000-7f998cafe000 r--p 00002000 fd:01 33863279 /usr/lib64/libnss_winbind.so.2
7f998cafe000-7f998cb05000 rw-p 00000000 00:00 0
7f998cb05000-7f998cb10000 r-xp 00000000 fd:01 33597217 /usr/lib64/libnss_ldap.so.2
7f998cb10000-7f998cd10000 ---p 0000b000 fd:01 33597217 /usr/lib64/libnss_ldap.so.2
7f998cd10000-7f998cd11000 r--p 0000b000 fd:01 33597217 /usr/lib64/libnss_ldap.so.2
7f998cd11000-7f998cd12000 rw-p 0000c000 fd:01 33597217 /usr/lib64/libnss_ldap.so.2
7f998cd12000-7f998cd1e000 r-xp 00000000 fd:01 34296052 /usr/lib64/libnss_files-2.17.so
7f998cd1e000-7f998cf1d000 ---p 0000c000 fd:01 34296052 /usr/lib64/libnss_files-2.17.so
7f998cf1d000-7f998cf1e000 r--p 0000b000 fd:01 34296052 /usr/lib64/libnss_files-2.17.so
7f998cf1e000-7f998cf1f000 rw-p 0000c000 fd:01 34296052 /usr/lib64/libnss_files-2.17.so
7f998cf1f000-7f99d35a5000 rw-p 00000000 00:00 0
7f99d35a5000-7f99d35a8000 r-xp 00000000 fd:01 33762228 /usr/lib64/libkeyutils.so.1.5
7f99d35a8000-7f99d37a7000 ---p 00003000 fd:01 33762228 /usr/lib64/libkeyutils.so.1.5
7f99d37a7000-7f99d37a8000 r--p 00002000 fd:01 33762228 /usr/lib64/libkeyutils.so.1.5
7f99d37a8000-7f99d37a9000 rw-p 00003000 fd:01 33762228 /usr/lib64/libkeyutils.so.1.5
7f99d37a9000-7f99d37b7000 r-xp 00000000 fd:01 34099408 /usr/lib64/libkrb5support.so.0.1
7f99d37b7000-7f99d39b7000 ---p 0000e000 fd:01 34099408 /usr/lib64/libkrb5support.so.0.1
7f99d39b7000-7f99d39b8000 r--p 0000e000 fd:01 34099408 /usr/lib64/libkrb5support.so.0.1
7f99d39b8000-7f99d39b9000 rw-p 0000f000 fd:01 34099408 /usr/lib64/libkrb5support.so.0.1
7f99d39b9000-7f99d39c8000 r-xp 00000000 fd:01 34335768 /usr/lib64/libbz2.so.1.0.6
7f99d39c8000-7f99d3bc7000 ---p 0000f000 fd:01 34335768 /usr/lib64/libbz2.so.1.0.6
7f99d3bc7000-7f99d3bc8000 r--p 0000e000 fd:01 34335768 /usr/lib64/libbz2.so.1.0.6
7f99d3bc8000-7f99d3bc9000 rw-p 0000f000 fd:01 34335768 /usr/lib64/libbz2.so.1.0.6
7f99d3bc9000-7f99d3be0000 r-xp 00000000 fd:01 33680559 /usr/lib64/libelf-0.172.so
7f99d3be0000-7f99d3ddf000 ---p 00017000 fd:01 33680559 /usr/lib64/libelf-0.172.so
7f99d3ddf000-7f99d3de0000 r--p 00016000 fd:01 33680559 /usr/lib64/libelf-0.172.so
7f99d3de0000-7f99d3de1000 rw-p 00017000 fd:01 33680559 /usr/lib64/libelf-0.172.so
7f99d3de1000-7f99d3e41000 r-xp 00000000 fd:01 33669631 /usr/lib64/libpcre.so.1.2.0
7f99d3e41000-7f99d4041000 ---p 00060000 fd:01 33669631 /usr/lib64/libpcre.so.1.2.0
7f99d4041000-7f99d4042000 r--p 00060000 fd:01 33669631 /usr/lib64/libpcre.so.1.2.0
7f99d4042000-7f99d4043000 rw-p 00061000 fd:01 33669631 /usr/lib64/libpcre.so.1.2.0
7f99d4043000-7f99d4047000 r-xp 00000000 fd:01 33697046 /usr/lib64/libattr.so.1.1.0
7f99d4047000-7f99d4246000 ---p 00004000 fd:01 33697046 /usr/lib64/libattr.so.1.1.0
7f99d4246000-7f99d4247000 r--p 00003000 fd:01 33697046 /usr/lib64/libattr.so.1.1.0
7f99d4247000-7f99d4248000 rw-p 00004000 fd:01 33697046 /usr/lib64/libattr.so.1.1.0
7f99d4248000-7f99d4279000 r-xp 00000000 fd:01 33596341 /usr/lib64/libk5crypto.so.3.1
7f99d4279000-7f99d4478000 ---p 00031000 fd:01 33596341 /usr/lib64/libk5crypto.so.3.1
7f99d4478000-7f99d447a000 r--p 00030000 fd:01 33596341 /usr/lib64/libk5crypto.so.3.1
7f99d447a000-7f99d447b000 rw-p 00032000 fd:01 33596341 /usr/lib64/libk5crypto.so.3.1
7f99d447b000-7f99d447e000 r-xp 00000000 fd:01 33598649 /usr/lib64/libcom_err.so.2.1
7f99d447e000-7f99d467d000 ---p 00003000 fd:01 33598649 /usr/lib64/libcom_err.so.2.1
7f99d467d000-7f99d467e000 r--p 00002000 fd:01 33598649 /usr/lib64/libcom_err.so.2.1
7f99d467e000-7f99d467f000 rw-p 00003000 fd:01 33598649 /usr/lib64/libcom_err.so.2.1
7f99d467f000-7f99d4758000 r-xp 00000000 fd:01 33596476 /usr/lib64/libkrb5.so.3.3
7f99d4758000-7f99d4957000 ---p 000d9000 fd:01 33596476 /usr/lib64/libkrb5.so.3.3
7f99d4957000-7f99d4965000 r--p 000d8000 fd:01 33596476 /usr/lib64/libkrb5.so.3.3
7f99d4965000-7f99d4968000 rw-p 000e6000 fd:01 33596476 /usr/lib64/libkrb5.so.3.3
7f99d4968000-7f99d49b2000 r-xp 00000000 fd:01 33764986 /usr/lib64/libgssapi_krb5.so.2.2
7f99d49b2000-7f99d4bb2000 ---p 0004a000 fd:01 33764986 /usr/lib64/libgssapi_krb5.so.2.2
7f99d4bb2000-7f99d4bb3000 r--p 0004a000 fd:01 33764986 /usr/lib64/libgssapi_krb5.so.2.2
7f99d4bb3000-7f99d4bb5000 rw-p 0004b000 fd:01 33764986 /usr/lib64/libgssapi_krb5.so.2.2
7f99d4bb5000-7f99d4bca000 r-xp 00000000 fd:01 34151745 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f99d4bca000-7f99d4dc9000 ---p 00015000 fd:01 34151745 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f99d4dc9000-7f99d4dca000 r--p 00014000 fd:01 34151745 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f99d4dca000-7f99d4dcb000 rw-p 00015000 fd:01 34151745 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f99d4dcb000-7f99d4e17000 r-xp 00000000 fd:01 33765721 /usr/lib64/libdw-0.172.so
7f99d4e17000-7f99d5017000 ---p 0004c000 fd:01 33765721 /usr/lib64/libdw-0.172.so
7f99d5017000-7f99d5019000 r--p 0004c000 fd:01 33765721 /usr/lib64/libdw-0.172.so
7f99d5019000-7f99d501a000 rw-p 0004e000 fd:01 33765721 /usr/lib64/libdw-0.172.so
7f99d501a000-7f99d5030000 r-xp 00000000 fd:01 34296062 /usr/lib64/libresolv-2.17.so
7f99d5030000-7f99d522f000 ---p 00016000 fd:01 34296062 /usr/lib64/libresolv-2.17.so
7f99d522f000-7f99d5230000 r--p 00015000 fd:01 34296062 /usr/lib64/libresolv-2.17.so
7f99d5230000-7f99d5231000 rw-p 00016000 fd:01 34296062 /usr/lib64/libresolv-2.17.so
7f99d5231000-7f99d5233000 rw-p 00000000 00:00 0
7f99d5233000-7f99d5237000 r-xp 00000000 fd:01 33697212 /usr/lib64/libgpg-error.so.0.10.0
7f99d5237000-7f99d5436000 ---p 00004000 fd:01 33697212 /usr/lib64/libgpg-error.so.0.10.0
7f99d5436000-7f99d5437000 r--p 00003000 fd:01 33697212 /usr/lib64/libgpg-error.so.0.10.0
7f99d5437000-7f99d5438000 rw-p 00004000 fd:01 33697212 /usr/lib64/libgpg-error.so.0.10.0
7f99d5438000-7f99d54b5000 r-xp 00000000 fd:01 34044836 /usr/lib64/libgcrypt.so.11.8.2
7f99d54b5000-7f99d56b4000 ---p 0007d000 fd:01 34044836 /usr/lib64/libgcrypt.so.11.8.2
7f99d56b4000-7f99d56b5000 r--p 0007c000 fd:01 34044836 /usr/lib64/libgcrypt.so.11.8.2
7f99d56b5000-7f99d56b8000 rw-p 0007d000 fd:01 34044836 /usr/lib64/libgcrypt.so.11.8.2
7f99d56b8000-7f99d56b9000 rw-p 00000000 00:00 0
7f99d56b9000-7f99d56cd000 r-xp 00000000 fd:01 33779980 /usr/lib64/liblz4.so.1.7.5
7f99d56cd000-7f99d58cc000 ---p 00014000 fd:01 33779980 /usr/lib64/liblz4.so.1.7.5
7f99d58cc000-7f99d58cd000 r--p 00013000 fd:01 33779980 /usr/lib64/liblz4.so.1.7.5
7f99d58cd000-7f99d58ce000 rw-p 00014000 fd:01 33779980 /usr/lib64/liblz4.so.1.7.5
7f99d58ce000-7f99d58f2000 r-xp 00000000 fd:01 33684614 /usr/lib64/libselinux.so.1
7f99d58f2000-7f99d5af1000 ---p 00024000 fd:01 33684614 /usr/lib64/libselinux.so.1
7f99d5af1000-7f99d5af2000 r--p 00023000 fd:01 33684614 /usr/lib64/libselinux.so.1
7f99d5af2000-7f99d5af3000 rw-p 00024000 fd:01 33684614 /usr/lib64/libselinux.so.1
7f99d5af3000-7f99d5af5000 rw-p 00000000 00:00 0
7f99d5af5000-7f99d5afc000 r-xp 00000000 fd:01 34328001 /usr/lib64/librt-2.17.so
7f99d5afc000-7f99d5cfb000 ---p 00007000 fd:01 34328001 /usr/lib64/librt-2.17.so
7f99d5cfb000-7f99d5cfc000 r--p 00006000 fd:01 34328001 /usr/lib64/librt-2.17.so
7f99d5cfc000-7f99d5cfd000 rw-p 00007000 fd:01 34328001 /usr/lib64/librt-2.17.so
7f99d5cfd000-7f99d5d01000 r-xp 00000000 fd:01 33598623 /usr/lib64/libcap.so.2.22
7f99d5d01000-7f99d5f00000 ---p 00004000 fd:01 33598623 /usr/lib64/libcap.so.2.22
7f99d5f00000-7f99d5f01000 r--p 00003000 fd:01 33598623 /usr/lib64/libcap.so.2.22
7f99d5f01000-7f99d5f02000 rw-p 00004000 fd:01 33598623 /usr/lib64/libcap.so.2.22
7f99d5f02000-7f99d5f04000 r-xp 00000000 fd:01 33597140 /usr/lib64/libfreebl3.so
7f99d5f04000-7f99d6103000 ---p 00002000 fd:01 33597140 /usr/lib64/libfreebl3.so
7f99d6103000-7f99d6104000 r--p 00001000 fd:01 33597140 /usr/lib64/libfreebl3.so
7f99d6104000-7f99d6105000 rw-p 00002000 fd:01 33597140 /usr/lib64/libfreebl3.so
7f99d6105000-7f99d62c7000 r-xp 00000000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d62c7000-7f99d64c7000 ---p 001c2000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d64c7000-7f99d64cb000 r--p 001c2000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d64cb000-7f99d64cd000 rw-p 001c6000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d64cd000-7f99d64d2000 rw-p 00000000 00:00 0
7f99d64d2000-7f99d65d3000 r-xp 00000000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d65d3000-7f99d67d2000 ---p 00101000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d67d2000-7f99d67d3000 r--p 00100000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d67d3000-7f99d67d4000 rw-p 00101000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d67d4000-7f99d68bd000 r-xp 00000000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d68bd000-7f99d6abc000 ---p 000e9000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d6abc000-7f99d6ac4000 r--p 000e8000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d6ac4000-7f99d6ac6000 rw-p 000f0000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d6ac6000-7f99d6adb000 rw-p 00000000 00:00 0
7f99d6adb000-7f99d6add000 r-xp 00000000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6add000-7f99d6cdd000 ---p 00002000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6cdd000-7f99d6cde000 r--p 00002000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6cde000-7f99d6cdf000 rw-p 00003000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6cdf000-7f99d6f15000 r-xp 00000000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d6f15000-7f99d7115000 ---p 00236000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d7115000-7f99d7131000 r--p 00236000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d7131000-7f99d713e000 rw-p 00252000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d713e000-7f99d7142000 rw-p 00000000 00:00 0
7f99d7142000-7f99d71a9000 r-xp 00000000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d71a9000-7f99d73a9000 ---p 00067000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d73a9000-7f99d73ad000 r--p 00067000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d73ad000-7f99d73b4000 rw-p 0006b000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d73b4000-7f99d73e3000 r-xp 00000000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d73e3000-7f99d75e3000 ---p 0002f000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d75e3000-7f99d75e4000 r--p 0002f000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d75e4000-7f99d75e5000 rw-p 00030000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d75e5000-7f99d75ed000 r-xp 00000000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d75ed000-7f99d77ec000 ---p 00008000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d77ec000-7f99d77ed000 r--p 00007000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d77ed000-7f99d77ee000 rw-p 00008000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d6104000-7f99d6105000 rw-p 00002000 fd:01 33597140 /usr/lib64/libfreebl3.so
7f99d6105000-7f99d62c7000 r-xp 00000000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d62c7000-7f99d64c7000 ---p 001c2000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d64c7000-7f99d64cb000 r--p 001c2000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d64cb000-7f99d64cd000 rw-p 001c6000 fd:01 33630260 /usr/lib64/libc-2.17.so
7f99d64cd000-7f99d64d2000 rw-p 00000000 00:00 0
7f99d64d2000-7f99d65d3000 r-xp 00000000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d65d3000-7f99d67d2000 ---p 00101000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d67d2000-7f99d67d3000 r--p 00100000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d67d3000-7f99d67d4000 rw-p 00101000 fd:01 34151789 /usr/lib64/libm-2.17.so
7f99d67d4000-7f99d68bd000 r-xp 00000000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d68bd000-7f99d6abc000 ---p 000e9000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d6abc000-7f99d6ac4000 r--p 000e8000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d6ac4000-7f99d6ac6000 rw-p 000f0000 fd:01 33596627 /usr/lib64/libstdc++.so.6.0.19
7f99d6ac6000-7f99d6adb000 rw-p 00000000 00:00 0
7f99d6adb000-7f99d6add000 r-xp 00000000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6add000-7f99d6cdd000 ---p 00002000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6cdd000-7f99d6cde000 r--p 00002000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6cde000-7f99d6cdf000 rw-p 00003000 fd:01 34151787 /usr/lib64/libdl-2.17.so
7f99d6cdf000-7f99d6f15000 r-xp 00000000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d6f15000-7f99d7115000 ---p 00236000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d7115000-7f99d7131000 r--p 00236000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d7131000-7f99d713e000 rw-p 00252000 fd:01 33680561 /usr/lib64/libcrypto.so.1.0.2k
7f99d713e000-7f99d7142000 rw-p 00000000 00:00 0
7f99d7142000-7f99d71a9000 r-xp 00000000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d71a9000-7f99d73a9000 ---p 00067000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d73a9000-7f99d73ad000 r--p 00067000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d73ad000-7f99d73b4000 rw-p 0006b000 fd:01 33838887 /usr/lib64/libssl.so.1.0.2k
7f99d73b4000-7f99d73e3000 r-xp 00000000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d73e3000-7f99d75e3000 ---p 0002f000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d75e3000-7f99d75e4000 r--p 0002f000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d75e4000-7f99d75e5000 rw-p 00030000 fd:01 33630657 /usr/lib64/libsystemd.so.0.6.0
7f99d75e5000-7f99d75ed000 r-xp 00000000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d75ed000-7f99d77ec000 ---p 00008000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d77ec000-7f99d77ed000 r--p 00007000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d77ed000-7f99d77ee000 rw-p 00008000 fd:01 33665949 /usr/lib64/libcrypt-2.17.so
7f99d77ee000-7f99d781c000 rw-p 00000000 00:00 0
7f99d781c000-7f99d7831000 r-xp 00000000 fd:01 33697034 /usr/lib64/libz.so.1.2.7
7f99d7831000-7f99d7a30000 ---p 00015000 fd:01 33697034 /usr/lib64/libz.so.1.2.7
7f99d7a30000-7f99d7a31000 r--p 00014000 fd:01 33697034 /usr/lib64/libz.so.1.2.7
7f99d7a31000-7f99d7a32000 rw-p 00015000 fd:01 33697034 /usr/lib64/libz.so.1.2.7
7f99d7a32000-7f99d7a33000 r-xp 00000000 fd:01 34044870 /usr/lib64/libaio.so.1.0.1
7f99d7a33000-7f99d7c32000 ---p 00001000 fd:01 34044870 /usr/lib64/libaio.so.1.0.1
7f99d7c32000-7f99d7c33000 r--p 00000000 fd:01 34044870 /usr/lib64/libaio.so.1.0.1
7f99d7c33000-7f99d7c34000 rw-p 00001000 fd:01 34044870 /usr/lib64/libaio.so.1.0.1
7f99d7c34000-7f99d7c59000 r-xp 00000000 fd:01 33697099 /usr/lib64/liblzma.so.5.2.2
7f99d7c59000-7f99d7e58000 ---p 00025000 fd:01 33697099 /usr/lib64/liblzma.so.5.2.2
7f99d7e58000-7f99d7e59000 r--p 00024000 fd:01 33697099 /usr/lib64/liblzma.so.5.2.2
7f99d7e59000-7f99d7e5a000 rw-p 00025000 fd:01 33697099 /usr/lib64/liblzma.so.5.2.2
7f99d7e5a000-7f99d7e71000 r-xp 00000000 fd:01 34296060 /usr/lib64/libpthread-2.17.so
7f99d7e71000-7f99d8070000 ---p 00017000 fd:01 34296060 /usr/lib64/libpthread-2.17.so
7f99d8070000-7f99d8071000 r--p 00016000 fd:01 34296060 /usr/lib64/libpthread-2.17.so
7f99d8071000-7f99d8072000 rw-p 00017000 fd:01 34296060 /usr/lib64/libpthread-2.17.so
7f99d8072000-7f99d8076000 rw-p 00000000 00:00 0
7f99d8076000-7f99d80fa000 r-xp 00000000 fd:01 34029542 /usr/lib64/liboneagentproc.so
7f99d80fa000-7f99d82f9000 ---p 00084000 fd:01 34029542 /usr/lib64/liboneagentproc.so
7f99d82f9000-7f99d82fc000 r--p 00083000 fd:01 34029542 /usr/lib64/liboneagentproc.so
7f99d82fc000-7f99d82fd000 rw-p 00086000 fd:01 34029542 /usr/lib64/liboneagentproc.so
7f99d82fd000-7f99d83e7000 rw-p 00000000 00:00 0
7f99d83e7000-7f99d8409000 r-xp 00000000 fd:01 33597130 /usr/lib64/ld-2.17.so
7f99d8411000-7f99d8413000 rw-p 00000000 00:00 0
7f99d8413000-7f99d8414000 rw-s 00000000 00:0b 17277991 /[aio] (deleted)
7f99d8414000-7f99d8417000 rw-s 00000000 00:0b 17277990 /[aio] (deleted)
7f99d8417000-7f99d841a000 rw-s 00000000 00:0b 17277989 /[aio] (deleted)
7f99d841a000-7f99d841d000 rw-s 00000000 00:0b 17277988 /[aio] (deleted)
7f99d841d000-7f99d8420000 rw-s 00000000 00:0b 17277987 /[aio] (deleted)
7f99d8420000-7f99d8423000 rw-s 00000000 00:0b 17277986 /[aio] (deleted)
7f99d8423000-7f99d8426000 rw-s 00000000 00:0b 17277985 /[aio] (deleted)
7f99d8426000-7f99d8429000 rw-s 00000000 00:0b 17277984 /[aio] (deleted)
7f99d8429000-7f99d84e8000 rw-p 00000000 00:00 0
7f99d84e8000-7f99d84e9000 r--p 00000000 00:00 0
7f99d84e9000-7f99d85fc000 rw-p 00000000 00:00 0
7f99d85fc000-7f99d85ff000 rw-s 00000000 00:0b 17277983 /[aio] (deleted)
7f99d85ff000-7f99d8602000 rw-s 00000000 00:0b 17277982 /[aio] (deleted)
7f99d8602000-7f99d8605000 rw-s 00000000 00:0b 17277981 /[aio] (deleted)
7f99d8605000-7f99d8606000 rw-s 00000000 00:0b 17277980 /[aio] (deleted)
7f99d8606000-7f99d8608000 rw-p 00000000 00:00 0
7f99d8608000-7f99d8609000 r--p 00021000 fd:01 33597130 /usr/lib64/ld-2.17.so
7f99d8609000-7f99d860a000 rw-p 00022000 fd:01 33597130 /usr/lib64/ld-2.17.so
7f99d860a000-7f99d860b000 rw-p 00000000 00:00 0
7ffe4ce8a000-7ffe4ceab000 rw-p 00000000 00:00 0 [stack]
7ffe4cfd1000-7ffe4cfd3000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
191029 23:11:45 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
:
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.4.8-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=3002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6737182 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
addr2line: 'mysqld': No such file
mysqld(my_print_stacktrace+0x2e)[0x55ada5b6e83e]
mysqld(handle_fatal_signal+0x30f)[0x55ada5604e0f]
sigaction.c:0(__restore_rt)[0x7f99d7e695d0]
:0(__GI_raise)[0x7f99d613b2c7]
:0(__GI_abort)[0x7f99d613c9b8]
:0(__libc_message)[0x7f99d617de17]
malloc.c:0(_int_free)[0x7f99d6186609]
bits/stl_set.h:582(std::set<void*, std::less<void*>, std::allocator<void*> >::erase(void* const&))[0x7f998b65418b]
src/gcache_mem_store.hpp:81(gcache::MemStore::free(gcache::BufferHeader*))[0x7f998b651c0e]
src/GCache_seqno.cpp:207(gcache::GCache::seqno_release(long))[0x7f998b63b852]
src/galera_service_thd.cpp:76(galera::ServiceThd::thd_func(void*))[0x7f998b7509db]
pthread_create.c:0(start_thread)[0x7f99d7e61dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f99d620302d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /db/data3
Resource Limits:
Fatal signal 11 while backtracing
(END)
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.4.8-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=3002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6737182 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
addr2line: 'mysqld': No such file
mysqld(my_print_stacktrace+0x2e)[0x55ada5b6e83e]
mysqld(handle_fatal_signal+0x30f)[0x55ada5604e0f]
sigaction.c:0(__restore_rt)[0x7f99d7e695d0]
:0(__GI_raise)[0x7f99d613b2c7]
:0(__GI_abort)[0x7f99d613c9b8]
:0(__libc_message)[0x7f99d617de17]
malloc.c:0(_int_free)[0x7f99d6186609]
bits/stl_set.h:582(std::set<void*, std::less<void*>, std::allocator<void*> >::erase(void* const&))[0x7f998b65418b]
src/gcache_mem_store.hpp:81(gcache::MemStore::free(gcache::BufferHeader*))[0x7f998b651c0e]
src/GCache_seqno.cpp:207(gcache::GCache::seqno_release(long))[0x7f998b63b852]
src/galera_service_thd.cpp:76(galera::ServiceThd::thd_func(void*))[0x7f998b7509db]
pthread_create.c:0(start_thread)[0x7f99d7e61dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f99d620302d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /db/data3
Resource Limits:
Fatal signal 11 while backtracing

Comment by Teemu Ollakka [ 2020-03-16 ]

Would it be possible to get wsrep_provider_options settings from MariaDB configuration file?

I could reproduce the crash by setting gcache.mem_size=1M in wsrep_provider_options. After removing the gcache.mem_size setting the crash did not happen anymore.

Comment by Alexey [ 2020-07-22 ]

Hi Julien. This issue is most likely a result of GCache memory corruption due to which writeset metadata were rendered unusable. The causes can be many, but I think that upgrade from Galera 3.x to Galera 4.x is a smoking gun: GCache metadata format has changed from 3.x to 4.x and given some bugs present in Galera before versions 3.30/4.5 the writeset histories in GCache could get mixed.

Seeing more of the error log, what was happening before the shutdown started, could help to diagnose that better.

Comment by Hartmut Holzgraefe [ 2020-08-25 ]

The only additional log snippet I have is

2019-10-29 23:11:33 0 [Note] mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2019-10-29 23:11:33 0 [Note] WSREP: Shutdown replication
2019-10-29 23:11:33 0 [Note] WSREP: Server status change synced -> disconnecting
2019-10-29 23:11:33 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:33 0 [Note] WSREP: Closing send monitor...
2019-10-29 23:11:33 0 [Note] WSREP: Closed send monitor.
2019-10-29 23:11:33 0 [Note] WSREP: gcomm: terminating thread
2019-10-29 23:11:33 0 [Note] WSREP: gcomm: joining thread
2019-10-29 23:11:33 0 [Note] WSREP: gcomm: closing backend
2019-10-29 23:11:36 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') connection to peer f6e2057e with addr tcp://10.142.8.142:4333 timed out, no messages seen in PT3S
2019-10-29 23:11:36 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') turning message relay requesting on, nonlive peers: tcp://10.142.8.142:4333
2019-10-29 23:11:38 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') reconnecting to f6e2057e (tcp://10.142.8.142:4333), attempt 0
2019-10-29 23:11:41 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') connection to peer f6e2057e with addr tcp://10.142.8.142:4333 timed out, no messages seen in PT3S
2019-10-29 23:11:43 0 [Note] WSREP: (eb0b5481, 'tcp://0.0.0.0:4333') connection established to f6e2057e tcp://10.142.8.142:4333
2019-10-29 23:11:45 0 [Note] WSREP: evs::proto(eb0b5481, LEAVING, view_id(REG,eb0b5481,6)) suspecting node: f6e2057e
2019-10-29 23:11:45 0 [Note] WSREP: evs::proto(eb0b5481, LEAVING, view_id(REG,eb0b5481,6)) suspected node without join message, declaring inactive
2019-10-29 23:11:45 0 [Note] WSREP: view(view_id(NON_PRIM,eb0b5481,6) memb {
eb0b5481,0
} joined {
} left {
} partitioned {
f6e2057e,0
})
2019-10-29 23:11:45 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2019-10-29 23:11:45 0 [Note] WSREP: view((empty))
2019-10-29 23:11:45 0 [Note] WSREP: gcomm: closed
2019-10-29 23:11:45 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2019-10-29 23:11:45 0 [Note] WSREP: Writing down CC checksum: 39b2e180 6e10a4de 74927600 a6dff24f at offset 120
2019-10-29 23:11:45 0 [Note] WSREP: Flow-control interval: [64, 64]
2019-10-29 23:11:45 0 [Note] WSREP: Trying to continue unpaused monitor
2019-10-29 23:11:45 0 [Note] WSREP: Received NON-PRIMARY.
2019-10-29 23:11:45 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 9)
2019-10-29 23:11:45 0 [Note] WSREP: Received self-leave message.
2019-10-29 23:11:45 0 [Note] WSREP: Writing down CC checksum: 94d1c656 7a51b027 67573338 31d1e2be at offset 64
2019-10-29 23:11:45 0 [Note] WSREP: Flow-control interval: [0, 0]
2019-10-29 23:11:45 0 [Note] WSREP: Trying to continue unpaused monitor
2019-10-29 23:11:45 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2019-10-29 23:11:45 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
2019-10-29 23:11:45 2 [Note] WSREP: ####### processing CC -1, local, ordered
2019-10-29 23:11:45 0 [Note] WSREP: RECV thread exiting 0: Success
2019-10-29 23:11:45 0 [Note] WSREP: recv_thread() joined.
2019-10-29 23:11:45 0 [Note] WSREP: Closing replication queue.
2019-10-29 23:11:45 0 [Note] WSREP: Closing slave action queue.
2019-10-29 23:11:45 2 [Note] WSREP: ####### drain monitors upto 9
2019-10-29 23:11:45 2 [Note] WSREP: ####### My UUID: eb0b5481-fa31-11e9-b396-3e828533d8bb
2019-10-29 23:11:45 2 [Note] WSREP: ####### ST not required
2019-10-29 23:11:45 2 [Note] WSREP: ================================================
View:
id: eb0c6714-fa31-11e9-8759-afff06cd3fd6:-1
status: non-primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: eb0b5481-fa31-11e9-b396-3e828533d8bb, rndmdc401
=================================================
2019-10-29 23:11:45 2 [Note] WSREP: Non-primary view
2019-10-29 23:11:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:45 2 [Note] WSREP: ####### processing CC -1, local, ordered
2019-10-29 23:11:45 2 [Note] WSREP: ####### drain monitors upto 9
2019-10-29 23:11:45 2 [Note] WSREP: ####### My UUID: eb0b5481-fa31-11e9-b396-3e828533d8bb
2019-10-29 23:11:45 2 [Note] WSREP: ####### ST not required
2019-10-29 23:11:45 2 [Note] WSREP: ================================================
View:
id: eb0c6714-fa31-11e9-8759-afff06cd3fd6:-1
status: non-primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: yes
own_index: -1
members(0):
=================================================
2019-10-29 23:11:45 2 [Note] WSREP: Non-primary view
2019-10-29 23:11:45 2 [Note] WSREP: Server status change disconnecting -> disconnected
2019-10-29 23:11:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:45 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-10-29 23:11:45 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
2019-10-29 23:11:45 1 [Note] WSREP: rollbacker thread exiting 1
2019-10-29 23:11:45 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
2019-10-29 23:11:45 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
2019-10-29 23:11:45 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 9
2019-10-29 23:11:45 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply
2019-10-29 23:11:45 0 [Note] Event Scheduler: Stopped
2019-10-29 23:11:45 0 [Note] Event Scheduler: Purging the queue. 0 events
2019-10-29 23:11:45 0 [Note] InnoDB: FTS optimize thread exiting.
2019-10-29 23:11:45 0 [Note] WSREP: dtor state: CLOSED
2019-10-29 23:11:45 0 [Note] WSREP: MemPool(TrxHandleSlave): hit ratio: 0, misses: 3, in use: 3, in pool: 0
2019-10-29 23:11:45 0 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
2019-10-29 23:11:45 0 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
2019-10-29 23:11:45 0 [Note] WSREP: mon: entered 21 oooe fraction 0 oool fraction 0
2019-10-29 23:11:45 0 [Note] WSREP: cert index usage at exit 1
2019-10-29 23:11:45 0 [Note] WSREP: cert trx map usage at exit 3
2019-10-29 23:11:45 0 [Note] WSREP: deps set usage at exit 0
2019-10-29 23:11:45 0 [Note] WSREP: avg deps dist 1
2019-10-29 23:11:45 0 [Note] WSREP: avg cert interval 0
2019-10-29 23:11:45 0 [Note] WSREP: cert index size 1
*** Error in `mysqld': free(): invalid pointer: 0x00007f9703fff500 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81609)[0x7f99d6186609]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6gcache8MemStore7discardEPNS_12BufferHeaderE+0x1b)[0x7f998b65418b]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6gcache6GCache11free_commonEPNS_12BufferHeaderE+0x5e)[0x7f998b651c0e]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6gcache6GCache13seqno_releaseEl+0x122)[0x7f998b63b852]
/usr/lib64/galera-4/libgalera_smm.so(_ZN6galera10ServiceThd8thd_funcEPv+0x14b)[0x7f998b7509db]
/lib64/libpthread.so.0(+0x7dd5)[0x7f99d7e61dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f99d620302d]

Generated at Thu Feb 08 09:03:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.