[MDEV-30548] munmap_chunk(): invalid pointer Created: 2023-02-02  Updated: 2023-07-28  Resolved: 2023-07-28

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.5.18
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Lars Timmann Assignee: Julius Goryavsky
Resolution: Incomplete Votes: 1
Labels: galera
Environment:

Ubuntu 20.04.5

Attachments: File maria2021-ham-1.log     File maria2021-ham-2.bt_full     File maria2021-ham-2.stacktrace     File mariadb.log    

 Description   

Hi MariDB-Team,
since a reboot one of my four mariadb galera nodes reports munmap_chunk(): invalid pointer and mysqld ends up with SIGABRT (signal 6).

At the moment the other three nodes still work as a cluster and I can keep this one in this state for further investigations. So if you want me to inspect something this can be done.
Greetings,
Lars

 Comments   
Comment by Daniel Black [ 2023-02-02 ]

Did it generate a core file in /GALERANFS_HAM_DATA/data/mariadb? If so a full backtrace would be appreciated. If you think this is containing private data (potentially some of the variables) https://mariadb.com/kb/en/meta/mariadb-ftp-server/ can be used to upload this file.

If you have the logs of the donor (maria2021-ham-1) that might be useful.

My look at the stack trace seem to be making the gcache go back a transaction sequence number that that's causing the assert. There hasn't been any fixes in gcache recently.

Comment by Lars Timmann [ 2023-02-03 ]

Hi Daniel,
there is no core file at maria2021-ham-2 after crash, but I attached what maria2021-ham-1 is logging during the startup of maria2021-ham-2.
Greetings,
Lars

Comment by Lars Timmann [ 2023-02-03 ]

I installed systemd-coredump and attached two outputs:

  1. coredumpctl list /usr/sbin/mariadbd
    TIME PID UID GID SIG COREFILE EXE
    Fri 2023-02-03 10:24:01 CET 211804 123 128 6 present /usr/sbin/mariadbd
  2. coredumpctl dump -o mariadbd.core 211804 > maria2021-ham-2.stacktrace 2>&1

and

  1. gdb /usr/sbin/mariadbd mariadbd.core -batch -ex 'thread apply all bt full' > maria2021-ham-2.bt_full

Hope that helps.
Lars

Comment by Daniel Black [ 2023-02-03 ]

lollypop that's great. Can you keep the core file around. I (or another engineer assigned here) might have some more requests of information to extract out of it.

If willing, completely optional, you could upload the core to the ftp server above. The buffer pools are excluded from the coredump but there still might be microfragments of user data (though the stack trace didn't).

There's still a bit to go. We see the current state, just need to work out how it got there.

Comment by Julius Goryavsky [ 2023-06-22 ]

lollypop Hi, the issue cannot be reproduced and a reproducible test case is needed. Can you test the reproducibility of this bug with current releases like 10.5.21+ and also with galera library 26.4.14? We have made several fixes in all parts of the system, but we cannot reproduce this crash, so it would be interesting to know if it is repeated with the current version of the server and the galera library?

Generated at Thu Feb 08 10:17:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.