[MDEV-32116] Server suddenly crashed Created: 2023-09-07  Updated: 2023-11-04  Resolved: 2023-11-04

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 11.0.2
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Kamil Cukrowski Assignee: Marko Mäkelä
Resolution: Cannot Reproduce Votes: 0
Labels: SEGV, segfault
Environment:

Mariadb: docker mariadb:11.0.2
Running in Nomad v.1.6.1 with no memory limit
On host: ArchLinux latest 6.1.51-1-lts #1 SMP PREEMPT_DYNAMIC Sat, 02 Sep 2023 14:51:30 +0000 x86_64 GNU/Linux


Attachments: Text File dmesg.txt     Text File stderr.txt     Text File stdout.txt    
Issue Links:
Relates
relates to MDEV-31767 InnoDB tables are being flagged as co... Closed

 Description   

Thanks for making mariadb, it is great!

FYI Mysql received a segfault. Nomad started auto-restarting it, and then it couldn't come up. I do not know for how long it was.

I will update to 11.0.3, so you can ignore. Thanks!



 Comments   
Comment by Marko Mäkelä [ 2023-09-07 ]

Hi! Thank you for the report. The reports of corruption could match MDEV-31767. Can you check if an upgrade to MariaDB Server 11.0.3 would fix this?

Comment by Marko Mäkelä [ 2023-09-07 ]

The crashes in stderr.txt probably occur somewhere inside InnoDB, during an INSERT operation. Unfortunately, the built-in stack trace report does not correctly resolve InnoDB symbols for some reason.

Can you please try to follow the instructions https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ to produce a full stack trace of the crash? danblack should be able to assist you with setting up and using a debug image. We successfully used that to analyze MDEV-32049.

Even though an upgrade to 11.0.3 may make the error go away, I would like to find and fix that crash, because InnoDB is not supposed to crash anymore, no matter what, in the spirit of MDEV-13542.

Comment by Daniel Black [ 2023-09-07 ]

First, can you build your own image from:

Dockerfile

FROM mariadb:11.0.2
 
RUN sed -i -e 's/main/main main\/debug/' /etc/apt/sources.list.d/mariadb.list && \
    apt-get update \
        && apt-get install -y linux-tools-common gdbserver gdb curl mariadb-server-core-dbgsym=11.0.2 ; \
        rm -rf /var/lib/apt/lists/*

buildah bud --tag m11.0.2debug .

Other dockerfile builders obviously acceptable.

podman run --user mysql -v volumename:/var/lib/mysql  m11.0.2debug   gdb -ex r -ex 'thread apply all bt -frame-arguments all full' --args  mariadbd

Just translate the podman/docker cmd into how nomad runs things.

Comment by Kamil Cukrowski [ 2023-10-04 ]

As I updated to 11.0.3 and I do not have a way to reproduce, I believe this can be closed to cleanup. Thanks!

Generated at Thu Feb 08 10:28:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.