[MDEV-32467] Coredumps were not generated in data directory in RHEL8 Created: 2023-10-13  Updated: 2023-11-21

Status: Confirmed
Project: MariaDB Server
Component/s: Documentation
Affects Version/s: 10.6
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Pandikrishnan Gurusamy Assignee: Ian Gilfillan
Resolution: Unresolved Votes: 0
Labels: None
Environment:

RHEL 8



 Description   

In RHEL 8 , the core dumps were not generated on the data directory with the default value in kernel.core_pattern

MariaDB [(none)]> select @@version,@@core_file;
+------------------------------+-------------+
| @@version                    | @@core_file |
+------------------------------+-------------+
| 10.6.14-9-MariaDB-enterprise |           1 |
+------------------------------+-------------+

[root@ip-172-31-32-162]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 (Ootpa)
[root@ip-172-31-32-162]# sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
 
[root@ip-172-31-32-162 ~]# coredumpctl list
No coredumps found.
[root@ip-172-31-32-162 ~]# ls -lrth /var/lib/mysql/core*
[root@ip-172-31-32-162 ~]# kill -6 `pidof mariadbd`
[root@ip-172-31-32-162 mysql]# ls -lrth /var/lib/mysql/
total 121M
drwx------. 2 mysql mysql 4.0K Oct 13 03:27 mysql
drwx------. 2 mysql mysql   20 Oct 13 03:27 performance_schema
drwx------. 2 mysql mysql 8.0K Oct 13 03:27 sys
-rw-r--r--. 1 mysql mysql   17 Oct 13 03:27 mysql_upgrade_info
-rw-rw----. 1 mysql mysql    0 Oct 13 03:45 multi-master.info
-rw-rw----. 1 mysql mysql  910 Oct 13 08:09 ib_buffer_pool
-rw-rw----. 1 mysql mysql  12M Oct 13 08:09 ibdata1
-rw-rw----. 1 mysql mysql   52 Oct 13 08:09 aria_log_control
-rw-rw----. 1 mysql mysql 408K Oct 13 08:09 aria_log.00000001
-rw-rw----. 1 mysql mysql  12M Oct 13 08:17 ibtmp1
-rw-rw----. 1 mysql mysql    9 Oct 13 08:17 ddl_recovery.log
-rw-rw----. 1 mysql mysql    9 Oct 13 08:17 ddl_recovery-backup.log
srwxrwxrwx. 1 mysql mysql    0 Oct 13 08:17 mysql.sock
-rw-rw----. 1 mysql mysql    5 Oct 13 08:17 ip-172-31-32-162.pid
-rw-rw----. 1 mysql mysql  22K Oct 13 08:17 error.log
-rw-rw----. 1 mysql mysql  96M Oct 13 08:17 ib_logfile0
 
[root@ip-172-31-32-162]# coredumpctl list
TIME                            PID   UID   GID SIG COREFILE  EXE
Fri 2023-10-13 08:17:53 UTC    1533   994   991   6 present   /usr/sbin/mariadbd
 
[root@ip-172-31-32-162]#  ls -lrth /var/lib/systemd/coredump/ 
total 2.4M
-rw-r-----. 1 root root 2.4M Oct 13 08:17 core.mariadbd.994.38608d77e33b4886a68706b8dfa975f5.1533.1697185069000000.lz4
 

Error log:

Version: '10.6.14-9-MariaDB-enterprise'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Enterprise Server
231013  8:17:49 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.6.14-9-MariaDB-enterprise source revision: 0c6b89c694a88aae764b152045bf5bdb234d7edf
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467982 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5588c425ce4e]
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5588c3d21bb5]
/lib64/libpthread.so.0(+0x12cf0)[0x7f1492c57cf0]
/lib64/libc.so.6(__poll+0x51)[0x7f14920890e1]
/usr/sbin/mariadbd(_Z26handle_connections_socketsv+0x1da)[0x5588c39fe60a]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x1080)[0x5588c39ff930]
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7f1491f9ed85]
/usr/sbin/mariadbd(_start+0x2e)[0x5588c39f3eae]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             2928                 2928                 processes 
Max open files            32768                32768                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       2928                 2928                 signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
 
Kernel version: Linux version 4.18.0-425.13.1.el8_7.x86_64 (mockbuild@x86-vm-07.build.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-16) (GCC)) #1 SMP Thu Feb 2 13:01:45 EST 2023

Our KB is not clear about this setting kernel.core_pattern in RHEL 8 what settings need to be change to generate core dump in data directory.
https://mariadb.com/kb/en/enabling-core-dumps/#extracting-linux-core-dumps-with-systemd-coredump



 Comments   
Comment by Valerii Kravchuk [ 2023-10-13 ]

I think our KB actually documented this case in enough details:

https://mariadb.com/kb/en/enabling-core-dumps/#extracting-linux-core-dumps-with-systemd-coredump

Comment by Sergei Golubchik [ 2023-10-13 ]

pandi.gurusamy, what exactly do you think is unclear in the documentation?

Comment by Valerii Kravchuk [ 2023-10-13 ]

I think we should list in KB those supported Linux versions where systemd is used by default to manage core dumps. With a warning about a change in behavior for those upgrading from older versions.

Generated at Thu Feb 08 10:31:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.