Details
-
Bug
-
Status: Confirmed (View Workflow)
-
Critical
-
Resolution: Unresolved
-
10.6
-
None
-
RHEL 8
Description
In RHEL 8 , the core dumps were not generated on the data directory with the default value in kernel.core_pattern
MariaDB [(none)]> select @@version,@@core_file;
|
+------------------------------+-------------+
|
| @@version | @@core_file |
|
+------------------------------+-------------+
|
| 10.6.14-9-MariaDB-enterprise | 1 |
|
+------------------------------+-------------+
|
[root@ip-172-31-32-162]# cat /etc/redhat-release
|
Red Hat Enterprise Linux release 8.7 (Ootpa)
|
[root@ip-172-31-32-162]# sysctl kernel.core_pattern
|
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
|
|
|
[root@ip-172-31-32-162 ~]# coredumpctl list
|
No coredumps found.
|
[root@ip-172-31-32-162 ~]# ls -lrth /var/lib/mysql/core*
|
[root@ip-172-31-32-162 ~]# kill -6 `pidof mariadbd`
|
[root@ip-172-31-32-162 mysql]# ls -lrth /var/lib/mysql/
|
total 121M
|
drwx------. 2 mysql mysql 4.0K Oct 13 03:27 mysql
|
drwx------. 2 mysql mysql 20 Oct 13 03:27 performance_schema
|
drwx------. 2 mysql mysql 8.0K Oct 13 03:27 sys
|
-rw-r--r--. 1 mysql mysql 17 Oct 13 03:27 mysql_upgrade_info
|
-rw-rw----. 1 mysql mysql 0 Oct 13 03:45 multi-master.info
|
-rw-rw----. 1 mysql mysql 910 Oct 13 08:09 ib_buffer_pool
|
-rw-rw----. 1 mysql mysql 12M Oct 13 08:09 ibdata1
|
-rw-rw----. 1 mysql mysql 52 Oct 13 08:09 aria_log_control
|
-rw-rw----. 1 mysql mysql 408K Oct 13 08:09 aria_log.00000001
|
-rw-rw----. 1 mysql mysql 12M Oct 13 08:17 ibtmp1
|
-rw-rw----. 1 mysql mysql 9 Oct 13 08:17 ddl_recovery.log
|
-rw-rw----. 1 mysql mysql 9 Oct 13 08:17 ddl_recovery-backup.log
|
srwxrwxrwx. 1 mysql mysql 0 Oct 13 08:17 mysql.sock
|
-rw-rw----. 1 mysql mysql 5 Oct 13 08:17 ip-172-31-32-162.pid
|
-rw-rw----. 1 mysql mysql 22K Oct 13 08:17 error.log
|
-rw-rw----. 1 mysql mysql 96M Oct 13 08:17 ib_logfile0
|
|
|
[root@ip-172-31-32-162]# coredumpctl list
|
TIME PID UID GID SIG COREFILE EXE
|
Fri 2023-10-13 08:17:53 UTC 1533 994 991 6 present /usr/sbin/mariadbd
|
|
|
[root@ip-172-31-32-162]# ls -lrth /var/lib/systemd/coredump/
|
total 2.4M
|
-rw-r-----. 1 root root 2.4M Oct 13 08:17 core.mariadbd.994.38608d77e33b4886a68706b8dfa975f5.1533.1697185069000000.lz4
|
|
|
|
Error log:
Version: '10.6.14-9-MariaDB-enterprise' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Enterprise Server
|
231013 8:17:49 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.14-9-MariaDB-enterprise source revision: 0c6b89c694a88aae764b152045bf5bdb234d7edf
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=153
|
thread_count=0
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467982 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5588c425ce4e]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5588c3d21bb5]
|
/lib64/libpthread.so.0(+0x12cf0)[0x7f1492c57cf0]
|
/lib64/libc.so.6(__poll+0x51)[0x7f14920890e1]
|
/usr/sbin/mariadbd(_Z26handle_connections_socketsv+0x1da)[0x5588c39fe60a]
|
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x1080)[0x5588c39ff930]
|
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7f1491f9ed85]
|
/usr/sbin/mariadbd(_start+0x2e)[0x5588c39f3eae]
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 2928 2928 processes
|
Max open files 32768 32768 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 2928 2928 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
|
|
|
Kernel version: Linux version 4.18.0-425.13.1.el8_7.x86_64 (mockbuild@x86-vm-07.build.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-16) (GCC)) #1 SMP Thu Feb 2 13:01:45 EST 2023
|
|
Our KB is not clear about this setting kernel.core_pattern in RHEL 8 what settings need to be change to generate core dump in data directory.
https://mariadb.com/kb/en/enabling-core-dumps/#extracting-linux-core-dumps-with-systemd-coredump
Attachments
Activity
pandi.gurusamy, what exactly do you think is unclear in the documentation?
I think we should list in KB those supported Linux versions where systemd is used by default to manage core dumps. With a warning about a change in behavior for those upgrading from older versions.
There are on https://mariadb.com/kb/en/enabling-core-dumps/#extracting-linux-core-dumps-with-systemd-coredump for extracting a core dumps saved by systemd's coredump handler.
Usually putting them into the datadir isn't the goal, but extracting information from it, hence "coredumpctl debug ... " is there too.
I think our KB actually documented this case in enough details:
https://mariadb.com/kb/en/enabling-core-dumps/#extracting-linux-core-dumps-with-systemd-coredump