Red Hat Enterprise Linux release 8.7 (Ootpa)
MariaDB-server-10.6.11-1.el8.x86_64

Additional note:
Differences in config between one intel and one AMD server slave:
Intel / AMD
innodb_buffer_pool_size 343597383680 / 549755813888
large_pages OFF / ON

Along with AMD servers being run as "ExecStart=numactl --cpunodebind=1 --interleave=all ...." due to suspected severe performance problems when 2 CPUs are used.

Hyperthreading was turned off all the time on AMD servers(as they have a total 128 cores we didn't think it would help much).

Can you please try to identify the crashing instruction, similar to how it was done in MDEV-24745?

As a starting point, you should install the debug symbols for the server. I think that in rpm-based packages, the package name has -debuginfo in it. Then, try to produce a full stack trace using a debugger. In GDB, I would additionally suggest

info registers

in the crashing thread (usually but not always it is thread 1).

Also, which location did you download the package from? I think that it will be necessary to examine the exact same binary code to figure out what is going on.

> Can you please try to identify the crashing instruction, similar to how it was done
I am not familar with the technical details in the ticket you linked to, it seems out of my depth.
Does that require retrying the bug? Of note our DB is 1.5 TB.
I have backed up the data just in case something is needed for reply, and are copying the binary data from an intel slave back to AMD slave just in case it is broken to be on the safe side.
I'll keep the old mariadb data for a few days if there is something you need.

> Also, which location did you download the package from? I think that it will be necessary to examine the exact same binary code to figure out what is going on.
This is easy, we used the following offical MariaDB repo:
/etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = http://archive.mariadb.org/mariadb-10.6.11/yum/rhel8-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

One question is do you have any opinion or comments regarding the performance problems we suspect on AMD EPYC dual 7773X servers? Do you think we would have better performance on single EPYC cpu servers or recommend just using Intel servers?

Tha nkyou.

FK, the problem is that the built-in stack trace reporter is garbage, at least in anything above mysql_alter_table(). Something apparently crashes during a table rebuild that is being executed during OPTIMIZE TABLE.

InnoDB used to crash on corrupted data until MDEV-13542 was fixed, typically with an assertion failure. Now something crashes with SIGSEGV, dereferencing an invalid address (typically a null pointer). Without knowing where it crashes, the bug can’t be fixed.

I downloaded the following two packages and will figure out how to extract the files on my Debian system:

MariaDB-server-10.6.11-1.el8.x86_64.rpm
MariaDB-server-debuginfo-10.6.11-1.el8.x86_64.rpm

But, I will need the correct address of the crashing instruction as a starting point. I am afraid that the hexadecimal addresses in the log output are subject to ASLR.

I deleted one comment, that I found unrelated to this problem and due to oom-killer.

This bug in this ticket had no such error.

Maybe this will help, the following log was noted in /var/log/messages(of note the segfault line and next kernel Code line is same in both servers):

Nov 29 16:48:27 f95 kernel: mariadbd[4095472]: segfault at 7efa413d6000 ip 0000564074ccff0c sp 00007efb10076ab0 error 4 in mariadbd[56407
3f96000+1581000]
Nov 29 16:48:27 f95 kernel: Code: ff 03 00 00 48 89 50 10 48 8b 55 98 50 41 52 e8 ba f4 ff ff 48 8b 4d b8 89 01 58 5a e9 a3 01 00 00 0f 1
f 44 00 00 48 8b 45 10 <0f> b6 40 01 48 83 f8 7f 0f 87 06 05 00 00 48 8b 4d 10 48 83 c1 02
Nov 29 16:48:27 f95 systemd[1]: Started Process Core Dump (PID 741170/UID 0).
Nov 29 16:48:27 f95 systemd-coredump[741171]: Resource limits disable core dumping for process 2378126 (mariadbd).
Nov 29 16:48:27 f95 systemd-coredump[741171]: Process 2378126 (mariadbd) of user 767 dumped core.
Nov 29 16:48:27 f95 systemd[1]: systemd-coredump@0-741170-0.service: Succeeded.
Nov 29 16:48:28 f95 systemd[1]: mariadb.service: Main process exited, code=dumped, status=11/SEGV
Nov 29 16:48:28 f95 systemd[1]: mariadb.service: Failed with result 'core-dump'.
Nov 29 16:48:33 f95 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
Nov 29 16:48:33 f95 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Nov 29 16:48:33 f95 systemd[1]: Stopped MariaDB 10.6.11 database server.
Nov 29 16:48:33 f95 systemd[1]: Starting MariaDB 10.6.11 database server...
Nov 29 16:48:33 f95 numactl[744695]: 2022-11-29 16:48:33 0 [Note] /usr/sbin/mariadbd (server 10.6.11-MariaDB-log) starting as process 744
695 ...
Nov 29 16:48:33 f95 numactl[744695]: 2022-11-29 16:48:33 0 [Warning] Could not increase number of max_open_files to more than 300000 (req
uest: 337711)

I think I can get somewhere with that: ip - base_address (0x564073f96000) = 0xD39F0C

$ rpm2cpio MariaDB-server-10.6.11-1.el8.x86_64.rpm | cpio -idmv

$ rpm2cpio MariaDB-server-debuginfo-10.6.11-1.el8.x86_64.rpm  | cpio -idmv

gdb ./usr/sbin/mariadbd

(gdb)  directory ./usr/src/debug/MariaDB-/src_0/

Source directories searched: /home/dan/Downloads/./usr/src/debug/MariaDB-/src_0:/home/dan/Downloads:$cdir:$cwd

(gdb) set debug-file-directory /home/dan/Downloads//usr/lib/debug

(gdb)   show debug-file-directory

The directory where separate debug symbols are searched for is "/home/dan/Downloads//usr/lib/debug".

(gdb) file ./usr/sbin/mariadbd

Reading symbols from ./usr/sbin/mariadbd...

Reading symbols from /home/dan/Downloads/usr/lib/debug/usr/sbin/mariadbd-10.6.11-1.el8.x86_64.debug...

(gdb) list *0xD39F0C

0xd39f0c is in row_log_table_apply_op(que_thr_t*, ulint, row_merge_dup_t*, dberr_t*, mem_heap_t*, mem_heap_t*, mrec_t const*, mrec_t const*, rec_offs*) (/usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0log.cc:2173).

2168		default:

2169			ut_ad(0);

2170			*error = DB_CORRUPTION;

2171			return(NULL);

2172		case ROW_T_INSERT:

2173			extra_size = *mrec++;

2174	

2175			if (extra_size >= 0x80) {

2176				/* Read another byte of extra_size. */

2177	

   0x0000000000d39efe <+654>:	jmp    0xd3a0a6 <_ZL22row_log_table_apply_opP9que_thr_tmP15row_merge_dup_tP7dberr_tP16mem_block_info_tS6_PKhS8_Pt+1078>

   0x0000000000d39f03 <+659>:	nopl   0x0(%rax,%rax,1)

   0x0000000000d39f08 <+664>:	mov    0x10(%rbp),%rax

   0x0000000000d39f0c <+668>:	movzbl 0x1(%rax),%eax

   0x0000000000d39f10 <+672>:	cmp    $0x7f,%rax

   0x0000000000d39f14 <+676>:	ja     0xd3a420 <_ZL22row_log_table_apply_opP9que_thr_tmP15row_merge_dup_tP7dberr_tP16mem_block_info_tS6_PKhS8_Pt+1968>

   0x0000000000d39f1a <+682>:	mov    0x10(%rbp),%rcx

   0x0000000000d39f1e <+686>:	add    $0x2,%rcx

   0x0000000000d39f22 <+690>:	add    %rcx,%rax

On the crash it appears systemd-coredump has captured the coredump.

After installing debuginfo packages (dnf install install MariaDB-server-debuginfo), (note: they don't need to be installed at the time of the crash, only at debugging)

Assuming gdb is installed too we'll look at the backtrace and the code generated. Then we can look for AMD incompatible code and out build chain.

coredumpctl debug

(gdb) bt full

...

(gdb)   disassemble

....

(gdb) registers

...

/* then "up" until in row_log_table_apply_op */

(gdb) info locals

...

On performance measurements, suggest using perf record -g -p $(pidof mariadb) / perf report, comparing, maybe with some flame graphs between the two.

Note systemd v243 onwards has better numa options - https://www.freedesktop.org/software/systemd/man/systemd.exec.html#NUMAPolicy=

Just a note: no core dump file.

[root@[REDACTED] ~]# coredumpctl
TIME PID UID GID SIG COREFILE EXE
Tue 2022-11-29 16:38:44 KST 1215950 767 767 11 none /usr/sbin/mariadbd
[root@[REDACTED] ~]# coredumpctl debug
PID: 1215950 (mariadbd)
UID: 767 (mysql)
GID: 767 (mysql)
Signal: 11 (SEGV)
Timestamp: Tue 2022-11-29 16:38:44 KST (24h ago)
Command Line: /usr/sbin/mariadbd
Executable: /usr/sbin/mariadbd
Control Group: /system.slice/mariadb.service
Unit: mariadb.service
Slice: system.slice
Boot ID: af2078efd2534a9ea45dc4e4333a55f7
Machine ID: cb5211815ad64289b825c6b1a6069cda
Hostname: [REDACTED]
Storage: none
Message: Process 1215950 (mariadbd) of user 767 dumped core.

Coredump entry has no core attached (neither internally in the journal nor externally on disk).

Additional note:
The following test had no problem.

Same DB and table was binary copied from non-crashed intel server(more binlogs applied and after the optimize was done successfully on the intel server).
MariaDB was downgraded to 10.6.10
Table had more binlog applied since last crash, table had same indexes but did not have one index added and dropped(which was one the reasons we ran the optimize table).
Server had large pages disabled and booted.
During current server test, it had no other queries sent to it, while during the initial crash other select queries were going to it.
Both were slaves and replicating with no slave lag.

We track all config changes through monitoring tool.
These are the changes that were made when the server crashed and when the server didn't with the same optimize command:
innodb_io_capacity 10000 -> 20000
innodb_io_capacity_max 20000 -> 40000 (this we did not touch, but mariadb modified automatically)
innodb_read_io_threads 4 -> 16
innodb_version 10.6.11 -> 10.6.10 (downgraded)
innodb_write_io_threads 4 -> 16
large_pages ON -> OFF
version 10.6.11-MariaDB-log -> 10.6.10-MariaDB-log
version_source_revision efa28fb234b6603ea1e1aa3951925c98146686eb -> fd0bdd3180a7d5f4b9804d372d6a63b6a202818c

One additional note regarding "First of all we noticed the AMD servers have much poor performance compared to Intel servers(when used as master, the slave are really unneccessary and have little load), nearly being unresponsive with high load, while intel servers having much lower specs were much more stable(when used as master)."

When changing from Intel to AMD servers we did three major changes other than hardware and bigger innodb buffer.
Two of them would be changing large_pages to ON and numa setting tests.
But another third change I missed initially was that we upgraded 10.6.10 to 10.6.11.
We have tens of thousands of queries per second on the master(slave is mostly just for testing/backup and migration to new servers and low load), suggesting that https://jira.mariadb.org/browse/MDEV-29988 might be related to our performance problems.

However, the "[ERROR] mysqld got signal 11" error was not found on any of the 3 Intel servers running 10.6.11 and only on all 2 AMD servers running 10.6.11. I do not know if that is coincidence or not.

FK, thanks to danblack’s help, we now know that the crash occurs during an online table rebuild, while applying log that was written by concurrently running DML operations. Somehow the log file is corrupted. I do not think that we have seen this in our internal testing.

Have you tried to run sudo memtester on the affected system?

This could still be a software bug, requiring some (mis)fortune to reproduce it. Were you able to identify the OPTIMIZE TABLE statement that was running during the crash? Could we get the SHOW CREATE TABLE output for that table? You can obfuscate the names of the table, the indexes, and the columns, but not the data types and the indexes.

MDEV-28880 might be a duplicate of MDEV-24412.

> Have you tried to run sudo memtester on the affected system?
These servers had memtester burn in tests for at least a few days before being installed.
Also the error occured on two AMD servers at the same time.

This is the schema with redacted names:

CREATE TABLE `[TABLENAME]` (
[REDACTEDc] bigint(11) NOT NULL,
[REDACTEDd] bigint(11) NOT NULL,
[REDACTEDh] bigint(11) NOT NULL,
[REDACTEDa] bigint(11) NOT NULL,
[REDACTEDm] bigint(11) NOT NULL,
[REDACTEDr] bigint(14) NOT NULL DEFAULT 0,
[REDACTEDd] bigint(11) NOT NULL,
PRIMARY KEY ([REDACTEDc]),
KEY `[REDACEDK1]` ([REDACTEDd],[REDACTEDh],[REDACTEDa]),
KEY `[REDACEDD1]` ([REDACTEDm],[REDACTEDr]),
KEY [REDACTEDh] ([REDACTEDh])
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

This is the command used:
OPTIMIZE TABLE [TABLENAME]

Another note, I made a typo above
"suggesting that https://jira.mariadb.org/browse/MDEV-28880 might be related to our performance problems."

should be https://jira.mariadb.org/browse/MDEV-29988
(I changed the text in the comment)

FK, the table structure looks rather basic. Only NOT NULL integer columns, no virtual columns. I hope that you can enable core dumps so that when it occurs again, we will have something more to debug on. Note: ever since MDEV-10814 was implemented in MariaDB Server 10.3, core dumps do not include large InnoDB data structures, such as the buffer pool.

Do you have a recommended method for doing so for Mariadb (as I am not familiar with core dumps).
If I have time in the future, I might try to recreate the issue as I saved the database state after the core dump before rsyncing data from other slave servers to be safe.

danblack should be able to guide you through reproducing the problem. As you can find in https://mariadb.org/contribute/, a better option than this ticket system could be Zulip chat. His approximate time zone is UTC+1100.

I would really appreciate the effort, so that we can solve the mystery.

I just tested with the backup data I saved during crash.
However, some conditions must have changed, preventing me from reproducing the issue.
Of note, two major changes compared to crash for my attempt to recreate the issue:

• The data used is backup after crash (could not get data before crash).
• The MariaDB version used was 10.6.10 for current test, while the crash was on 10.6.9

This ticket can be closed, as I failed to reproduce the issue.
Thank you.
Have a nice day.

You too FK. Thanks for reporting back.