OS: CentOS Linux release 7.5.1804 (Core)
HW Architecture: x86_64

There have been 3.5 years of 10.4 updates since its first GA release on 10.4.6.

I'd suggest restoring from a backup with a latest 10.4 version with innodb_change_buffering=none.

This bug has similarities to MDEV-29276 and MDEV-28457 and MDEV-13542. Look to upgrade to a 10.6+ version in the near future.

Last 3 years its running fine without any problem. How suddenly in all 3 instances crash happened at same time.

The following in mariadb_error_log_1-jan-2023.txt suggests that this bug is a duplicate of MDEV-19916, which was fixed in 10.4.7 almost 2½ years ago.

2023-01-01 12:00:00 2 [ERROR] InnoDB: Page old data size 9736 new data size 16057, page old max ins size 6472 new max ins size 151

2023-01-01 12:00:00 2 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2023-01-01 12:00:00 2 [ERROR] [FATAL] InnoDB: Rec offset 99, cur1 offset 4705, cur2 offset 16048

…

Server version: 10.4.6-MariaDB-log

I think that the only fix is to rebuild all tables on which ALTER TABLE…ADD COLUMN or ALTER TABLE…DROP COLUMN was invoked, or columns were reordered in the past, without rebuilding the table. The bug is triggered by a DML operation, such as a massive DELETE, which forces the index tree to shrink. You may have to restore the affected tables from a backup.

When it comes to crashing on this corruption, recent versions of MariaDB 10.6 or later series no longer do that, thanks to MDEV-13542 and some related changes.

In MariaDB 10.4.13 version also similar crashing observed. But its recovers after crash.

toku_memory_startup: Assertion `result == 0' failed (errno=0)

230926 3:37:23 [ERROR] mysqld got signal 6 ;

Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f4af4ab2e50 thread_stack 0x49000

• • • buffer overflow detected ***: /db/mariadb/bin/mysqld terminated mariadb_10.4.13_error_log.txt

Workaround:

Removed mariadb/lib/plugin/ha_tokudb.so file from tar ball installation. Its working without any crash.

Similar & related Issue
https://jira.mariadb.org/browse/MDEV-15174