Static hostname: DB11
Icon name: computer-vm
Chassis: vm
Machine ID: 7da95cd7e7c341bc9163e2746197b5cd
Boot ID: 4f4c20d51d8246dab1463f40591e902c
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1127.18.2.el7.x86_64
Architecture: x86-64

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

AdamStark, it looks like there is a deadlock between InnoDB threads that leads to an InnoDB watchdog to kill the process.

To debug the hang, we would need the stack traces of the process. Please see https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ and especially the section on getting stack traces while the process is running. I would like to see some samples of stack traces already when the server shows the first symptoms of hanging (the first "long semaphore wait" message appears).

At the moment it appears that setting innodb_adaptive_hash_index to 0 has resolved the semaphore issue. Since these DB's are crashing in production I am hesitant to install additional debugging and profiling tools. If we experience this again I will install these tools and generate a full stack trace to supply additional details.

Feel free to add more information when you have it — we'll reopen the issue then

I got a very similar looking crash on 10.3.28-MariaDB
server thread appeared to be up but had to be killed to be able to restart.
err.txt

Similar crash happen for MariaDB 10.6.9

corrupted size vs. prev_size while consolidating

230620  6:51:45 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.6.9

key_buffer_size=67108864

read_buffer_size=131072

max_used_connections=212

max_threads=3002

thread_count=80

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6676289 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f352401b5d8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f353c0ecc18 thread_stack 0x40000

2023-06-20  7:35:20 0 [Note] mariadbd: Aria engine: starting recovery

recovered pages: 0% 100% (0.0 seconds); tables to flush: 1 0

 (0.0 seconds);