Assertion Failure results in crash

2023-05-11 09:05:12 0x7fbdda556700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.29/storage/innobase/row/row0ins.cc line 219

InnoDB: Failing assertion: !cursor->index->is_committed()

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/

InnoDB: about forcing recovery.

230511  9:05:12 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.4.29-MariaDB-log source revision: 2594da7a33580bf03590502a011679c878487d0c

key_buffer_size=268435456

read_buffer_size=1048576

max_used_connections=3

max_threads=2002

thread_count=49

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4411983 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fba180008c8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7fbdda555cf0 thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c3608c156e]

/usr/sbin/mysqld(handle_fatal_signal+0x317)[0x55c360313857]

/lib64/libpthread.so.0(+0xf630)[0x7ff1665ab630]

/lib64/libc.so.6(gsignal+0x37)[0x7ff1659f6387]

/lib64/libc.so.6(abort+0x148)[0x7ff1659f7a78]

/usr/sbin/mysqld(+0x5d7040)[0x55c360014040]

/usr/sbin/mysqld(+0xc63055)[0x55c3606a0055]

/usr/sbin/mysqld(+0xc636d3)[0x55c3606a06d3]

/usr/sbin/mysqld(+0xc63cc2)[0x55c3606a0cc2]

/usr/sbin/mysqld(+0xc75c30)[0x55c3606b2c30]

/usr/sbin/mysqld(+0xbb36ed)[0x55c3605f06ed]

/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPKh+0x2af)[0x55c36032130f]

/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x6a)[0x55c3600e71ba]

/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x1284)[0x55c3600edba4]

/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3ae0)[0x55c36011d6e0]

/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x55c36012250b]

/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1956)[0x55c360124e26]

/usr/sbin/mysqld(_Z10do_commandP3THD+0x112)[0x55c360126332]

/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x229)[0x55c36020aab9]

/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c36020ab7d]

pthread_create.c:0(start_thread)[0x7ff1665a3ea5]

/lib64/libc.so.6(clone+0x6d)[0x7ff165abeb0d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7fba1805ae40): insert into director.proc (......

Connection ID (thread ID): 1804685

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /data/mysql

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units     

Max cpu time              unlimited            unlimited            seconds   

Max file size             unlimited            unlimited            bytes     

Max data size             unlimited            unlimited            bytes     

Max stack size            8388608              unlimited            bytes     

Max core file size        0                    unlimited            bytes     

Max resident set          unlimited            unlimited            bytes     

Max processes             1029879              1029879              processes 

Max open files            32768                32768                files     

Max locked memory         65536                65536                bytes     

Max address space         unlimited            unlimited            bytes     

Max file locks            unlimited            unlimited            locks     

Max pending signals       1029879              1029879              signals   

Max msgqueue size         819200               819200               bytes     

Max nice priority         0                    0                    

Max realtime priority     0                    0                    

Max realtime timeout      unlimited            unlimited            us        

Kernel version: Linux version 3.10.0-1160.81.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Fri Dec 16 17:29:43 UTC 2022

[mysqld]

tmpdir=/tmp # /data/tmp if the linux drive is too small 

datadir=/data/mysql

port            = 3306

socket          = /var/lib/mysql/mysql.sock

skip-external-locking

key_buffer_size = 256M

max_allowed_packet=1G

table_open_cache = 256

sort_buffer_size = 1M

read_buffer_size = 1M

read_rnd_buffer_size = 4M

myisam_sort_buffer_size = 64M

thread_cache_size = 8

query_cache_size= 100M

lower_case_table_names=1

default_storage_engine=InnoDB

tmp_table_size = 4096M #64m for each GB

max_heap_table_size = 4096M

# replication

server-id = 100              

#multi source

#master-info-repository=TABLE

#relay-log-info-repository=TABLE

#Addition for GTID

#binlog_format=ROW

#gtid-domain-id=1 #up one on each master

#enforce-gtid-consistency

#log-slave-updates

#lower usage of network link

slave_compressed_protocol = 1

# on master

log-bin=mysql-bin

log-bin=/data/mysql/mysql-bin

expire_logs_days=10                                           

innodb_flush_log_at_trx_commit=1

sync_binlog=1

# for master master

auto_increment_increment=2

auto_increment_offset=1 

relay-log=ctc-80-relay-bin     

#parallel replication

slave_parallel_threads=32

slave_parallel_mode=optimistic

#to ensure that users with problems do not kill other users interaction

max_connections=2000 # number of directors times max_user_connections

max_user_connections=250

#for fulltext search

ft_min_word_len=2

# Innodb parameters

innodb_buffer_pool_size = 200G #70% of physical memory

innodb_log_file_size = 4G

#innodb_log_group_home_dir=

innodb_file_per_table

innodb_lock_wait_timeout=100

# Calculate index stats more often

innodb_stats_sample_pages=100

innodb_log_buffer_size=4M

innodb_thread_concurrency = 128  

innodb_io_capacity=400

innodb_io_capacity_max=4000

innodb_strict_mode=on