InnoDB: ibuf cursor restoration fails error in MariaDB 10.6.20

We are experiencing an issue with MariaDB 10.6.20 involving an ibuf cursor restoration fails error within the InnoDB storage engine.

This error occurs during routine database operations and appears to be related to the insert buffer for secondary indexes. The error log output suggests a failure in restoring the ibuf cursor due to operations involving a specific tablespace and page.

This seems to stall the thread and eventually fills up connection slots rendering the server effectively down. Any task perform on this table, even something as simple as checking it in phpmyadmin yields the issue, dropping the table also hangs.

–
4Dec 2 23:25:06 server mariadbd[1234861]: PHYSICAL RECORD: n_fields 6; 1-byte offsets; info bits 0
Dec 2 23:25:06 server mariadbd[1234861]: 0: len 4; hex 000008b6; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 1: len 1; hex 00; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 2: len 4; hex 00000785; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 3: len 16; hex 00000101830700048008860800088000; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 4: len 4; hex 6749d362; asc gI b;;
Dec 2 23:25:06 server mariadbd[1234861]: 5: len 8; hex 0000000000036523; asc e#;;
Dec 2 23:25:06 server mariadbd[1234861]: PHYSICAL RECORD: n_fields 6; 1-byte offsets; info bits 0
Dec 2 23:25:06 server mariadbd[1234861]: 0: len 4; hex 000008b6; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 1: len 1; hex 00; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 2: len 4; hex 00000785; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 3: len 16; hex 00000101830700048008860800088000; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 4: len 4; hex 6749d362; asc gI b;;
Dec 2 23:25:06 server mariadbd[1234861]: 5: len 8; hex 0000000000036523; asc e#;;
Dec 2 23:25:06 server mariadbd[1234861]: DATA TUPLE: 3 fields;
Dec 2 23:25:06 server mariadbd[1234861]: 0: len 4; hex 0001129c; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 1: len 1; hex 00; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 2: len 4; hex 000004ce; asc ;;
Dec 2 23:25:06 server mariadbd[1234861]: 2024-12-02 23:25:06 0 [ERROR] InnoDB: ibuf cursor restoration fails! ibuf record inserted to page [page id: space=70300, page number=1230] in file ./wp_xxx/rank_math_analytics_gsc.ibd
Dec 2 23:25:06 server mariadbd[1234861]: 2024-12-02 23:25:06 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
–

MariaDB [psa]> show engine innodb status;

=====================================
2024-12-02 23:05:38 0x7ff8d404b700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 14 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 1706 srv_idle
srv_master_thread log flush and writes: 1706
----------
SEMAPHORES
----------
------------
TRANSACTIONS
------------
Trx id counter 3008590727
Purge done for trx's n:o < 3004362410 undo n:o < 118 state: running
History list length 1837625
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION (0x7ff8c79c6380), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c1680), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c0b80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c5880), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79ca580), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c4d80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c8f80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79cd180), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c6e80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c2c80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79dd980), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79d3f80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c4280), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79d4a80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c2180), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79c3780), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79d6b80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79ce780), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x7ff8c79cc680), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
--------
FILE I/O
--------
Pending flushes (fsync) log: 0; buffer pool: 0
131030 OS file reads, 66026 OS file writes, 39928 OS fsyncs
28.93 reads/s, 16262 avg bytes/read, 23.50 writes/s, 23.50 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 843, free list len 39714, seg size 40558, 74 merges
merged operations:
insert 920, delete mark 2294, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
—
LOG
—
Log sequence number 8664023453370
Log flushed up to 8664023453370
Pages flushed up to 8663888798835
Last checkpoint at 8663888798835
0 pending log flushes, 0 pending chkp writes
47384 log i/o's done, 23.86 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 2986344448
Dictionary memory allocated 21635096
Buffer pool size 178464
Free buffers 26345
Database pages 151753
Old database pages 55998
Modified db pages 13715
Percent of dirty pages(LRU & free pages): 7.701
Max dirty pages percent: 90.000
Pending reads 0
Pending writes: LRU 0, flush list 0
Pages made young 76164, not young 7092762
43.28 youngs/s, 1375.62 non-youngs/s
Pages read 144138, created 8232, written 27544
28.64 reads/s, 4.57 creates/s, 0.00 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 8 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 151753, unzip_LRU len: 765
I/O sum[1544]:cur[398], unzip sum[3]:cur[0]
--------------
ROW OPERATIONS
--------------
0 read views open inside InnoDB
Process ID=0, Main thread ID=0, state: sleeping
Number of rows inserted 79686, updated 25150, deleted 11806, read 1810781482
6.64 inserts/s, 13.43 updates/s, 3.79 deletes/s, 1344444.90 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

1 row in set (0.345 sec)

–
max_allowed_packet=1G
performance_schema=ON
performance-schema-instrument='stage/%=ON'
performance-schema-consumer-events-stages-current=ON
performance-schema-consumer-events-stages-history=ON
performance-schema-consumer-events-stages-history-long=ON
query_cache_type=1
query_cache_size=134217728
query_cache_limit=16777216
query_cache_min_res_unit=4096
thread_cache_size=0
key_buffer_size=8388608
sort_buffer_size=2097152
read_rnd_buffer_size=262144
bulk_insert_buffer_size=8388608
myisam_sort_buffer_size=8388608
innodb_buffer_pool_chunk_size=134217728
max_heap_table_size=16777216
tmp_table_size=16777216
max_connections=451
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16777216
innodb_write_io_threads=4
innodb_read_io_threads=4
innodb_file_per_table=1
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
innodb_purge_threads=4
innodb_change_buffering = changes
innodb_change_buffer_max_size = 15
thread_cache_size = 0
innodb_buffer_pool_size = 2952790016
innodb_log_file_size = 738197504
myisam_sort_buffer_size = 8388608
join_buffer_size = 8388608
table_open_cache = 2048
table_definition_cache = 1408
optimizer_search_depth = 0
thread_handling = pool-of-threads
thread_pool_size = 3
–