MariaDB 10.5.29 / Aria – Significant performance degradation over time on a table with ~2 billion rows

Hello MariaDB Support,

we are experiencing a significant and reproducible performance degradation with MariaDB 10.5.29 using the Aria storage engine and would appreciate your advice on identifying the bottleneck and optimizing the current configuration.

Environment
MariaDB 10.5.29
Aria storage engine
Red Hat Enterprise Linux (RHEL) 8.10
VMware virtual machine
32 vCPUs
~125 GiB RAM
4 GiB swap
XFS filesystem
Database and application are running on separate VMs
performance_schema is currently disabled

The main affected table contains approximately 2 billion rows (~2,050,000,000 rows):

Data size: ~556 GB
Index size: ~52 GB
Row format: PAGE
TRANSACTIONAL=0
Relevant MariaDB / Aria configuration

The most relevant current settings are:

aria_block_size = 8192
aria_pagecache_buffer_size = 68719476736 # 64 GiB
aria_pagecache_division_limit = 100
aria_pagecache_file_hash_size = 512
aria_group_commit = soft
aria_group_commit_interval = 0

key_buffer_size = 1 GiB
join_buffer_size = 16 MiB
sort_buffer_size = 256 MiB
read_buffer_size = 4 MiB
read_rnd_buffer_size = 256 MiB

tmp_table_size = 16 GiB
max_heap_table_size = 16 GiB

table_open_cache = 10240
table_definition_cache = 10240
table_open_cache_instances = 1
thread_cache_size = 128
max_connections = 128
open_files_limit = 32768

The Aria page cache is therefore configured with 64 GiB on a server with approximately 125 GiB RAM.

Workload

The workload consists mainly of parallel Java migration jobs performing point lookups and row-by-row UPDATE/INSERT operations.

The relevant UPDATE lookup uses a unique index on (ITEMID, VERSIONID). EXPLAIN confirms that this index is used and normally estimates one matching row.

Problem

Performance is initially good after restarting the server/MariaDB but degrades significantly after the system has been running under this workload for several days.

For example, with two identical parallel jobs:

Shortly after restart: approximately 1.09 million rows/hour per job
After several days: approximately 0.5–0.6 million rows/hour per job

Historically, the same application and database environment has also achieved substantially higher throughput.

A restart therefore has a significant positive effect, but the performance gradually degrades again.

Observations / Tests

We have already investigated the main OS and database resource bottlenecks:

CPU is not saturated; the server is mostly idle.
Storage is not saturated and I/O latency is low.
No significant system I/O wait.
Aria page-cache hit rate remains very high (>99.8%).
Index lookup appears efficient; Handler_read_key, Com_update and application activity show no indication of table scans.
Aria table-level locking is visible with two writers (Waiting for table level lock), but the lock-wait ratio during the degraded state is similar to measurements taken when performance was significantly better.
Swap usage increased over time and approximately 2.5 GiB of MariaDB memory had been swapped out. However, we tested swapoff while keeping MariaDB and the jobs running. Performance improved only slightly, so swap does not appear to be the primary cause.
After restart, the Aria page cache is initially only partially populated. Over time it becomes heavily/full populated while performance decreases.
In a recent degraded-state measurement, the Aria page-cache hit rate was approximately 99.89%, while CPU and storage still had substantial available capacity.

This makes us suspect that the degradation may be related to an internal Aria runtime/page-cache/concurrency or synchronization effect rather than a simple CPU, disk, index, or cache-capacity bottleneck.

Questions

Could you please advise:

Are there any known Aria performance issues in MariaDB 10.5.29 involving long-running UPDATE workloads, very large Aria tables, a large/full Aria page cache, or concurrent writers?
Are there specific Aria status variables, diagnostics, logging options, or other measurements you recommend to identify internal Aria contention or page-cache bottlenecks?
Are the above Aria and memory settings appropriate for this workload and server size?
Would you recommend changing aria_pagecache_buffer_size or any of the other Aria-related parameters for this workload?
Is there any known reason why performance could degrade as the Aria page cache becomes fully populated even though the measured cache hit rate remains above 99.8%?
Are there additional tests you recommend to distinguish Aria page-cache/internal synchronization issues from table-level locking or physical table-layout effects?
Upgrade constraint

The server currently runs Red Hat Enterprise Linux 8.10. Due to the operating-system/platform constraints in our environment, we currently cannot upgrade MariaDB to a newer major version.

Therefore, we need to identify the cause of the performance degradation and, if possible, optimize the workload and configuration using the currently installed MariaDB 10.5.29 version.

We can provide the complete SHOW GLOBAL STATUS, SHOW VARIABLES, table definition, SHOW TABLE STATUS, process lists and OS statistics (vmstat, iostat, memory statistics) as required.

Thank you for your support.