Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
11.8.9
-
None
-
- `mariadb:11.8` (`11.8.9-MariaDB-ubu2404-log`), official Docker image
- Galera cluster, 3 nodes, Kubernetes-hosted (containerd), `wsrep_provider = libgalera_smm.so`, `wsrep_sst_method = mariabackup`
- `wsrep_max_ws_size` / `repl.max_ws_size`: default, `2147483647` (2GiB - 1)
- `wsrep_max_ws_rows`: `0` (unlimited)
- `innodb_buffer_pool_size`: 8GiB (fixed)
- Per-pod memory limit (cgroup v2, `kubepods-burstable` slice): currently 15872Mi; historically cycled between 15Gi/16Gi/17Gi/15872Mi over the past 14 months trying to work around this- `mariadb:11.8` (`11.8.9-MariaDB-ubu2404-log`), official Docker image - Galera cluster, 3 nodes, Kubernetes-hosted (containerd), `wsrep_provider = libgalera_smm.so`, `wsrep_sst_method = mariabackup` - `wsrep_max_ws_size` / `repl.max_ws_size`: default, `2147483647` (2GiB - 1) - `wsrep_max_ws_rows`: `0` (unlimited) - `innodb_buffer_pool_size`: 8GiB (fixed) - Per-pod memory limit (cgroup v2, `kubepods-burstable` slice): currently 15872Mi; historically cycled between 15Gi/16Gi/17Gi/15872Mi over the past 14 months trying to work around this
-
Can result in hang or crash
-
Description
-
- How it's triggered
A background job (in our case, a third-party JMAP server's account-deletion task, but the trigger is generic: any sufficiently large single transaction) attempts to `DELETE` several hundred thousand rows belonging to one account in one transaction. The resulting writeset exceeds `wsrep_max_ws_size` and MariaDB correctly rejects it:
```
ERROR 1105 (HY000): Maximum writeset size exceeded
```
The caller retries the same (still oversized) transaction on a backoff schedule. Over a ~70-minute window we saw this rejection recur 4 times across our 3 nodes, and 2 of those retries ended in the node's `mariadbd` being OOM-killed by the kernel.
-
- Observed behavior (evidence)
Kernel OOM-killer report (`dmesg -T`) from two independent nodes, for two independent occurrences of the same rejected transaction, ~52 minutes apart:
```
Node A: Memory cgroup out of memory: Killed process (mariadbd) total-vm:8602138600kB, anon-rss:16152468kB, file-rss:13940kB
Node B: Memory cgroup out of memory: Killed process (mariadbd) total-vm:8602111008kB, anon-rss:16130300kB, file-rss:13552kB
```
Note how close these are - anon-rss within 0.14% of each other, file-rss within 2.8%, on two different physical hosts. That's a deterministic memory footprint tied to this specific operation, not noise.
The `anon-rss`/`file-rss` split is the interesting part. Contrast with a live, healthy `mariadbd` on the same node right now, same workload otherwise:
```
Normal: RssAnon: 11581684 kB (~11.0 GiB) RssFile: 2729380 kB (~2.6 GiB) total ~13.65 GiB
At OOM kill: anon-rss: ~16.14 GiB file-rss: ~13.6 MiB total ~15.4 GiB (= cgroup limit)
```
In normal operation there's a healthy mix of anonymous and file-backed (page-cache) resident memory. At the moment of the OOM kill, file-backed RSS had been reduced to almost nothing - the kernel's own reclaim was clearly working hard right up to the kill (`pgscan`/`pgsteal` counters in the OOM report show tens of millions of pages scanned/reclaimed), but couldn't free enough because nearly all resident memory was anonymous (private, application-owned, not reclaimable by the kernel on its own). That anonymous allocation grew to just under the container's memory limit and stayed there until killed.
`total_vm` (~8.6TB) is not unusual by itself - it matches this same node's normal steady-state `VmSize` right now, so that's just baseline glibc/InnoDB address-space reservation, not part of the problem.
-
- What doesn't work
We've raised the container memory limit for this cluster repeatedly over rouecifically to get past recurring OOM kills (15Gi -> 16Gi -> 17Gi -> back down to 15872Mi as the currently active value) - each time, the same class of OOM eventually recurs once a large-enough transaction/writeset is attempted again, right at whatever the
current limit is. We've also hit this same pattern from our own application'rge-transaction load in the past, independent of ever hitting the 1105 limitspecifically - raising memory has never durably fixed it. That's what makes us suspect this isn't "just needs more RAM" but a genuine unbounded-growth/non-release issue somewhere in
writeset construction or certification-failure handling.
-
- Expected behavior
When a writeset is rejected for exceeding `wsrep_max_ws_size` (or, more genetransaction fails/rolls back), the memory allocated while building it should bereleased back to the process/OS, the same way we assume it is for a transaction that completes successfully. Right now it looks like that release either doesn't happen, or doesn't
happen completely, on the rejection path.
-
- Notes
We don't have wsrep/InnoDB-internal instrumentation (heap profiling, valgrinctual incident - what's above is kernel-level cgroup/OOM evidence, not asource-level diagnosis. Happy to provide the full `dmesg` OOM reports, or try to build a minimal, deliberately-oversized-transaction reproduction against a memory-limited container if that would help - let us know what would be most useful.