Intermittent segmentation fault in CPython GC when traversing a SQLAlchemy _ConnectionRecord wrapping a mariadb connection

Summary

Intermittent segmentation fault (SIGSEGV) inside CPython's cyclic garbage collector while it traverses a SQLAlchemy _ConnectionRecord object that wraps a pooled mariadb connection. The crash happens on the main thread, seemingly at random points in unrelated application code, whenever a generation-2 GC pass happens to run. Confirmed across numerous independent CI occurrences (well over a handful) with an identical native backtrace every time.

Because the crash occurs randomly and infrequently per test run, we ran a large repeated CI matrix (both MariaDB 10.6 and 11.4, both drivers, many parallel/repeated jobs per pipeline) specifically to gather enough samples to tell the two drivers apart with confidence, rather than relying on a single run of each. Across that matrix, the crash only ever occurred with mariadb (mariadb+mariadbconnector://) and never with pymysql (mariadb+pymysql://), strongly indicating the corruption originates in this connector (or the MariaDB Connector/C library it wraps).

Update: we re-ran the same repeated CI matrix pinned to the 2.0.0rc2 prerelease of mariadb instead of 1.1.14, with no other changes. That full run produced zero failures, consistent with this being fixed already in the unreleased 2.0 line. See "Additional test" below for details/caveats.

Environment

Description

We run a Flask + SQLAlchemy + Flask-Migrate test suite (~655 tests) in GitLab CI. Each test creates a fresh Flask app / SQLAlchemy Engine (function-scoped fixture), which means the process churns through a large number of connection-pool checkouts/creations over the run (pool_pre_ping=True means effectively every checkout pings the connection first).

Somewhere between roughly 4 and a few hundred tests into a run, the process segfaults. The crash is not at a fixed point in application code - we've seen it happen during Flask blueprint/route registration, during do_ping/pool checkout, and during unrelated generator/deque iteration in pytest's own internals. What is always identical is the actual native crash site: CPython's cyclic GC (generation 2) is in the middle of decref'ing an object referenced by a _ConnectionRecord (sqlalchemy/pool/base.py), and that reference turns out to be a dangling/invalid pointer.

This is consistent with memory corruption (or premature freeing) of a Python object owned by this connector - most likely the connection object itself - that only surfaces later, whenever CPython's generational GC happens to traverse it, which explains why the apparent crash location looked "random" across different runs.

Reproduction

We were unable to build a small, deterministic standalone repro (the underlying corruption event seems rare per-connection-lifecycle, but with a ~650-test suite doing constant connection churn, the aggregate odds of hitting it within a single CI run are high enough that it fails a meaningful fraction of the time).

What did reliably isolate the cause, after enough repeated runs, was swapping only the driver in an otherwise identical CI matrix:

• SQLALCHEMY_DATABASE_URI=mariadb+mariadbconnector://... → crashes (every failing run, identical backtrace below)
• SQLALCHEMY_DATABASE_URI=mariadb+pymysql://... → never crashes, across the same matrix of MariaDB 10.6/11.4 servers and the same test suite

We also ruled out:

• MariaDB server version as a factor - identical crash on both 10.6 and 11.4.
• Heap corruption detectable by glibc/pymalloc - ran with MALLOC_CHECK_=3 and PYTHONMALLOC=debug across several real occurrences; neither ever produced a diagnostic message, so this doesn't look like a classic buffer-overflow/double-free caught by allocator bookkeeping. It looks more like a dangling pointer / use-after-free of a whole object.
• Unrelated native extensions in the same process (numpy/scipy/OpenBLAS) - present in the process but never on the crashing thread's stack; the OpenBLAS worker threads are idle in pthread_cond_wait in every capture.

Backtrace (native, via gdb; consistent across all observed occurrences)

Thread 1 "python" received signal SIGSEGV, Segmentation fault.

_PyObject_IS_GC (obj=<unknown at remote 0x...>) at ./Include/object.h:969

#0  _PyObject_IS_GC (obj=<unknown at remote 0x...>) at ./Include/object.h:969

#1  visit_decref (op=<unknown at remote 0x...>, parent=<_ConnectionRecord>) at Modules/gcmodule.c:465

#2  traverse_slots (self=<_ConnectionRecord() at remote 0x...>, visit=visit_decref, type=...) at Objects/typeobject.c:1760

    i = 1

    n = 6

#3  subtype_traverse (self=<_ConnectionRecord() at remote 0x...>, visit=visit_decref, arg=...) at Objects/typeobject.c:1781

#4  subtract_refs (containers=...) at Modules/gcmodule.c:491

    op = <_ConnectionRecord() at remote 0x...>

#5  deduce_unreachable (unreachable=..., base=...) at Modules/gcmodule.c:1116

#6  gc_collect_main (tstate=..., generation=2, ...) at Modules/gcmodule.c:1242

#7  gc_collect_with_callback (generation=2, tstate=...) at Modules/gcmodule.c:1426

#8  gc_collect_generations (tstate=...) at Modules/gcmodule.c:1481

#9  _Py_RunGC (tstate=...) at Modules/gcmodule.c:2292

#10 _Py_HandlePending (tstate=...) at Python/ceval_gil.c:1045

#11 _PyEval_EvalFrameDefault (...) at Python/ceval.c:834

    ... (ordinary Python bytecode evaluation frames follow, varying between

    occurrences - blueprint registration, pool checkout, or unrelated

    generator/deque iteration in pytest internals, depending on when the

    generation-2 GC threshold happened to be hit)

obj/op at frame #0/#1 is reported by gdb as <unknown at remote 0x...> - i.e. not a pointer to a recognizable/valid Python object - while parent is a live _ConnectionRecord instance (the SQLAlchemy pool's wrapper around a pooled DBAPI connection). traverse_slots shows it was visiting slot index 1 of 6 when it crashed.

Additional test: appears fixed in 2.0.0rc2

We pinned mariadb==2.0.0rc2 (no other changes to the test suite, CI matrix, or SQLAlchemy/pool settings) and re-ran the same repeated CI matrix described above (both MariaDB 10.6/11.4, many parallel/repeated jobs). Every job passed - no segfaults. We have not yet accumulated as many repeated runs on 2.0.0rc2 as we did while confirming the original bug on 1.1.14, so we'd treat this as a strong positive signal rather than absolute proof, given how intermittent the crash is - but it's consistent with the underlying issue already being fixed in the 2.0 line. Happy to run further repeated matrices against 2.0.0rc2 (or a later prerelease) if that's useful confirmation for triage.

Expected behavior

_ConnectionRecord and the objects it references should remain valid for as long as the pool holds a reference to them; the connector should not corrupt or prematurely free memory backing a Python object it owns (most likely its own connection object) while the SQLAlchemy pool still references it.

Additional notes

• We have not (yet) observed this in production, which uses a much smaller connection churn rate and different topology (Galera cluster via a comma-separated host connect arg, TLS, pool_recycle=10, pool_pre_ping=True). Absence of evidence there is not strong evidence of absence, given production's gunicorn workers auto-restart on crash and we only checked ~30 days of logs.
• We're able to run further diagnostics/test additional scenarios on request (e.g. against a specific mariadb version, or with different pool settings), and can share relevant excerpts from pyproject.toml/poetry.lock (dependency versions) if that would help. We'd rather not publish full CI job logs or the full lockfile, since our pipeline output and dependency list include internal infrastructure details (internal package/host names, registries) beyond what's relevant here.