Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6696

MariaDB-columnstore-engine RPM only weakly depends on jemalloc: PrimProc silently runs on glibc malloc and gets OOM-killed under load

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 23.02.25
    • 23.02, 25.10
    • Build
    • None
    • 2026-12

    Description

      Problem

      PrimProc gets jemalloc only via LD_PRELOAD in the mcs-primproc systemd unit. If ldconfig can't find libjemalloc, the unit logs "Please install jemalloc to avoid ColumnStore performance degradation and unexpected service interruptions." and starts PrimProc on glibc malloc anyway.

      The RPM doesn't guarantee jemalloc is installed:

      • Server-side storage/columnstore/CMakeLists.txt declares it as Recommends: (weak dependency) since 979af00b7c0 (2020-08-04).
      • 23.02 engine cmake/cpack_manage.cmake (82b9046b1) adds Requires: snappy jemalloc procps-ng gawk, but that doesn't reach the built package. Verified on the signed ES RPM MariaDB-columnstore-engine-10.6.15_10_23.02.25-1.el9: rpm -qR MariaDB-columnstore-engine | grep jemalloc returns nothing.
      • Community MariaDB-columnstore-engine-11.8.9_25.10.6-1.el9 has Recommends: jemalloc, no Requires.
      • cmake/cpackEngineRPM.cmake also lists jemalloc but isn't included anywhere.

      A weak dependency isn't installed when install_weak_deps=False, and dnf can remove it when nothing hard-requires it.

      Impact (see MCOL-6694)

      3-node cluster with the identical 23.02.25 RPM on all nodes. On the one node where jemalloc had gone missing during an upgrade/downgrade:

      • Parallel wide GROUP BY rollups pushed PrimProc RSS past the 95% watchdog before disk-based aggregation spilled.
      • Result: CAL0044 14x, OOM kills 12x, 2 core dumps within hours.

      The other two nodes, which have jemalloc, ran the same workload without crashing. Installing jemalloc on the bad node and restarting stopped the OOMs.

      Expected

      1. A ColumnStore installation can't end up without jemalloc: a hard Requires: jemalloc (RPM) / Depends: (DEB) in the shipped engine package for 23.02 and 25.10, community and enterprise. This means revisiting the 2020 decision in 979af00b7c0 with the server packaging owners.
      2. Fix or remove the 23.02 cpack_manage.cmake dependency logic so the declared dependencies actually reach the package, and drop the dead cpackEngineRPM.cmake.
      3. (Under discussion with the team, not yet agreed.) Running without jemalloc should be visible, not just one journal line. For example, CMAPI / mcs cluster status reports it, or PrimProc refuses to start unless an explicit override is set.

      How to check an installed system

      rpm -qR MariaDB-columnstore-engine | grep jemalloc
      grep -c jemalloc /proc/$(pidof PrimProc)/maps   # should be > 0

      Related: MCOL-5649 (closed Won't Do), MCOL-4554.

      Attachments

        Issue Links

          Activity

            People

              tturenko Timofey Turenko
              stasusov Stanislav Usov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 2d 6h 29m
                  2d 6h 29m
                  Remaining:
                  Remaining Estimate - 2d 6h 29m
                  2d 6h 29m
                  Logged:
                  Time Spent - Not Specified
                  Not Specified

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.