Details
-
Bug
-
Status: Open (View Workflow)
-
Blocker
-
Resolution: Unresolved
-
25.10.4
-
None
-
2026-9
Description
Symptoms
When installing MariaDB Server and ColumnStore packages on Fedora 44 through Buildbot, the ColumnStore installation incorrectly selects the non-systemd path and attempts to use mysqld_safe.
Failure URL: https://buildbot.mariadb.org/#/builders/1272/builds/4
ColumnStore affected code: https://github.com/mariadb-corporation/mariadb-columnstore-engine/blob/962e861c6a11e375cd95c78a517ab2b5d50e0076/oam/install_scripts/columnstore-post-install.in#L77-L88
The Buildbot failure during MariaDB service startup is a side effect of ColumnStore not cleaning up the server process after selecting the non-systemd installation path.
This is not a Buildbot-specific issue. Buildbot only exposes the problem because its worker process has no valid terminal environment. Any CI runner, container, cron job, systemd service, SSH automation, or other headless environment can trigger it when TERM is absent or invalid.
Packages install correctly:
- On Fedora 43 through Buildbot.
- On Fedora 44 when installation is performed interactively.
- On Fedora 44 Buildbot workers when the systemd pager is disabled or replaced. See for example this run where i deliberately provide env SYSTEMD_PAGER=cat to dnf install.
Order of events:
- systemctl starts less as its pager.
- less fails immediately because TERM=unknown has no terminfo entry.
- systemctl writes into the closed pager pipe.
- systemctl receives SIGPIPE and returns 141.
- The ColumnStore script interprets the nonzero result as “systemd/MariaDB service unavailable.”
- ColumnStore selects the non-systemd/mysqld_safe installation path.
What I propose as a fix:
Make mariadb.service detection independent of the environment e.g.:
systemctl --no-pager cat mariadb.service >/dev/null 2>&1 |
Root cause:
The Buildbot execution environment has:
TERM=unknown
|
The ColumnStore installation uses the command from below to probe
if the mariadb service file is installed:
systemctl cat mariadb.service >/dev/null 2>&1 |
The failure can easily be demonstrated on a Fedora 44 environment
by enforcing TERM and observing that systemctl is terminated by SIGPIPE.
[buildbot@bb-ro-apexis-bbw03-x64-fedora-44-amd64 ~]$ test_case() {
|
local name=$1 |
shift |
|
|
printf '%-25s ' "$name" |
printf '' | bash -c "$*; printf 'rc=%s\n' \"\$?\"" |
}
|
|
|
test_case "normal TERM" \ |
'systemctl cat auditd.service >/dev/null 2>&1' |
|
|
test_case "TERM=unknown" \ |
'env TERM=unknown systemctl cat auditd.service >/dev/null 2>&1' |
|
|
test_case "unknown + no pager" \ |
'env TERM=unknown systemctl --no-pager cat auditd.service >/dev/null 2>&1' |
|
|
test_case "unknown + pager cat" \ |
'env TERM=unknown SYSTEMD_PAGER=cat systemctl cat auditd.service >/dev/null 2>&1' |
normal TERM rc=0
|
TERM=unknown rc=141
|
unknown + no pager rc=0
|
unknown + pager cat rc=0 |
We can observe that systemctl attempts to write the service contents to the pager pipe after its reader has exited:
[buildbot@bb-ro-apexis-bbw03-x64-fedora-44-amd64 ~]$ trace_dir=/tmp/systemctl-cat-trace |
rm -rf "$trace_dir" |
mkdir -p "$trace_dir" |
|
|
printf '' | |
env TERM=unknown \ |
strace -ff \ |
-tt \
|
-s 1024 \
|
-yy \
|
-o "$trace_dir/trace" \ |
-e trace=execve,clone,clone3,fork,vfork,pipe,pipe2,dup,dup2,dup3,close,read,write,writev,wait4,waitid,signal \ |
/usr/bin/systemctl cat auditd.service \ |
>/dev/null 2>&1 |
|
|
rc=$?
|
|
|
echo "rc=$rc" |
echo "trace_dir=$trace_dir" |
|
|
grep -HnE \ |
'execve|less|pager|pipe|EPIPE|SIGPIPE|exited|killed' \ |
"$trace_dir"/trace* |
rc=141
|
trace_dir=/tmp/systemctl-cat-trace |
/tmp/systemctl-cat-trace/trace.1287:1:10:13:58.514383 execve("/usr/bin/systemctl", ["/usr/bin/systemctl", "cat", "auditd.service"], 0x7ffc20456058 /* 33 vars */) = 0 |
/tmp/systemctl-cat-trace/trace.1287:40:10:13:58.525318 pipe2([5<pipe:[12265]>, 6<pipe:[12265]>], O_CLOEXEC) = 0 |
/tmp/systemctl-cat-trace/trace.1287:41:10:13:58.525403 pipe2([7<pipe:[12268]>, 8<pipe:[12268]>], O_CLOEXEC) = 0 |
/tmp/systemctl-cat-trace/trace.1287:47:10:13:58.525965 dup2(6<pipe:[12265]>, 1</dev/null<char 1:3>>) = 1<pipe:[12265]> |
/tmp/systemctl-cat-trace/trace.1287:48:10:13:58.526079 dup2(6<pipe:[12265]>, 2</dev/null<char 1:3>>) = 2<pipe:[12265]> |
/tmp/systemctl-cat-trace/trace.1287:49:10:13:58.526141 close(8<pipe:[12268]>) = 0 |
/tmp/systemctl-cat-trace/trace.1287:50:10:13:58.526261 read(7<pipe:[12268]>, "pager\0", 4096) = 6 |
/tmp/systemctl-cat-trace/trace.1287:51:10:13:58.534079 read(7<pipe:[12268]>, "less\0", 4096) = 5 |
/tmp/systemctl-cat-trace/trace.1287:52:10:13:58.534365 read(7<pipe:[12268]>, "", 4096) = 0 |
/tmp/systemctl-cat-trace/trace.1287:53:10:13:58.535028 close(7<pipe:[12268]>) = 0 |
/tmp/systemctl-cat-trace/trace.1287:54:10:13:58.535098 close(5<pipe:[12265]>) = 0 |
/tmp/systemctl-cat-trace/trace.1287:55:10:13:58.535155 close(6<pipe:[12265]>) = 0 |
/tmp/systemctl-cat-trace/trace.1287:103:10:13:58.548901 write(1<pipe:[12265]>, "\33[0;1;34m# \33]8;;file://bb-ro-apexis-bbw03-x64-fedora-44-amd64/usr/lib/systemd/system/auditd.service\33\\/usr/lib/systemd/system/auditd.service\33]8;;\33\\\33[0m\n\33[0;1;36m[Unit]\33[0m\n\33[0;1;32mDescription=\33[0mSecurity Audit Logging Service\n\33[0;1;32mConditionKernelCommandLine=\33[0m!audit=0\n\33[0;1;32mConditionKernelCommandLine=\33[0m!audit=off\n\33[0;1;32mDefaultDependencies=\33[0mno\n\n\33[0;1;38:5:245m## The following is a \"Wants\" so that if rules don't load, it will\33[0m\n\33[0;1;38:5:245m## not fail starting the main service. This indicates a weaker dependency.\33[0m\n\33[0;1;32mWants=\33[0maudit-rules.service\n\n\33[0;1;38:5:245m## If auditd is sending or receiving remote logging, copy this file to\33[0m\n\33[0;1;38:5:245m## /etc/systemd/system/auditd.service and comment out the first After and\33[0m\n\33[0;1;38:5:245m## uncomment the second so that network-online.target is part of After.\33[0m\n\33[0;1;38:5:245m## then comment the first Before and uncomment the second Before to remove\33[0m\n\33[0;1;38:5:245m## sysinit.target from \"Before\". Finish by removing every"..., 3824) = -1 EPIPE (Broken pipe) |
/tmp/systemctl-cat-trace/trace.1287:104:10:13:58.548952 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=1287, si_uid=1000} --- |
/tmp/systemctl-cat-trace/trace.1287:105:10:13:58.549838 +++ killed by SIGPIPE +++ |
/tmp/systemctl-cat-trace/trace.1288:15:10:13:58.527400 rt_sigaction(SIGPIPE, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7efdd261f470}, NULL, 8) = 0 |
/tmp/systemctl-cat-trace/trace.1288:68:10:13:58.532099 dup2(5<pipe:[12265]>, 0<pipe:[11252]>) = 0<pipe:[12265]> |
/tmp/systemctl-cat-trace/trace.1288:69:10:13:58.532192 close(5<pipe:[12265]>) = 0 |
/tmp/systemctl-cat-trace/trace.1288:70:10:13:58.532260 close(6<pipe:[12265]>) = 0 |
/tmp/systemctl-cat-trace/trace.1288:77:10:13:58.533918 write(8<pipe:[12268]>, "pager\0", 6) = 6 |
/tmp/systemctl-cat-trace/trace.1288:78:10:13:58.534030 execve("/usr/local/bin/pager", ["pager"], 0x55de8499de30 /* 35 vars */) = -1 ENOENT (No such file or directory) |
/tmp/systemctl-cat-trace/trace.1288:79:10:13:58.534161 execve("/usr/bin/pager", ["pager"], 0x55de8499de30 /* 35 vars */) = -1 ENOENT (No such file or directory) |
/tmp/systemctl-cat-trace/trace.1288:80:10:13:58.534236 write(8<pipe:[12268]>, "less\0", 5) = 5 |
/tmp/systemctl-cat-trace/trace.1288:81:10:13:58.534327 execve("/usr/local/bin/less", ["less"], 0x55de8499de30 /* 35 vars */) = -1 ENOENT (No such file or directory) |
/tmp/systemctl-cat-trace/trace.1288:82:10:13:58.534413 execve("/usr/bin/less", ["less"], 0x55de8499de30 /* 35 vars */) = 0 |
/tmp/systemctl-cat-trace/trace.1288:107:10:13:58.547716 +++ exited with 1 +++ |
|
It seems that between Fedora 43 and Fedora 44
the behavioral change comes from a less upgrade from 685 to 691.
At least this is the observed behaviour when probing less rc:
:~$ docker run -it --rm quay.io/mariadb-foundation/bb-worker:dev_fedora43 bash |
[buildbot@3777cfafe084 ~]$ printf 'test\n' | |
env TERM=unknown \ |
less \ |
>/dev/null 2>&1 |
|
|
echo "less rc=$?" |
less rc=0 |
[buildbot@3777cfafe084 ~]$ exit |
exit
|
:~$ docker run -it --rm quay.io/mariadb-foundation/bb-worker:dev_fedora44 bash |
[buildbot@9a5bc7996338 ~]$ printf 'test\n' | |
env TERM=unknown \ |
less \ |
>/dev/null 2>&1 |
|
|
echo "less rc=$?" |
less rc=1 |
|
Attachments
Issue Links
- links to
