[MXS-4227] MaxCtrl incompatibility with MemoryDenyWriteExecute=true is not documented Created: 2022-07-30  Updated: 2022-09-01  Resolved: 2022-09-01

Status: Closed
Project: MariaDB MaxScale
Component/s: Documentation
Affects Version/s: 6.4.1
Fix Version/s: 2.5.22, 6.4.3

Type: Bug Priority: Minor
Reporter: Nikita Borisenkov Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

Debian GNU/Linux 11 (bullseye)


 Description   

I ran into a problem that maxscale logs are not rotated

syslog:
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: # Fatal error in , line 0
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: # Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #FailureMessage Object: 0x7fff7eaecc20
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  1: 0x94cca1  [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  2: 0x1389ce9 V8_Fatal(char const*, ...) [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  3: 0xbfce67 v8::internal::MemoryChunk::DecrementWriteUnprotectCounterAndMaybeSetPermissions(v8::PageAllocator::Permission) [/usr/b>
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  4: 0xc19cb5 v8::internal::PagedSpace::SetReadAndExecutable() [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  5: 0xb3ae2e  [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  6: 0xef7d4e  [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  7: 0xa2143e v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  8: 0x921f66 node::NodeMainInstance::NodeMainInstance(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, std::vec>
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  9: 0x8bf299 node::Start(int, char**) [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: 10: 0x7f8c87a58d0a __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 
июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: 11: 0x842a71  [/usr/bin/maxctrl]
 
июл 28 00:00:00 dc01-maxscale02 kernel: traps: maxctrl[32914] trap invalid opcode ip:83a61b sp:7fff7eaecbf8 error:0 in maxctrl[821000+12fa000]

As a result, a new /var/log/maxscale/query_audit file are not created

cat /proc/cpuinfo
 
...
 
processor       : 3
 
vendor_id       : GenuineIntel
 
cpu family      : 15
 
model           : 6
 
model name      : Common KVM processor
 
stepping        : 1
 
microcode       : 0x1
 
cpu MHz         : 2600.042
 
cache size      : 16384 KB
 
physical id     : 1
 
siblings        : 2
 
core id         : 1
 
cpu cores       : 2
 
apicid          : 3
 
initial apicid  : 3
 
fpu             : yes
 
fpu_exception   : yes
 
cpuid level     : 13
 
wp              : yes
 
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
 
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
 
bogomips        : 5200.08
 
clflush size    : 64
 
cache_alignment : 128
 
address sizes   : 40 bits physical, 48 bits virtual
 
power management:


systemctl cat logrotate.service 
# /lib/systemd/system/logrotate.service
 
[Unit]
 
Description=Rotate log files
 
Documentation=man:logrotate(8) man:logrotate.conf(5)
 
RequiresMountsFor=/var/log
 
ConditionACPower=true
 
 
 
[Service]
 
Type=oneshot
 
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
 
 
 
# performance options
 
Nice=19
 
IOSchedulingClass=best-effort
 
IOSchedulingPriority=7
 
 
 
# hardening options
 
#  details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
 
#  no ProtectHome for userdir logs
 
#  no PrivateNetwork for mail deliviery
 
#  no NoNewPrivileges for third party rotate scripts
 
#  no RestrictSUIDSGID for creating setgid directories
 
LockPersonality=true
 
MemoryDenyWriteExecute=true
 
PrivateDevices=true
 
PrivateTmp=true
 
ProtectClock=true
 
ProtectControlGroups=true
 
ProtectHostname=true
 
ProtectKernelLogs=true
 
ProtectKernelModules=true
 
ProtectKernelTunables=true
 
ProtectSystem=full
 
RestrictNamespaces=true
 
RestrictRealtime=true


cat /etc/logrotate.d/maxscale
 
/var/log/maxscale/query_audit.unified{
 
  hourly
 
  nocreate
 
  dateformat -%Y%m%d%H%M
 
  rotate 12
 
  missingok
 
  nocompress
 
  dateext
 
  sharedscripts
 
  postrotate
 
    test -r /var/run/maxscale/maxscale.pid && /usr/bin/maxctrl rotate logs
 
  endscript
 
}

maxscale version: 6.4.1~bullseye-1
Debian GNU/Linux 11 (bullseye)
systemd 247 (247.3-7)

I did a little research and found out that the problem is in the "MemoryDenyWriteExecute=true" parameter in the "/lib/systemd/system/logrotate.service" file.

From man systemd.exec:

MemoryDenyWriteExecute=
 
Takes a boolean argument.
 
If set, attempts to create memory mappings that are writable and executable at the same time, or to change existing memory mappings to become executable, or mapping shared memory segments as executable are prohibited.
 
Specifically, a system call filter is added that rejects mmap(2) system calls with both PROT_EXEC and PROT_WRITE set, mprotect(2) or pkey_mprotect(2) system calls with PROT_EXEC set and shmat(2) system calls with SHM_EXEC set.
 
Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers.
 
This option improves service security, as it makes harder for software exploits to change running code dynamically.

How to reproduce this issue:
You can create a simple systemd service that forces logrotate to run, or temporarily edit the current one:

 systemctl edit --full logrotate.service
 
 add -f to ExecStart command: "ExecStart=/usr/sbin/logrotate -f /etc/logrotate.conf"
 
 systemctl daemon-reload
 
 systemctl start logrotate.service

Accordingly, there should be a configuration for maxscale log rotation and the log file must exist.

Workaround: Set "MemoryDenyWriteExecute=false" in logrotate systemd unit

 Comments   
Comment by markus makela [ 2022-08-01 ]

This might be expected behavior as MaxCtrl is based on NodeJS:

Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers.

If I'd have to guess, I'd say all MaxCtrl commands will behave exactly the same and this has nothing to do with the rotate logs command in particular.

This also seems like something you can't even use with NodeJS in general:

[markusjm@monolith system-test]$ sudo systemd-run -t -p MemoryDenyWriteExecute=true node
 
Running as unit: run-u179.service
 
Press ^] three times within 1s to disconnect TTY.
 
 
 
 
 
#
 
# Fatal error in , line 0
 
# Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).
 
#
 
#
 
#
 
#FailureMessage Object: 0x7ffe03b9d0f0
 
 1: 0x7fef1c2d7689  [/lib64/libnode.so.93]
 
 2: 0x7fef1d02dc4e V8_Fatal(char const*, ...) [/lib64/libnode.so.93]
 
 3: 0x7fef1ca556f3 v8::internal::MemoryChunk::DecrementWriteUnprotectCounterAndMaybeSetPermissions(v8::PageAllocator::Permission) [/lib64/libnode.so.93]
 
 4: 0x7fef1ca66b10 v8::internal::PagedSpace::SetReadAndExecutable() [/lib64/libnode.so.93]
 
 5: 0x7fef1c99c9d9 v8::internal::Isolate::Init(v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) [/lib64/libnode.so.93]
 
 6: 0x7fef1cd3be4c v8::internal::Snapshot::Initialize(v8::internal::Isolate*) [/lib64/libnode.so.93]
 
 7: 0x7fef1c86e069 v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [/lib64/libnode.so.93]
 
 8: 0x7fef1c2a52c1 node::NodeMainInstance::NodeMainInstance(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<unsigned long, std::allocator<unsigned long> > const*) [/lib64/libnode.so.93]
 
 9: 0x7fef1c21254c node::Start(int, char**) [/lib64/libnode.so.93]
 
10: 0x7fef1b36c550  [/lib64/libc.so.6]
 
11: 0x7fef1b36c609 __libc_start_main [/lib64/libc.so.6]
 
12: 0x55db42a180f5 _start [/usr/bin/node]

So far the only solution is to not use it and to docuent this limitation in the MaxCtrl documentation.

Comment by markus makela [ 2022-08-30 ]

We'll add a note to the MaxCtrl documentation about this limitation in NodeJS.

Generated at Thu Feb 08 04:27:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.