Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-4227

MaxCtrl incompatibility with MemoryDenyWriteExecute=true is not documented

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 6.4.1
    • 2.5.22, 6.4.3
    • Documentation
    • None
    • Debian GNU/Linux 11 (bullseye)

    Description

      I ran into a problem that maxscale logs are not rotated

      syslog:
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: # Fatal error in , line 0
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: # Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: #FailureMessage Object: 0x7fff7eaecc20
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  1: 0x94cca1  [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  2: 0x1389ce9 V8_Fatal(char const*, ...) [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  3: 0xbfce67 v8::internal::MemoryChunk::DecrementWriteUnprotectCounterAndMaybeSetPermissions(v8::PageAllocator::Permission) [/usr/b>
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  4: 0xc19cb5 v8::internal::PagedSpace::SetReadAndExecutable() [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  5: 0xb3ae2e  [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  6: 0xef7d4e  [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  7: 0xa2143e v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  8: 0x921f66 node::NodeMainInstance::NodeMainInstance(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, std::vec>
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]:  9: 0x8bf299 node::Start(int, char**) [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: 10: 0x7f8c87a58d0a __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
      июл 28 00:00:00 dc01-maxscale02 logrotate[32914]: 11: 0x842a71  [/usr/bin/maxctrl]
      июл 28 00:00:00 dc01-maxscale02 kernel: traps: maxctrl[32914] trap invalid opcode ip:83a61b sp:7fff7eaecbf8 error:0 in maxctrl[821000+12fa000]
      

      As a result, a new /var/log/maxscale/query_audit file are not created

      cat /proc/cpuinfo
      ...
      processor       : 3
      vendor_id       : GenuineIntel
      cpu family      : 15
      model           : 6
      model name      : Common KVM processor
      stepping        : 1
      microcode       : 0x1
      cpu MHz         : 2600.042
      cache size      : 16384 KB
      physical id     : 1
      siblings        : 2
      core id         : 1
      cpu cores       : 2
      apicid          : 3
      initial apicid  : 3
      fpu             : yes
      fpu_exception   : yes
      cpuid level     : 13
      wp              : yes
      flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
      bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
      bogomips        : 5200.08
      clflush size    : 64
      cache_alignment : 128
      address sizes   : 40 bits physical, 48 bits virtual
      power management:
      

      systemctl cat logrotate.service 
      # /lib/systemd/system/logrotate.service
      [Unit]
      Description=Rotate log files
      Documentation=man:logrotate(8) man:logrotate.conf(5)
      RequiresMountsFor=/var/log
      ConditionACPower=true
       
      [Service]
      Type=oneshot
      ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
       
      # performance options
      Nice=19
      IOSchedulingClass=best-effort
      IOSchedulingPriority=7
       
      # hardening options
      #  details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
      #  no ProtectHome for userdir logs
      #  no PrivateNetwork for mail deliviery
      #  no NoNewPrivileges for third party rotate scripts
      #  no RestrictSUIDSGID for creating setgid directories
      LockPersonality=true
      MemoryDenyWriteExecute=true
      PrivateDevices=true
      PrivateTmp=true
      ProtectClock=true
      ProtectControlGroups=true
      ProtectHostname=true
      ProtectKernelLogs=true
      ProtectKernelModules=true
      ProtectKernelTunables=true
      ProtectSystem=full
      RestrictNamespaces=true
      RestrictRealtime=true
      

      cat /etc/logrotate.d/maxscale
      /var/log/maxscale/query_audit.unified{
        hourly
        nocreate
        dateformat -%Y%m%d%H%M
        rotate 12
        missingok
        nocompress
        dateext
        sharedscripts
        postrotate
          test -r /var/run/maxscale/maxscale.pid && /usr/bin/maxctrl rotate logs
        endscript
      }
      

      maxscale version: 6.4.1~bullseye-1
      Debian GNU/Linux 11 (bullseye)
      systemd 247 (247.3-7)

      I did a little research and found out that the problem is in the "MemoryDenyWriteExecute=true" parameter in the "/lib/systemd/system/logrotate.service" file.

      From man systemd.exec:

      MemoryDenyWriteExecute=
      Takes a boolean argument.
      If set, attempts to create memory mappings that are writable and executable at the same time, or to change existing memory mappings to become executable, or mapping shared memory segments as executable are prohibited.
      Specifically, a system call filter is added that rejects mmap(2) system calls with both PROT_EXEC and PROT_WRITE set, mprotect(2) or pkey_mprotect(2) system calls with PROT_EXEC set and shmat(2) system calls with SHM_EXEC set.
      Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers.
      This option improves service security, as it makes harder for software exploits to change running code dynamically.
      

      How to reproduce this issue:
      You can create a simple systemd service that forces logrotate to run, or temporarily edit the current one:

       systemctl edit --full logrotate.service
       add -f to ExecStart command: "ExecStart=/usr/sbin/logrotate -f /etc/logrotate.conf"
       systemctl daemon-reload
       systemctl start logrotate.service
      

      Accordingly, there should be a configuration for maxscale log rotation and the log file must exist.

      Workaround: Set "MemoryDenyWriteExecute=false" in logrotate systemd unit

      Attachments

        Activity

          People

            markus makela markus makela
            n.borisenkov Nikita Borisenkov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.