Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-5265

Strange behavior of admin_pam_readwrite_service

Details

    • Bug
    • Status: Stalled (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 24.02.3
    • N/A
    • None
    • None
    • MXS-SPRINT-218, MXS-SPRINT-230

    Description

      I'm trying to have PAM authentication for the REST API that only grants to users from a certain group.

      On the MariaDB server side this could be done with our own pam_user_map PAM module and PROXY privileges, but as no actual user mapping is needed here, just group based authentication, I was trying to rather just use the standard pam_succeed_if module like this:

      auth required pam_unix.so audit
      auth required pam_succeed_if.so debug user ingroup mysql
      account required pam_unix.so audit
      

      I've also created two unix users unix_user and unix_mysql, with only the later being in the mysql group, and made sure that user maxscale is in the shadow group to be able to read /etc/shadow for password verification. (I see that there is PAM sandboxing process support now in 23.08, but only for listeners, not for the REST APIs admin_pam_readwrite_service)

      Both the MariaDB and Maxscale test servers are running on the same machine, so both are using the same PAM setup and have the same unix users and groups.

      When testing the above PAM service with MariaDB server I can log in as user unix_mysql but not as unix_user, as expected. In /var/log/audit I see respective messages a expected due to audit and debug options used in the service file:

      Successful login with unix_mysql

      Sep 15 18:23:55 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): 'user' resolves to 'unix_mysql'
      Sep 15 18:23:55 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): requirement "user ingroup mysql" was met by user "unix_mysql"
      

      Failed login due to missing mysql group membership for unix_user:

      Sep 15 18:24:02 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): 'user' resolves to 'unix_user'
      Sep 15 18:24:02 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): requirement "user ingroup mysql" not met by user "unix_user"
      

      When trying to use the same PAM service with maxscale as admin_pam_readwrite_service=mariadb_unix I see different behavior though.

      First of all both user_mysql and user_unix can both log in, the pam_succeed_if ingroup check seems to get ignored totally by Maxscale here.

      Also I do not find the reqested audit and debug output anywhere. Not in syslog, not in
      /var/log/audit, not in the maxscale log file in /var/log/maxscale/maxscale.log

      Only when trying an actually invalid login with wrong user or password I'm seeing

      2024-09-15 18:47:40   warning: PAM authentication of user 'foo' to service 'mariadb_unix' failed: 'Authentication failure'.
      2024-09-15 18:47:40   warning: Authentication failed for 'foo', using password. Request: GET /auth
      

      in the maxscale log file and

      Sep 15 18:47:38 ubuntu-focal maxscale[4113]: pam_unix(mariadb_unix:auth): check pass; user unknown
      Sep 15 18:47:38 ubuntu-focal maxscale[4113]: pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost=
      

      in the syslog / systemd journal.

      Attachments

        Activity

          hholzgra Hartmut Holzgraefe added a comment -

          > As far as I know, the implementations can't be 100% the same due to licensing reasons unless we rewrite the one in the MariaDB server with 100% new code that's owned by MariaDB.

          As far as I can tell both `auth_pam` v1, and especially the v2 `auth_pam_tool` external helper process, source files clearly have MariaDB copyright headers only, no mention of MySQL or Oracle anywhere in server/plugin/auth_pam/ at all

          > As Markus said, the REST-API pam implementation calls the same functions as the sql authenticator.

          I was referring to "I see that there is PAM sandboxing process support now in 23.08, but only for listeners, not for the REST APIs admin_pam_readwrite_service" when claiming different implementations. Yes, the same PAM library functions are called in the end, but that was not what I was referring to

          > I tested this with MaxScale and it worked correctly.

          I can still reproduce that maxscale REST api PAM authentication behaves differently than the servers client authentication using the same PAM service.

          Client connecting to MariaDB server directly, with user in correct group:

          noformat
          $ mysql --host=mariadb --port=3306 --user=unix_mysql --password=secret test
          Welcome to the MariaDB monitor. Commands end with ; or \g.
          Your MariaDB connection id is 39
          Server version: 10.6.21-MariaDB-ubu2004 mariadb.org binary distribution
          ...
          --------
          Mar 5 23:10:45 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): 'user' resolves to 'unix_mysql'
          Mar 5 23:10:45 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): requirement "user ingroup mysql" was met by user "unix_mysql"
          noformat

          Client connecting to the server directly.

          noformat
          $ mysql --host=mariadb --port=3306 --user=unix_user --password=secret test
          ERROR 1045 (28000): Access denied for user 'unix_user'@'mariadb' (using password: NO)


          Mar 5 23:21:03 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): 'user' resolves to 'unix_user'
          Mar 5 23:21:03 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): requirement "user ingroup mysql" not met by user "unix_user"
          noformat

          Doing the same via maxscale both logins seem to work at first, but it turns out that no actual client session exists as maxscale logs

          noformat
          2025-03-05 23:22:43 info : (2) [readwritesplit] (Read-Write-Service); Server 'server1' failed: Authentication to 'server1' failed: 1045, #28000: Access denied for user 'unix_user'@'localhost' (using password: NO) (server1, session=2, conn_id=72, idle=0s)
          2025-03-05 23:22:43 error : (2) (Read-Write-Service); All backends are now permanently unusable, closing session.

          { Name: server1 Open: Broken Status: Master, Running }

          Error from 'server1': Authentication to 'server1' failed: 1045, #28000: Access denied for user 'unix_user'@'localhost' (using password: NO) (server1, session=2, conn_id=72, idle=0s)
          noformat

          and any attempt to run actual queries ends in ERROR 2006 (HY000): Server has gone away

          authenticator=PAMAuth and authenticator=PAMBackendAuth had been added to the listener and server sections.

          Now turning to maxctrl:

          Both the user in the correct group and the one not being a member of the required group can connect:

          maxctrl --user=unix_mysql --password=secret list servers

          and

          maxctrl --user=unix_user --password=secret list servers

          both work. And all the log output I'm getting is

          2025-03-05 23:27:58 info : Accept authentication from 'unix_user', using password. Request: /v1/servers

          in the maxscale log, in /var/log/auth.log there's nothing in either case. Only when using a nonexisting user, or a wrong password, access fails and there is

          Mar 5 23:30:26 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost= user=unix_user

          in the auth.log for the attempt with wrong password, and

          Mar  5 23:30:18 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): check pass; user unknown
          Mar  5 23:30:18 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost= 
          Mar  5 23:30:19 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): check pass; user unknown
          Mar  5 23:30:19 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost= 
          

          (yes, twice) for the attempt with an unknown user.

          So why is there no debug/audit log info in auth.log coming from maxscale, just error output, while the server shows debug/audit output just fine; and why does the server correctly fail on wrong group membership, but when PAM checks are initiated by maxscale it looks as if the pam_succeed_if ... ingroup ... is apparently ignored completely?

          I even tried with both the old and new server plugins for PAM to see whether this maybe only works with the new extra `auth_pam_tool` process approach, but found that it works fine the same on the server side with both versions.

          Versions I'm currently testing with:

          MariaDB 10.6.21
          Maxscale 24.02.4
          OS: Ubuntu 20.04.6 LTS

           

          hholzgra Hartmut Holzgraefe added a comment - > As far as I know, the implementations can't be 100% the same due to licensing reasons unless we rewrite the one in the MariaDB server with 100% new code that's owned by MariaDB. As far as I can tell both `auth_pam` v1, and especially the v2 `auth_pam_tool` external helper process, source files clearly have MariaDB copyright headers only, no mention of MySQL or Oracle anywhere in server/plugin/auth_pam/ at all > As Markus said, the REST-API pam implementation calls the same functions as the sql authenticator. I was referring to "I see that there is PAM sandboxing process support now in 23.08, but only for listeners, not for the REST APIs admin_pam_readwrite_service" when claiming different implementations. Yes, the same PAM library functions are called in the end, but that was not what I was referring to > I tested this with MaxScale and it worked correctly. I can still reproduce that maxscale REST api PAM authentication behaves differently than the servers client authentication using the same PAM service. Client connecting to MariaDB server directly, with user in correct group: noformat $ mysql --host=mariadb --port=3306 --user=unix_mysql --password=secret test Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 39 Server version: 10.6.21-MariaDB-ubu2004 mariadb.org binary distribution ... -------- Mar 5 23:10:45 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): 'user' resolves to 'unix_mysql' Mar 5 23:10:45 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): requirement "user ingroup mysql" was met by user "unix_mysql" noformat Client connecting to the server directly. noformat $ mysql --host=mariadb --port=3306 --user=unix_user --password=secret test ERROR 1045 (28000): Access denied for user 'unix_user'@'mariadb' (using password: NO) Mar 5 23:21:03 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): 'user' resolves to 'unix_user' Mar 5 23:21:03 ubuntu-focal auth_pam_tool: pam_succeed_if(mariadb_unix:auth): requirement "user ingroup mysql" not met by user "unix_user" noformat Doing the same via maxscale both logins seem to work at first, but it turns out that no actual client session exists as maxscale logs noformat 2025-03-05 23:22:43 info : (2) [readwritesplit] (Read-Write-Service); Server 'server1' failed: Authentication to 'server1' failed: 1045, #28000: Access denied for user 'unix_user'@'localhost' (using password: NO) (server1, session=2, conn_id=72, idle=0s) 2025-03-05 23:22:43 error : (2) (Read-Write-Service); All backends are now permanently unusable, closing session. { Name: server1 Open: Broken Status: Master, Running } Error from 'server1': Authentication to 'server1' failed: 1045, #28000: Access denied for user 'unix_user'@'localhost' (using password: NO) (server1, session=2, conn_id=72, idle=0s) noformat and any attempt to run actual queries ends in ERROR 2006 (HY000): Server has gone away authenticator=PAMAuth and authenticator=PAMBackendAuth had been added to the listener and server sections. Now turning to maxctrl : Both the user in the correct group and the one not being a member of the required group can connect: maxctrl --user=unix_mysql --password=secret list servers and maxctrl --user=unix_user --password=secret list servers both work. And all the log output I'm getting is 2025-03-05 23:27:58 info : Accept authentication from 'unix_user', using password. Request: /v1/servers in the maxscale log, in /var/log/auth.log there's nothing in either case. Only when using a nonexisting user, or a wrong password, access fails and there is Mar 5 23:30:26 ubuntu-focal maxscale [2420] : pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost= user=unix_user in the auth.log for the attempt with wrong password, and Mar 5 23:30:18 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): check pass; user unknown Mar 5 23:30:18 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost= Mar 5 23:30:19 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): check pass; user unknown Mar 5 23:30:19 ubuntu-focal maxscale[2420]: pam_unix(mariadb_unix:auth): authentication failure; logname= uid=996 euid=996 tty= ruser= rhost= (yes, twice) for the attempt with an unknown user. So why is there no debug/audit log info in auth.log coming from maxscale, just error output, while the server shows debug/audit output just fine; and why does the server correctly fail on wrong group membership, but when PAM checks are initiated by maxscale it looks as if the pam_succeed_if ... ingroup ... is apparently ignored completely? I even tried with both the old and new server plugins for PAM to see whether this maybe only works with the new extra `auth_pam_tool` process approach, but found that it works fine the same on the server side with both versions. Versions I'm currently testing with: MariaDB 10.6.21 Maxscale 24.02.4 OS: Ubuntu 20.04.6 LTS  
          hholzgra Hartmut Holzgraefe added a comment -

          forgot to mention:

          everything is running on the same machine, and I made sure to have both the mysql and maxscale user in the "shadow" group so that both the server and maxscale are able to access /etc/shadow

          hholzgra Hartmut Holzgraefe added a comment - forgot to mention: everything is running on the same machine, and I made sure to have both the mysql and maxscale user in the "shadow" group so that both the server and maxscale are able to access /etc/shadow

          People

            esa.korhonen Esa Korhonen
            hholzgra Hartmut Holzgraefe
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.