[MDEV-19878] pam v2: pam password authentication doesn't work at all Created: 2019-06-27  Updated: 2020-08-25  Resolved: 2019-07-02

Status: Closed
Project: MariaDB Server
Component/s: Authentication and Privilege System, Plugin - pam
Affects Version/s: 10.4.6
Fix Version/s: 10.4.7

Type: Bug Priority: Major
Reporter: Geoff Montee (Inactive) Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
Problem/Incident
is caused by MDEV-7032 new pam plugin with a suid wrapper Closed
is caused by MDEV-15473 Isolate/sandbox PAM modules, so that ... Closed
Relates
relates to MDEV-19879 server can send empty error message t... Closed
relates to MDEV-19880 pam v1: pam password authentication d... Closed
relates to MDEV-19881 pam plugin from MariaDB 10.3 doesn't ... Open
relates to MDEV-19876 pam v2: auth_pam_tool_dir and auth_pa... Closed
relates to MDEV-19877 pam v2: auth_pam_tool input format is... Open
relates to MDEV-19882 pam v2: auth_pam_tool truncates passw... Closed
relates to MDEV-19898 PAM plugin testing Stalled

 Description   

In MariaDB 10.4, version 2 of the pam plugin is provided. Unfortunately, password authentication seems to be completely broken.

To reproduce, simply do the following:

Create a Unix user account and set a password for the user:

sudo useradd alice
 
sudo passwd alice

Create the PAM service configuration:

sudo tee /etc/pam.d/mariadb <<EOF
 
auth required pam_unix.so audit
 
account required pam_unix.so audit
 
EOF

Then in MariaDB, install the plugin:

INSTALL SONAME 'auth_pam';

And then create the user account:

CREATE USER 'alice'@'localhost' IDENTIFIED VIA pam USING 'mariadb';

And then you might need to execute some commands to work around MDEV-19876:

sudo chmod 0755 /usr/lib64/mysql/plugin/auth_pam_tool_dir/
 
sudo chmod 4755 /usr/lib64/mysql/plugin/auth_pam_tool_dir/auth_pam_tool

And then, try to authenticate as the Unix account (while using the workaround for MDEV-19807):

$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin
 
ERROR 1045 (28000): Access denied for user 'alice'@'localhost' (using password: NO)

I thought that this might be caused by MDEV-19882, but the client does not even seem to be prompting for a password, so this bug seems different.

The syslog shows the following:

Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): unexpected response from failed conversation function
 
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): conversation failed
 
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): unable to obtain a password
 
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): auth could not identify password for [alice]

And the strace output for the process running the auth_pam_tool utility shows the following:

read(0, "\0", 1)                        = 1
 
read(0, "\0\5", 2)                      = 2
 
read(0, "alice", 5)                     = 5
 
read(0, "\0\7", 2)                      = 2
 
read(0, "mariadb", 7)                   = 7
 
...
 
write(1, "C", 1)                        = 1
 
write(1, "\0\v", 2)                     = 2
 
write(1, "\4Password: ", 11)            = 11
 
read(0, "\0\0", 2)                      = 2
 
read(0, "", 0)                          = 0
 
...
 
sendto(3, "<84>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): unexpected response from failed conversation function", 112, MSG_NOSIGNAL, NULL, 0) = 112
 
sendto(3, "<83>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): conversation failed", 78, MSG_NOSIGNAL, NULL, 0) = 78
 
sendto(3, "<87>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): unable to obtain a password", 86, MSG_NOSIGNAL, NULL, 0) = 86
 
sendto(3, "<82>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): auth could not identify password for [alice]", 103, MSG_NOSIGNAL, NULL, 0) = 103

And the strace output for the process running the client connection shows the following:

recvfrom(44, "\244\0\0\1", 4, MSG_DONTWAIT, NULL, NULL) = 4
 
recvfrom(44, "\204\246\337 \0\0\0\1!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0alice\0\0mysql_native_password\0f\3_os\5Linux\f_client_name\nlibmariadb\4_pid\0044750\17_client_version\0053.1.2\t_platform\6x86_64\fprogram_name\5mysql", 164, MSG_DONTWAIT, NULL, NULL) = 164
 
pipe([45, 46])                          = 0
 
pipe([47, 48])                          = 0
 
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f74e81329d0) = 4752
 
close(45)                               = 0
 
close(48)                               = 0
 
sendto(44, "\10\0\0\2\376dialog\0", 12, MSG_DONTWAIT, NULL, 0) = 12
 
recvfrom(44, "\1\0\0\3", 4, MSG_DONTWAIT, NULL, NULL) = 4
 
recvfrom(44, "\0", 1, MSG_DONTWAIT, NULL, NULL) = 1
 
write(46, "\0", 1)                      = 1
 
write(46, "\0\5", 2)                    = 2
 
write(46, "alice", 5)                   = 5
 
write(46, "\0\7", 2)                    = 2
 
write(46, "mariadb", 7)                 = 7
 
read(47, "C", 1)                        = 1
 
read(47, "\0\v", 2)                     = 2
 
read(47, "\4Password: ", 11)            = 11
 
write(46, "\0\0", 2)                    = 2
 
write(46, "", 0)                        = 0
 
read(47, "", 1)                         = 0
 
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=4752, si_uid=997, si_status=255, si_utime=0, si_stime=0} ---
 
close(46)                               = 0
 
close(47)                               = 0
 
clock_gettime(CLOCK_REALTIME, {1561615579, 586285461}) = 0
 
write(2, "2019-06-27  6:06:19 9 [Warning] Access denied for user 'alice'@'localhost' (using password: NO)\n", 96) = 96
 
sendto(44, "H\0\0\4\377\25\4#28000Access denied for user 'alice'@'localhost' (using password: NO)", 76, MSG_DONTWAIT, NULL, 0) = 76

In the strace output for the client connection, it looks to me like the client is sending an empty password for some reason. This made me want to try testing how it worked if I provided the password on the command line. Even that fails:

$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin -palicemariadb
 
ERROR 1045 (28000): Access denied for user 'alice'@'localhost' (using password: NO)

And the strace output for that does show that the client connection received the proper password:

recvfrom(44, "\270\0\0\1", 4, MSG_DONTWAIT, NULL, NULL) = 4
 
recvfrom(44, "\204\246\337 \0\0\0\1!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0alice\0\24JQ\316\275'\353u\201\320\360S;\26Z\354H\301\317\266\24mysql_native_password\0f\3_os\5Linux\f_client_name\nlibmariadb\4_pid\0044917\17_client_version\0053.1.2\t_platform\6x86_64\fprogram_name\5mysql", 184, MSG_DONTWAIT, NULL, NULL) = 184
 
pipe([45, 46])                          = 0
 
pipe([47, 48])                          = 0
 
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f89c037e9d0) = 4919
 
close(45)                               = 0
 
close(48)                               = 0
 
sendto(44, "\10\0\0\2\376dialog\0", 12, MSG_DONTWAIT, NULL, 0) = 12
 
recvfrom(44, "\r\0\0\3", 4, MSG_DONTWAIT, NULL, NULL) = 4
 
recvfrom(44, "alicemariadb\0", 13, MSG_DONTWAIT, NULL, NULL) = 13
 
write(46, "\0", 1)                      = 1
 
write(46, "\0\5", 2)                    = 2
 
write(46, "alice", 5)                   = 5
 
write(46, "\0\7", 2)                    = 2
 
write(46, "mariadb", 7)                 = 7
 
read(47, "C", 1)                        = 1
 
read(47, "\0\v", 2)                     = 2
 
read(47, "\4Password: ", 11)            = 11
 
write(46, "\0\0", 2)                    = 2
 
write(46, "", 0)                        = 0
 
read(47, "", 1)                         = 0
 
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=4919, si_uid=997, si_status=255, si_utime=0, si_stime=0} ---
 
close(46)                               = 0
 
close(47)                               = 0
 
clock_gettime(CLOCK_REALTIME, {1561616363, 716309324}) = 0
 
write(2, "2019-06-27  6:19:23 9 [Warning] Access denied for user 'alice'@'localhost' (using password: NO)\n", 96) = 96
 
sendto(44, "H\0\0\4\377\25\4#28000Access denied for user 'alice'@'localhost' (using password: NO)", 76, MSG_DONTWAIT, NULL, 0) = 76

It just doesn't seem to be passing the proper password to the auth_pam_tool process.

 Comments   
Comment by Taylor Davis [ 2019-06-27 ]

We ran into the same issue at Wiland while testing 10.4.6. We are using kerberos to authenticate off an AD server so my messaging in /var/log/secure is different, but I was able to replicate the problem when using the workaround for MDEV-19807. If I can provide any logs to assist let me know.

Comment by Geoff Montee (Inactive) [ 2019-06-28 ]

After some code inspection, I wonder if the problem is due to this block:

        if (!pkt || (buf[0] >> 1) != 2)
 
        {
 
          PAM_DEBUG((stderr, "PAM: sending CONV string.\n"));
 
          if (vio->write_packet(vio, buf, buf_len))
 
            goto error_ret;
 
 
 
          PAM_DEBUG((stderr, "PAM: reading CONV answer.\n"));
 
          if ((pkt_len= vio->read_packet(vio, &pkt)) < 0)
 
            goto error_ret;
 
        }

https://github.com/MariaDB/server/blob/mariadb-10.4.6/plugin/auth_pam/auth_pam.c#L160

This code seems to be checking for the data sent by auth_pam_tool here:

      param->buf[0] = msg[i]->msg_style == PAM_PROMPT_ECHO_ON ? 2 : 4;
 
      PAM_DEBUG((stderr, "PAM: conv: send(%.*s)\n",
 
                (int)(param->ptr - param->buf - 1), param->buf));
 
      pkt_len= roundtrip(param, param->buf, param->ptr - param->buf - 1, &pkt);
 
      if (pkt_len < 0)
 
        return PAM_CONV_ERR;

https://github.com/MariaDB/server/blob/mariadb-10.4.6/plugin/auth_pam/auth_pam_base.c#L111

This code seems to be saying that if PAM needs a password with a prompt, then the "message style" is 2. If you need a password without a prompt, then the "message style" is 4.

In the Linux-PAM API, PAM_PROMPT_ECHO_OFF is 1 and PAM_PROMPT_ECHO_ON is 2, so MariaDB seems to be using non-standard values here:

https://github.com/linux-pam/linux-pam/blob/955b3e2f100205be2db4358e9c812de2ae453b8e/libpam/include/security/_pam_types.h#L223

Anyway, the plugin itself is only asking for a password if (buf[0] >> 1) != 2). Since this is right-shifted one bit, this seems equivalent to buf[0] != 4). This means that the plugin will only ask the user for a password if PAM is saying that it needs a password without a prompt.

We can see from the strace output that the client connection is receiving a message style "4" from auth_pam_tool with a prompt of "Password: ":

read(47, "C", 1)                        = 1
 
read(47, "\0\v", 2)                     = 2
 
read(47, "\4Password: ", 11)            = 11
 
write(46, "\0\0", 2)                    = 2
 
write(46, "", 0)                        = 0

And since the plugin only asks for a password if the message style != 4, this would seem to explain why the plugin isn't asking the client for the password.

It seems like the relevant block in the plugin code should actually be something like this:

        if (!pkt)
 
        {
 
			// Send a prompt to the client if PAM_PROMPT_ECHO_ON is needed
 
			if (buf[0] == 4)
 
			{
 
				PAM_DEBUG((stderr, "PAM: sending CONV string.\n"));
 
				if (vio->write_packet(vio, buf, buf_len))
 
					goto error_ret;
 
			}
 
 
 
			// Read the password from the client if PAM_PROMPT_ECHO_OFF or PAM_PROMPT_ECHO_ON is needed
 
			if (buf[0] == 2 || buf[0] == 4)
 
			{
 
				PAM_DEBUG((stderr, "PAM: reading CONV answer.\n"));
 
				if ((pkt_len= vio->read_packet(vio, &pkt)) < 0)
 
					goto error_ret;
 
			}
 
        }

Comment by Geoff Montee (Inactive) [ 2019-07-09 ]

I tested out the fix for this using tarbuildnum #27339 from hasky for RHEL 7:

http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64/builds/8839

http://hasky.askmonty.org/archive/10.4/build-27339/kvm-rpm-centos74-amd64/rpms/

Password authentication with pam v2 still fails for me, even though the same password works just fine with pam v1 after MDEV-19880.

For example, here's the user account:

MariaDB [(none)]> SHOW CREATE USER 'alice'@'localhost';
 
+--------------------------------------------------------------------+
 
| CREATE USER for alice@localhost                                    |
 
+--------------------------------------------------------------------+
 
| CREATE USER 'alice'@'localhost' IDENTIFIED VIA pam USING 'mariadb' |
 
+--------------------------------------------------------------------+
 
1 row in set (0.000 sec)

Now let's say that I install pam v2:

MariaDB [(none)]> INSTALL SONAME 'auth_pam';
 
Query OK, 0 rows affected (0.001 sec)

And then I try to log in as my "alice" user account, but it fails:

$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin -palicemariadb
 
ERROR 1045 (28000): Access denied for user 'alice'@'localhost' (using password: NO)

So then I uninstall pam v2 and install pam v1:

MariaDB [(none)]> UNINSTALL SONAME 'auth_pam';
 
Query OK, 0 rows affected (0.001 sec)
 
 
 
MariaDB [(none)]> INSTALL SONAME 'auth_pam_v1';
 
Query OK, 0 rows affected (0.001 sec)

And now the "alice" user account can login just fine:

$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin -palicemariadb
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 16
 
Server version: 10.4.7-MariaDB-log MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SELECT USER(), CURRENT_USER();
 
+-----------------+-----------------+
 
| USER()          | CURRENT_USER()  |
 
+-----------------+-----------------+
 
| alice@localhost | alice@localhost |
 
+-----------------+-----------------+
 
1 row in set (0.000 sec)

For the pam v2 failure, the syslog only shows a generic "password check failed" error:

Jul  9 00:18:15 ip-172-30-0-123 unix_chkpwd[7223]: password check failed for user (alice)
 
Jul  9 00:18:15 ip-172-30-0-123 mysqld: pam_unix(mariadb:auth): authentication failure; logname= uid=997 euid=997 tty= ruser= rhost=  user=alice

But you can see from the above output that I used the exact same password when using both pam v1 and pam v2, and the password check only failed with pam v2. This indicates that the password is not the problem.

I do see that the systemd service file has the relevant capabilities that are required for pam v2:

$ grep "CapabilityBoundingSet" /usr/lib/systemd/system/mariadb.service
 
CapabilityBoundingSet=CAP_IPC_LOCK CAP_DAC_OVERRIDE CAP_AUDIT_WRITE

Comment by Geoff Montee (Inactive) [ 2019-07-09 ]

The cause of the previous failures was MDEV-19876. I've re-opened that issue.

Generated at Thu Feb 08 08:55:03 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.