Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.4.6
-
None
Description
In MariaDB 10.4, version 2 of the pam plugin is provided. Unfortunately, password authentication seems to be completely broken.
To reproduce, simply do the following:
Create a Unix user account and set a password for the user:
sudo useradd alice
|
sudo passwd alice
|
Create the PAM service configuration:
sudo tee /etc/pam.d/mariadb <<EOF
|
auth required pam_unix.so audit
|
account required pam_unix.so audit
|
EOF
|
Then in MariaDB, install the plugin:
INSTALL SONAME 'auth_pam';
|
And then create the user account:
CREATE USER 'alice'@'localhost' IDENTIFIED VIA pam USING 'mariadb';
|
And then you might need to execute some commands to work around MDEV-19876:
sudo chmod 0755 /usr/lib64/mysql/plugin/auth_pam_tool_dir/
|
sudo chmod 4755 /usr/lib64/mysql/plugin/auth_pam_tool_dir/auth_pam_tool
|
And then, try to authenticate as the Unix account (while using the workaround for MDEV-19807):
$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin
|
ERROR 1045 (28000): Access denied for user 'alice'@'localhost' (using password: NO)
|
I thought that this might be caused by MDEV-19882, but the client does not even seem to be prompting for a password, so this bug seems different.
The syslog shows the following:
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): unexpected response from failed conversation function
|
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): conversation failed
|
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): unable to obtain a password
|
Jun 27 06:06:19 ip-172-30-0-123 auth_pam_tool: pam_unix(mariadb:auth): auth could not identify password for [alice]
|
And the strace output for the process running the auth_pam_tool utility shows the following:
read(0, "\0", 1) = 1
|
read(0, "\0\5", 2) = 2
|
read(0, "alice", 5) = 5
|
read(0, "\0\7", 2) = 2
|
read(0, "mariadb", 7) = 7
|
...
|
write(1, "C", 1) = 1
|
write(1, "\0\v", 2) = 2
|
write(1, "\4Password: ", 11) = 11
|
read(0, "\0\0", 2) = 2
|
read(0, "", 0) = 0
|
...
|
sendto(3, "<84>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): unexpected response from failed conversation function", 112, MSG_NOSIGNAL, NULL, 0) = 112
|
sendto(3, "<83>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): conversation failed", 78, MSG_NOSIGNAL, NULL, 0) = 78
|
sendto(3, "<87>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): unable to obtain a password", 86, MSG_NOSIGNAL, NULL, 0) = 86
|
sendto(3, "<82>Jun 27 06:06:19 auth_pam_tool: pam_unix(mariadb:auth): auth could not identify password for [alice]", 103, MSG_NOSIGNAL, NULL, 0) = 103
|
And the strace output for the process running the client connection shows the following:
recvfrom(44, "\244\0\0\1", 4, MSG_DONTWAIT, NULL, NULL) = 4
|
recvfrom(44, "\204\246\337 \0\0\0\1!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0alice\0\0mysql_native_password\0f\3_os\5Linux\f_client_name\nlibmariadb\4_pid\0044750\17_client_version\0053.1.2\t_platform\6x86_64\fprogram_name\5mysql", 164, MSG_DONTWAIT, NULL, NULL) = 164
|
pipe([45, 46]) = 0
|
pipe([47, 48]) = 0
|
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f74e81329d0) = 4752
|
close(45) = 0
|
close(48) = 0
|
sendto(44, "\10\0\0\2\376dialog\0", 12, MSG_DONTWAIT, NULL, 0) = 12
|
recvfrom(44, "\1\0\0\3", 4, MSG_DONTWAIT, NULL, NULL) = 4
|
recvfrom(44, "\0", 1, MSG_DONTWAIT, NULL, NULL) = 1
|
write(46, "\0", 1) = 1
|
write(46, "\0\5", 2) = 2
|
write(46, "alice", 5) = 5
|
write(46, "\0\7", 2) = 2
|
write(46, "mariadb", 7) = 7
|
read(47, "C", 1) = 1
|
read(47, "\0\v", 2) = 2
|
read(47, "\4Password: ", 11) = 11
|
write(46, "\0\0", 2) = 2
|
write(46, "", 0) = 0
|
read(47, "", 1) = 0
|
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=4752, si_uid=997, si_status=255, si_utime=0, si_stime=0} ---
|
close(46) = 0
|
close(47) = 0
|
clock_gettime(CLOCK_REALTIME, {1561615579, 586285461}) = 0
|
write(2, "2019-06-27 6:06:19 9 [Warning] Access denied for user 'alice'@'localhost' (using password: NO)\n", 96) = 96
|
sendto(44, "H\0\0\4\377\25\4#28000Access denied for user 'alice'@'localhost' (using password: NO)", 76, MSG_DONTWAIT, NULL, 0) = 76
|
In the strace output for the client connection, it looks to me like the client is sending an empty password for some reason. This made me want to try testing how it worked if I provided the password on the command line. Even that fails:
$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin -palicemariadb
|
ERROR 1045 (28000): Access denied for user 'alice'@'localhost' (using password: NO)
|
And the strace output for that does show that the client connection received the proper password:
recvfrom(44, "\270\0\0\1", 4, MSG_DONTWAIT, NULL, NULL) = 4
|
recvfrom(44, "\204\246\337 \0\0\0\1!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0alice\0\24JQ\316\275'\353u\201\320\360S;\26Z\354H\301\317\266\24mysql_native_password\0f\3_os\5Linux\f_client_name\nlibmariadb\4_pid\0044917\17_client_version\0053.1.2\t_platform\6x86_64\fprogram_name\5mysql", 184, MSG_DONTWAIT, NULL, NULL) = 184
|
pipe([45, 46]) = 0
|
pipe([47, 48]) = 0
|
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f89c037e9d0) = 4919
|
close(45) = 0
|
close(48) = 0
|
sendto(44, "\10\0\0\2\376dialog\0", 12, MSG_DONTWAIT, NULL, 0) = 12
|
recvfrom(44, "\r\0\0\3", 4, MSG_DONTWAIT, NULL, NULL) = 4
|
recvfrom(44, "alicemariadb\0", 13, MSG_DONTWAIT, NULL, NULL) = 13
|
write(46, "\0", 1) = 1
|
write(46, "\0\5", 2) = 2
|
write(46, "alice", 5) = 5
|
write(46, "\0\7", 2) = 2
|
write(46, "mariadb", 7) = 7
|
read(47, "C", 1) = 1
|
read(47, "\0\v", 2) = 2
|
read(47, "\4Password: ", 11) = 11
|
write(46, "\0\0", 2) = 2
|
write(46, "", 0) = 0
|
read(47, "", 1) = 0
|
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=4919, si_uid=997, si_status=255, si_utime=0, si_stime=0} ---
|
close(46) = 0
|
close(47) = 0
|
clock_gettime(CLOCK_REALTIME, {1561616363, 716309324}) = 0
|
write(2, "2019-06-27 6:19:23 9 [Warning] Access denied for user 'alice'@'localhost' (using password: NO)\n", 96) = 96
|
sendto(44, "H\0\0\4\377\25\4#28000Access denied for user 'alice'@'localhost' (using password: NO)", 76, MSG_DONTWAIT, NULL, 0) = 76
|
It just doesn't seem to be passing the proper password to the auth_pam_tool process.
Attachments
Issue Links
- is caused by
-
MDEV-7032 new pam plugin with a suid wrapper
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Open
-
-
-
- Closed
-
-
-
- Open
-
-
-
- Closed
-
-
-
- Stalled
-
Activity
After some code inspection, I wonder if the problem is due to this block:
if (!pkt || (buf[0] >> 1) != 2)
|
{
|
PAM_DEBUG((stderr, "PAM: sending CONV string.\n"));
|
if (vio->write_packet(vio, buf, buf_len))
|
goto error_ret;
|
|
|
PAM_DEBUG((stderr, "PAM: reading CONV answer.\n"));
|
if ((pkt_len= vio->read_packet(vio, &pkt)) < 0)
|
goto error_ret;
|
}
|
https://github.com/MariaDB/server/blob/mariadb-10.4.6/plugin/auth_pam/auth_pam.c#L160
This code seems to be checking for the data sent by auth_pam_tool here:
param->buf[0] = msg[i]->msg_style == PAM_PROMPT_ECHO_ON ? 2 : 4;
|
PAM_DEBUG((stderr, "PAM: conv: send(%.*s)\n",
|
(int)(param->ptr - param->buf - 1), param->buf));
|
pkt_len= roundtrip(param, param->buf, param->ptr - param->buf - 1, &pkt);
|
if (pkt_len < 0)
|
return PAM_CONV_ERR;
|
https://github.com/MariaDB/server/blob/mariadb-10.4.6/plugin/auth_pam/auth_pam_base.c#L111
This code seems to be saying that if PAM needs a password with a prompt, then the "message style" is 2. If you need a password without a prompt, then the "message style" is 4.
In the Linux-PAM API, PAM_PROMPT_ECHO_OFF is 1 and PAM_PROMPT_ECHO_ON is 2, so MariaDB seems to be using non-standard values here:
Anyway, the plugin itself is only asking for a password if (buf[0] >> 1) != 2). Since this is right-shifted one bit, this seems equivalent to buf[0] != 4). This means that the plugin will only ask the user for a password if PAM is saying that it needs a password without a prompt.
We can see from the strace output that the client connection is receiving a message style "4" from auth_pam_tool with a prompt of "Password: ":
read(47, "C", 1) = 1
|
read(47, "\0\v", 2) = 2
|
read(47, "\4Password: ", 11) = 11
|
write(46, "\0\0", 2) = 2
|
write(46, "", 0) = 0
|
And since the plugin only asks for a password if the message style != 4, this would seem to explain why the plugin isn't asking the client for the password.
It seems like the relevant block in the plugin code should actually be something like this:
if (!pkt)
|
{
|
// Send a prompt to the client if PAM_PROMPT_ECHO_ON is needed
|
if (buf[0] == 4)
|
{
|
PAM_DEBUG((stderr, "PAM: sending CONV string.\n"));
|
if (vio->write_packet(vio, buf, buf_len))
|
goto error_ret;
|
}
|
|
|
// Read the password from the client if PAM_PROMPT_ECHO_OFF or PAM_PROMPT_ECHO_ON is needed
|
if (buf[0] == 2 || buf[0] == 4)
|
{
|
PAM_DEBUG((stderr, "PAM: reading CONV answer.\n"));
|
if ((pkt_len= vio->read_packet(vio, &pkt)) < 0)
|
goto error_ret;
|
}
|
}
|
I tested out the fix for this using tarbuildnum #27339 from hasky for RHEL 7:
http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64/builds/8839
http://hasky.askmonty.org/archive/10.4/build-27339/kvm-rpm-centos74-amd64/rpms/
Password authentication with pam v2 still fails for me, even though the same password works just fine with pam v1 after MDEV-19880.
For example, here's the user account:
MariaDB [(none)]> SHOW CREATE USER 'alice'@'localhost';
|
+--------------------------------------------------------------------+
|
| CREATE USER for alice@localhost |
|
+--------------------------------------------------------------------+
|
| CREATE USER 'alice'@'localhost' IDENTIFIED VIA pam USING 'mariadb' |
|
+--------------------------------------------------------------------+
|
1 row in set (0.000 sec)
|
Now let's say that I install pam v2:
MariaDB [(none)]> INSTALL SONAME 'auth_pam';
|
Query OK, 0 rows affected (0.001 sec)
|
And then I try to log in as my "alice" user account, but it fails:
$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin -palicemariadb
|
ERROR 1045 (28000): Access denied for user 'alice'@'localhost' (using password: NO)
|
So then I uninstall pam v2 and install pam v1:
MariaDB [(none)]> UNINSTALL SONAME 'auth_pam';
|
Query OK, 0 rows affected (0.001 sec)
|
|
|
MariaDB [(none)]> INSTALL SONAME 'auth_pam_v1';
|
Query OK, 0 rows affected (0.001 sec)
|
And now the "alice" user account can login just fine:
$ mysql -u alice --plugin-dir=/usr/lib64/mysql/plugin -palicemariadb
|
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 16
|
Server version: 10.4.7-MariaDB-log MariaDB Server
|
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
|
|
|
MariaDB [(none)]> SELECT USER(), CURRENT_USER();
|
+-----------------+-----------------+
|
| USER() | CURRENT_USER() |
|
+-----------------+-----------------+
|
| alice@localhost | alice@localhost |
|
+-----------------+-----------------+
|
1 row in set (0.000 sec)
|
For the pam v2 failure, the syslog only shows a generic "password check failed" error:
Jul 9 00:18:15 ip-172-30-0-123 unix_chkpwd[7223]: password check failed for user (alice)
|
Jul 9 00:18:15 ip-172-30-0-123 mysqld: pam_unix(mariadb:auth): authentication failure; logname= uid=997 euid=997 tty= ruser= rhost= user=alice
|
But you can see from the above output that I used the exact same password when using both pam v1 and pam v2, and the password check only failed with pam v2. This indicates that the password is not the problem.
I do see that the systemd service file has the relevant capabilities that are required for pam v2:
$ grep "CapabilityBoundingSet" /usr/lib/systemd/system/mariadb.service
|
CapabilityBoundingSet=CAP_IPC_LOCK CAP_DAC_OVERRIDE CAP_AUDIT_WRITE
|
The cause of the previous failures was MDEV-19876. I've re-opened that issue.
We ran into the same issue at Wiland while testing 10.4.6. We are using kerberos to authenticate off an AD server so my messaging in /var/log/secure is different, but I was able to replicate the problem when using the workaround for
MDEV-19807. If I can provide any logs to assist let me know.