[MDEV-29910] mysql_install_db fails with libpam-tmpdir Created: 2022-10-28  Updated: 2023-11-28

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.6.10
Fix Version/s: 10.6

Type: Bug Priority: Major
Reporter: Sunil Mohan Adapa Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: bug, regression
Environment:

OS: Debian GNU/Linux (unstable)
Package: mariadb-server-10.6 (version 1:10.6.10-1)



 Description   

On Debian GNU/Linux, when the package libpam-tmpdir is installed, mysql_install_db script fails during post install setup. As a result, mariadb daemon fails to start. The following error message is shown:

rm -rf /var/lib/mysql ; mysql_install_db --rpm --cross-bootstrap --user=mysql --disable-log-bin --skip-test-db

2022-10-28 19:33:00 0 [ERROR] mariadbd: Can't create/write to file '/tmp/user/0/ib2C7oNS' (Errcode: 13 "Permission denied")
2022-10-28 19:33:00 0 [ERROR] InnoDB: Unable to create temporary file; errno: 13
2022-10-28 19:33:00 0 [ERROR] mariadbd: Can't create/write to file '/tmp/user/0/ibykVtxz' (Errcode: 13 "Permission denied")
2022-10-28 19:33:00 0 [ERROR] InnoDB: Unable to create temporary file; errno: 13
2022-10-28 19:33:00 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.
2022-10-28 19:33:00 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-10-28 19:33:00 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-10-28 19:33:00 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-10-28 19:33:00 0 [ERROR] Aborting
 
Installation of system tables failed!  Examine the logs in
/var/lib/mysql for more information.

Environment

On FreedomBox (a pure blend of Debian), several applications that depend on mariadb fail to install when running on Debian testing/unstable. This is due to mariadb not running soon after installation. FreedomBox installs that package libpam-tmpdir by default. If this package is removed, mariadb server is running successfully after install.

This bug was reproduced on Debian unstable (as of 2022-10-28) with mariadb-server package version 1:10.6.10-1+b1.

Workarounds

  1. If libpam-tmpdir package is removed, the installation and daemon start succeed.
  2. When the environment variable TMPDIR is set to empty value, the mysql_install_db command succeeds. Example:

    rm -rf /var/lib/mysql ; TMPDIR= mysql_install_db --rpm --cross-bootstrap --user=mysql --disable-log-bin --skip-test-db

  3. When mysql_install_db is not run are root, the problem is not observed. Example:

    rm -rf /var/lib/mysql ; mkdir /var/lib/mysql; chown mysql:mysql /var/lib/mysql/ ; sudo -u mysql mysql_install_db --rpm --cross-bootstrap --user=mysql --disable-log-bin --skip-test-db

Regression:

This error does not occur on Debian stable (bullseye) where mariadb package version is 1:10.5.15-0+deb11u1. Hence this is a regression since that version.

Analysis

According to pam-tmpdir: "Many programs use $TMPDIR for storing temporary files. Not all of them are good at securing the permissions of those files. libpam-tmpdir sets $TMPDIR and $TMP for PAM sessions and sets the permissions quite tight. This helps system security by having an extra layer of security, making such symlink attacks and other /tmp based attacks harder or impossible".

Errors like the one being reported are typically seen when directories/files are created by root user in the $TMPDIR and later a non-root user tries to access those files without any further permission changes. libpam-tmpdir tries to ensure that temporary files created by one user are not accidentally accessible to unauthorized users.

During 10.6.x release cycle a change was introduced that makes this mistake. It creates files as 'root' and then tries to access them as 'mysql' user. The problem can be fixed by:

  1. Copying the files temporarily created by 'root' user to a location accessible to the 'mysql' user and then setting proper ownership, or by
  2. Creating all the temporary files with 'mysql' user to start with.


 Comments   
Comment by Daniel Black [ 2022-10-28 ]

Because mysql_install_db passes --user mysql to the mariadbd server process, and the server process does setuid to the mysql user, the $TMPDIR isn't changed from the root user.

Running su -l mysql mysql_install_db ...

Might trigger the pam bits to set $TMPDIR to the path for the mysql user. If you could test that it would be appreciated.

Comment by Sunil Mohan Adapa [ 2022-10-28 ]

I believe

su -l mysql ...

is not working (perhaps because mysql user is not allowed to have a shell). My test:

root@testing:~# su -l mysql -c 'echo test'
root@testing:~# echo $?
1
root@testing:~# su -l root -c 'echo test'
test
root@testing:~# getent passwd mysql
mysql:x:116:120:MySQL Server,,,:/nonexistent:/bin/false

root@testing:~# rm -rf /var/lib/mysql ; su -l mysql -c 'mysql_install_db --rpm --cross-bootstrap --user=mysql --disable-log-bin --skip-test-db'
root@testing:~# echo $?

However, bypassing the disabled shell for mysql user using sudo works as I tried in one of the workarounds:

root@testing:~# rm -rf /var/lib/mysql ; mkdir /var/lib/mysql; chown mysql:mysql /var/lib/mysql/ ; sudo -u mysql mysql_install_db --rpm --cross-bootstrap --user=mysql --disable-log-bin --skip-test-db
2022-10-28 22:43:44 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.

Comment by Otto Kekäläinen [ 2022-10-29 ]

A MR to fix this in Debian directly was submitted at https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/23 the the same author. Thanks Sunil!

Comment by Daniel Black [ 2022-10-31 ]

Thanks for testing. I was wondering about locked accounts. sudo probably isn't always available.

Setting TMPDIR to empty is a nice, probably portable fix for pushing into mysql_install_db.

Comment by Tuukka Pasanen [ 2022-11-07 ]

Thank you for good bug report sunil.

After little bit digging this I like to ask some questions if I understood correctly. does I understand correctly that pam-tmpdir does this:

  1. Create dir

    /tmp/users

    with chmod 0711 (rwx-x-x) which is little problematic but if it's always run by root when creating new tmp-dir then it works

  2. After that it create

    /tmp/users/uid

    in this case as we use root it's

    /tmp/users/0

    with chmod 0700 (rwx------)

  3. Then try to write to

    /tmp/users/0

    with user mysql

  4. Fail with error message

I would see that TMPDIR should be filled with

mkstemp

neither than just unset it.

Comment by Sunil Mohan Adapa [ 2022-11-07 ]

1. Create dir

/tmp/users

with chmod 0711 (rwx-x-x) which is little problematic but if it's always run by root when creating new tmp-dir then it works

It looks like /tmp/users is created by the PAM module with superuser privileges. I did the following to confirm:

root@freedombox:~# rm -rf /tmp/user
root@freedombox:~# exit

(SSH into machine as user fbx)
fbx@freedombox:~$ ls -lad /tmp/user/
drwx-x-x 3 root root 60 Nov 7 10:22 /tmp/user/
fbx@freedombox:~$ ls -lad /tmp/user/$UID
drwx------ 2 fbx root 40 Nov 7 10:22 /tmp/user/10000

Comment by Tuukka Pasanen [ 2022-11-09 ]

Thank you for fast reply. Need little bit digging and testing on my side to get this straight. danblack do you have opinion on this?

Comment by Daniel Black [ 2022-11-09 ]

Opinion on "I would see that TMPDIR should be filled with mkstemp neither than just unset it.":

  • If you're thinking of something like TMPDIR=$(mktemp -d) in the mysql_install_db then that would create a root:root 700 permission directory that --user=mysql wouldn't be able to write to. chown to user, problematic, path might be not +x, and selinux/apparmor restrictions could apply.
  • Or are you thinking of EPERM errors in the server code to call mkstemp and changing the env TMPDIR (or DEFAULT_TMPDIR) there to the created directory? Messy, but plausible.
Comment by Sunil Mohan Adapa [ 2022-11-10 ]

I don't know the exact code we are talking about here but in order for 'mysql' user to get a proper value for TMPDIR, one could start a proper PAM session. So, instead of doing sometime like 'setuid ; do task' one could do 'su -c task'.

Generated at Thu Feb 08 10:12:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.