OmniOS r151020 - Illumos derivative.
Solaris 11.3 (maybe earlier).

Tried this quick patch and I can confirm that my server now works. This isn't the proper fix, just a quick test - the semantics of the various operating systems need taking into account.

--- mariadb-10.1.22.distrib/mysys/mysys_priv.h  Sat Mar 11 19:09:10 2017

+++ mariadb-10.1.22/mysys/mysys_priv.h  Mon Mar 20 13:27:02 2017

@@ -108,9 +108,13 @@

 void my_error_unregister_all(void);

+#if !defined(O_PATH) && defined(O_SEARCH) /* Illumos */

+#define O_PATH O_SEARCH

+#else

 #if !defined(O_PATH) && defined(O_EXEC) /* FreeBSD */

 #define O_PATH O_EXEC

 #endif

+#endif

 #ifdef O_PATH

 #define HAVE_OPEN_PARENT_DIR_NOSYMLINKS

I've done some reading around on this and it seems that O_PATH is Linux specific whereas O_SEARCH/O_EXEC are defined by POSIX. They are almost equivalent except that when opening a symlink with O_NOFOLLOW:

• Linux O_PATH|O_NOFOLLOW opens a file descriptor referring to the symlink inode itself.
• POSIX O_NOFOLLOW with O_SEARCH or O_EXEC forces failure if the pathname refers to a symlink.

These are the results of running a simple test program using my_open_parent_dir_nosymlinks() on Linux and illumos where the illumos variant is using O_SEARCH:

Linux:

% ls -li ~/test

7871180 lrwxrwxrwx 1 root root    5 Mar 23 10:56 fred -> wilma/

7871179 drwxr-xr-x 2 root root 4.0K Mar 23 10:56 wilma/

% ./pdfd ~/test/wilma/barney.txt

RET 3 (inode: 7871179), barney.txt

% ./pdfd ~/test/fred/barney.txt

RET 3 (inode: 7871180), barney.txt

illumos (same directory structure):

% ./pdfd ~/test/wilma/barney.txt

RET 4 (inode: 70302), barney.txt

% ./pdfd ~/test/fred/barney.txt

RET 4927 (inode: -1), NULL

I don't have access to *BSD at present but I would expect the use of O_EXEC|O_NOFOLLOW to follow POSIX and therefore behave as O_SEARCH does on illumos.

Which behaviour is really wanted here? The POSIX result seems better from my limited understanding of the code where this is used (still reading...)

[1] https://sourceware.org/ml/libc-alpha/2013-08/msg00016.html
[2] https://lkml.org/lkml/2013/8/2/856

We don't care what behavior of these two will be, this code doesn't rely on linux specific O_PATH|O_NOFOLLOW.

What I need is openat() to fail if there's a symlink involved. On linux it'll open a symlink, and it will fail on the next openat(), because symlink is not a directory, and symlink's fd cannot be used as a first argument for openat(). With POSIX behavior it'll fail right away. Which is just fine too.

The logic of my_open_parent_dir_nosymlinks() is as follows — the path argument comes from realpath(), there must be no symlinks in it. If a symlink is found, it means someone replaced a path component with a symlink between realpath() and my_open_parent_dir_nosymlinks() calls, clearly an attempt to exploit a TOCTOU race condition.

Thanks for the explanation. It would seem that you should use O_SEARCH if available, O_EXEC if not and fallback to O_PATH, although the *BSD behaviour falls into 'undefined' as per POSIX.

Here's my updated local patch:

diff -ru mariadb-10.1.22.distrib/mysys/my_symlink.c mariadb-10.1.22/mysys/my_symlink.c

--- mariadb-10.1.22.distrib/mysys/my_symlink.c  Sat Mar 11 19:09:10 2017

+++ mariadb-10.1.22/mysys/my_symlink.c  Fri Mar 24 08:31:32 2017

@@ -244,7 +244,7 @@

       return pathname + (s - buf);

     }

-    fd = openat(dfd, s, O_NOFOLLOW | O_PATH);

+    fd = openat(dfd, s, O_NOFOLLOW | O_SEARCH);

     if (fd < 0)

       goto err;

diff -ru mariadb-10.1.22.distrib/mysys/mysys_priv.h mariadb-10.1.22/mysys/mysys_priv.h

--- mariadb-10.1.22.distrib/mysys/mysys_priv.h  Sat Mar 11 19:09:10 2017

+++ mariadb-10.1.22/mysys/mysys_priv.h  Fri Mar 24 08:31:18 2017

@@ -108,11 +108,15 @@

 void my_error_unregister_all(void);

-#if !defined(O_PATH) && defined(O_EXEC) /* FreeBSD */

-#define O_PATH O_EXEC

+#ifndef O_SEARCH       /* POSIX */

+#if defined(O_EXEC)    /* FreeBSD */

+#define O_SEARCH O_EXEC

+#elif defined(O_PATH)  /* Linux */

+#define O_SEARCH O_PATH

 #endif

+#endif /* !defined(O_SEARCH) */

-#ifdef O_PATH

+#ifdef O_SEARCH

 #define HAVE_OPEN_PARENT_DIR_NOSYMLINKS

 const char *my_open_parent_dir_nosymlinks(const char *pathname, int *pdfd);

 #define NOSYMLINK_FUNCTION_BODY(AT,NOAT)                                \