[MDEV-5090]  troubles on smp/multicore systems with mariadb+pam+ldap+openssl Created: 2013-10-01  Updated: 2018-12-03  Resolved: 2018-12-03

Status: Closed
Project: MariaDB Server
Component/s: Plugin - pam
Affects Version/s: 5.5.33
Fix Version/s: 10.4.0

Type: Bug Priority: Major
Reporter: Dmitry Bakshaev Assignee: Alexey Botchkov
Resolution: Fixed Votes: 0
Labels: crash
Environment:

gentoo linux:
Linux bark 3.8.13-gentoo #1 SMP Mon Aug 5 21:51:43 MSK 2013 x86_64 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz GenuineIntel GNU/Linux

software versions:
dev-db/mariadb 5.5.33
sys-auth/pam_ldap 186
net-nds/openldap 2.4.35
sys-libs/pam 1.1.6



 Description   

Version: '5.5.31-MariaDB' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution
* glibc detected * /usr/sbin/mysqld: corrupted double-linked list: 0x00007f2088006a80 *
= Backtrace:
/lib64/libc.so.6(+0x793d6)[0x7f20ed3063d6]
/lib64/libc.so.6(+0x79789)[0x7f20ed306789]
/lib64/libc.so.6(+0x7a035)[0x7f20ed307035]
/usr/lib64/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f2091d65254]
/usr/lib64/libssl.so.1.0.0(SSL_CTX_free+0x180)[0x7f2092034928]
/usr/lib64/libldap-2.4.so.2(ldap_int_tls_destroy+0x15)[0x7f20926ae936]
/lib64/ld-linux-x86-64.so.2(+0x12966)[0x7f20eeabb966]
/lib64/ld-linux-x86-64.so.2(+0x13423)[0x7f20eeabc423]
/lib64/ld-linux-x86-64.so.2(+0xde3f)[0x7f20eeab6e3f]
/lib64/libdl.so.2(+0x15cf)[0x7f20ede345cf]
/lib64/libdl.so.2(dlclose+0x1f)[0x7f20ede3416f]
/lib64/libpam.so.0(+0x59d6)[0x7f20932009d6]
/lib64/libpam.so.0(pam_end+0x33)[0x7f20931fe503]
/usr/lib64/mysql/plugin/auth_pam.so(+0xb6c)[0x7f2093408b6c]
/usr/sbin/mysqld[0x54bfc5]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDjj+0x895)[0x55edd3]
/usr/sbin/mysqld[0x634615]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x38)[0x63562c]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x21)[0x635c20]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x136)[0x635fa8]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x636095]
/lib64/libpthread.so.0(+0x8bbe)[0x7f20ee895bbe]
/lib64/libc.so.6(clone+0x6d)[0x7f20ed36ec9d]

details:

/etc/mysql/my.cnf:

...
[mysqld]
plugin-load=auth_pam.so
...

/etc/pam.d/mysql:

auth required pam_ldap.so config=/etc/ldap_mysql.conf debug
account required pam_ldap.so config=/etc/ldap_mysql.conf debug

/etc/ldap_mysql.conf:

uri ldaps:ldap.xxx ldaps:ldap2.xxx
ssl on
scope sub

stress test

:
#!/bin/sh
...
for((i=0;i<100;i++)); do
 for((y=0;y<10;y++)); do
  echo 'show processlist;'|mysql -u uuu --password=ppp -h hhh &
 done
done



 Comments   
Comment by John W Smith [ 2014-07-17 ]

I think we just encountered this (or a very similar issue) with 5.5.38 running on Ubuntu 12.04 LTS.

Server version: 5.5.38-MariaDB-1~precise-log
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=9
max_threads=202
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1692418 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f1f9c424000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2853701e20 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa7499b]
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6ccf88]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f28564b2cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f2854c88425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f2854c8bb8b]
/lib/x86_64-linux-gnu/libc.so.6(+0x7439e)[0x7f2854cc639e]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f2854d5cf47]
/lib/x86_64-linux-gnu/libc.so.6(+0x109e40)[0x7f2854d5be40]
/lib/x86_64-linux-gnu/libc.so.6(+0x10aefe)[0x7f2854d5cefe]
/lib/x86_64-linux-gnu/security/pam_ldap.so(+0x4587)[0x7f2575cfd587]
/lib/x86_64-linux-gnu/security/pam_ldap.so(+0x1859)[0x7f2575cfa859]
/lib/x86_64-linux-gnu/security/pam_ldap.so(pam_sm_authenticate+0x12c)[0x7f2575cfb48c]
/lib/x86_64-linux-gnu/libpam.so.0(+0x2b45)[0x7f285480db45]
/lib/x86_64-linux-gnu/libpam.so.0(pam_authenticate+0x38)[0x7f285480d3c8]
/usr/lib/mysql/plugin/auth_pam.so(+0xcae)[0x7f1f881f6cae]
/usr/sbin/mysqld[0x53f619]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDjj+0x9da)[0x556b1a]
/usr/sbin/mysqld[0x651259]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x47)[0x652be7]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x12a)[0x65316a]
/usr/sbin/mysqld(handle_one_connection+0x4c)[0x6532bc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f28564aae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f2854d463fd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 6340
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

Comment by Dmitry Bakshaev [ 2015-08-18 ]

i found workaround for this problem:

replace pam_ldap (http://www.padl.com/OSS/pam_ldap.html) with nss-pam-ldapd (http://arthurdejong.org/nss-pam-ldapd/)

maybe this is not mariadb bug? pam_ldap related? close it?

Comment by Elena Stepanova [ 2015-08-19 ]

JWSmith, did you also find any kind of a workaround? Does the one suggested by dab1818 work for you as well?

Comment by John W Smith [ 2015-08-19 ]

We moved to MariaDB v10.0.x series, which does not exhibit the same issue. MariaDB v10.0.x series has worked flawlessly for several months in our development environment. We currently use the OS version of Centrify on our production MariaDB 5.5.x DB server. We plan on migrating production to Maria v10.0.x series soon, and plan to migrate those systems to pam_ldap at that time.

Comment by Dmitry Bakshaev [ 2015-08-20 ]

retested on mariadb-10.0.21 with pam_ldap-186.
this bug reproduced in our environment.
"dangerous mix" of openldap+openssl+threads?
we move to nss-pam-ldapd...
and nss-pam-ldapd has more other advantages for us.

p.s.: also i need retest auth_pam with sssd (https://fedorahosted.org/sssd/)

Comment by Sergei Golubchik [ 2018-12-03 ]

pam_ldap is apparently not thread-safe. But after MDEV-7032 MariaDB won't use pam modules from different threads,so the issue should go away.

Generated at Thu Feb 08 07:01:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.