Elena Stepanova added a comment - 2016-07-11 20:50 serg , would you look at this? GeoffMontee , I don't know if serg will need any of this, but here are first questions that came to my mind after seeing this: Did the crash happen more than once? Had anything changed on the user's side before the crash(es) happened? System upgrade, configuration change, anything like that? Assuming the crash happened more than once, is it still happening, or has it stopped, and if it's stopped, what the user had done to achieve this? Which system does the server run on, and which versions of pam libraries are installed? Was there anything suspicious in the system logs around the time of the crash?

Geoff Montee (Inactive) added a comment - 2016-07-11 20:58 Did the crash happen more than once? The crash happened several times. Assuming the crash happened more than once, is it still happening, or has it stopped, and if it's stopped, what the user had done to achieve this? The user was trying to log in to a MariaDB server from SQLyog using a user account that is configured to use PAM authentication. I have asked the user to clarify if the crash happens every time the user logs in with that user account from SQLyog. I have also asked the user if pam_use_cleartext_plugin is enabled on the server. If SQLyog doesn't support the dialog plugin, could that lead to crashes? I will try to get more details.

Sergei Golubchik added a comment - 2016-07-12 08:45 The crash is happening, as you can see, not in auth_pam, but in pam_securid.so pam module. So the first question is — does the user need to use pam_securid for MariaDB? I mean, do they actually authenticate their database users with pam_securid?

Geoff Montee (Inactive) added a comment - 2016-07-12 16:14 I mean, do they actually authenticate their database users with pam_securid? It does appear so. I believe the intention is to use some kind of RSA token in addition to passwords to authenticate DBA user accounts. Here's the PAM policy for the user's MariaDB server: #%PAM-1.0 auth required pam_securid.so debug #account include password-auth #password include password-auth account required pam_securid.so debug password required pam_securid.so debug And pam_use_cleartext_plugin is turned OFF.

Sergei Golubchik added a comment - 2016-07-12 18:14 We've had a similar case before. A user was complaining about a crash, in pam_fprintd.so , if I'm not mistaken. It turned out that this particular PAM module was not safe for concurrent use, so when many users tried to connect (and authenticate) at the same time, it crashed. In that case, though, the user did not want to authenticate database users with fingerprints, he simply reused some global policy file. So the workaround was to remove pam_fprintd.so from MariaDB PAM policy. It looks like pam_securid.so is the reason here, too. But I don't know if it's concurrency or something else that triggers the crash. Do you know how to cause the crash? Is it deterministic?

Geoff Montee (Inactive) added a comment - 2016-07-12 18:41 - edited Do you know how to cause the crash? Is it deterministic? The user says that the server crashes every time that they try to authenticate with an account configured to use an RSA token, but I am trying to get him to clarify if the crash happens when connecting from any client, or just when connecting from SQLyog. I've also attached /var/log/messages from the user's server.

Geoff Montee (Inactive) added a comment - 2016-07-12 20:42 The user clarified that it does work without crashing when connecting with the mysql command-line client. So far, it only seems to crash when connecting via SQLyog.

Geoff Montee (Inactive) added a comment - 2016-07-13 18:05 The user's version of pam_securid.so: > rpm -qf /usr/lib64/security/pam_securid.so rsaauthagent-7.0-0.x86_64 This version looks like it might be kind of old, based on the versions listed here: https://www.rsa.com/en-us/products-services/identity-access-management/securid/authentication-agents/authentication-agents-for-pam I've asked the user to try upgrading.

Elena Stepanova added a comment - 2016-07-13 18:23 If the problem is indeed in the module, I'm not sure the upgrade will help – I've seen a similar complaint about 7.1.0.1: https://www.redhat.com/archives/pam-list/2015-April/msg00003.html Unfortunately, I couldn't find any follow-up.

Geoff Montee (Inactive) added a comment - 2016-07-15 16:28 It looks like you were right about that. The user upgraded to this version: > rpm -qf /usr/lib64/security/pam_securid.so rsaauthagent-7.1.0.1.25-0.x86_64 But it still crashed: 160715 10:41:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=5 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1233659 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x7fe6562f0008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff390da5d30 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0x7ff395ff836e] sql/signal_handler.cc:160(handle_fatal_signal)[0x7ff395b266ad] /lib64/libpthread.so.0(+0xf100)[0x7ff395148100] /usr/lib64/security/pam_securid.so(+0x13357)[0x7ff38f8b0357] /usr/lib64/security/pam_securid.so(+0x147e4)[0x7ff38f8b17e4] /usr/lib64/security/pam_securid.so(pam_sm_authenticate+0x17bc)[0x7ff38f8b382d] /lib64/libpam.so.0(+0x2f6a)[0x7ff385eadf6a] /lib64/libpam.so.0(pam_authenticate+0x30)[0x7ff385ead830] /usr/lib64/mysql/plugin/auth_pam.so(+0xf1e)[0x7ff3860baf1e] sql/sql_acl.cc:12263(do_auth_once)[0x7ff39593b341] sql/sql_acl.cc:12374(acl_authenticate(THD*, unsigned int))[0x7ff39595df5d] sql/sql_connect.cc:1024(check_connection)[0x7ff395a738df] sql/sql_connect.cc:1092(login_connection(THD*))[0x7ff395a75275] sql/sql_connect.cc:1270(thd_prepare_connection(THD*))[0x7ff395a75be9] sql/sql_connect.cc:1340(do_handle_one_connection(THD*))[0x7ff395a75db0] sql/sql_connect.cc:1263(handle_one_connection)[0x7ff395a75fb0] /lib64/libpthread.so.0(+0x7dc5)[0x7ff395140dc5] /lib64/libc.so.6(clone+0x6d)[0x7ff39356228d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 13 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on

Sergei Golubchik added a comment - 2016-07-18 11:37 There isn't much I can do without RSA SecurID token and SQLyog. I can, I suppose, build a debugging version of auth_pam module that logs all the data to a file and then to try to analyze the differences between SQLyog and mysql cli cases. Note that this log might contain passwords and other sensitive data.

Geoff Montee (Inactive) added a comment - 2016-07-18 16:49 Since elenst found record of a similar crash in pam_securid.so with no mention of MariaDB, it seems likely to me that this could be caused by some bug in pam_securid.so. I asked the user if they have some support channel at RSA that they can use to ask them for help.

Geoff Montee (Inactive) added a comment - 2016-07-19 16:36 RSA was not very helpful: Unfortunately, We do not support SQLyog application and it is not in our scope. In RHEL, we do support SSH, Telnet, SU, Login, rlogin, ftp, sudo and gdm. I would request you to reach out to RHEL Team or SQLyog team to check about this.

Sergei Golubchik added a comment - 2016-07-26 10:57 I've pushed a commit that adds debug output to pam plugin into bb-10.1-serg branch. When it'll build, you can find binaries at the usual location.

Sergei Golubchik added a comment - 2017-09-12 15:20 It turned out to be an issue in SQLyog, not Connector/C or MariaDB.

Geoff Montee (Inactive) added a comment - 2017-11-29 21:56 The user who reported this issue says that SQLyog still causes MariaDB to crash, and that the problem goes away if SQLyog is built with MySQL's connector instead of MariaDB Connector/C. This appears to be a bug in MariaDB and/or MariaDB Connector/C.

Geoff Montee (Inactive) added a comment - 2018-03-14 19:56 I'm closing this, because this problem probably reflects a bug in pam_securid.so which would have to be fixed by RSA rather than MariaDB.