serg, would you look at this?

GeoffMontee,

I don't know if serg will need any of this, but here are first questions that came to my mind after seeing this:

• Did the crash happen more than once?
• Had anything changed on the user's side before the crash(es) happened? System upgrade, configuration change, anything like that?
• Assuming the crash happened more than once, is it still happening, or has it stopped, and if it's stopped, what the user had done to achieve this?
• Which system does the server run on, and which versions of pam libraries are installed?
• Was there anything suspicious in the system logs around the time of the crash?

Did the crash happen more than once?

The crash happened several times.

Assuming the crash happened more than once, is it still happening, or has it stopped, and if it's stopped, what the user had done to achieve this?

The user was trying to log in to a MariaDB server from SQLyog using a user account that is configured to use PAM authentication.

I have asked the user to clarify if the crash happens every time the user logs in with that user account from SQLyog.

I have also asked the user if pam_use_cleartext_plugin is enabled on the server. If SQLyog doesn't support the dialog plugin, could that lead to crashes?

I will try to get more details.

The crash is happening, as you can see, not in auth_pam, but in pam_securid.so pam module.
So the first question is — does the user need to use pam_securid for MariaDB? I mean, do they actually authenticate their database users with pam_securid?

I mean, do they actually authenticate their database users with pam_securid?

It does appear so. I believe the intention is to use some kind of RSA token in addition to passwords to authenticate DBA user accounts. Here's the PAM policy for the user's MariaDB server:

#%PAM-1.0

auth       required     pam_securid.so debug

#account    include    password-auth

#password   include    password-auth 

account    required pam_securid.so debug

password   required pam_securid.so debug

And pam_use_cleartext_plugin is turned OFF.

We've had a similar case before. A user was complaining about a crash, in pam_fprintd.so, if I'm not mistaken. It turned out that this particular PAM module was not safe for concurrent use, so when many users tried to connect (and authenticate) at the same time, it crashed. In that case, though, the user did not want to authenticate database users with fingerprints, he simply reused some global policy file. So the workaround was to remove pam_fprintd.so from MariaDB PAM policy.

It looks like pam_securid.so is the reason here, too. But I don't know if it's concurrency or something else that triggers the crash. Do you know how to cause the crash? Is it deterministic?

Do you know how to cause the crash? Is it deterministic?

The user says that the server crashes every time that they try to authenticate with an account configured to use an RSA token, but I am trying to get him to clarify if the crash happens when connecting from any client, or just when connecting from SQLyog.

I've also attached /var/log/messages from the user's server.

The user clarified that it does work without crashing when connecting with the mysql command-line client. So far, it only seems to crash when connecting via SQLyog.

The user's version of pam_securid.so:

> rpm -qf /usr/lib64/security/pam_securid.so

rsaauthagent-7.0-0.x86_64

This version looks like it might be kind of old, based on the versions listed here:

https://www.rsa.com/en-us/products-services/identity-access-management/securid/authentication-agents/authentication-agents-for-pam

I've asked the user to try upgrading.

If the problem is indeed in the module, I'm not sure the upgrade will help – I've seen a similar complaint about 7.1.0.1:
https://www.redhat.com/archives/pam-list/2015-April/msg00003.html
Unfortunately, I couldn't find any follow-up.

It looks like you were right about that.

The user upgraded to this version:

> rpm -qf /usr/lib64/security/pam_securid.so

rsaauthagent-7.1.0.1.25-0.x86_64

But it still crashed:

160715 10:41:11 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.1.10-MariaDB-log

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=6

max_threads=502

thread_count=5

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1233659 K bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7fe6562f0008

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7ff390da5d30 thread_stack 0x48000

mysys/stacktrace.c:247(my_print_stacktrace)[0x7ff395ff836e]

sql/signal_handler.cc:160(handle_fatal_signal)[0x7ff395b266ad]

/lib64/libpthread.so.0(+0xf100)[0x7ff395148100]

/usr/lib64/security/pam_securid.so(+0x13357)[0x7ff38f8b0357]

/usr/lib64/security/pam_securid.so(+0x147e4)[0x7ff38f8b17e4]

/usr/lib64/security/pam_securid.so(pam_sm_authenticate+0x17bc)[0x7ff38f8b382d]

/lib64/libpam.so.0(+0x2f6a)[0x7ff385eadf6a]

/lib64/libpam.so.0(pam_authenticate+0x30)[0x7ff385ead830]

/usr/lib64/mysql/plugin/auth_pam.so(+0xf1e)[0x7ff3860baf1e]

sql/sql_acl.cc:12263(do_auth_once)[0x7ff39593b341]

sql/sql_acl.cc:12374(acl_authenticate(THD*, unsigned int))[0x7ff39595df5d]

sql/sql_connect.cc:1024(check_connection)[0x7ff395a738df]

sql/sql_connect.cc:1092(login_connection(THD*))[0x7ff395a75275]

sql/sql_connect.cc:1270(thd_prepare_connection(THD*))[0x7ff395a75be9]

sql/sql_connect.cc:1340(do_handle_one_connection(THD*))[0x7ff395a75db0]

sql/sql_connect.cc:1263(handle_one_connection)[0x7ff395a75fb0]

/lib64/libpthread.so.0(+0x7dc5)[0x7ff395140dc5]

/lib64/libc.so.6(clone+0x6d)[0x7ff39356228d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): is an invalid pointer

Connection ID (thread ID): 13

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on

There isn't much I can do without RSA SecurID token and SQLyog.

I can, I suppose, build a debugging version of auth_pam module that logs all the data to a file and then to try to analyze the differences between SQLyog and mysql cli cases. Note that this log might contain passwords and other sensitive data.

Since elenst found record of a similar crash in pam_securid.so with no mention of MariaDB, it seems likely to me that this could be caused by some bug in pam_securid.so. I asked the user if they have some support channel at RSA that they can use to ask them for help.

RSA was not very helpful:

Unfortunately, We do not support SQLyog application and it is not in our scope. In RHEL, we do support SSH, Telnet, SU, Login, rlogin, ftp, sudo and gdm. I would request you to reach out to RHEL Team or SQLyog team to check about this.

I've pushed a commit that adds debug output to pam plugin into bb-10.1-serg branch. When it'll build, you can find binaries at the usual location.

It turned out to be an issue in SQLyog, not Connector/C or MariaDB.

The user who reported this issue says that SQLyog still causes MariaDB to crash, and that the problem goes away if SQLyog is built with MySQL's connector instead of MariaDB Connector/C. This appears to be a bug in MariaDB and/or MariaDB Connector/C.

I'm closing this, because this problem probably reflects a bug in pam_securid.so which would have to be fixed by RSA rather than MariaDB.