[MDEV-16578] Repeated mysql_library_init()+mysql_library_end() calls freeze running process Created: 2018-06-25  Updated: 2024-01-30  Resolved: 2024-01-30

Status: Closed
Project: MariaDB Server
Component/s: Embedded Server
Affects Version/s: 10.1
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Pali Assignee: Sergei Golubchik
Resolution: Cannot Reproduce Votes: 0
Labels: None

Issue Links:
Relates
relates to CONC-336 Repeated mysql_library_init()+mysql_l... Closed

 Description   

When test test-init.c from CONC-336 is linked with `mysql_config --cflags --libmysqld-libs` then second run cause deadlock in inline_mysql_cond_destroy.

$ gcc -W -Wall -O2 -g test-init.c -o test-init `mysql_config --cflags --libmysqld-libs`
$ gdb --args ./test-init 2 127.0.0.1 3306 pali pali
...
(gdb) r
...
^C
Program received signal SIGINT, Interrupt.
__lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
95      ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Adresár alebo súbor neexistuje.
(gdb) bt
#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007ffff6184eb5 in __pthread_cond_destroy (cond=cond@entry=0x7ffff7568ae0 <COND_thread_count>) at pthread_cond_destroy.c:34
#2  0x00007ffff67436c0 in inline_mysql_cond_destroy (that=0x7ffff7568ae0 <COND_thread_count>) at ./include/mysql/psi/mysql_thread.h:1131
#3  clean_up_mutexes () at ./libmysqld/../sql/mysqld.cc:2264
#4  0x00007ffff6748974 in end_embedded_server () at ./libmysqld/lib_sql.cc:648
#5  0x00007ffff674c7ec in mysql_server_end () at ./libmysql/libmysql.c:213
#6  0x00005555555549e4 in main (argc=<optimized out>, argv=<optimized out>) at test-init.c:49
(gdb)

This is from MariaDB version 10.1.26, which is available in current Debian stable.

So problem is in function mysql_server_end which cause that deadlock.

Originally reported in comment: https://jira.mariadb.org/browse/CONC-336?focusedCommentId=111921&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-111921



 Comments   
Comment by Elena Stepanova [ 2018-07-02 ]

Did you see georg's comment after yours in CONC-336?

Please also note that there is no need to initialize the client library multiple times. If you don't call mysql_library_init explicitly a call to mysql_init will automatically initialize the library.

Is the hang still reproducible if you don't initialize the library multiple times?

Comment by Pali [ 2018-07-03 ]

Yes, I saw that comment, but it does not apply here. If you initialize library without embedded server and then you want to start embedded server you need to first deinitialize it and then initialize it again (with embedded server args). Similarly when you initialized library with embedded server, then embedded server is running. And when you want to stop embedded server you can do that only via deinitializing library (mysql_library_end() resp. mysql_server_end()). Please note that this logic is used in Perl DBI driver, which provides uniform upper-layer database interface for Perl applications and implements library access to mysql/mariadb client. Therefore if upper-layer ask for closing embedded server connection, then DBI driver needs to call mysql_server_end(). And after that if upper-layer ask for new remote connection, DBI driver needs to call mysql_library_init()/mysql_server_init() again. Important part is that everything is running in one process in which is loaded Perl interpreter and this cannot be (easily) changed. JDBC or other similar interfaces can use one process per connection or per driver to workaround those limitations, but this is not the case of Perl's DBI.

Comment by Pali [ 2018-09-28 ]

Hi elenst! Do you have any update for this issue?

Comment by Elena Stepanova [ 2018-10-04 ]

I'll leave it to georg to comment on it.

Comment by Georg Richter [ 2018-10-09 ]

I think this affects embedded elenst

Comment by Pali [ 2019-06-18 ]

Yes, if affects embedded server. elenst, any update to this issue?

Comment by Pali [ 2020-02-02 ]

Hello sanja! Could you please look at this issue? Do you need more details or logs for this issue?

Comment by Oleksandr Byelkin [ 2020-02-02 ]

I looked (for embedded) for me it tries to take uninitialized mutex. Looks quite different from original bug.

in any case what I lack is time to fix, sorry (the bug is in queue)

Comment by Pali [ 2023-08-23 ]

Hello sanja, any updates about this issue?

Comment by Sergei Golubchik [ 2023-09-01 ]

10.1 is long out of support. What version are you using now? Is the bug still there?

I'm getting different symptoms on 11.3

Comment by Pali [ 2023-10-03 ]

I have tried testing application with mariadb 11.3.0 and it still crashes.

Why you have closed this issue as incomplete? What is missing from me? I have provided description with simple testing application which reproduce it. So what is incomplete?

Comment by Sergei Golubchik [ 2023-10-04 ]

I've closed it as incomplete, because there was no answer for a month, so the assumption was that the original reporter (you) has lost interest and moved on, and I couldn't repeat the issue, so there was nothing we could do here.

Now that you replied and clearly shown that the assumption was wrong, I've reopened it

Comment by Pali [ 2023-10-04 ]

So, if you cannot reproduce it, does it mean that it is working for you? That is quite strange as I just took source code package of 11.3.0 version, compiled it locally, linked it into the attached testing application and run it. And it still crashed.

Comment by Pali [ 2023-11-20 ]

Hello, any progress on this issue?

Comment by Pali [ 2023-11-20 ]

Just to note that I have run testing application test-init.c from the description with mariadb 11.3.0 and it still crashes.

Comment by Sergei Golubchik [ 2023-11-29 ]

I cannot repeat it (still):

$ gcc -Iinclude -Llibmysqld  -lmysqld -lpcre2-8 -lcrypt -llz4 -llzo2 -llzma -lbz2 -lsnappy -lz -lm -lssl -lcrypto -ldl test-init.c -DEMBEDDED_LIBRARY=1
$ LD_LIBRARY_PATH=libmysqld ./a.out 10 127.0.0.1 16000 root ''
$ LD_LIBRARY_PATH=libmysqld ./a.out 100 127.0.0.1 16000 root ''
$ LD_LIBRARY_PATH=libmysqld ./a.out 1000 127.0.0.1 16000 root ''
$ LD_LIBRARY_PATH=libmysqld ./a.out 10000 127.0.0.1 16000 root ''
$

doesn't crash. (LD_LIBRARY_PATH because I didn't install libmysqld.so globally)

it was the latest 10.6, but as you said it crashes on 10.1 and on 11.3, I suppose 10.6 should crash too

Comment by Otto Kekäläinen [ 2023-12-01 ]

I also tested this on building and running on Debian Bookworm with these build dependencies installed and using these commands:

noformat
dpkg -l | grep mariadb
ii libmariadb-dev 1:10.11.4-1~deb12u1
ii libmariadb-dev-compat 1:10.11.4-1~deb12u1
ii libmariadb3:amd64 1:10.11.4-1~deb12u1
ii mariadb-common 1:10.11.4-1~deb12u1

gcc -W -Wall -O2 -g test-init.c -o test-init -I/usr/include/mariadb -I/usr/include/mariadb/mysql -L/usr/lib/x86_64-linux-gnu/ -lmariadb -lz -ldl -lm -lssl -lcrypto

mariadb -Bse 'STATUS' -u mysql -pasdf234
--------------
mariadb Ver 15.1 Distrib 10.11.6-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper

Connection id: 182
Current database:
Current user: mysql@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8mb4
Db characterset: utf8mb4
Client characterset: latin1
Conn. characterset: latin1
UNIX socket: /run/mysqld/mysqld.sock

./test-init 2 localhost 3306 mysql asdf234
noformat

I was not able to reproduce a crash.

Can you Pali please provide a reproducible case that is detailed enough so it can be reproduced?

It can be as simple as a copy-paste of the commands you ran and their outputs.

Comment by Georg Richter [ 2023-12-01 ]

otto You link against libmariadb, but the problem appears with libmysqld (embedded).

Comment by Pali [ 2023-12-25 ]

Hello, sorry for a longer delay. Now I have tested it with new 10.11.6 version (last 10 released at this time of testing) and the result is that this version is not affected by this bug. I'm going to test also other versions to figure out which are broken and which works. Because something must have been fixed in this 10.x version branch as I'm really sure that it was crashing.

Here are steps which I used for future reproducing:

$ wget https://archive.mariadb.org/mariadb-10.11.6/source/mariadb-10.11.6.tar.gz
$ tar -xf mariadb-10.11.6.tar.gz
$ cd mariadb-10.11.6/
$ mkdir build
$ cd build/
$ cmake -DWITH_EMBEDDED_SERVER=ON -DWITH_UNIT_TESTS=OFF ..
$ cd libmysqld/
$ make -j8
$ wget https://jira.mariadb.org/secure/attachment/45689/test-init.c
$ gcc -W -Wall -O2 -g test-init.c -o test-init -I../../libmariadb/include/ -I../libmariadb/include/ ./libmysqld.so -Wl,-rpath=.
$ ./test-init 2 127.0.0.1 3306 test ''

(MariaDB server is running on localhost at port 3306 and user named test can be used without password)

Please check the steps if are correct and if I did not some mistake there.

To ensure that correct library (libmysql*d*.so is symlink to libmariadb*d*.so) was used here is output from ldd:

$ ldd ./test-init
        linux-vdso.so.1 (0x00007ffc949ef000)
        libmariadbd.so.19 => ./libmariadbd.so.19 (0x00007fc8630cc000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc862ec7000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fc862e8d000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc862e6f000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fc862ddc000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fc862af0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc862ae9000)
        libaio.so.1 => /usr/lib/x86_64-linux-gnu/libaio.so.1 (0x00007fc862ae4000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc862960000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc8627dd000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc8627c3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc8627a2000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc8650b7000)

And cmake showed following features details (in case some feature may affect this problem):

The following features have been enabled:
 
 * WSREP, Server plugin STATIC
 * ARCHIVE, Storage Engine MODULE
 * BLACKHOLE, Storage Engine MODULE
 * CONNECT_VCT, Support for VCT in the CONNECT storage engine
 * CONNECT_LIBXML2, Support for LIBXML2 in the CONNECT storage engine
 * CONNECT_ODBC, Support for ODBC in the CONNECT storage engine
 * CONNECT_ZIP, Support for ZIP in the CONNECT storage engine
 * CONNECT_REST, Support for REST API in the CONNECT storage engine
 * CONNECT_XMAP, Support for index file mapping in the CONNECT storage engine
 * CONNECT, Storage Engine MODULE
 * CSV, Storage Engine STATIC
 * EXAMPLE, Storage Engine MODULE
 * FEDERATED, Storage Engine MODULE
 * FEDERATEDX, Storage Engine MODULE
 * HEAP, Storage Engine STATIC
 * INNODB_AHI, InnoDB Adaptive Hash Index
 * INNODB_ROOT_GUESS, Cache index root block descriptors in InnoDB
 * INNOBASE, Storage Engine STATIC
 * MARIABACKUP, MariaDB Backup Utility
 * ARIA, Storage Engine STATIC
 * S3, Storage Engine MODULE
 * MROONGA, Storage Engine MODULE
 * MYISAM, Storage Engine STATIC
 * MYISAMMRG, Storage Engine STATIC
 * PERFSCHEMA, Storage Engine STATIC
 * ROCKSDB, Storage Engine MODULE
 * ROCKSDB_LZ4, LZ4 Compression in the RocksDB storage engine
 * ROCKSDB_BZip2, BZip2 Compression in the RocksDB storage engine
 * ROCKSDB_Snappy, Snappy Compression in the RocksDB storage engine
 * ROCKSDB_ZSTD, ZSTD Compression in the RocksDB storage engine
 * ROCKSDB_ZLIB, zlib Compression in the RocksDB storage engine
 * SEQUENCE, Storage Engine STATIC
 * SPHINX, Storage Engine MODULE
 * SPIDER, Storage Engine MODULE
 * TEST_SQL_DISCOVERY, Storage Engine MODULE
 * AUDIT_NULL, Server plugin MODULE
 * AUTH_ED25519, Server plugin MODULE
 * DIALOG_EXAMPLES, Server plugin MODULE
 * AUTH_TEST_PLUGIN, Server plugin MODULE
 * QA_AUTH_INTERFACE, Server plugin MODULE
 * QA_AUTH_SERVER, Server plugin MODULE
 * QA_AUTH_CLIENT, Server plugin MODULE
 * AUTH_0X0100, Server plugin MODULE
 * AUTH_SOCKET, Server plugin STATIC
 * DAEMON_EXAMPLE, Server plugin MODULE
 * DEBUG_KEY_MANAGEMENT, Server plugin MODULE
 * DISKS, Server plugin MODULE
 * EXAMPLE_KEY_MANAGEMENT, Server plugin MODULE
 * FEEDBACK, Server plugin STATIC
 * FILE_KEY_MANAGEMENT, Server plugin MODULE
 * FTEXAMPLE, Server plugin MODULE
 * FUNC_TEST, Server plugin MODULE
 * HANDLERSOCKET, Server plugin MODULE
 * HASHICORP_KEY_MANAGEMENT, Hashicorp Key Management Plugin
 * LOCALES, Server plugin MODULE
 * METADATA_LOCK_INFO, Server plugin MODULE
 * PASSWORD_REUSE_CHECK, Server plugin MODULE
 * PROVIDER_BZIP2, Server plugin MODULE
 * PROVIDER_LZMA, Server plugin MODULE
 * PROVIDER_LZO, Server plugin MODULE
 * QUERY_CACHE_INFO, Server plugin MODULE
 * QUERY_RESPONSE_TIME, Server plugin MODULE
 * SERVER_AUDIT, Server plugin MODULE
 * SIMPLE_PASSWORD_CHECK, Server plugin MODULE
 * SQL_ERRLOG, Server plugin MODULE
 * TEST_SQL_SERVICE, Server plugin MODULE
 * TYPE_GEOM, Server plugin STATIC
 * TYPE_INET, Server plugin STATIC
 * TYPE_MYSQL_JSON, Server plugin MODULE
 * TYPE_MYSQL_TIMESTAMP, Server plugin MODULE
 * TYPE_TEST, Server plugin MODULE
 * TYPE_UUID, Server plugin STATIC
 * USER_VARIABLES, Server plugin STATIC
 * USERSTAT, Server plugin STATIC
 * TEST_VERSIONING, Server plugin MODULE
 * WSREP_INFO, Server plugin MODULE
 * THREAD_POOL_INFO, Server plugin STATIC
 * PARTITION, Storage Engine STATIC
 * SQL_SEQUENCE, Storage Engine STATIC
 * EMBEDDED_SERVER, Embedded MariaDB Server Library
 
-- The following OPTIONAL packages have been found:
 
 * Git
 * LibXml2
 * CURL
 * Boost (required version >= 1.40.0)
   Required for the OQGraph storage engine
 * BZip2
 * LibLZMA
 * LZO
 * BISON (required version >= 2.4)
 
-- The following RECOMMENDED packages have been found:
 
 * OpenSSL
 
-- The following REQUIRED packages have been found:
 
 * Curses
 * Threads
 
-- The following features have been disabled:
 
 * LIBWRAP, Support for tcp wrappers
 * SYSTEMD, Systemd scripts and notification support
 * CONNECT_JDBC, Support for JDBC in the CONNECT storage engine
 * CONNECT_MONGODB, Support for MongoDB in the CONNECT storage engine
 * NUMA, NUMA memory allocation policy
 * INNODB_EXTRA_DEBUG, Extra InnoDB debug checks
 * AWS_KEY_MANAGEMENT, AWS Encryption Key Management Plugin
 
-- The following OPTIONAL packages have not been found:
 
 * Java (required version >= 1.6)
   Required for the CONNECT_JDBC feature
 * JNI
   Required for the CONNECT_JDBC feature
 * PMEM
 * Judy
   Required for the OQGraph storage engine
 * GSSAPI
 * LZ4 (required version >= 1.6)
 * Snappy

Comment by Pali [ 2023-12-25 ]

Now I take MariaDB version 10.1.26 from the sources: https://archive.mariadb.org/mariadb-10.1.26/source/mariadb-10.1.26.tar.gz

Then I used same steps for compiling as in the previous post but into cmake I added another parameter -DWITH_SSL=bundled because such old MariaDB version failed to compile with new OpenSSL.

And testing application passed without any crash and without any deadlock.

So I have feeling that this issue is related to some setup or something other. Becuase for sure the version from the description (10.1.26) caused this issue and I put into description backtrace where it hanged.

So can be end_embedded_server() or its init function affected by some SSL library? Or some system / additional plugin? Or something other? If you have some idea or what can I test then please let me know. I think I do not have original machine with original system on which it was crashing.

Comment by Sergei Golubchik [ 2023-12-25 ]

I see, thanks. Yes, it could be an SSL library, could be a specific plugin — there could be many things that don't like being initialized and deinitialized repeatedly. I mean, it's not a given that some piece of code can deinitialize itself back to its initial state.

Still, it's likely some external library. I suspect your recent test most probably had exactly the same set of plugins of exactly same versions as your earlier tests, that exhibited the crash.

Comment by Pali [ 2023-12-25 ]

Now I tried another thing: I took Debian version of MariaDB 10.1.26 (which has patches) from archive:
https://snapshot.debian.org/package/mariadb-10.1/10.1.26-1/

Unpacked it via:

$ dpkg-source -x mariadb-10.1_10.1.26-1.dsc

And then applied same build steps as before. But results are same. It is working.

I do not know what else I can try...

Comment by Sergei Golubchik [ 2024-01-30 ]

pali, I'll close it as Can't repeat then. If you'll stumble upon it again, please, just add a comment here and I'll reopen

Generated at Thu Feb 08 08:29:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.