[MDEV-16693] libmariadbclient not threadsafe anymore Created: 2018-06-25  Updated: 2019-06-05  Resolved: 2019-06-05

Status: Closed
Project: MariaDB Server
Component/s: libmariadb
Affects Version/s: 10.1.34
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Noel Kuntze Assignee: Sergei Golubchik
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Linux

Attachments: File core.charon-systemd.0.ae09eee7fbd0460fa2c78720d957a226.37439.1529943447000000.lz4     File core.charon-systemd.0.ae09eee7fbd0460fa2c78720d957a226.37681.1529943473000000.lz4     File strongswan-5.6.3-1-x86_64.pkg.tar.xz     Text File valgrind.txt    

 Description   

libmariadbclient connections are not thread safe anymore.
This is a problem shared with libmysqlclient, under which it is reproducible.
The application with the issue is strongSwan, which is heavily multithreaded and reuses connections. The code worked fine previously in heavily loaded environments, but not anymore since at least the last version. The relevant code can be seen under this URL: https://github.com/strongswan/strongswan/blob/master/src/libstrongswan/plugins/mysql/mysql_database.c

The stack traces are reproducible and always occur with either memory access or memory allocation in libmysqlclient.so of libmariadbclient.

The coredumps are attached and so are the built packages for Arch, which contain all relevant libraries. The packages are too large to upload here, so they are available under this URL: https://thermi.consulting/~thermi~private/00354388-8950-4ffe-baaf-dbb6aa6dfa35/

Attached is also a valgrind output of a run of strongSwan in which many invalid reads or writes in the mysql libraries occur. This should make it easier to figure out on your end.
I do not remember to which crash the file relates. I tested this many times, so don't bother trying to do that.

gcc: gcc 8.1.1+20180531
mariadb: 10.1.34

Please let me know if you need anything else.

Stack Trace 1 from a scenario where two clients work with the connection objects: 

                Stack trace of thread 40388:                   
                #0  0x00007f21df95d73b raise (libc.so.6)                                                                     
                #1  0x00007f21df948406 abort (libc.so.6)       
                #2  0x00005603242bf84b n/a (charon-systemd)    
                #3  0x00007f21dfcf4a50 __restore_rt (libpthread.so.0)
 
                #4  0x00007f21dfcddc90 main_arena (libc.so.6)                                                                
                #5  0x00007f21d7fb1814 my_net_read_packet (libmysqlclient.so.18)
 
                #6  0x00007f21d7fa3160 cli_safe_read (libmysqlclient.so.18)
 
                #7  0x00007f21d7fa3552 cli_read_rows (libmysqlclient.so.18)                                                  
                #8  0x00007f21d7f9f51d cli_read_prepare_result (libmysqlclient.so.18)
 
                #9  0x00007f21d7fa17bd mysql_stmt_prepare (libmysqlclient.so.18)
 
                #10 0x00007f21d85083e3 n/a (libstrongswan-mysql.so)  
                #11 0x00007f21d85091af n/a (libstrongswan-mysql.so)
 
                #12 0x00007f21d7230e17 n/a (libstrongswan-attr-sql.so)                                                       
                #13 0x00007f21e042e49d n/a (libstrongswan.so.0)                                                              
                #14 0x00007f21e042e5e5 enumerator_enumerate_default (libstrongswan.so.0)                                     
                #15 0x00007f21e01d5506 n/a (libcharon.so.0)                 
                #16 0x00007f21e01c8a08 n/a (libcharon.so.0)
 
                #17 0x00007f21e01b7ae7 n/a (libcharon.so.0)
 
                #18 0x00007f21e01b15df n/a (libcharon.so.0)    
                #19 0x00007f21e044ee13 n/a (libstrongswan.so.0)
 
                #20 0x00007f21e0460b8e n/a (libstrongswan.so.0)                                                              
                #21 0x00007f21dfcea075 start_thread (libpthread.so.0)
 
                #22 0x00007f21dfa1f2cf __clone (libc.so.6)

Stack Trace 2, here with multiple threads involved

                Stack trace of thread 37626:
 
                #0  0x00007f336c56c73b raise (libc.so.6)
 
                #1  0x00007f336c557406 abort (libc.so.6)
 
                #2  0x00007f336c5ae6a8 __libc_message (libc.so.6)
 
                #3  0x00007f336c5b4aea malloc_printerr (libc.so.6)
 
                #4  0x00007f336c5b655c _int_free (libc.so.6)
 
                #5  0x00007f3364be28ee vio_delete (libmysqlclient.so.18)
 
                #6  0x00007f3364bb2066 end_server (libmysqlclient.so.18)
 
                #7  0x00007f3364bb22a0 cli_safe_read (libmysqlclient.so.18)
 
                #8  0x00007f3364bb7bd5 cli_advanced_command (libmysqlclient.so.18)
 
                #9  0x00007f3364bad7a1 mysql_ping (libmysqlclient.so.18)
 
                #10 0x00007f3365117da5 n/a (libstrongswan-mysql.so)
 
                #11 0x00007f336511816e n/a (libstrongswan-mysql.so)
 
                #12 0x00007f3363e3ff2d n/a (libstrongswan-attr-sql.so)
 
                #13 0x00007f3363e40574 n/a (libstrongswan-attr-sql.so)
 
                #14 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #15 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #16 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #17 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #18 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #19 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #20 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #21 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #22 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37618:
 
                #0  0x00007f336c90299e recv (libpthread.so.0)
 
                #1  0x00007f3364be3994 recv (libmysqlclient.so.18)
 
                #2  0x00007f3364be3a1e vio_read_buff (libmysqlclient.so.18)
 
                #3  0x00007f3364bbf9f6 my_real_read (libmysqlclient.so.18)
 
                #4  0x00007f3364bc0814 my_net_read_packet (libmysqlclient.so.18)
 
                #5  0x00007f3364bb2160 cli_safe_read (libmysqlclient.so.18)
 
                #6  0x00007f3364bb7bd5 cli_advanced_command (libmysqlclient.so.18)
 
                #7  0x00007f3364bad7a1 mysql_ping (libmysqlclient.so.18)
 
                #8  0x00007f3365117da5 n/a (libstrongswan-mysql.so)
 
                #9  0x00007f3365117e9e n/a (libstrongswan-mysql.so)
 
                #10 0x00007f3363e3facc n/a (libstrongswan-attr-sql.so)
 
                #11 0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #12 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #13 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #14 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #15 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #16 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #17 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #18 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #19 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #20 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37439:
 
                #0  0x00007f336c56d4ec __sigtimedwait (libc.so.6)
 
                #1  0x0000559b2e7ee3cb n/a (charon-systemd)
 
                #2  0x00007f336c558f4b __libc_start_main (libc.so.6)
 
                #3  0x0000559b2e7ee5fa n/a (charon-systemd)
 
 
 
                Stack trace of thread 37622:
 
                #0  0x00007f336c63b5ab __lll_lock_wait_private (libc.so.6)
 
                #1  0x00007f336c5b6676 _int_free (libc.so.6)
 
                #2  0x00007f3364bbf8c6 net_end (libmysqlclient.so.18)
 
                #3  0x00007f3364bb2102 end_server (libmysqlclient.so.18)
 
                #4  0x00007f3364bb22a0 cli_safe_read (libmysqlclient.so.18)
 
                #5  0x00007f3364bb2552 cli_read_rows (libmysqlclient.so.18)
 
                #6  0x00007f3364bae51d cli_read_prepare_result (libmysqlclient.so.18)
 
                #7  0x00007f3364bb07bd mysql_stmt_prepare (libmysqlclient.so.18)
 
                #8  0x00007f33651173e3 n/a (libstrongswan-mysql.so)
 
                #9  0x00007f33651181af n/a (libstrongswan-mysql.so)
 
                #10 0x00007f3363e3ff2d n/a (libstrongswan-attr-sql.so)
 
                #11 0x00007f3363e40574 n/a (libstrongswan-attr-sql.so)
 
                #12 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #13 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #14 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #15 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #16 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #17 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #18 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #19 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #20 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37623:
 
                #0  0x00007f336c90299e recv (libpthread.so.0)
 
                #1  0x00007f3364be3994 recv (libmysqlclient.so.18)
 
                #2  0x00007f3364be3a1e vio_read_buff (libmysqlclient.so.18)
 
                #3  0x00007f3364bbf9f6 my_real_read (libmysqlclient.so.18)
 
                #4  0x00007f3364bc0814 my_net_read_packet (libmysqlclient.so.18)
 
                #5  0x00007f3364bb2160 cli_safe_read (libmysqlclient.so.18)
 
                #6  0x00007f3364bb3828 cli_read_query_result (libmysqlclient.so.18)
 
                #7  0x00007f3364bae13f execute (libmysqlclient.so.18)
 
                #8  0x00007f3364bae91d cli_stmt_execute (libmysqlclient.so.18)
 
                #9  0x00007f3364bb0ee6 mysql_stmt_execute (libmysqlclient.so.18)
 
                #10 0x00007f33651174ed n/a (libstrongswan-mysql.so)
 
                #11 0x00007f3365118072 n/a (libstrongswan-mysql.so)
 
                #12 0x00007f3363e3fb6f n/a (libstrongswan-attr-sql.so)
 
                #13 0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #14 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #15 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #16 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #17 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #18 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #19 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #20 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #21 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #22 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37625:
 
                #0  0x00007f336c902b1e __send (libpthread.so.0)
 
                #1  0x00007f3364be3c68 inline_mysql_socket_send (libmysqlclient.so.18)
 
                #2  0x00007f3364bbfcbe net_real_write (libmysqlclient.so.18)
 
                #3  0x00007f3364bbffc4 net_flush (libmysqlclient.so.18)
 
                #4  0x00007f3364bc0616 net_write_command (libmysqlclient.so.18)
 
                #5  0x00007f3364bb7b35 cli_advanced_command (libmysqlclient.so.18)
 
                #6  0x00007f3364bb0a4b mysql_stmt_close (libmysqlclient.so.18)
 
                #7  0x00007f336511809f n/a (libstrongswan-mysql.so)
 
                #8  0x00007f3363e3fb6f n/a (libstrongswan-attr-sql.so)
 
                #9  0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #10 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #11 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #12 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #13 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #14 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #15 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #16 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #17 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #18 0x00007f336c62e2cf __clone (libc.so.6)
 
 
 
                Stack trace of thread 37631:
 
                #0  0x00007f336c8ff3c6 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
 
                #1  0x00007f336d070589 n/a (libstrongswan.so.0)
 
                #2  0x00007f336d05e663 n/a (libstrongswan.so.0)
 
                #3  0x00007f336d05d4a2 n/a (libstrongswan.so.0)
 
                #4  0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #5  0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #6  0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #7  0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37621:
 
                #0  0x00007f336c8fefcc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
 
                #1  0x00007f336cdbf3b8 n/a (libcharon.so.0)
 
                #2  0x00007f336d05d4a2 n/a (libstrongswan.so.0)
 
                #3  0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #4  0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #5  0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #6  0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37628:
 
                #0  0x00007f336c63b5f4 __libc_enable_asynccancel (libc.so.6)
 
                #1  0x00007f336c623c26 __poll (libc.so.6)
 
                #2  0x00007f3364bbfbfb poll (libmysqlclient.so.18)
 
                #3  0x00007f3364bb7b1d cli_advanced_command (libmysqlclient.so.18)
 
                #4  0x00007f3364bb0a4b mysql_stmt_close (libmysqlclient.so.18)
 
                #5  0x00007f336511809f n/a (libstrongswan-mysql.so)
 
                #6  0x00007f3363e3fb6f n/a (libstrongswan-attr-sql.so)
 
                #7  0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #8  0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #9  0x00007f336cde4217 n/a (libcharon.so.0)
 
                #10 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #11 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #12 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #13 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #14 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #15 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #16 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37620:
 
                #0  0x00007f336c5b7e90 _int_malloc (libc.so.6)
 
                #1  0x00007f336c5b941a malloc (libc.so.6)
 
                #2  0x00007f336cdb734f ike_header_create (libcharon.so.0)
 
                #3  0x00007f336cdb20b8 n/a (libcharon.so.0)
 
                #4  0x00007f336cdaeb64 n/a (libcharon.so.0)
 
                #5  0x00007f336cdbeb1f n/a (libcharon.so.0)
 
                #6  0x00007f336d05d4a2 n/a (libstrongswan.so.0)
 
                #7  0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #8  0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #9  0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #10 0x00007f336c62e2cf __clone (libc.so.6)
 
 
 
                Stack trace of thread 37629:
 
                #0  0x00007f3364bebd25 free_root (libmysqlclient.so.18)
 
                #1  0x00007f3364bb0e30 reset_stmt_handle (libmysqlclient.so.18)
 
                #2  0x00007f33651174ed n/a (libstrongswan-mysql.so)
 
                #3  0x00007f3365118072 n/a (libstrongswan-mysql.so)
 
                #4  0x00007f3363e3fb6f n/a (libstrongswan-attr-sql.so)
 
                #5  0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #6  0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #7  0x00007f336cde4217 n/a (libcharon.so.0)
 
                #8  0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #9  0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #10 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #11 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #12 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #13 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #14 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37630:
 
                #0  0x00007f3364bb28a9 cli_read_rows (libmysqlclient.so.18)
 
                #1  0x00007f3364bb398e cli_read_query_result (libmysqlclient.so.18)
 
                #2  0x00007f3364bae13f execute (libmysqlclient.so.18)
 
                #3  0x00007f3364bae91d cli_stmt_execute (libmysqlclient.so.18)
 
                #4  0x00007f3364bb0ee6 mysql_stmt_execute (libmysqlclient.so.18)
 
                #5  0x00007f33651174ed n/a (libstrongswan-mysql.so)
 
                #6  0x00007f33651181af n/a (libstrongswan-mysql.so)
 
                #7  0x00007f3363e3fb08 n/a (libstrongswan-attr-sql.so)
 
                #8  0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #9  0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #10 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #11 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #12 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #13 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #14 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #15 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #16 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #17 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37627:
 
                #0  0x00007f336c5b8001 _int_malloc (libc.so.6)
 
                #1  0x00007f336c5b941a malloc (libc.so.6)
 
                #2  0x00007f3364bf0767 my_malloc (libmysqlclient.so.18)
 
                #3  0x00007f3364bebaa3 alloc_root (libmysqlclient.so.18)
 
                #4  0x00007f3364bb261e cli_read_rows (libmysqlclient.so.18)
 
                #5  0x00007f3364bb398e cli_read_query_result (libmysqlclient.so.18)
 
                #6  0x00007f3364bae13f execute (libmysqlclient.so.18)
 
                #7  0x00007f3364bae91d cli_stmt_execute (libmysqlclient.so.18)
 
                #8  0x00007f3364bb0ee6 mysql_stmt_execute (libmysqlclient.so.18)
 
                #9  0x00007f33651174ed n/a (libstrongswan-mysql.so)
 
                #10 0x00007f33651181af n/a (libstrongswan-mysql.so)
 
                #11 0x00007f3363e3fb08 n/a (libstrongswan-attr-sql.so)
 
                #12 0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #13 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #14 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #15 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #16 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #17 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #18 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #19 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #20 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #21 0x00007f336c62e2cf __clone (libc.so.6)
 
                Stack trace of thread 37632:
 
                #0  0x00007f3364be3ad6 vio_read_buff (libmysqlclient.so.18)
 
                #1  0x00007f3364bbf9f6 my_real_read (libmysqlclient.so.18)
 
                #2  0x00007f3364bc0814 my_net_read_packet (libmysqlclient.so.18)
 
                #3  0x00007f3364bb2160 cli_safe_read (libmysqlclient.so.18)
 
                #4  0x00007f3364bb2787 cli_read_rows (libmysqlclient.so.18)
 
                #5  0x00007f3364bae51d cli_read_prepare_result (libmysqlclient.so.18)
 
                #6  0x00007f3364bb07bd mysql_stmt_prepare (libmysqlclient.so.18)
 
                #7  0x00007f33651173e3 n/a (libstrongswan-mysql.so)
 
                #8  0x00007f33651181af n/a (libstrongswan-mysql.so)
 
                #9  0x00007f3363e3fb08 n/a (libstrongswan-attr-sql.so)
 
                #10 0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #11 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #12 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #13 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #14 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #15 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #16 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #17 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #18 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #19 0x00007f336c62e2cf __clone (libc.so.6)
 
                
                Stack trace of thread 37624:
 
                #0  0x00007f336c5b8001 _int_malloc (libc.so.6)
 
                #1  0x00007f336c5b941a malloc (libc.so.6)
 
                #2  0x00007f336c5a483d _IO_file_doallocate (libc.so.6)
 
                #3  0x00007f336c5b25c0 _IO_doallocbuf (libc.so.6)
 
                #4  0x00007f336c5b1635 _IO_file_underflow@@GLIBC_2.2.5 (libc.so.6)
 
                #5  0x00007f336c5b2672 _IO_default_uflow (libc.so.6)
 
                #6  0x00007f336c5acff8 _IO_getc (libc.so.6)
 
                #7  0x00007f336d06d6a2 n/a (libstrongswan.so.0)
 
                #8  0x00007f336d06da3d n/a (libstrongswan.so.0)
 
                #9  0x0000559b2e7ee80c n/a (charon-systemd)
 
                #10 0x00007f336c903a50 __restore_rt (libpthread.so.0)
 
                #11 0x00007f3364bbfabc my_real_read (libmysqlclient.so.18)
 
                #12 0x00007f3364bc0814 my_net_read_packet (libmysqlclient.so.18)
 
                #13 0x00007f3364bb2160 cli_safe_read (libmysqlclient.so.18)
 
                #14 0x00007f3364bae4c7 cli_read_prepare_result (libmysqlclient.so.18)
 
                #15 0x00007f3364bb07bd mysql_stmt_prepare (libmysqlclient.so.18)
 
                #16 0x00007f33651173e3 n/a (libstrongswan-mysql.so)
 
                #17 0x00007f3365118072 n/a (libstrongswan-mysql.so)
 
                #18 0x00007f3363e3fb6f n/a (libstrongswan-attr-sql.so)
 
                #19 0x00007f3363e4052f n/a (libstrongswan-attr-sql.so)
 
                #20 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #21 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #22 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #23 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #24 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #25 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #26 0x00007f336d06fb8e n/a (libstrongswan.so.0)
 
                #27 0x00007f336c8f9075 start_thread (libpthread.so.0)
 
                #28 0x00007f336c62e2cf __clone (libc.so.6)
 
 
 
                Stack trace of thread 37633:
 
                #0  0x00007f336c90299e recv (libpthread.so.0)
 
                #1  0x00007f3364be3994 recv (libmysqlclient.so.18)
 
                #2  0x00007f3364be3a1e vio_read_buff (libmysqlclient.so.18)
 
                #3  0x00007f3364bbf9f6 my_real_read (libmysqlclient.so.18)
 
                #4  0x00007f3364bc0814 my_net_read_packet (libmysqlclient.so.18)
 
                #5  0x00007f3364bb2160 cli_safe_read (libmysqlclient.so.18)
 
                #6  0x00007f3364bae4c7 cli_read_prepare_result (libmysqlclient.so.18)
 
                #7  0x00007f3364bb07bd mysql_stmt_prepare (libmysqlclient.so.18)
 
                #8  0x00007f33651173e3 n/a (libstrongswan-mysql.so)
 
                #9  0x00007f33651181af n/a (libstrongswan-mysql.so)
 
                #10 0x00007f3363e3ff2d n/a (libstrongswan-attr-sql.so)
 
                #11 0x00007f3363e40574 n/a (libstrongswan-attr-sql.so)
 
                #12 0x00007f336cda24d3 n/a (libcharon.so.0)
 
                #13 0x00007f336cde4217 n/a (libcharon.so.0)
 
                #14 0x00007f336cdd7a08 n/a (libcharon.so.0)
 
                #15 0x00007f336cdc6ae7 n/a (libcharon.so.0)
 
                #16 0x00007f336cdc05df n/a (libcharon.so.0)
 
                #17 0x00007f336d05de13 n/a (libstrongswan.so.0)
 
                #18 0x00007f336d06fb8e n/a (libstrongswan.so.0)



 Comments   
Comment by Noel Kuntze [ 2018-07-04 ]

It'd be great if this could be taken care of. This is an extreme regression in the code base of mariadb and the impact of this (set of) bugs is global

Comment by Georg Richter [ 2018-07-05 ]

Hi,

can you please provide the Connector/C version number?

Thanks!

Comment by Noel Kuntze [ 2018-07-05 ]

Hello,

It's not specific to the connector, but happens deep in libmysqlclient.so. It's probably a double free. Maybe it's better if it was moved/copied to the mariadb project? I opened it here because it's not specific to the server either.

Comment by Georg Richter [ 2018-07-05 ]

After checking the stack trace I noticed that the crash happens in libmysql (likely from a server package < 10.2). Connector/C is not used.

Comment by Sergei Golubchik [ 2018-07-05 ]

Do you remember on what version it used to work?

Comment by Noel Kuntze [ 2018-07-05 ]

That must have been a version from around mid 2017, I don't have any recordings or memories regarding the exact version. On Arch, the mariadb version around that time was 10.2.6[1].

As far as I can tell, it's a double free and it occurs when there is more than one connection.

[1] https://git.archlinux.org/svntogit/packages.git/log/trunk?h=packages/mariadb

Comment by Noel Kuntze [ 2018-07-05 ]

It also happens with mariadb 10.3.7 and mysql 5.7.20.

Comment by Sergei Golubchik [ 2018-07-06 ]

Hmm. mariadb-10.1 and 10.3 use two different client libraries, client library in 10.3 was written from scratch. And mysql 5.7 uses a heavily evolved library (but with the same origin as mariadb-10.1 library).

Given that three significantly different client libraries suddenly stopped working, the logical question is, are you sure it wasn't caused by strongSwan? Like, they've changed something in the mariadb/mysql related code and it started failing on all client libraries?

Comment by Noel Kuntze [ 2018-07-07 ]

No, the functional parts of the code wasn't changed in 5 years and even if, then the mysql library shouldn't permit actions that cause double frees or segfaults in it.

A short description of the usage model:

A pool of connections to the database is used. The connections are created on-demand and cleaned up, if they are not needed. One connection can only be used for one query/transaction at a time. When data is read, the connection can not be used by any other thread or any other action until that is done and the connection is not used anymore. The same happens analogously for writes (transactions). I debugged this and found out that this is indeed the case and there is no simultaneous usage happening when the problems occur.

Curiously, with 10.3.8, it seems the types of the variables are incorrect when the statements are created, at least that's what the library complains about. At some point, it segfaults, too.

Comment by Noel Kuntze [ 2018-07-11 ]

Hello,

What is the status on this ticket?

Comment by Sergei Golubchik [ 2018-07-11 ]

So far the status is (it's on the very top of the page) "Open". Eventually it'll either change to "Confirmed" or "Closed" (if this will turn out to be not MariaDB bug). If confirmed, it'll get something in the "Fix Version/s" field and then it'll be fixed according to the release schedule (on the start jira.mariadb.org page).

So, let's try to get it confirmed.

  • What file is the valgrind output, strongswan-5.6.3-1-x86_64.pkg.tar.xz ?
  • What error do you get? Your stack traces end in asserts inside glibc malloc. What does glibc say before crashing?

Later I might ask (if nothing else would help) to try to use -O0 -g3 compiled libmysqlclient or/and use MALLOC_CHECK_ or/and use debug-compiled libmysqlclient. But for now, just two questions above.

Comment by Noel Kuntze [ 2018-07-12 ]

Well, yes, I knew of the formal "status" in the bug tracker. I was more inquiring towards how this will go on and if you need anything from me.

I forgot to attach the valgrind output previously. It's attached now.

Glibc tells me one of the following, if it errors out with any memory allocation related fucnctions:

  • double free or corruption (out)
  • double free or corruption (!prev)

The applications and libraries are compiled with -g3. I can try with -O0 later and with the settings you mention. I didn't find the information on how to do that myself yet. Would you kindly give me a link to the instructions, so I can compile it with the settings you want?

Comment by Noel Kuntze [ 2018-08-31 ]

I have worked with the code and fixed a problem in the code that does the locking. Now the double free is left.

Comment by Noel Kuntze [ 2019-06-04 ]

Please close this issue. It's resolved.

Generated at Thu Feb 08 08:30:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.