[MDEV-4890] Valgrind warnings on shutdown on a build with openSSL Created: 2013-08-12  Updated: 2013-11-19  Resolved: 2013-11-19

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.0.4, 5.5.32, 5.1.67, 5.3.12
Fix Version/s: 10.0.5, 5.5.34, 5.1.73, 5.3.13

Type: Bug Priority: Minor
Reporter: Elena Stepanova Assignee: Alexander Barkov
Resolution: Fixed Votes: 0
Labels: valgrind
Environment:

Ubuntu 12.04 x86_64


 Description   

I built the server with openSSL (WITH_SSL=system instead of bundled) and started getting valgrind warnings on server shutdown:

perl ./mtr alias --valgrind-mysqld
 

==29019== 24 bytes in 1 blocks are still reachable in loss record 1 of 600
 
==29019==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==29019==    by 0x5DCE7F3: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5E4928A: lh_insert (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5DD149B: OBJ_NAME_add (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5E53714: EVP_add_cipher (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5B54690: SSL_library_init (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
 
==29019==    by 0xEBE487: check_ssl_init (viosslfactories.c:155)
 
==29019==    by 0xEBE5AA: new_VioSSLFd (viosslfactories.c:189)
 
==29019==    by 0xEBEB66: new_VioSSLAcceptorFd (viosslfactories.c:336)
 
==29019==    by 0x5841CF: init_ssl() (mysqld.cc:4412)
 
==29019==    by 0x585888: mysqld_main(int, char**) (mysqld.cc:5224)
 
==29019==    by 0x57C4C3: main (main.cc:25)
 
==29019== 
==29019== 24 bytes in 1 blocks are still reachable in loss record 2 of 600
 
==29019==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==29019==    by 0x5DCE7F3: CRYPTO_malloc (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5E4928A: lh_insert (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5DD149B: OBJ_NAME_add (in /lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
 
==29019==    by 0x5B54690: SSL_library_init (in /lib/x86_64-linux-gnu/libssl.so.1.0.0)
 
==29019==    by 0xEBE487: check_ssl_init (viosslfactories.c:155)
 
==29019==    by 0xEBE5AA: new_VioSSLFd (viosslfactories.c:189)
 
==29019==    by 0xEBEB66: new_VioSSLAcceptorFd (viosslfactories.c:336)
 
==29019==    by 0x5841CF: init_ssl() (mysqld.cc:4412)
 
==29019==    by 0x585888: mysqld_main(int, char**) (mysqld.cc:5224)
 
==29019==    by 0x57C4C3: main (main.cc:25)
 
==29019== 
...

I'm not sure whether they're essential or not, and if they are, whether it's our fault, maybe it's a bad valgrind version or a 3rd-party library bug.

ldd ../sql/mysqld
 
	linux-vdso.so.1 =>  (0x00007fff7578d000)
 
	libodbc.so.1 => /usr/lib/x86_64-linux-gnu/libodbc.so.1 (0x00007f1731474000)
 
	libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007f1731272000)
 
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f173105a000)
 
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1730e52000)
 
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1730c35000)
 
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f17309fb000)
 
	libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f173079d000)
 
	libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f17303d5000)
 
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f17301d0000)
 
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f172fed0000)
 
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f172fbd4000)
 
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f172f9bd000)
 
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f172f5fe000)
 
	libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f172f3f4000)
 
	/lib64/ld-linux-x86-64.so.2 (0x00007f17316ed000)
 

valgrind --version
 
valgrind-3.7.0



 Comments   
Comment by Alexander Barkov [ 2013-08-13 ]

The same problem is observed with:

  • Fedora 17 (64bit)
  • openssl-1.0.0k-1.fc17.x86_64
  • valgrind-3.8.1
Comment by Alexander Barkov [ 2013-08-13 ]

The problem is repeatable with:

  • 5.5 bzr tree built with -DWITH_SSL=system
  • 5.3 bzr tree built with ./BUILD/compile-pentium-debug-openssl
  • 5.1 bzr tree built with ./BUILD/compile-pentium-debug-openssl
Comment by Alexander Barkov [ 2013-10-11 ]

Still reproducible with openssl-1.0.1e-4.fc17.x86_64.

Comment by Alexander Barkov [ 2013-10-16 ]

Fixed in 5.1, 5.2, 5.3, 5.5, 10.0-base.

Comment by Pavel Ivanov [ 2013-11-17 ]

The change in vio_end() made here http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/2502.565.54 causes mysqltest_embedded to hang on exit. So in 10.0.5 tarball testing of embedded server doesn't work at all when it's compiled with openssl.

I think the problem is that mysql_server_end() in case of compilation with embedded server calls both end_embedded_server() (which calls clean_up() which calls vio_end() ) and vio_end(). But apparently ERR_remove_state(0) can be called only once. On the second call it hangs inside openssl trying to lock the mutex.

Comment by Elena Stepanova [ 2013-11-17 ]

Hi Alexander,

Re-opening to draw your attention to the above. I think for the sake of changelog history and such, it will be better to open another bug report and close this one again, but I'll leave it to you to decide.

Comment by Alexander Barkov [ 2013-11-18 ]

Pavel, can you please post how-to-repeat instructions?

I can't repeat the hang on my FC18 64bit box.
I tried both debug and release builds:

cmake . -DCMAKE_INSTALL_PREFIX=/opt/mariadb-10.0 -DWITH_SSL=system -DWITH_EMBEDDED_SERVER=yes

and

cmake . -DCMAKE_INSTALL_PREFIX=/opt/mariadb-10.0 -DWITH_SSL=system -DWITH_EMBEDDED_SERVER=yes -DCMAKE_BUILD_TYPE=Debug

It does not hang.

Note, I can confirm that there is a problem in the code with vio_end() executed twice.
But it would be nice to repeat the hang anyway.
Thanks.

Comment by Pavel Ivanov [ 2013-11-18 ]

Hm... Sorry, it turned out that I was linking with a locally built openssl library that included some hacks which were actually causing the hang. I can't reproduce with Ubuntu's standard openssl library either.

Comment by Alexander Barkov [ 2013-11-19 ]

Thanks for checking! I opened a separate issue for this:
https://mariadb.atlassian.net/browse/MDEV-5311
The double call for vio_end() still needs to be fixed.

Generated at Thu Feb 08 06:59:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.