[MDEV-12749] multiple tests fail in buildbot with ASAN and embedded Created: 2017-05-09  Updated: 2024-01-31

Status: Stalled
Project: MariaDB Server
Component/s: Embedded Server, Tests
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2, 11.3
Fix Version/s: 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2, 11.3

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Unresolved Votes: 0
Labels: ASAN, leak, race

Issue Links:
Relates
relates to MDEV-26463 Memory leak in embedded server bootstrap Open

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest-big/builds/886/steps/test/logs/stdio

innodb.innodb_mysql 'innodb'             w1 [ fail ]
 
        Test ended at 2017-05-08 12:29:36
 
 
 
CURRENT_TEST: innodb.innodb_mysql
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/cacert.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-cert.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-key.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
 
=================================================================
 
==1333==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100009ffc0 at pc 0x55c5e13d2f40 bp 0x7ffc3e27f5b0 sp 0x7ffc3e27f5a0
 
READ of size 8 at 0x61100009ffc0 thread T0
 
    #0 0x55c5e13d2f3f in lf_pinbox_real_free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353
 
    #1 0x55c5e13d360d in lf_pinbox_put_pins /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:237
 
    #2 0x55c5e1338cb9 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1584
 
    #3 0x55c5e1339680 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1636
 
    #4 0x55c5e0edec24 in emb_free_embedded_thd /home/buildbot/buildbot/build/mariadb-10.2.6/libmysqld/lib_sql.cc:438
 
    #5 0x55c5e0ec469f in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.6/sql-common/client.c:3968
 
    #6 0x55c5e0e5fcfa in do_close_connection(st_command*) /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:5623
 
    #7 0x55c5e0e17485 in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9245
 
    #8 0x7fbfd904282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
    #9 0x55c5e0e44da8 in _start (/mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded+0x697da8)
 
 
 
0x61100009ffc0 is located 192 bytes inside of 208-byte region [0x61100009ff00,0x61100009ffd0)
 
freed by thread T26 here:
 
    #0 0x7fbfdb0a12ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
 
    #1 0x55c5e0e8abff in my_thread_end /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:389
 
    #2 0x55c5e0e44fca in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:908
 
    #3 0x7fbfdadf36b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
 
 
previously allocated by thread T26 here:
 
    #0 0x7fbfdb0a179a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
 
    #1 0x55c5e0e89f9f in my_thread_init /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:295
 
    #2 0x55c5e0e44f2d in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:865
 
    #3 0x7fbfdadf36b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
 
 
Thread T26 created by T0 here:
 
    #0 0x7fbfdb03f253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
 
    #1 0x55c5e0e61608 in init_connection_thd /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1027
 
    #2 0x55c5e0e61608 in do_connect(st_command*) /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:6023
 
    #3 0x55c5e0e1751f in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9240
 
    #4 0x7fbfd904282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353 lf_pinbox_real_free
 
Shadow bytes around the buggy address:
 
  0x0c228000bfa0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000bfb0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
 
  0x0c228000bfc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c228000bfd0: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000bfe0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
=>0x0c228000bff0: fd fd fd fd fd fd fd fd[fd]fd fa fa fa fa fa fa
 
  0x0c228000c000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000c010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000c020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000c030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000c040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Heap right redzone:      fb
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack partial redzone:   f4
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
==1333==ABORTING


Too many tests(10) failed! Terminating...
 
 
 
Failing test(s): innodb.innodb_mysql innodb.innodb_stats_del_mark innodb_zip.large_blob innodb_zip.16k innodb.sp_temp_table parts.part_supported_sql_func_innodb parts.partition_alter2_1_2_innodb parts.partition_alter2_2_2_innodb parts.partition_decimal_innodb

More:

main.sum_distinct-big 'innodb'           w2 [ fail ]
 
        Test ended at 2017-05-08 12:42:38
 
 
 
CURRENT_TEST: main.sum_distinct-big
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/cacert.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-cert.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-key.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
 
=================================================================
 
==1686==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100008bfc0 at pc 0x55860c19ff40 bp 0x7ffc4ca3fbe0 sp 0x7ffc4ca3fbd0
 
READ of size 8 at 0x61100008bfc0 thread T0
 
    #0 0x55860c19ff3f in lf_pinbox_real_free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353
 
    #1 0x55860c1a060d in lf_pinbox_put_pins /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:237
 
    #2 0x55860c105cb9 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1584
 
    #3 0x55860c106680 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1636
 
    #4 0x55860bcabc24 in emb_free_embedded_thd /home/buildbot/buildbot/build/mariadb-10.2.6/libmysqld/lib_sql.cc:438
 
    #5 0x55860bc9169f in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.6/sql-common/client.c:3968
 
    #6 0x55860bc12c0b in close_connections() /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1403
 
    #7 0x55860bc154bc in free_used_memory() /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1451
 
    #8 0x55860bc1575f in cleanup_and_exit /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1491
 
    #9 0x55860bbe3085 in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9681
 
    #10 0x7f097547582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
    #11 0x55860bc11da8 in _start (/mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded+0x697da8)
 
 
 
0x61100008bfc0 is located 192 bytes inside of 208-byte region [0x61100008bf00,0x61100008bfd0)
 
freed by thread T24 here:
 
    #0 0x7f09774d42ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
 
    #1 0x55860bc57bff in my_thread_end /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:389
 
    #2 0x55860bc11fca in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:908
 
    #3 0x7f09772266b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
 
 
previously allocated by thread T24 here:
 
    #0 0x7f09774d479a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
 
    #1 0x55860bc56f9f in my_thread_init /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:295
 
    #2 0x55860bc11f2d in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:865
 
    #3 0x7f09772266b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
 
 
Thread T24 created by T0 here:
 
    #0 0x7f0977472253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
 
    #1 0x55860bbe2c67 in init_connection_thd /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1027
 
    #2 0x55860bbe2c67 in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9116
 
    #3 0x7f097547582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353 lf_pinbox_real_free
 
Shadow bytes around the buggy address:
 
  0x0c22800097a0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
 
  0x0c22800097b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c22800097c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c22800097d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c22800097e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
=>0x0c22800097f0: fd fd fd fd fd fd fd fd[fd]fd fa fa fa fa fa fa
 
  0x0c2280009800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2280009810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2280009820: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2280009830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2280009840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Heap right redzone:      fb
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack partial redzone:   f4
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
==1686==ABORTING



 Comments   
Comment by Alice Sherepa [ 2017-08-21 ]

recent one http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest-big/builds/1158/steps/test/logs/stdio

Comment by Elena Stepanova [ 2017-09-22 ] 

innodb_zip.innodb_prefix_index_liftedlimit 'innodb_plugin' w1 [ fail ]
 
        Test ended at 2017-09-20 00:33:41
 
 
 
CURRENT_TEST: innodb_zip.innodb_prefix_index_liftedlimit
 
Warning: /mnt/buildbot/build/mariadb-10.1.27/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.1.27/mysql-test/std_data/cacert.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.1.27/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.1.27/mysql-test/std_data/client-cert.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.1.27/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.1.27/mysql-test/std_data/client-key.pem'
 
Warning: /mnt/buildbot/build/mariadb-10.1.27/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
 
=================================================================
 
==1512==ERROR: AddressSanitizer: heap-use-after-free on address 0x6110000bdfb8 at pc 0x56064db316e6 bp 0x7ffdfeb10940 sp 0x7ffdfeb10930
 
READ of size 8 at 0x6110000bdfb8 thread T0
 
    #0 0x56064db316e5 in lf_pinbox_real_free /home/buildbot/buildbot/build/mariadb-10.1.27/mysys/lf_alloc-pin.c:348
 
    #1 0x56064db320d0 in lf_pinbox_put_pins /home/buildbot/buildbot/build/mariadb-10.1.27/mysys/lf_alloc-pin.c:238
 
    #2 0x56064d95a5c0 in MDL_context::destroy() /home/buildbot/buildbot/build/mariadb-10.1.27/sql/mdl.cc:871
 
    #3 0x56064dacba78 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.1.27/sql/sql_class.cc:1662
 
    #4 0x56064dacc18c in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.1.27/sql/sql_class.cc:1714
 
    #5 0x56064d714a61 in emb_free_embedded_thd /home/buildbot/buildbot/build/mariadb-10.1.27/libmysqld/lib_sql.cc:437
 
    #6 0x56064d6f8770 in mysql_close /home/buildbot/buildbot/build/mariadb-10.1.27/sql-common/client.c:3969
 
    #7 0x56064d6a1a9f in do_close_connection(st_command*) /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:5670
 
    #8 0x56064d6a69c1 in main /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:9295
 
    #9 0x7fe8b367682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
    #10 0x56064d6887a8 in _start (/mnt/buildbot/build/mariadb-10.1.27/libmysqld/examples/mysqltest_embedded+0x56b7a8)
 
 
 
0x6110000bdfb8 is located 184 bytes inside of 200-byte region [0x6110000bdf00,0x6110000bdfc8)
 
freed by thread T20 here:
 
    #0 0x7fe8b56d52ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
 
    #1 0x56064d6c6a03 in my_thread_end /home/buildbot/buildbot/build/mariadb-10.1.27/mysys/my_thr_init.c:389
 
    #2 0x56064d6e52f2 in mysql_thread_end /home/buildbot/buildbot/build/mariadb-10.1.27/libmysql/libmysql.c:253
 
    #3 0x56064d688ec8 in connection_thread /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:906
 
    #4 0x7fe8b54276b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
 
 
previously allocated by thread T20 here:
 
    #0 0x7fe8b56d579a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
 
    #1 0x56064d6c61a1 in my_thread_init /home/buildbot/buildbot/build/mariadb-10.1.27/mysys/my_thr_init.c:295
 
    #2 0x56064d6e52e7 in mysql_thread_init /home/buildbot/buildbot/build/mariadb-10.1.27/libmysql/libmysql.c:248
 
    #3 0x56064d688b16 in connection_thread /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:863
 
    #4 0x7fe8b54276b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
 
 
Thread T20 created by T0 here:
 
    #0 0x7fe8b5673253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
 
    #1 0x56064d68fb3c in init_connection_thd /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:1025
 
    #2 0x56064d6a2b47 in do_connect(st_command*) /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:6070
 
    #3 0x56064d6a69a1 in main /home/buildbot/buildbot/build/mariadb-10.1.27/client/mysqltest.cc:9290
 
    #4 0x7fe8b367682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.1.27/mysys/lf_alloc-pin.c:348 lf_pinbox_real_free
 
Shadow bytes around the buggy address:
 
  0x0c228000fba0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
 
  0x0c228000fbb0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
 
  0x0c228000fbc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
  0x0c228000fbd0: 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000fbe0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
=>0x0c228000fbf0: fd fd fd fd fd fd fd[fd]fd fa fa fa fa fa fa fa
 
  0x0c228000fc00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000fc10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000fc20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000fc30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c228000fc40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Heap right redzone:      fb
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack partial redzone:   f4
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
==1512==ABORTING

Comment by Sergey Vojtovich [ 2017-10-25 ]

The problem here is that MYSQL and specifically MYSQL::thd are being shared between 2 threads:

  • mysqltest thread: does mysql_init()/mysql_close()/etc
  • mysqltest connection_thread: does mysql_send_query()/etc

This looks fragile as such, but may be alright if we properly guard concurrent access to these structures.

What happens here is we use LF_PINS::stack_ends_here pointing to THR_KEY_mysys of connection_thread, which was freed by the time of mysql_close() call in mysqltest thread.

The fix is probably to move at least mysql_close() to connection_thread. Ideally we should move all references to MYSQL into this thread.

Comment by Marko Mäkelä [ 2023-12-19 ]

This leads to test case timeouts on mandatory builders and makes ./mtr --embedded totally unusable in cmake -DWITH_ASAN=ON builds. Here is an example from a non-ASAN failure of a mandatory builder:

10.5 2c60d43d7df134c3875b3760a5d54ac72da01f5f plus an unrelated change

 
main.ctype_nopad_8bit                    w9 [ fail ]  timeout after 900 seconds
 
…
 
#2  0x000055cd18ffd31c in signal_handler (sig=6) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/client/mysqltest.cc:9726
 
#3  <signal handler called>
 
#4  0x00007ff7bd20996f in poll () from /lib/x86_64-linux-gnu/libc.so.6
 
#5  0x000055cd1962f657 in addr_resolve (ptr=0x7411da, loc=0x7ffc7befdec0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/my_addr_resolve.c:245
 
#6  0x000055cd1962f882 in my_addr_resolve (ptr=0x55cd18fec1da <_start+42>, loc=0x7ffc7befdec0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/my_addr_resolve.c:337
 
#7  0x000055cd1907c4aa in print_stack (frame=0x55cd1c89e7d8) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/safemalloc.c:240
 
#8  0x000055cd1907cae3 in sf_report_leaked_memory (id=0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/safemalloc.c:389
 
#9  0x000055cd1907cb5d in sf_terminate () at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/safemalloc.c:403
 
#10 0x00007ff7bd1554d7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
#11 0x00007ff7bd15567a in exit () from /lib/x86_64-linux-gnu/libc.so.6
 
#12 0x000055cd18fed8a8 in cleanup_and_exit (exit_code=0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/client/mysqltest.cc:1545
 
#13 0x000055cd18ffef17 in main (argc=18, argv=0x7ffc7befe328) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/client/mysqltest.cc:10532

and another from the same run:

10.5 2c60d43d7df134c3875b3760a5d54ac72da01f5f plus an unrelated change

 
main.precedence                          w15 [ fail ]  timeout after 900 seconds
 
…
 
#2  0x0000558e5329931c in signal_handler (sig=6) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/client/mysqltest.cc:9726
 
#3  <signal handler called>
 
#4  0x00007f30a026296f in poll () from /lib/x86_64-linux-gnu/libc.so.6
 
#5  0x0000558e538cb657 in addr_resolve (ptr=0x7411da, loc=0x7ffcabe872a0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/my_addr_resolve.c:245
 
#6  0x0000558e538cb882 in my_addr_resolve (ptr=0x558e532881da <_start+42>, loc=0x7ffcabe872a0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/my_addr_resolve.c:337
 
#7  0x0000558e533184aa in print_stack (frame=0x558e55cf2788) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/safemalloc.c:240
 
#8  0x0000558e53318ae3 in sf_report_leaked_memory (id=0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/safemalloc.c:389
 
#9  0x0000558e53318b5d in sf_terminate () at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/mysys/safemalloc.c:403
 
#10 0x00007f30a01ae4d7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
#11 0x00007f30a01ae67a in exit () from /lib/x86_64-linux-gnu/libc.so.6
 
#12 0x0000558e532898a8 in cleanup_and_exit (exit_code=0) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/client/mysqltest.cc:1545
 
#13 0x0000558e5329af17 in main (argc=18, argv=0x7ffcabe87708) at /home/buildbot/amd64-debian-11-debug-ps-embedded/build/client/mysqltest.cc:10532

Comment by Marko Mäkelä [ 2023-12-19 ]

MDEV-26463 has been filed for a memory leak in the embedded server test driver.

Comment by Marko Mäkelä [ 2024-01-31 ]

This is still causing massive failures on any embedded server tests on buildbot.mariadb.org.

Generated at Thu Feb 08 08:00:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.