Details
-
Bug
-
Status: Stalled (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.11, 11.0(EOL), 11.1(EOL), 11.2(EOL), 11.3(EOL)
Description
http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest-big/builds/886/steps/test/logs/stdio
innodb.innodb_mysql 'innodb' w1 [ fail ]
|
Test ended at 2017-05-08 12:29:36
|
|
|
CURRENT_TEST: innodb.innodb_mysql
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/cacert.pem'
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-cert.pem'
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-key.pem'
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
|
=================================================================
|
==1333==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100009ffc0 at pc 0x55c5e13d2f40 bp 0x7ffc3e27f5b0 sp 0x7ffc3e27f5a0
|
READ of size 8 at 0x61100009ffc0 thread T0
|
#0 0x55c5e13d2f3f in lf_pinbox_real_free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353
|
#1 0x55c5e13d360d in lf_pinbox_put_pins /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:237
|
#2 0x55c5e1338cb9 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1584
|
#3 0x55c5e1339680 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1636
|
#4 0x55c5e0edec24 in emb_free_embedded_thd /home/buildbot/buildbot/build/mariadb-10.2.6/libmysqld/lib_sql.cc:438
|
#5 0x55c5e0ec469f in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.6/sql-common/client.c:3968
|
#6 0x55c5e0e5fcfa in do_close_connection(st_command*) /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:5623
|
#7 0x55c5e0e17485 in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9245
|
#8 0x7fbfd904282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
|
#9 0x55c5e0e44da8 in _start (/mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded+0x697da8)
|
|
|
0x61100009ffc0 is located 192 bytes inside of 208-byte region [0x61100009ff00,0x61100009ffd0)
|
freed by thread T26 here:
|
#0 0x7fbfdb0a12ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
|
#1 0x55c5e0e8abff in my_thread_end /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:389
|
#2 0x55c5e0e44fca in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:908
|
#3 0x7fbfdadf36b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
|
|
|
previously allocated by thread T26 here:
|
#0 0x7fbfdb0a179a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
|
#1 0x55c5e0e89f9f in my_thread_init /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:295
|
#2 0x55c5e0e44f2d in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:865
|
#3 0x7fbfdadf36b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
|
|
|
Thread T26 created by T0 here:
|
#0 0x7fbfdb03f253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
|
#1 0x55c5e0e61608 in init_connection_thd /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1027
|
#2 0x55c5e0e61608 in do_connect(st_command*) /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:6023
|
#3 0x55c5e0e1751f in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9240
|
#4 0x7fbfd904282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
|
|
|
SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353 lf_pinbox_real_free
|
Shadow bytes around the buggy address:
|
0x0c228000bfa0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c228000bfb0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
|
0x0c228000bfc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
|
0x0c228000bfd0: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c228000bfe0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
|
=>0x0c228000bff0: fd fd fd fd fd fd fd fd[fd]fd fa fa fa fa fa fa
|
0x0c228000c000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c228000c010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c228000c020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c228000c030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c228000c040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
Shadow byte legend (one shadow byte represents 8 application bytes):
|
Addressable: 00
|
Partially addressable: 01 02 03 04 05 06 07
|
Heap left redzone: fa
|
Heap right redzone: fb
|
Freed heap region: fd
|
Stack left redzone: f1
|
Stack mid redzone: f2
|
Stack right redzone: f3
|
Stack partial redzone: f4
|
Stack after return: f5
|
Stack use after scope: f8
|
Global redzone: f9
|
Global init order: f6
|
Poisoned by user: f7
|
Container overflow: fc
|
Array cookie: ac
|
Intra object redzone: bb
|
ASan internal: fe
|
==1333==ABORTING
|
Too many tests(10) failed! Terminating...
|
|
|
Failing test(s): innodb.innodb_mysql innodb.innodb_stats_del_mark innodb_zip.large_blob innodb_zip.16k innodb.sp_temp_table parts.part_supported_sql_func_innodb parts.partition_alter2_1_2_innodb parts.partition_alter2_2_2_innodb parts.partition_decimal_innodb
|
More:
main.sum_distinct-big 'innodb' w2 [ fail ]
|
Test ended at 2017-05-08 12:42:38
|
|
|
CURRENT_TEST: main.sum_distinct-big
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/cacert.pem'
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-cert.pem'
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.2.6/mysql-test/std_data/client-key.pem'
|
Warning: /mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
|
=================================================================
|
==1686==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100008bfc0 at pc 0x55860c19ff40 bp 0x7ffc4ca3fbe0 sp 0x7ffc4ca3fbd0
|
READ of size 8 at 0x61100008bfc0 thread T0
|
#0 0x55860c19ff3f in lf_pinbox_real_free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353
|
#1 0x55860c1a060d in lf_pinbox_put_pins /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:237
|
#2 0x55860c105cb9 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1584
|
#3 0x55860c106680 in THD::~THD() /home/buildbot/buildbot/build/mariadb-10.2.6/sql/sql_class.cc:1636
|
#4 0x55860bcabc24 in emb_free_embedded_thd /home/buildbot/buildbot/build/mariadb-10.2.6/libmysqld/lib_sql.cc:438
|
#5 0x55860bc9169f in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.6/sql-common/client.c:3968
|
#6 0x55860bc12c0b in close_connections() /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1403
|
#7 0x55860bc154bc in free_used_memory() /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1451
|
#8 0x55860bc1575f in cleanup_and_exit /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1491
|
#9 0x55860bbe3085 in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9681
|
#10 0x7f097547582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
|
#11 0x55860bc11da8 in _start (/mnt/buildbot/build/mariadb-10.2.6/libmysqld/examples/mysqltest_embedded+0x697da8)
|
|
|
0x61100008bfc0 is located 192 bytes inside of 208-byte region [0x61100008bf00,0x61100008bfd0)
|
freed by thread T24 here:
|
#0 0x7f09774d42ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
|
#1 0x55860bc57bff in my_thread_end /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:389
|
#2 0x55860bc11fca in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:908
|
#3 0x7f09772266b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
|
|
|
previously allocated by thread T24 here:
|
#0 0x7f09774d479a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
|
#1 0x55860bc56f9f in my_thread_init /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/my_thr_init.c:295
|
#2 0x55860bc11f2d in connection_thread /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:865
|
#3 0x7f09772266b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
|
|
|
Thread T24 created by T0 here:
|
#0 0x7f0977472253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
|
#1 0x55860bbe2c67 in init_connection_thd /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:1027
|
#2 0x55860bbe2c67 in main /home/buildbot/buildbot/build/mariadb-10.2.6/client/mysqltest.cc:9116
|
#3 0x7f097547582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
|
|
|
SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.6/mysys/lf_alloc-pin.c:353 lf_pinbox_real_free
|
Shadow bytes around the buggy address:
|
0x0c22800097a0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
|
0x0c22800097b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c22800097c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c22800097d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c22800097e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
|
=>0x0c22800097f0: fd fd fd fd fd fd fd fd[fd]fd fa fa fa fa fa fa
|
0x0c2280009800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c2280009810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c2280009820: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c2280009830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
0x0c2280009840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
|
Shadow byte legend (one shadow byte represents 8 application bytes):
|
Addressable: 00
|
Partially addressable: 01 02 03 04 05 06 07
|
Heap left redzone: fa
|
Heap right redzone: fb
|
Freed heap region: fd
|
Stack left redzone: f1
|
Stack mid redzone: f2
|
Stack right redzone: f3
|
Stack partial redzone: f4
|
Stack after return: f5
|
Stack use after scope: f8
|
Global redzone: f9
|
Global init order: f6
|
Poisoned by user: f7
|
Container overflow: fc
|
Array cookie: ac
|
Intra object redzone: bb
|
ASan internal: fe
|
==1686==ABORTING
|
Attachments
Issue Links
- is part of
-
MDEV-33073 always green buildbot
-
- Stalled
-
- relates to
-
-
- Open
-
Activity
This is still causing massive failures on any embedded server tests on buildbot.mariadb.org.
I am observing 100% CPU load in ASAN builds when this bug triggers. It does not always trigger. If I invoke
./mtr --rr=-h --embedded main.ctype_nopad_8bit
|
the process would sometimes appear to hang (but consume CPU cycles).
In one set of stack traces I observed something interesting:
|
10.6 53c6c823dc7cafefffdc93c79661cfb146ff8641 |
Thread 10 (Thread 0x7f345ef416c0 (LWP 186512) "mysqltest_embed"):
|
#0 0x00007f346d448f26 in __asan::FakeStack::Allocate (this=0x7f345d938000, stack_size_log=20, class_id=4, real_stack=139862908062448) at ../../../../src/libsanitizer/asan/asan_fake_stack.cpp:105
|
#1 __asan::OnMalloc (class_id=4, size=<optimized out>) at ../../../../src/libsanitizer/asan/asan_fake_stack.cpp:229
|
#2 __asan_stack_malloc_4 (size=<optimized out>) at ../../../../src/libsanitizer/asan/asan_fake_stack.cpp:277
|
#3 0x0000556f369b089b in dfield_check_typed (field=0x516000301670) at /mariadb/10.6/storage/innobase/data/data0data.cc:194, even though I used
|
According to perf top there is quite a bit of context switching going on:
23,32% [kernel] [k] syscall_exit_to_user_mode
|
8,34% [kernel] [k] entry_SYSCALL_64
|
7,99% [kernel] [k] syscall_return_via_sysret
|
3,52% [kernel] [k] update_curr
|
2,73% [kernel] [k] __schedule
|
2,56% [kernel] [k] __calc_delta.constprop.0
|
2,49% [kernel] [k] entry_SYSCALL_64_after_hwframe
|
2,38% [kernel] [k] pick_next_task_fair
|
2,15% [kernel] [k] __pick_eevdf
|
1,95% [kernel] [k] update_min_vruntime
|
1,71% [kernel] [k] entity_eligible
|
1,67% libasan.so.8.0.0 [.] __sanitizer::internal_sched_yield()
|
1,57% [kernel] [k] rcu_note_context_switch
|
I could not get any reports from AddressSanitizer this time. What I see that running
./mtr --embedded --parallel=5 --suite=main
|
will eventually stop to make progress and increase the loadavg to about 5. I repeated the same also with 60 concurrent workers.
So above change appears to have fixed 10.6 where it has been merged so far while 10.5 and 10.11+ have remained failing.
cr confirms (same for main.ctype_nopad_8bit test)
innodb.innodb_mysql hits are from 10.6 brances before the above commit.
innodb_zip.large_blob - recent - but different stack trace
Haven't occurred for a long time:
innodb_zip.innodb_prefix_index_liftedlimit innodb.innodb_stats_del_mark parts.partition_decimal_innodb
parts.* (or are different stack traces)
Can a 10.5 backport of above commit happen?
MDEV-26463 has been filed for a memory leak in the embedded server test driver.