[MDEV-18656] innodb.innodb_bulk_create_index fails in buildbot with ASAN unknown-crash in trx_undo_rec_get_pars Created: 2019-02-20  Updated: 2019-03-27  Resolved: 2019-03-27

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Tests
Affects Version/s: 10.3, 10.4
Fix Version/s: 10.2.24, 10.1.39, 10.3.14, 10.4.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-18417 innodb.innodb_simulate_comp_failures ... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest-big/builds/2459

innodb.innodb_bulk_create_index '16k,innodb' w1 [ fail ]
        Test ended at 2019-02-18 08:22:12
 
CURRENT_TEST: innodb.innodb_bulk_create_index
mysqltest: In included file "./suite/innodb/include/innodb_bulk_create_index.inc": 
included from /mnt/buildbot/build/mariadb-10.3.13/mysql-test/suite/innodb/t/innodb_bulk_create_index.test at line 23:
At line 118: query '/* Drop column. */
ALTER TABLE t1 DROP COLUMN content' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
class	id	title	content
10	10	a10	NULL
SELECT * FROM t1 WHERE id = 5000;
class	id	title	content
5000	5000	a5000	NULL
SELECT * FROM t1 WHERE title = 'a5000';
class	id	title	content
5000	5000	a5000	NULL
SELECT * FROM t1 WHERE id = 10000;
class	id	title	content
10000	10000	a10000	NULL
SELECT * FROM t1 WHERE title = 'a10000';
class	id	title	content
10000	10000	a10000	NULL
SELECT * FROM t1 WHERE id = 10010;
class	id	title	content
SELECT * FROM t1 WHERE title = 'a10010';
class	id	title	content
/* Drop column. */
ALTER TABLE t1 DROP COLUMN content;
 
More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.3.13/mysql-test/var/1/log/innodb_bulk_create_index.log
 
 
Server [mysqld.1 - pid: 13031, winpid: 13031, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
...
=================================================================
==13032==ERROR: AddressSanitizer: unknown-crash on address 0x61300000bf84 at pc 0x55a9e14bda2f bp 0x7f6d2a03b370 sp 0x7f6d2a03b360
READ of size 4 at 0x61300000bf84 thread T19
    #0 0x55a9e14bda2e in trx_undo_rec_get_pars(unsigned char*, unsigned long*, unsigned long*, bool*, unsigned long*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0rec.cc:577
    #1 0x55a9e13f6557 in row_purge_parse_undo_rec /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:1046
    #2 0x55a9e13f6557 in row_purge /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:1254
    #3 0x55a9e13f6557 in row_purge_step(que_thr_t*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:1343
    #4 0x55a9e134ad9a in que_thr_step /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/que/que0que.cc:1042
    #5 0x55a9e134ad9a in que_run_threads_low /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/que/que0que.cc:1104
    #6 0x55a9e134ad9a in que_run_threads(que_thr_t*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/que/que0que.cc:1144
    #7 0x55a9e1464998 in srv_task_execute /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0srv.cc:2449
    #8 0x55a9e1464998 in srv_worker_thread /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0srv.cc:2497
    #9 0x7f6d3cf306b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #10 0x7f6d3c5db82c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)
 
0x61300000bf84 is located 132 bytes inside of 368-byte region [0x61300000bf00,0x61300000c070)
allocated by thread T0 here:
    #0 0x7f6d3e2ec602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x55a9e12c03de in mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/mem/mem0mem.cc:269
    #2 0x55a9e13ef7a5 in mem_heap_create_func /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/include/mem0mem.ic:484
    #3 0x55a9e13ef7a5 in row_purge_node_create(que_thr_t*, mem_block_info_t*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:80
    #4 0x55a9e14a34e6 in purge_graph_build /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0purge.cc:151
    #5 0x55a9e14a34e6 in purge_sys_t::create() /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0purge.cc:166
    #6 0x55a9e1503a9b in trx_lists_init_at_db_start() /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0trx.cc:705
    #7 0x55a9e147494a in srv_start(bool) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0start.cc:1973
    #8 0x55a9e11ac571 in innodb_init /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/handler/ha_innodb.cc:4266
    #9 0x55a9e0acbd3c in ha_initialize_handlerton(st_plugin_int*) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/handler.cc:523
    #10 0x55a9e05a1be5 in plugin_initialize /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1432
    #11 0x55a9e05a310c in plugin_init(int*, char**, int) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1714
    #12 0x55a9e035912c in init_server_components /home/buildbot/buildbot/build/mariadb-10.3.13/sql/mysqld.cc:5385
    #13 0x55a9e0364919 in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/mysqld.cc:5998
    #14 0x7f6d3c4f582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
Thread T19 created by T0 here:
    #0 0x7f6d3e28a253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x55a9e12ef2b7 in os_thread_create_func(void* (*)(void*), void*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/os/os0thread.cc:132
    #2 0x55a9e1473de3 in srv_start(bool) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0start.cc:2418
    #3 0x55a9e11ac571 in innodb_init /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/handler/ha_innodb.cc:4266
    #4 0x55a9e0acbd3c in ha_initialize_handlerton(st_plugin_int*) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/handler.cc:523
    #5 0x55a9e05a1be5 in plugin_initialize /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1432
    #6 0x55a9e05a310c in plugin_init(int*, char**, int) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1714
    #7 0x55a9e035912c in init_server_components /home/buildbot/buildbot/build/mariadb-10.3.13/sql/mysqld.cc:5385
    #8 0x55a9e0364919 in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/mysqld.cc:5998
    #9 0x7f6d3c4f582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
 
SUMMARY: AddressSanitizer: unknown-crash /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0rec.cc:577 trx_undo_rec_get_pars(unsigned char*, unsigned long*, unsigned long*, bool*, unsigned long*, unsigned long*)
Shadow bytes around the buggy address:
  0x0c267fff97a0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c267fff97b0: 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff97c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff97d0: f7 f7 f7 f7 f7 f7 fa fa fa fa fa fa fa fa fa fa
  0x0c267fff97e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c267fff97f0:[07]00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7
  0x0c267fff9800: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 fa fa
  0x0c267fff9810: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c267fff9820: 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff9830: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff9840: f7 f7 f7 f7 f7 f7 fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==13032==ABORTING
----------SERVER LOG END-------------



 Comments   
Comment by Elena Stepanova [ 2019-03-26 ]

From March 26th, getting the same failure massively in my tests on Xenial, server built with gcc 5.4.0:

10.4 8b480df6 ASAN

==7920==ERROR: AddressSanitizer: unknown-crash on address 0x61300000cf44 at pc 0x55a712812cbe bp 0x7f99a7a63610 sp 0x7f99a7a63600
READ of size 4 at 0x61300000cf44 thread T24
    #0 0x55a712812cbd in trx_undo_rec_get_pars(unsigned char*, unsigned long*, unsigned long*, bool*, unsigned long*, unsigned long*) /home/vsts/src/storage/innobase/trx/trx0rec.cc:580
    #1 0x55a71276c062 in row_purge_parse_undo_rec /home/vsts/src/storage/innobase/row/row0purge.cc:1025
    #2 0x55a71276c062 in row_purge /home/vsts/src/storage/innobase/row/row0purge.cc:1251
    #3 0x55a71276c062 in row_purge_step(que_thr_t*) /home/vsts/src/storage/innobase/row/row0purge.cc:1315
    #4 0x55a7126ccab8 in que_thr_step /home/vsts/src/storage/innobase/que/que0que.cc:1042
    #5 0x55a7126ccab8 in que_run_threads_low /home/vsts/src/storage/innobase/que/que0que.cc:1104
    #6 0x55a7126ccab8 in que_run_threads(que_thr_t*) /home/vsts/src/storage/innobase/que/que0que.cc:1144
    #7 0x55a7127c67f1 in srv_task_execute /home/vsts/src/storage/innobase/srv/srv0srv.cc:2437
    #8 0x55a7127c67f1 in srv_worker_thread /home/vsts/src/storage/innobase/srv/srv0srv.cc:2485
    #9 0x7f99d82246b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #10 0x7f99d74a941c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
0x61300000cf44 is located 132 bytes inside of 368-byte region [0x61300000cec0,0x61300000d030)
allocated by thread T0 here:
    #0 0x7f99d98c5602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x55a7126561a8 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long) /home/vsts/src/storage/innobase/mem/mem0mem.cc:269
Thread T24 created by T0 here:
    #0 0x7f99d9863253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x55a71267b636 in os_thread_create_func(void* (*)(void*), void*, unsigned long*) /home/vsts/src/storage/innobase/os/os0thread.cc:132
SUMMARY: AddressSanitizer: unknown-crash /home/vsts/src/storage/innobase/trx/trx0rec.cc:580 trx_undo_rec_get_pars(unsigned char*, unsigned long*, unsigned long*, bool*, unsigned long*, unsigned long*)
Shadow bytes around the buggy address:
  0x0c267fff9990: 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c267fff99a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f7 f7
  0x0c267fff99b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff99c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 fa fa
  0x0c267fff99d0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x0c267fff99e0: 00 00 00 00 00 00 00 00[07]00 00 00 00 00 00 00
  0x0c267fff99f0: 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff9a00: f7 f7 f7 f7 f7 f7 fa fa fa fa fa fa fa fa fa fa
  0x0c267fff9a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f7 f7
  0x0c267fff9a20: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c267fff9a30: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==7920==ABORTING

Not reproducible locally, but the MTR test from the description still fails readily on kvm-xenial-build, fulltest-big-like 10.3 build.

Comment by Marko Mäkelä [ 2019-03-27 ]

It seems to me that GCC 5.4 on Ubuntu Xenial is emitting invalid code WITH_ASAN for -O2 or -O3. I fixed this by appending -O1 to the compilation flags of trx0rec.cc if the compiler is GCC and older than 6.0.0.

Generated at Thu Feb 08 08:45:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.