[MDEV-19199] main.bootstrap fails with ASAN unknown-crash in mysql_rm_tmp_tables when run with --parallel Created: 2019-04-06  Updated: 2019-05-08  Resolved: 2019-05-02

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.1, 10.2, 10.3, 10.4
Fix Version/s: 10.2.24, 5.5.65, 10.1.41, 10.3.15, 10.4.5

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Aleksey Midenkov
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-19349 mysql_install_db: segfault at tmp_fil... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/809

10.2 b4a7bde7

 
main.bootstrap                           w2 [ fail ]
 
        Test ended at 2019-04-05 21:09:06
 
 
 
CURRENT_TEST: main.bootstrap
 
mysqltest: At line 18: exec of '/home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld --no-defaults --disable-getopt-prefix-matching --bootstrap --basedir=/home/buildbot/buildbot/build/mariadb-10.2.24 --datadir=/dev/shm/var/install.db --plugin-dir=/dev/shm/var/plugins --default-storage-engine=myisam --loose-skip-plugin-aria --loose-skip-plugin-feedback --loose-skip-plugin-innodb --loose-skip-plugin-innodb-buffer-page --loose-skip-plugin-innodb-buffer-page-lru --loose-skip-plugin-innodb-buffer-pool-stats --loose-skip-plugin-innodb-cmp --loose-skip-plugin-innodb-cmp-per-index --loose-skip-plugin-innodb-cmp-per-index-reset --loose-skip-plugin-innodb-cmp-reset --loose-skip-plugin-innodb-cmpmem --loose-skip-plugin-innodb-cmpmem-reset --loose-skip-plugin-innodb-ft-being-deleted --loose-skip-plugin-innodb-ft-config --loose-skip-plugin-innodb-ft-default-stopword --loose-skip-plugin-innodb-ft-deleted --loose-skip-plugin-innodb-ft-index-cache --loose-skip-plugin-innodb-ft-index-table --loose-skip-plugin-innodb-lock-waits --loose-skip-plugin-innodb-locks --loose-skip-plugin-innodb-metrics --loose-skip-plugin-innodb-mutexes --loose-skip-plugin-innodb-sys-columns --loose-skip-plugin-innodb-sys-datafiles --loose-skip-plugin-innodb-sys-fields --loose-skip-plugin-innodb-sys-foreign --loose-skip-plugin-innodb-sys-foreign-cols --loose-skip-plugin-innodb-sys-indexes --loose-skip-plugin-innodb-sys-semaphore-waits --loose-skip-plugin-innodb-sys-tables --loose-skip-plugin-innodb-sys-tablespaces --loose-skip-plugin-innodb-sys-tablestats --loose-skip-plugin-innodb-sys-virtual --loose-skip-plugin-innodb-tablespaces-encryption --loose-skip-plugin-innodb-tablespaces-scrubbing --loose-skip-plugin-innodb-trx --loose-skip-plugin-partition --loose-skip-plugin-sequence --loose-skip-plugin-user-variables --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/dev/shm/var/tmp/ --core-file --console --lc-messages-dir=/home/buildbot/buildbot/build/mariadb-10.2.24/sql/share/ --character-sets-dir=/home/buildbot/buildbot/build/mariadb-10.2.24/sql/share/charsets --datadir=/dev/shm/var/2/mysqld.1/data/ --default-storage-engine=MyISAM --loose-skip-innodb < /dev/shm/var/2/tmp/bootstrap_test.sql >> /dev/shm/var/2/tmp/bootstrap.log 2>&1' failed, error: 34304, status: 134, errno: 2
 
Output from before failure:
 
drop table if exists t1;
 
 
 
 
 
 
 
The result from queries just before the failure was:
 
drop table if exists t1;

Not reproducible locally for me, but reproducible on buildbot VMs when the test is run with --parallel, even although it's the only test:

10.4 eb872ceb

 
perl ./mtr main.bootstrap --parallel=2
 
 
 
==841==ERROR: AddressSanitizer: unknown-crash on address 0x62d00001e48a at pc 0x7f8f2e4826ff bp 0x7ffd9aade800 sp 0x7ffd9aaddfb0
 
READ of size 4 at 0x62d00001e48a thread T0
 
    #0 0x7f8f2e4826fe  (/usr/lib/x86_64-linux-gnu/libasan.so.3+0x8d6fe)
 
    #1 0x55f2bb986661 in mysql_rm_tmp_tables() /home/buildbot/10.4/sql/sql_base.cc:8793
 
    #2 0x55f2bb834a70 in mysqld_main(int, char**) /home/buildbot/10.4/sql/mysqld.cc:5741
 
    #3 0x55f2bb81f7cf in main /home/buildbot/10.4/sql/main.cc:25
 
    #4 0x7f8f2c2522b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
 
    #5 0x55f2bb81f6a9 in _start (/home/buildbot/10.4/sql/mysqld+0xee86a9)
 
 
 
0x62d00001e48a is located 138 bytes inside of 32908-byte region [0x62d00001e400,0x62d00002648c)
 
allocated by thread T0 here:
 
    #0 0x7f8f2e4b6d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
 
    #1 0x55f2bd2f82f3 in sf_malloc /home/buildbot/10.4/mysys/safemalloc.c:118
 
    #2 0x55f2bd2ca47d in my_malloc /home/buildbot/10.4/mysys/my_malloc.c:101
 
    #3 0x55f2bd2a97a0 in init_alloc_root /home/buildbot/10.4/mysys/my_alloc.c:81
 
    #4 0x55f2bd2c8de6 in my_dir /home/buildbot/10.4/mysys/my_lib.c:135
 
    #5 0x55f2bb98656d in mysql_rm_tmp_tables() /home/buildbot/10.4/sql/sql_base.cc:8784
 
    #6 0x55f2bb834a70 in mysqld_main(int, char**) /home/buildbot/10.4/sql/mysqld.cc:5741



 Comments   
Comment by Marko Mäkelä [ 2019-04-09 ]

I see this on 10.2 built with clang-7, -O2, -DWITH_ASAN=ON -DWITH_SAFEMALLOC=OFF. I invoke the test as

ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto --force --retry=0 --max-test-fail=0

10.2 725579c0f47f424578520c7252680c53ed1583eb

 
CURRENT_TEST: main.bootstrap
 
mysqltest: At line 18: exec of '/dev/shm/10.2c/sql/mysqld … >> …/tmp/bootstrap.log 2>&1' failed, error: 34304, status: 134, errno: 2

The file var/log/main.bootstrap/tmp/bootstrap.log contains the following:

10.2 725579c0f47f424578520c7252680c53ed1583eb

 
2019-04-09 17:52:46 140025798089856 [Warning] setrlimit could not change the size of core files to 'infinity';  We may not be able to generate a core file on signals
 
2019-04-09 17:52:46 140025798089856 [Note] Plugin 'SEQUENCE' is disabled.
 
2019-04-09 17:52:46 140025798089856 [Note] Plugin 'partition' is disabled.
 
2019-04-09 17:52:46 140025798089856 [Note] Plugin 'InnoDB' is disabled.
 
2019-04-09 17:52:46 140025798089856 [Note] Plugin 'Aria' is disabled.
 
…
 
2019-04-09 17:52:46 140025798089856 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
 
=================================================================
 
==14037==ERROR: AddressSanitizer: unknown-crash on address 0x62d000014423 at pc 0x5614ac803956 bp 0x7ffef1001c70 sp 0x7ffef10013f8
 
READ of size 4 at 0x62d000014423 thread T0
 
    #0 0x5614ac803955 in memcmp (/dev/shm/10.2c/sql/mysqld+0xc6d955)
 
    #1 0x5614ac9fab2c in mysql_rm_tmp_tables() /mariadb/10.2/sql/sql_base.cc:8394:12
 
    #2 0x5614ac8d1091 in mysqld_main(int, char**) /mariadb/10.2/sql/mysqld.cc:5945:7
 
    #3 0x7f5a4c0ae09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
    #4 0x5614ac7e8df9 in _start (/dev/shm/10.2c/sql/mysqld+0xc52df9)
 
 
 
0x62d000014423 is located 35 bytes inside of 32800-byte region [0x62d000014400,0x62d00001c420)
 
allocated by thread T0 here:
 
    #0 0x5614ac890c43 in __interceptor_malloc (/dev/shm/10.2c/sql/mysqld+0xcfac43)
 
    #1 0x5614ae49cfc6 in my_malloc /mariadb/10.2/mysys/my_malloc.c:101:10
 
    #2 0x5614ae480593 in init_alloc_root /mariadb/10.2/mysys/my_alloc.c:78:22
 
    #3 0x5614ae49ba02 in my_dir /mariadb/10.2/mysys/my_lib.c:135:3
 
    #4 0x5614ac9faa98 in mysql_rm_tmp_tables() /mariadb/10.2/sql/sql_base.cc:8385:18
 
    #5 0x5614ac8d1091 in mysqld_main(int, char**) /mariadb/10.2/sql/mysqld.cc:5945:7
 
    #6 0x7f5a4c0ae09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
 
 
SUMMARY: AddressSanitizer: unknown-crash (/dev/shm/10.2c/sql/mysqld+0xc6d955) in memcmp
 
Shadow bytes around the buggy address:
 
  0x0c5a7fffa830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c5a7fffa840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c5a7fffa850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c5a7fffa860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c5a7fffa870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
=>0x0c5a7fffa880: 00 00 00 00[03]03 03 03 03 03 03 03 03 03 03 03
 
  0x0c5a7fffa890: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03
 
  0x0c5a7fffa8a0: 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03 03
 
  0x0c5a7fffa8b0: 03 03 02 03 02 02 03 03 02 02 03 02 02 02 02 03
 
  0x0c5a7fffa8c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
 
  0x0c5a7fffa8d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
  Shadow gap:              cc
 
==14037==ABORTING

If I run the test alone, then it will pass. So, I think that the problem may be related to cleaning up files from previous tests.

Comment by Elena Stepanova [ 2019-04-30 ]

On 10.1: http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/1086

Comment by Marko Mäkelä [ 2019-05-02 ]

Fixed by MDEV-19349.

Generated at Thu Feb 08 08:49:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.