[MDEV-14800] main.mysqldump failed in buildbot - stack smashing detected in mysqlimport Created: 2017-12-29  Updated: 2020-06-10  Resolved: 2020-06-10

Status: Closed
Project: MariaDB Server
Component/s: Scripts & Clients, Tests
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Elena Stepanova Assignee: Oleksandr Byelkin
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

P8

Attachments: File main_mysqldump.log    
Issue Links:
Blocks

 Description   

http://buildbot.askmonty.org/buildbot/builders/p8-rhel6-bintar-debug/builds/1497/steps/test/logs/stdio

main.mysqldump 'innodb'                  w4 [ fail ]
 
        Test ended at 2017-11-28 14:00:40
 
 
 
CURRENT_TEST: main.mysqldump
 
mysqldump: Couldn't find table: "non_existing"
 
mysqldump: Got error: 1356: "View 'test.v1' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES
 
mysqldump: Couldn't execute 'SHOW FIELDS FROM `v1`': View 'test.v1' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them (1356)
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: user2 has insufficent privileges to SHOW CREATE PROCEDURE `sp1`!
 
mysqldump: Got error: 1146: "Table 'test.???????????????????????' doesn't exist" when using LOCK TABLES
 
/home/buildbot/maria-slave/p8-rhel6-bintar-debug/build/client/mysqlimport: Error: 1146, Table 'test.words' doesn't exist, when using table: words
 
*** stack smashing detected ***: /home/buildbot/maria-slave/p8-rhel6-bintar-debug/build/client/mysqlimport terminated
 
mysqltest: At line 1780: command "$MYSQL_IMPORT --silent --use-threads=2 test $MYSQLTEST_VARDIR/tmp/t1.txt $MYSQLTEST_VARDIR/tmp/t2.txt $MYSQLTEST_VARDIR/std_data/words.dat $MYSQLTEST_VARDIR/std_data/words2.dat" failed with wrong error: 0
 
 
 
The result from queries just before the failure was:
 
< snip >
 
abdomen
 
abdomens
 
abdominal
 
abduct
 
abducted
 
abduction
 
abductions
 
abductor
 
abductors
 
abducts
 
Abe
 
abed
 
Abel
 
Abelian
 
Abelson
 
Aberdeen
 
Abernathy
 
aberrant
 
aberration
 
drop table words;
 
 
 
More results from queries before failure can be found in /home/buildbot/maria-slave/p8-rhel6-bintar-debug/build/mysql-test/var/4/log/mysqldump.log



 Comments   
Comment by Marko Mäkelä [ 2019-01-25 ]

Possibly related to this, ASAN is reporting errors. I compile and run with

cmake -DWITH_ASAN=ON -DWITH_SAFEMALLOC=OFF …
 
ASAN_OPTIONS=abort_on_error=1 ./mtr main.mysqldump

Failures:

10.2 fab531a150ec588f87e8a636d2ba1ecbfffdb08e

 
CURRENT_TEST: main.mysqldump
 
mysqldump: Couldn't find table: "non_existing"
 
mysqldump: Got error: 1356: "View 'test.v1' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES
 
mysqldump: Couldn't execute 'SHOW FIELDS FROM `v1`': View 'test.v1' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them (1356)
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: user2 has insufficent privileges to SHOW CREATE PROCEDURE `sp1`!
 
mysqldump: Got error: 1146: "Table 'test.???????????????????????' doesn't exist" when using LOCK TABLES
 
/dev/shm/10.2/client/mysqlimport: Error: 1146, Table 'test.words' doesn't exist, when using table: words
 
=================================================================
 
==22234==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000000388 at pc 0x0000004fc98b bp 0x7ffe0d8d35b0 sp 0x7ffe0d8d35a8
 
READ of size 8 at 0x615000000388 thread T0
 
    #0 0x4fc98a in main /mariadb/10.2/client/mysqlimport.c:683:12
 
    #1 0x7efd00d3b09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
    #2 0x420a49 in _start (/dev/shm/10.2/client/mysqlimport+0x420a49)
 
 
 
0x615000000388 is located 136 bytes inside of 480-byte region [0x615000000300,0x6150000004e0)
 
freed by thread T3 here:
 
    #0 0x4c8512 in free (/dev/shm/10.2/client/mysqlimport+0x4c8512)
 
    #1 0x596b02 in my_free /mariadb/10.2/mysys/my_malloc.c:217:5
 
    #2 0x587bdf in free_root /mariadb/10.2/mysys/my_alloc.c:399:7
 
    #3 0x581cfa in free_defaults /mariadb/10.2/mysys/my_default.c:652:3
 
    #4 0x4fba1d in safe_exit /mariadb/10.2/client/mysqlimport.c:520:3
 
    #5 0x4fba1d in db_error_with_table /mariadb/10.2/client/mysqlimport.c:536
 
    #6 0x4fba1d in write_to_table /mariadb/10.2/client/mysqlimport.c:385
 
    #7 0x4faa3e in worker_thread /mariadb/10.2/client/mysqlimport.c:622:14
 
 
 
previously allocated by thread T0 here:
 
    #0 0x4c8893 in __interceptor_malloc (/dev/shm/10.2/client/mysqlimport+0x4c8893)
 
    #1 0x596756 in my_malloc /mariadb/10.2/mysys/my_malloc.c:101:10
 
    #2 0x586f03 in alloc_root /mariadb/10.2/mysys/my_alloc.c:242:30
 
    #3 0x580f3e in my_load_defaults /mariadb/10.2/mysys/my_default.c:587:21
 
    #4 0x4fbc64 in main /mariadb/10.2/client/mysqlimport.c:646:3
 
    #5 0x7efd00d3b09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
 
 
Thread T3 created by T0 here:
 
    #0 0x4b11ad in pthread_create (/dev/shm/10.2/client/mysqlimport+0x4b11ad)
 
    #1 0x4fc257 in main /mariadb/10.2/client/mysqlimport.c:697:11
 
    #2 0x7efd00d3b09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.2/client/mysqlimport.c:683:12 in main
 
Shadow bytes around the buggy address:
 
  0x0c2a7fff8020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8040: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
 
  0x0c2a7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2a7fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
=>0x0c2a7fff8070: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8090: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
 
  0x0c2a7fff80a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2a7fff80b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2a7fff80c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
  Shadow gap:              cc
 
==22234==ABORTING
 
Aborted
 
mysqltest: At line 1780: command "$MYSQL_IMPORT --silent --use-threads=2 test $MYSQLTEST_VARDIR/tmp/t1.txt $MYSQLTEST_VARDIR/tmp/t2.txt $MYSQLTEST_VARDIR/std_data/words.dat $MYSQLTEST_VARDIR/std_data/words2.dat" failed with wrong error: 134

10.3 e9ba165bcbb9b913411b9a366a5f21d18e313de2

 
CURRENT_TEST: main.mysqldump
 
mysqldump: Couldn't find table: "non_existing"
 
mysqldump: Got error: 1356: "View 'test.v1' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES
 
mysqldump: Couldn't execute 'SHOW FIELDS FROM `v1`': View 'test.v1' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them (1356)
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: Got error: 1083: "Field separator argument is not what is expected; check the manual" when executing 'SELECT INTO OUTFILE'
 
mysqldump: user2 has insufficent privileges to SHOW CREATE PROCEDURE `sp1`!
 
mysqldump: Got error: 1146: "Table 'test.???????????????????????' doesn't exist" when using LOCK TABLES
 
/dev/shm/10.3/client/mysqlimport: Error: 1146, Table 'test.words' doesn't exist, when using table: words
 
=================================================================
 
==3495==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000000390 at pc 0x0000004fc96b bp 0x7ffd110bfb90 sp 0x7ffd110bfb88
 
READ of size 8 at 0x615000000390 thread T0
 
    #0 0x4fc96a in main /mariadb/10.3/client/mysqlimport.c:683:12
 
    #1 0x7f62102f609a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
    #2 0x420a29 in _start (/dev/shm/10.3/client/mysqlimport+0x420a29)
 
 
 
0x615000000390 is located 144 bytes inside of 480-byte region [0x615000000300,0x6150000004e0)
 
freed by thread T3 here:
 
    #0 0x4c84f2 in free (/dev/shm/10.3/client/mysqlimport+0x4c84f2)
 
    #1 0x596e41 in my_free /mariadb/10.3/mysys/my_malloc.c:222:5
 
    #2 0x5882fb in free_root /mariadb/10.3/mysys/my_alloc.c:428:7
 
    #3 0x58216e in free_defaults /mariadb/10.3/mysys/my_default.c:652:3
 
    #4 0x4fb9fd in safe_exit /mariadb/10.3/client/mysqlimport.c:520:3
 
    #5 0x4fb9fd in db_error_with_table /mariadb/10.3/client/mysqlimport.c:536
 
    #6 0x4fb9fd in write_to_table /mariadb/10.3/client/mysqlimport.c:385
 
    #7 0x4faa1e in worker_thread /mariadb/10.3/client/mysqlimport.c:622:14
 
 
 
previously allocated by thread T0 here:
 
    #0 0x4c8873 in __interceptor_malloc (/dev/shm/10.3/client/mysqlimport+0x4c8873)
 
    #1 0x596a66 in my_malloc /mariadb/10.3/mysys/my_malloc.c:101:10
 
    #2 0x587563 in alloc_root /mariadb/10.3/mysys/my_alloc.c:250:30
 
    #3 0x581389 in my_load_defaults /mariadb/10.3/mysys/my_default.c:587:21
 
    #4 0x4fbc44 in main /mariadb/10.3/client/mysqlimport.c:646:3
 
    #5 0x7f62102f609a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
 
 
Thread T3 created by T0 here:
 
    #0 0x4b118d in pthread_create (/dev/shm/10.3/client/mysqlimport+0x4b118d)
 
    #1 0x4fc237 in main /mariadb/10.3/client/mysqlimport.c:697:11
 
    #2 0x7f62102f609a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.3/client/mysqlimport.c:683:12 in main
 
Shadow bytes around the buggy address:
 
  0x0c2a7fff8020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8040: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
 
  0x0c2a7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2a7fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
=>0x0c2a7fff8070: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c2a7fff8090: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
 
  0x0c2a7fff80a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2a7fff80b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c2a7fff80c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
  Shadow gap:              cc
 
==3495==ABORTING
 
Aborted
 
mysqltest: At line 1782: command "$MYSQL_IMPORT --silent --use-threads=2 test $MYSQLTEST_VARDIR/tmp/t1.txt $MYSQLTEST_VARDIR/tmp/t2.txt $MYSQLTEST_VARDIR/std_data/words.dat $MYSQLTEST_VARDIR/std_data/words2.dat" failed with wrong error: 134

The source code line in question is:

    for (; *argv != NULL; argv++) /* Loop through tables */

It seems that we copied the command line arguments to heap, then freed the heap possibly due to some error handling, and finally keep iterating through the freed command line arguments. Maybe the memory should be freed by the main thread, not by error handling in worker threads?

Comment by Oleksandr Byelkin [ 2020-04-01 ]

I do not see mysqldump problems with ASAN, on buildbot it fails om 32 bit systems (usually with bunch other tests, so probably something with setup).

tturenko how I can repeat the problem (on azur for example because of it priority was risen)?

Generated at Thu Feb 08 08:16:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.