Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13829

Server crash / ASAN errors in post_kill_notification

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.2, 10.3, 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2
    • 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2
    • Server
    • None

    Description

      SIGSEGV crashes have been seen, but the result is much easier to achieve on an ASAN build.

      10.4 64f44b22d9a3dab3d4c0b77addbcbdafde57b466

      ==3142241==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000112ad0 at pc 0x55f4affc829f bp 0x7fb859cdcd90 sp 0x7fb859cdcd88
      READ of size 8 at 0x615000112ad0 thread T41
          #0 0x55f4affc829e in post_kill_notification(THD*) /data/bld/10.4-asan/sql/scheduler.cc:118
          #1 0x55f4afaab92a in THD::awake_no_mutex(killed_state) /data/bld/10.4-asan/sql/sql_class.cc:1888
          #2 0x55f4afbf55b7 in kill_one_thread(THD*, unsigned long long, killed_state, killed_type) /data/bld/10.4-asan/sql/sql_parse.cc:9276
          #3 0x55f4afbf6199 in sql_kill /data/bld/10.4-asan/sql/sql_parse.cc:9399
          #4 0x55f4afbdde70 in mysql_execute_command(THD*) /data/bld/10.4-asan/sql/sql_parse.cc:5686
          #5 0x55f4afbeda6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/bld/10.4-asan/sql/sql_parse.cc:8060
          #6 0x55f4afbc3ae1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/bld/10.4-asan/sql/sql_parse.cc:1857
          #7 0x55f4afbc0650 in do_command(THD*) /data/bld/10.4-asan/sql/sql_parse.cc:1378
          #8 0x55f4affc6d14 in do_handle_one_connection(CONNECT*) /data/bld/10.4-asan/sql/sql_connect.cc:1419
          #9 0x55f4affc662b in handle_one_connection /data/bld/10.4-asan/sql/sql_connect.cc:1323
          #10 0x7fb88e4a8043 in start_thread nptl/pthread_create.c:442
          #11 0x7fb88e52861b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
       
      0x615000112ad0 is located 464 bytes inside of 504-byte region [0x615000112900,0x615000112af8)
      freed by thread T36 here:
          #0 0x7fb88eab76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
          #1 0x55f4b1767b71 in my_free /data/bld/10.4-asan/mysys/my_malloc.c:222
          #2 0x55f4b08e3fac in vio_delete /data/bld/10.4-asan/vio/vio.c:322
          #3 0x55f4afaa8fb6 in THD::free_connection() /data/bld/10.4-asan/sql/sql_class.cc:1593
          #4 0x55f4af8d331a in unlink_thd(THD*) /data/bld/10.4-asan/sql/mysqld.cc:2653
          #5 0x55f4af8d3d54 in one_thread_per_connection_end(THD*, bool) /data/bld/10.4-asan/sql/mysqld.cc:2789
          #6 0x55f4affc6edd in do_handle_one_connection(CONNECT*) /data/bld/10.4-asan/sql/sql_connect.cc:1430
          #7 0x55f4affc662b in handle_one_connection /data/bld/10.4-asan/sql/sql_connect.cc:1323
          #8 0x7fb88e4a8043 in start_thread nptl/pthread_create.c:442
       
      previously allocated by thread T0 here:
          #0 0x7fb88eab89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
          #1 0x55f4b1766fd2 in my_malloc /data/bld/10.4-asan/mysys/my_malloc.c:101
          #2 0x55f4b08e3919 in mysql_socket_vio_new /data/bld/10.4-asan/vio/vio.c:221
          #3 0x55f4af8e011b in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/bld/10.4-asan/sql/mysqld.cc:6437
          #4 0x55f4af8e11d9 in handle_connections_sockets() /data/bld/10.4-asan/sql/mysqld.cc:6622
          #5 0x55f4af8dee77 in mysqld_main(int, char**) /data/bld/10.4-asan/sql/mysqld.cc:5954
          #6 0x55f4af8c60b8 in main /data/bld/10.4-asan/sql/main.cc:25
          #7 0x7fb88e4461c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
       
      Thread T41 created by T0 here:
          #0 0x7fb88ea49726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
          #1 0x55f4b17c7a28 in spawn_thread_noop /data/bld/10.4-asan/mysys/psi_noop.c:187
          #2 0x55f4af8c7f89 in inline_mysql_thread_create /data/bld/10.4-asan/include/mysql/psi/mysql_thread.h:1275
          #3 0x55f4af8df714 in create_thread_to_handle_connection(CONNECT*) /data/bld/10.4-asan/sql/mysqld.cc:6296
          #4 0x55f4af8dfe5f in create_new_thread(CONNECT*) /data/bld/10.4-asan/sql/mysqld.cc:6366
          #5 0x55f4af8e032d in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/bld/10.4-asan/sql/mysqld.cc:6464
          #6 0x55f4af8e11d9 in handle_connections_sockets() /data/bld/10.4-asan/sql/mysqld.cc:6622
          #7 0x55f4af8dee77 in mysqld_main(int, char**) /data/bld/10.4-asan/sql/mysqld.cc:5954
          #8 0x55f4af8c60b8 in main /data/bld/10.4-asan/sql/main.cc:25
          #9 0x7fb88e4461c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
       
      Thread T36 created by T0 here:
          #0 0x7fb88ea49726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
          #1 0x55f4b17c7a28 in spawn_thread_noop /data/bld/10.4-asan/mysys/psi_noop.c:187
          #2 0x55f4af8c7f89 in inline_mysql_thread_create /data/bld/10.4-asan/include/mysql/psi/mysql_thread.h:1275
          #3 0x55f4af8df714 in create_thread_to_handle_connection(CONNECT*) /data/bld/10.4-asan/sql/mysqld.cc:6296
          #4 0x55f4af8dfe5f in create_new_thread(CONNECT*) /data/bld/10.4-asan/sql/mysqld.cc:6366
          #5 0x55f4af8e032d in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/bld/10.4-asan/sql/mysqld.cc:6464
          #6 0x55f4af8e11d9 in handle_connections_sockets() /data/bld/10.4-asan/sql/mysqld.cc:6622
          #7 0x55f4af8dee77 in mysqld_main(int, char**) /data/bld/10.4-asan/sql/mysqld.cc:5954
          #8 0x55f4af8c60b8 in main /data/bld/10.4-asan/sql/main.cc:25
          #9 0x7fb88e4461c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
       
      SUMMARY: AddressSanitizer: heap-use-after-free /data/bld/10.4-asan/sql/scheduler.cc:118 in post_kill_notification(THD*)
      Shadow bytes around the buggy address:
        0x0c2a8001a500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
        0x0c2a8001a510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c2a8001a520: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c2a8001a530: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c2a8001a540: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      =>0x0c2a8001a550: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fa
        0x0c2a8001a560: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c2a8001a570: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c2a8001a580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c2a8001a590: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c2a8001a5a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==3142241==ABORTING
      

      To reproduce (remember to set the correct basedir):

      git clone https://github.com/MariaDB/randgen --branch mdev13829 rqg-mdev13829
      cd rqg-mdev13829
      perl ./run.pl --threads=8 --duration=300 --queries=100M --grammar=mdev13829.yy --nometadata-reload --mysqld=--max-statement-time=5 --mysqld=--lock-wait-timeout=3 --mysqld=--innodb-lock-wait-timeout=2 --trials=5 --output="post_kill_notification" --vardir=/dev/shm/var-mdev13829 --basedir=/data/bld/10.4-asan 
      

      It fails within seconds for me on 10.4-11.2 ASAN builds. However, I couldn't reproduce it under rr in a limited number of attempts.

      The grammar (same as mdev13829.yy in the branch above):

      query_init:
        CREATE TABLE IF NOT EXISTS test.t (pk INT PRIMARY KEY);
       
      query:
        REPLACE INTO test.t (`pk`) VALUES (1) |
        SELECT ID INTO @kill FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER != 'system user' AND Command != 'Sleep' ORDER BY ID LIMIT 1 ;; KILL @kill ;
      

      Attachments

        Issue Links

          Activity

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.