Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18069

Server hang or crash in MDL_lock::incompatible_granted_types_bitmap or ASAN heap-use-after-free in MDL_ticket::has_stronger_or_equal_type

Details

    Description

      Note: It's most likely the same problem as MDEV-18067, but since the assertion is different, I'm filing it separately just in case.

      The following test case fails on an ASAN build:

      CREATE TABLE t1 (a INT);
      		 
      BACKUP STAGE START;
      		 
      FLUSH TABLES t1 WITH READ LOCK;
      		 
      UNLOCK TABLES;
      		 
      BACKUP STAGE BLOCK_COMMIT;
      		 
       
      		 
      # Cleanup
      		 
      BACKUP STAGE END;
      		 
      DROP TABLE t1;

      10.4 ASAN a8eb0c76bf

      		 
      ==13044==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060000059f8 at pc 0x563748f3c800 bp 0x7f9c8933d6d0 sp 0x7f9c8933d6c8
      		 
      READ of size 8 at 0x6060000059f8 thread T5
      		 
          #0 0x563748f3c7ff in MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const /data/src/10.4/sql/mdl.cc:1893
      		 
          #1 0x563748f3f60d in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) /data/src/10.4/sql/mdl.cc:2532
      		 
          #2 0x563749108f17 in backup_flush /data/src/10.4/sql/backup.cc:198
      		 
          #3 0x5637491087cb in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:110
      		 
          #4 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
      		 
          #5 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
      		 
          #6 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
      		 
          #7 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
      		 
          #8 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
      		 
          #9 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
      		 
          #10 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
      		 
          #11 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
      		 
          #12 0x7f9c93d7d93e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe893e)
      		 
       
      		 
      0x6060000059f8 is located 56 bytes inside of 64-byte region [0x6060000059c0,0x606000005a00)
      		 
      freed by thread T5 here:
      		 
          #0 0x7f9c959e7477 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x55477)
      		 
          #1 0x563748f4559f in MDL_ticket::~MDL_ticket() /data/src/10.4/sql/mdl.h:667
      		 
          #2 0x563748f3a7b8 in MDL_ticket::destroy(MDL_ticket*) /data/src/10.4/sql/mdl.cc:1017
      		 
          #3 0x563748f405cc in MDL_context::release_lock(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2836
      		 
          #4 0x563748f408af in MDL_context::release_locks_stored_before(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2883
      		 
          #5 0x563748f41664 in MDL_context::release_transactional_locks() /data/src/10.4/sql/mdl.cc:3056
      		 
          #6 0x563748bc4032 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5172
      		 
          #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
      		 
          #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
      		 
          #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
      		 
          #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
      		 
          #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
      		 
          #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
      		 
          #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
      		 
       
      		 
      previously allocated by thread T5 here:
      		 
          #0 0x7f9c959e723f in operator new(unsigned long, std::nothrow_t const&) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x5523f)
      		 
          #1 0x563748f3a720 in MDL_ticket::create(MDL_context*, enum_mdl_type, enum_mdl_duration) /data/src/10.4/sql/mdl.cc:1011
      		 
          #2 0x563748f3d6bc in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) /data/src/10.4/sql/mdl.cc:2100
      		 
          #3 0x563748f3e2d3 in MDL_context::acquire_lock(MDL_request*, double) /data/src/10.4/sql/mdl.cc:2263
      		 
          #4 0x563749108c87 in backup_start /data/src/10.4/sql/backup.cc:165
      		 
          #5 0x563749108794 in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:104
      		 
          #6 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
      		 
          #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
      		 
          #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
      		 
          #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
      		 
          #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
      		 
          #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
      		 
          #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
      		 
          #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
      		 
       
      		 
      Thread T5 created by T0 here:
      		 
          #0 0x7f9c959b5bba in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x23bba)
      		 
          #1 0x563749a89caf in spawn_thread_v1 /data/src/10.4/storage/perfschema/pfs.cc:1912
      		 
          #2 0x5637489087b6 in inline_mysql_thread_create /data/src/10.4/include/mysql/psi/mysql_thread.h:1268
      		 
          #3 0x56374891e9d4 in create_thread_to_handle_connection(CONNECT*) /data/src/10.4/sql/mysqld.cc:6436
      		 
          #4 0x56374891f0d9 in create_new_thread(CONNECT*) /data/src/10.4/sql/mysqld.cc:6506
      		 
          #5 0x56374891f469 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/src/10.4/sql/mysqld.cc:6623
      		 
          #6 0x5637489200b5 in handle_connections_sockets() /data/src/10.4/sql/mysqld.cc:6788
      		 
          #7 0x56374891de91 in mysqld_main(int, char**) /data/src/10.4/sql/mysqld.cc:6058
      		 
          #8 0x56374890663f in main /data/src/10.4/sql/main.cc:25
      		 
          #9 0x7f9c93cb52b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
      		 
       
      		 
      SUMMARY: AddressSanitizer: heap-use-after-free /data/src/10.4/sql/mdl.cc:1893 MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const
      		 
      Shadow bytes around the buggy address:
      		 
        0x0c0c7fff8ae0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      		 
        0x0c0c7fff8af0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      		 
        0x0c0c7fff8b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      		 
        0x0c0c7fff8b10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      		 
        0x0c0c7fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
      		 
      =>0x0c0c7fff8b30: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd[fd]
      		 
        0x0c0c7fff8b40: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
      		 
        0x0c0c7fff8b50: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
      		 
        0x0c0c7fff8b60: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
      		 
        0x0c0c7fff8b70: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
      		 
        0x0c0c7fff8b80: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
      		 
      Shadow byte legend (one shadow byte represents 8 application bytes):
      		 
        Addressable:           00
      		 
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
      		 
        Heap right redzone:      fb
      		 
        Freed heap region:       fd
      		 
        Stack left redzone:      f1
      		 
        Stack mid redzone:       f2
      		 
        Stack right redzone:     f3
      		 
        Stack partial redzone:   f4
      		 
        Stack after return:      f5
      		 
        Stack use after scope:   f8
      		 
        Global redzone:          f9
      		 
        Global init order:       f6
      		 
        Poisoned by user:        f7
      		 
        Contiguous container OOB:fc
      		 
        ASan internal:           fe
      		 
      ==13044==ABORTING

      On a non-ASAN build it fails at a later stage with MDEV-18067 or hangs with the stack trace hang_threads_full.

      More complicated but similar test cases can crash on non-ASAN builds with the stack trace below, while producing the same ASAN errors on ASAN builds.

      10.4 a8eb0c76bf

      		 
      #3  <signal handler called>
      		 
      #4  0x0000555c589fe4f9 in MDL_lock::incompatible_granted_types_bitmap (this=0x7efc581018a0) at /data/src/10.4/sql/mdl.cc:543
      		 
      #5  0x0000555c589faf63 in MDL_ticket::has_stronger_or_equal_type (this=0x7efc680915c0, type=MDL_SHARED) at /data/src/10.4/sql/mdl.cc:1893
      		 
      #6  0x0000555c589fc476 in MDL_context::upgrade_shared_lock (this=0x7efc68000c20, mdl_ticket=0x7efc680915c0, new_type=MDL_SHARED, lock_wait_timeout=2) at /data/src/10.4/sql/mdl.cc:2532
      		 
      #7  0x0000555c58ad1bb9 in backup_flush (thd=0x7efc68000b00) at /data/src/10.4/sql/backup.cc:198
      		 
      #8  0x0000555c58ad1861 in run_backup_stage (thd=0x7efc68000b00, stage=BACKUP_FLUSH) at /data/src/10.4/sql/backup.cc:110
      		 
      #9  0x0000555c5888f94b in mysql_execute_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:5233
      		 
      #10 0x0000555c588988e4 in mysql_parse (thd=0x7efc68000b00, rawbuf=0x7efc68013b88 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */", length=44, parser_state=0x7efccc85d650, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:8104
      		 
      #11 0x0000555c58885b0e in dispatch_command (command=COM_QUERY, thd=0x7efc68000b00, packet=0x7efc6800b321 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */ ", packet_length=45, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1851
      		 
      #12 0x0000555c58884532 in do_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:1396
      		 
      #13 0x0000555c589f06de in do_handle_one_connection (connect=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1402
      		 
      #14 0x0000555c589f0462 in handle_one_connection (arg=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1308
      		 
      #15 0x00007efce3dae494 in start_thread (arg=0x7efccc85e700) at pthread_create.c:333
      		 
      #16 0x00007efce23af93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

      Attachments

        Issue Links

          causes

          Task - A task that needs to be done. MDEV-20867 Perform careful review of "Server crashes with BACKUP STAGE and FLUSH TABLE table_name"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is caused by

          Task - A task that needs to be done. MDEV-5336 Implement BACKUP STAGE for safe external backups

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18067 Server crash in backup_end or assertion failure `ticket->m_duration == MDL_EXPLICIT' upon BACKUP STAGE END after FLUSH TABLE with locks

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18068 Assertion `this == ticket->get_ctx()' failed in MDL_context::release_lock upon BACKUP STAGE END

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15481 [Draft] Server crashed in I_P_List<MDL_ticket, I_P_List_adapter<MDL_ticket, &MDL_ticket::next_in_lock, &MDL_ticket::prev_in_lock>, I_P_List_null_counter, I_P_List_fast_push_back<MDL_ticket> >::remove

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            elenst Elena Stepanova created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Summary [Draft] Server crashes in MDL_lock::incompatible_granted_types_bitmap Server crash in MDL_lock::incompatible_granted_types_bitmap or ASAN heap-use-after-free in MDL_ticket::has_stronger_or_equal_type
            elenst Elena Stepanova made changes -
            Description {noformat:title=10.4 a8eb0c76bf}
            #3 <signal handler called>
            #4 0x0000555c589fe4f9 in MDL_lock::incompatible_granted_types_bitmap (this=0x7efc581018a0) at /data/src/10.4/sql/mdl.cc:543
            #5 0x0000555c589faf63 in MDL_ticket::has_stronger_or_equal_type (this=0x7efc680915c0, type=MDL_SHARED) at /data/src/10.4/sql/mdl.cc:1893
            #6 0x0000555c589fc476 in MDL_context::upgrade_shared_lock (this=0x7efc68000c20, mdl_ticket=0x7efc680915c0, new_type=MDL_SHARED, lock_wait_timeout=2) at /data/src/10.4/sql/mdl.cc:2532
            #7 0x0000555c58ad1bb9 in backup_flush (thd=0x7efc68000b00) at /data/src/10.4/sql/backup.cc:198
            #8 0x0000555c58ad1861 in run_backup_stage (thd=0x7efc68000b00, stage=BACKUP_FLUSH) at /data/src/10.4/sql/backup.cc:110
            #9 0x0000555c5888f94b in mysql_execute_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:5233
            #10 0x0000555c588988e4 in mysql_parse (thd=0x7efc68000b00, rawbuf=0x7efc68013b88 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */", length=44, parser_state=0x7efccc85d650, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:8104
            #11 0x0000555c58885b0e in dispatch_command (command=COM_QUERY, thd=0x7efc68000b00, packet=0x7efc6800b321 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */ ", packet_length=45, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1851
            #12 0x0000555c58884532 in do_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:1396
            #13 0x0000555c589f06de in do_handle_one_connection (connect=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1402
            #14 0x0000555c589f0462 in handle_one_connection (arg=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1308
            #15 0x00007efce3dae494 in start_thread (arg=0x7efccc85e700) at pthread_create.c:333
            #16 0x00007efce23af93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
            {noformat}

            {noformat:title=elenst-dev df3fbc8be48}
            perl ./runall-new.pl --basedir=/home/travis/server/10.4 --vardir=/home/travis/logs/vardir6 --duration=350 --threads=6 --seed=1545592040 --reporters=Backtrace,ErrorLog,Deadlock --redefine=conf/mariadb/alter_table.yy --redefine=conf/mariadb/instant_add.yy --redefine=conf/mariadb/sp.yy --redefine=conf/mariadb/bulk_insert.yy --redefine=conf/mariadb/sequences.yy --redefine=conf/mariadb/modules/admin.yy --redefine=conf/mariadb/modules/foreign_keys.yy --redefine=conf/mariadb/modules/locks-10.4.yy --mysqld=--log_output=FILE --mysqld=--max-statement-time=3 --mysqld=--lock-wait-timeout=2 --mysqld=--loose-innodb-lock-wait-timeout=1 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --grammar=conf/engines/many_indexes.yy --gendata=conf/engines/many_indexes.zz --skip-gendata --gendata-advanced
            {noformat}

            {color:red}Not reproducible easily, among other things MDEV-18067 is on the way.{color}             		_Note: It's most likely the same problem as MDEV-18067, but since the assertion is different, I'm filing it separately just in case._

            The following test case fails on an ASAN build:
            {code:sql}
            CREATE TABLE t1 (a INT);
            BACKUP STAGE START;
            FLUSH TABLES t1 WITH READ LOCK;
            UNLOCK TABLES;
            BACKUP STAGE BLOCK_COMMIT;

            # Cleanup
            BACKUP STAGE END;
            DROP TABLE t1;
            {code}
            {noformat:title=10.4 ASAN a8eb0c76bf}
            ==13044==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060000059f8 at pc 0x563748f3c800 bp 0x7f9c8933d6d0 sp 0x7f9c8933d6c8
            READ of size 8 at 0x6060000059f8 thread T5
                #0 0x563748f3c7ff in MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const /data/src/10.4/sql/mdl.cc:1893
                #1 0x563748f3f60d in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) /data/src/10.4/sql/mdl.cc:2532
                #2 0x563749108f17 in backup_flush /data/src/10.4/sql/backup.cc:198
                #3 0x5637491087cb in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:110
                #4 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
                #5 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #6 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #7 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #8 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #9 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #10 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #11 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
                #12 0x7f9c93d7d93e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe893e)

            0x6060000059f8 is located 56 bytes inside of 64-byte region [0x6060000059c0,0x606000005a00)
            freed by thread T5 here:
                #0 0x7f9c959e7477 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x55477)
                #1 0x563748f4559f in MDL_ticket::~MDL_ticket() /data/src/10.4/sql/mdl.h:667
                #2 0x563748f3a7b8 in MDL_ticket::destroy(MDL_ticket*) /data/src/10.4/sql/mdl.cc:1017
                #3 0x563748f405cc in MDL_context::release_lock(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2836
                #4 0x563748f408af in MDL_context::release_locks_stored_before(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2883
                #5 0x563748f41664 in MDL_context::release_transactional_locks() /data/src/10.4/sql/mdl.cc:3056
                #6 0x563748bc4032 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5172
                #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

            previously allocated by thread T5 here:
                #0 0x7f9c959e723f in operator new(unsigned long, std::nothrow_t const&) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x5523f)
                #1 0x563748f3a720 in MDL_ticket::create(MDL_context*, enum_mdl_type, enum_mdl_duration) /data/src/10.4/sql/mdl.cc:1011
                #2 0x563748f3d6bc in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) /data/src/10.4/sql/mdl.cc:2100
                #3 0x563748f3e2d3 in MDL_context::acquire_lock(MDL_request*, double) /data/src/10.4/sql/mdl.cc:2263
                #4 0x563749108c87 in backup_start /data/src/10.4/sql/backup.cc:165
                #5 0x563749108794 in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:104
                #6 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
                #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

            Thread T5 created by T0 here:
                #0 0x7f9c959b5bba in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x23bba)
                #1 0x563749a89caf in spawn_thread_v1 /data/src/10.4/storage/perfschema/pfs.cc:1912
                #2 0x5637489087b6 in inline_mysql_thread_create /data/src/10.4/include/mysql/psi/mysql_thread.h:1268
                #3 0x56374891e9d4 in create_thread_to_handle_connection(CONNECT*) /data/src/10.4/sql/mysqld.cc:6436
                #4 0x56374891f0d9 in create_new_thread(CONNECT*) /data/src/10.4/sql/mysqld.cc:6506
                #5 0x56374891f469 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/src/10.4/sql/mysqld.cc:6623
                #6 0x5637489200b5 in handle_connections_sockets() /data/src/10.4/sql/mysqld.cc:6788
                #7 0x56374891de91 in mysqld_main(int, char**) /data/src/10.4/sql/mysqld.cc:6058
                #8 0x56374890663f in main /data/src/10.4/sql/main.cc:25
                #9 0x7f9c93cb52b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

            SUMMARY: AddressSanitizer: heap-use-after-free /data/src/10.4/sql/mdl.cc:1893 MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const
            Shadow bytes around the buggy address:
              0x0c0c7fff8ae0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8af0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
            =>0x0c0c7fff8b30: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd[fd]
              0x0c0c7fff8b40: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c7fff8b50: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
              0x0c0c7fff8b60: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
              0x0c0c7fff8b70: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c7fff8b80: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable: 00
              Partially addressable: 01 02 03 04 05 06 07
              Heap left redzone: fa
              Heap right redzone: fb
              Freed heap region: fd
              Stack left redzone: f1
              Stack mid redzone: f2
              Stack right redzone: f3
              Stack partial redzone: f4
              Stack after return: f5
              Stack use after scope: f8
              Global redzone: f9
              Global init order: f6
              Poisoned by user: f7
              Contiguous container OOB:fc
              ASan internal: fe
            ==13044==ABORTING
            {noformat}

            On a non-ASAN build it fails at a later stage with MDEV-18067.

            More complicated but similar test cases can crash on non-ASAN builds with the stack trace below, while producing the same ASAN errors on ASAN builds.

            {noformat:title=10.4 a8eb0c76bf}
            #3 <signal handler called>
            #4 0x0000555c589fe4f9 in MDL_lock::incompatible_granted_types_bitmap (this=0x7efc581018a0) at /data/src/10.4/sql/mdl.cc:543
            #5 0x0000555c589faf63 in MDL_ticket::has_stronger_or_equal_type (this=0x7efc680915c0, type=MDL_SHARED) at /data/src/10.4/sql/mdl.cc:1893
            #6 0x0000555c589fc476 in MDL_context::upgrade_shared_lock (this=0x7efc68000c20, mdl_ticket=0x7efc680915c0, new_type=MDL_SHARED, lock_wait_timeout=2) at /data/src/10.4/sql/mdl.cc:2532
            #7 0x0000555c58ad1bb9 in backup_flush (thd=0x7efc68000b00) at /data/src/10.4/sql/backup.cc:198
            #8 0x0000555c58ad1861 in run_backup_stage (thd=0x7efc68000b00, stage=BACKUP_FLUSH) at /data/src/10.4/sql/backup.cc:110
            #9 0x0000555c5888f94b in mysql_execute_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:5233
            #10 0x0000555c588988e4 in mysql_parse (thd=0x7efc68000b00, rawbuf=0x7efc68013b88 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */", length=44, parser_state=0x7efccc85d650, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:8104
            #11 0x0000555c58885b0e in dispatch_command (command=COM_QUERY, thd=0x7efc68000b00, packet=0x7efc6800b321 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */ ", packet_length=45, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1851
            #12 0x0000555c58884532 in do_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:1396
            #13 0x0000555c589f06de in do_handle_one_connection (connect=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1402
            #14 0x0000555c589f0462 in handle_one_connection (arg=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1308
            #15 0x00007efce3dae494 in start_thread (arg=0x7efccc85e700) at pthread_create.c:333
            #16 0x00007efce23af93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
            {noformat}
            elenst Elena Stepanova made changes -
            Attachment mysql.log.gz [ 46876 ]
            elenst Elena Stepanova made changes -
            Attachment threads [ 46875 ]
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            Fix Version/s 10.4 [ 22408 ]
            Assignee Elena Stepanova [ elenst ] Michael Widenius [ monty ]
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            Labels affects-tests
            Priority Major [ 3 ] Critical [ 2 ]
            elenst Elena Stepanova made changes -
            Assignee Michael Widenius [ monty ] Vladislav Lesin [ vlad.lesin ]
            elenst Elena Stepanova made changes -
            Summary Server crash in MDL_lock::incompatible_granted_types_bitmap or ASAN heap-use-after-free in MDL_ticket::has_stronger_or_equal_type Server hang or crash in MDL_lock::incompatible_granted_types_bitmap or ASAN heap-use-after-free in MDL_ticket::has_stronger_or_equal_type
            elenst Elena Stepanova made changes -
            Description _Note: It's most likely the same problem as MDEV-18067, but since the assertion is different, I'm filing it separately just in case._

            The following test case fails on an ASAN build:
            {code:sql}
            CREATE TABLE t1 (a INT);
            BACKUP STAGE START;
            FLUSH TABLES t1 WITH READ LOCK;
            UNLOCK TABLES;
            BACKUP STAGE BLOCK_COMMIT;

            # Cleanup
            BACKUP STAGE END;
            DROP TABLE t1;
            {code}
            {noformat:title=10.4 ASAN a8eb0c76bf}
            ==13044==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060000059f8 at pc 0x563748f3c800 bp 0x7f9c8933d6d0 sp 0x7f9c8933d6c8
            READ of size 8 at 0x6060000059f8 thread T5
                #0 0x563748f3c7ff in MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const /data/src/10.4/sql/mdl.cc:1893
                #1 0x563748f3f60d in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) /data/src/10.4/sql/mdl.cc:2532
                #2 0x563749108f17 in backup_flush /data/src/10.4/sql/backup.cc:198
                #3 0x5637491087cb in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:110
                #4 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
                #5 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #6 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #7 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #8 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #9 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #10 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #11 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
                #12 0x7f9c93d7d93e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe893e)

            0x6060000059f8 is located 56 bytes inside of 64-byte region [0x6060000059c0,0x606000005a00)
            freed by thread T5 here:
                #0 0x7f9c959e7477 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x55477)
                #1 0x563748f4559f in MDL_ticket::~MDL_ticket() /data/src/10.4/sql/mdl.h:667
                #2 0x563748f3a7b8 in MDL_ticket::destroy(MDL_ticket*) /data/src/10.4/sql/mdl.cc:1017
                #3 0x563748f405cc in MDL_context::release_lock(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2836
                #4 0x563748f408af in MDL_context::release_locks_stored_before(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2883
                #5 0x563748f41664 in MDL_context::release_transactional_locks() /data/src/10.4/sql/mdl.cc:3056
                #6 0x563748bc4032 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5172
                #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

            previously allocated by thread T5 here:
                #0 0x7f9c959e723f in operator new(unsigned long, std::nothrow_t const&) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x5523f)
                #1 0x563748f3a720 in MDL_ticket::create(MDL_context*, enum_mdl_type, enum_mdl_duration) /data/src/10.4/sql/mdl.cc:1011
                #2 0x563748f3d6bc in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) /data/src/10.4/sql/mdl.cc:2100
                #3 0x563748f3e2d3 in MDL_context::acquire_lock(MDL_request*, double) /data/src/10.4/sql/mdl.cc:2263
                #4 0x563749108c87 in backup_start /data/src/10.4/sql/backup.cc:165
                #5 0x563749108794 in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:104
                #6 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
                #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

            Thread T5 created by T0 here:
                #0 0x7f9c959b5bba in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x23bba)
                #1 0x563749a89caf in spawn_thread_v1 /data/src/10.4/storage/perfschema/pfs.cc:1912
                #2 0x5637489087b6 in inline_mysql_thread_create /data/src/10.4/include/mysql/psi/mysql_thread.h:1268
                #3 0x56374891e9d4 in create_thread_to_handle_connection(CONNECT*) /data/src/10.4/sql/mysqld.cc:6436
                #4 0x56374891f0d9 in create_new_thread(CONNECT*) /data/src/10.4/sql/mysqld.cc:6506
                #5 0x56374891f469 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/src/10.4/sql/mysqld.cc:6623
                #6 0x5637489200b5 in handle_connections_sockets() /data/src/10.4/sql/mysqld.cc:6788
                #7 0x56374891de91 in mysqld_main(int, char**) /data/src/10.4/sql/mysqld.cc:6058
                #8 0x56374890663f in main /data/src/10.4/sql/main.cc:25
                #9 0x7f9c93cb52b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

            SUMMARY: AddressSanitizer: heap-use-after-free /data/src/10.4/sql/mdl.cc:1893 MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const
            Shadow bytes around the buggy address:
              0x0c0c7fff8ae0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8af0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
            =>0x0c0c7fff8b30: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd[fd]
              0x0c0c7fff8b40: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c7fff8b50: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
              0x0c0c7fff8b60: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
              0x0c0c7fff8b70: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c7fff8b80: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable: 00
              Partially addressable: 01 02 03 04 05 06 07
              Heap left redzone: fa
              Heap right redzone: fb
              Freed heap region: fd
              Stack left redzone: f1
              Stack mid redzone: f2
              Stack right redzone: f3
              Stack partial redzone: f4
              Stack after return: f5
              Stack use after scope: f8
              Global redzone: f9
              Global init order: f6
              Poisoned by user: f7
              Contiguous container OOB:fc
              ASan internal: fe
            ==13044==ABORTING
            {noformat}

            On a non-ASAN build it fails at a later stage with MDEV-18067.

            More complicated but similar test cases can crash on non-ASAN builds with the stack trace below, while producing the same ASAN errors on ASAN builds.

            {noformat:title=10.4 a8eb0c76bf}
            #3 <signal handler called>
            #4 0x0000555c589fe4f9 in MDL_lock::incompatible_granted_types_bitmap (this=0x7efc581018a0) at /data/src/10.4/sql/mdl.cc:543
            #5 0x0000555c589faf63 in MDL_ticket::has_stronger_or_equal_type (this=0x7efc680915c0, type=MDL_SHARED) at /data/src/10.4/sql/mdl.cc:1893
            #6 0x0000555c589fc476 in MDL_context::upgrade_shared_lock (this=0x7efc68000c20, mdl_ticket=0x7efc680915c0, new_type=MDL_SHARED, lock_wait_timeout=2) at /data/src/10.4/sql/mdl.cc:2532
            #7 0x0000555c58ad1bb9 in backup_flush (thd=0x7efc68000b00) at /data/src/10.4/sql/backup.cc:198
            #8 0x0000555c58ad1861 in run_backup_stage (thd=0x7efc68000b00, stage=BACKUP_FLUSH) at /data/src/10.4/sql/backup.cc:110
            #9 0x0000555c5888f94b in mysql_execute_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:5233
            #10 0x0000555c588988e4 in mysql_parse (thd=0x7efc68000b00, rawbuf=0x7efc68013b88 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */", length=44, parser_state=0x7efccc85d650, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:8104
            #11 0x0000555c58885b0e in dispatch_command (command=COM_QUERY, thd=0x7efc68000b00, packet=0x7efc6800b321 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */ ", packet_length=45, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1851
            #12 0x0000555c58884532 in do_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:1396
            #13 0x0000555c589f06de in do_handle_one_connection (connect=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1402
            #14 0x0000555c589f0462 in handle_one_connection (arg=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1308
            #15 0x00007efce3dae494 in start_thread (arg=0x7efccc85e700) at pthread_create.c:333
            #16 0x00007efce23af93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
            {noformat}
            		_Note: It's most likely the same problem as MDEV-18067, but since the assertion is different, I'm filing it separately just in case._

            The following test case fails on an ASAN build:
            {code:sql}
            CREATE TABLE t1 (a INT);
            BACKUP STAGE START;
            FLUSH TABLES t1 WITH READ LOCK;
            UNLOCK TABLES;
            BACKUP STAGE BLOCK_COMMIT;

            # Cleanup
            BACKUP STAGE END;
            DROP TABLE t1;
            {code}
            {noformat:title=10.4 ASAN a8eb0c76bf}
            ==13044==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060000059f8 at pc 0x563748f3c800 bp 0x7f9c8933d6d0 sp 0x7f9c8933d6c8
            READ of size 8 at 0x6060000059f8 thread T5
                #0 0x563748f3c7ff in MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const /data/src/10.4/sql/mdl.cc:1893
                #1 0x563748f3f60d in MDL_context::upgrade_shared_lock(MDL_ticket*, enum_mdl_type, double) /data/src/10.4/sql/mdl.cc:2532
                #2 0x563749108f17 in backup_flush /data/src/10.4/sql/backup.cc:198
                #3 0x5637491087cb in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:110
                #4 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
                #5 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #6 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #7 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #8 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #9 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #10 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #11 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
                #12 0x7f9c93d7d93e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe893e)

            0x6060000059f8 is located 56 bytes inside of 64-byte region [0x6060000059c0,0x606000005a00)
            freed by thread T5 here:
                #0 0x7f9c959e7477 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x55477)
                #1 0x563748f4559f in MDL_ticket::~MDL_ticket() /data/src/10.4/sql/mdl.h:667
                #2 0x563748f3a7b8 in MDL_ticket::destroy(MDL_ticket*) /data/src/10.4/sql/mdl.cc:1017
                #3 0x563748f405cc in MDL_context::release_lock(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2836
                #4 0x563748f408af in MDL_context::release_locks_stored_before(enum_mdl_duration, MDL_ticket*) /data/src/10.4/sql/mdl.cc:2883
                #5 0x563748f41664 in MDL_context::release_transactional_locks() /data/src/10.4/sql/mdl.cc:3056
                #6 0x563748bc4032 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5172
                #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

            previously allocated by thread T5 here:
                #0 0x7f9c959e723f in operator new(unsigned long, std::nothrow_t const&) (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x5523f)
                #1 0x563748f3a720 in MDL_ticket::create(MDL_context*, enum_mdl_type, enum_mdl_duration) /data/src/10.4/sql/mdl.cc:1011
                #2 0x563748f3d6bc in MDL_context::try_acquire_lock_impl(MDL_request*, MDL_ticket**) /data/src/10.4/sql/mdl.cc:2100
                #3 0x563748f3e2d3 in MDL_context::acquire_lock(MDL_request*, double) /data/src/10.4/sql/mdl.cc:2263
                #4 0x563749108c87 in backup_start /data/src/10.4/sql/backup.cc:165
                #5 0x563749108794 in run_backup_stage(THD*, backup_stages) /data/src/10.4/sql/backup.cc:104
                #6 0x563748bc4437 in mysql_execute_command(THD*) /data/src/10.4/sql/sql_parse.cc:5233
                #7 0x563748bd6580 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.4/sql/sql_parse.cc:8104
                #8 0x563748bb0ae4 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.4/sql/sql_parse.cc:1851
                #9 0x563748badb6c in do_command(THD*) /data/src/10.4/sql/sql_parse.cc:1396
                #10 0x563748f23f06 in do_handle_one_connection(CONNECT*) /data/src/10.4/sql/sql_connect.cc:1402
                #11 0x563748f23912 in handle_one_connection /data/src/10.4/sql/sql_connect.cc:1308
                #12 0x563749a896e7 in pfs_spawn_thread /data/src/10.4/storage/perfschema/pfs.cc:1862
                #13 0x7f9c9577c493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

            Thread T5 created by T0 here:
                #0 0x7f9c959b5bba in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x23bba)
                #1 0x563749a89caf in spawn_thread_v1 /data/src/10.4/storage/perfschema/pfs.cc:1912
                #2 0x5637489087b6 in inline_mysql_thread_create /data/src/10.4/include/mysql/psi/mysql_thread.h:1268
                #3 0x56374891e9d4 in create_thread_to_handle_connection(CONNECT*) /data/src/10.4/sql/mysqld.cc:6436
                #4 0x56374891f0d9 in create_new_thread(CONNECT*) /data/src/10.4/sql/mysqld.cc:6506
                #5 0x56374891f469 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/src/10.4/sql/mysqld.cc:6623
                #6 0x5637489200b5 in handle_connections_sockets() /data/src/10.4/sql/mysqld.cc:6788
                #7 0x56374891de91 in mysqld_main(int, char**) /data/src/10.4/sql/mysqld.cc:6058
                #8 0x56374890663f in main /data/src/10.4/sql/main.cc:25
                #9 0x7f9c93cb52b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

            SUMMARY: AddressSanitizer: heap-use-after-free /data/src/10.4/sql/mdl.cc:1893 MDL_ticket::has_stronger_or_equal_type(enum_mdl_type) const
            Shadow bytes around the buggy address:
              0x0c0c7fff8ae0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8af0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c7fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
            =>0x0c0c7fff8b30: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd[fd]
              0x0c0c7fff8b40: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c7fff8b50: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
              0x0c0c7fff8b60: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
              0x0c0c7fff8b70: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c7fff8b80: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable: 00
              Partially addressable: 01 02 03 04 05 06 07
              Heap left redzone: fa
              Heap right redzone: fb
              Freed heap region: fd
              Stack left redzone: f1
              Stack mid redzone: f2
              Stack right redzone: f3
              Stack partial redzone: f4
              Stack after return: f5
              Stack use after scope: f8
              Global redzone: f9
              Global init order: f6
              Poisoned by user: f7
              Contiguous container OOB:fc
              ASan internal: fe
            ==13044==ABORTING
            {noformat}

            On a non-ASAN build it fails at a later stage with MDEV-18067 or hangs with the stack trace [^hang_threads_full].

            More complicated but similar test cases can crash on non-ASAN builds with the stack trace below, while producing the same ASAN errors on ASAN builds.

            {noformat:title=10.4 a8eb0c76bf}
            #3 <signal handler called>
            #4 0x0000555c589fe4f9 in MDL_lock::incompatible_granted_types_bitmap (this=0x7efc581018a0) at /data/src/10.4/sql/mdl.cc:543
            #5 0x0000555c589faf63 in MDL_ticket::has_stronger_or_equal_type (this=0x7efc680915c0, type=MDL_SHARED) at /data/src/10.4/sql/mdl.cc:1893
            #6 0x0000555c589fc476 in MDL_context::upgrade_shared_lock (this=0x7efc68000c20, mdl_ticket=0x7efc680915c0, new_type=MDL_SHARED, lock_wait_timeout=2) at /data/src/10.4/sql/mdl.cc:2532
            #7 0x0000555c58ad1bb9 in backup_flush (thd=0x7efc68000b00) at /data/src/10.4/sql/backup.cc:198
            #8 0x0000555c58ad1861 in run_backup_stage (thd=0x7efc68000b00, stage=BACKUP_FLUSH) at /data/src/10.4/sql/backup.cc:110
            #9 0x0000555c5888f94b in mysql_execute_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:5233
            #10 0x0000555c588988e4 in mysql_parse (thd=0x7efc68000b00, rawbuf=0x7efc68013b88 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */", length=44, parser_state=0x7efccc85d650, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:8104
            #11 0x0000555c58885b0e in dispatch_command (command=COM_QUERY, thd=0x7efc68000b00, packet=0x7efc6800b321 "BACKUP STAGE FLUSH /* QNO 10058 CON_ID 15 */ ", packet_length=45, is_com_multi=false, is_next_command=false) at /data/src/10.4/sql/sql_parse.cc:1851
            #12 0x0000555c58884532 in do_command (thd=0x7efc68000b00) at /data/src/10.4/sql/sql_parse.cc:1396
            #13 0x0000555c589f06de in do_handle_one_connection (connect=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1402
            #14 0x0000555c589f0462 in handle_one_connection (arg=0x555c5cdc7050) at /data/src/10.4/sql/sql_connect.cc:1308
            #15 0x00007efce3dae494 in start_thread (arg=0x7efccc85e700) at pthread_create.c:333
            #16 0x00007efce23af93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
            {noformat}
            elenst Elena Stepanova made changes -
            Attachment hang_threads_full [ 49274 ]
            monty Michael Widenius made changes -
            Assignee Vladislav Lesin [ vlad.lesin ] Michael Widenius [ monty ]
            monty Michael Widenius made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            monty Michael Widenius added a comment -

            The problem was that "FLUSH TABLES table_name" combined with UNLOCK TABLES
            calls MDL_context::set_transaction_duration_for_all_locks(), which
            changed backup_locks from MDL_EXPLICT to MDL_TRANSACTION.

            Fixed by ensuring that set_transaction_duration_for_all_locks() doesn't
            touch BACKUP locks.

            monty Michael Widenius added a comment - The problem was that "FLUSH TABLES table_name" combined with UNLOCK TABLES calls MDL_context::set_transaction_duration_for_all_locks(), which changed backup_locks from MDL_EXPLICT to MDL_TRANSACTION. Fixed by ensuring that set_transaction_duration_for_all_locks() doesn't touch BACKUP locks.
            monty Michael Widenius made changes -
            issue.field.resolutiondate 2019-10-20 12:48:35.0 2019-10-20 12:48:35.169
            monty Michael Widenius made changes -
            Fix Version/s 10.4.10 [ 23907 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            svoj Sergey Vojtovich made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 10.4.9 [ 23906 ]
            Fix Version/s 10.4.10 [ 23907 ]
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 91383 ] MariaDB v4 [ 155397 ]

            People

              monty Michael Widenius
              elenst Elena Stepanova
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.