Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19415

multi_source.mdev-9544 , multi_source.info_logs failed in buildbot with AddressSanitizer: heap-use-after-free in strend / get_one_variable

Details

    Description

      http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/662

      10.2 1e9c2b23

      ==16830==ERROR: AddressSanitizer: heap-use-after-free on address 0x610000077cb0 at pc 0x558f088eb90e bp 0x7fe6d52bd640 sp 0x7fe6d52bd630
      READ of size 1 at 0x610000077cb0 thread T28
          #0 0x558f088eb90d in strend /home/buildbot/buildbot/build/mariadb-10.2.24/strings/strend.c:45
          #1 0x558f073abacb in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_show.cc:3430
          #2 0x558f073ac937 in show_status_array /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_show.cc:3573
          #3 0x558f073da7f0 in fill_variables(THD*, TABLE_LIST*, Item*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_show.cc:7496
          #4 0x558f073e276b in get_schema_tables_result(JOIN*, enum_schema_table_state) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_show.cc:8425
          #5 0x558f072e4e38 in JOIN::exec_inner() /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_select.cc:3587
          #6 0x558f072e316b in JOIN::exec() /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_select.cc:3418
          #7 0x558f072e64b8 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_select.cc:3818
          #8 0x558f072c5c7e in handle_select(THD*, LEX*, select_result*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_select.cc:376
          #9 0x558f0724d5ee in execute_sqlcom_select /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:6479
          #10 0x558f0723aab4 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:3537
          #11 0x558f07b98bc5 in sp_instr_stmt::exec_core(THD*, unsigned int*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sp_head.cc:3246
          #12 0x558f07b97848 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sp_head.cc:3009
          #13 0x558f07b984a2 in sp_instr_stmt::execute(THD*, unsigned int*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sp_head.cc:3162
          #14 0x558f07b8de43 in sp_head::execute(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sp_head.cc:1327
          #15 0x558f07b91c9a in sp_head::execute_procedure(THD*, List<Item>*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sp_head.cc:2116
          #16 0x558f07236f58 in do_execute_sp /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:2956
          #17 0x558f07248a89 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:5827
          #18 0x558f07255d52 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:8013
          #19 0x558f07231407 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:1832
          #20 0x558f0722e586 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:1386
          #21 0x558f07556367 in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_connect.cc:1335
          #22 0x558f07555d6f in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_connect.cc:1241
          #23 0x558f07d06a93 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.24/storage/perfschema/pfs.cc:1862
          #24 0x7fe6eca366b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
          #25 0x7fe6ebecb82c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)
       
      0x610000077cb0 is located 112 bytes inside of 188-byte region [0x610000077c40,0x610000077cfc)
      freed by thread T33 here:
          #0 0x7fe6ee2142ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
          #1 0x558f0882acdb in free_memory /home/buildbot/buildbot/build/mariadb-10.2.24/mysys/safemalloc.c:279
          #2 0x558f0882a365 in sf_free /home/buildbot/buildbot/build/mariadb-10.2.24/mysys/safemalloc.c:197
          #3 0x558f087faed8 in my_free /home/buildbot/buildbot/build/mariadb-10.2.24/mysys/my_malloc.c:218
          #4 0x558f07766fb5 in mysql_close_free_options /home/buildbot/buildbot/build/mariadb-10.2.24/sql-common/client.c:3828
          #5 0x558f07767afe in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.24/sql-common/client.c:3991
          #6 0x558f070b67e4 in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.2.24/sql/slave.cc:4651
          #7 0x558f07d06a93 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.24/storage/perfschema/pfs.cc:1862
          #8 0x7fe6eca366b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
       
      previously allocated by thread T33 here:
          #0 0x7fe6ee214602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
          #1 0x558f08829d36 in sf_malloc /home/buildbot/buildbot/build/mariadb-10.2.24/mysys/safemalloc.c:118
          #2 0x558f087fa63f in my_malloc /home/buildbot/buildbot/build/mariadb-10.2.24/mysys/my_malloc.c:101
          #3 0x558f087fb1a4 in my_strdup /home/buildbot/buildbot/build/mariadb-10.2.24/mysys/my_malloc.c:241
          #4 0x558f0776adbb in mysql_options /home/buildbot/buildbot/build/mariadb-10.2.24/sql-common/client.c:4337
          #5 0x558f070c214a in connect_to_master /home/buildbot/buildbot/build/mariadb-10.2.24/sql/slave.cc:6717
          #6 0x558f070c19a5 in safe_connect /home/buildbot/buildbot/build/mariadb-10.2.24/sql/slave.cc:6647
          #7 0x558f070b4ff4 in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.2.24/sql/slave.cc:4336
          #8 0x558f07d06a93 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.24/storage/perfschema/pfs.cc:1862
          #9 0x7fe6eca366b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
       
      Thread T28 created by T0 here:
          #0 0x7fe6ee1b2253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
          #1 0x558f07d06e80 in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.2.24/storage/perfschema/pfs.cc:1912
          #2 0x558f0703d38e in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.2.24/include/mysql/psi/mysql_thread.h:1239
          #3 0x558f07051995 in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld.cc:6482
          #4 0x558f07052095 in create_new_thread /home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld.cc:6552
          #5 0x558f070530d8 in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld.cc:6827
          #6 0x558f07050ee0 in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld.cc:6101
          #7 0x558f0703b79f in main /home/buildbot/buildbot/build/mariadb-10.2.24/sql/main.cc:25
          #8 0x7fe6ebde582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
       
      Thread T33 created by T28 here:
          #0 0x7fe6ee1b2253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
          #1 0x558f07d06e80 in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.2.24/storage/perfschema/pfs.cc:1912
          #2 0x558f0709e39b in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.2.24/include/mysql/psi/mysql_thread.h:1239
          #3 0x558f070a2de6 in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/slave.cc:931
          #4 0x558f070a36d8 in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/slave.cc:1053
          #5 0x558f072bb561 in start_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_repl.cc:3117
          #6 0x558f075424a3 in Master_info_index::start_all_slaves(THD*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/rpl_mi.cc:1644
          #7 0x558f0723d71d in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:4174
          #8 0x558f07255d52 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:8013
          #9 0x558f07231407 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:1832
          #10 0x558f0722e586 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_parse.cc:1386
          #11 0x558f07556367 in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_connect.cc:1335
          #12 0x558f07555d6f in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.24/sql/sql_connect.cc:1241
          #13 0x558f07d06a93 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.24/storage/perfschema/pfs.cc:1862
          #14 0x7fe6eca366b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
       
      SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.24/strings/strend.c:45 strend
      Shadow bytes around the buggy address:
        0x0c2080006f40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c2080006f50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c2080006f60: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
        0x0c2080006f70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
        0x0c2080006f80: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
      =>0x0c2080006f90: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
        0x0c2080006fa0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
        0x0c2080006fb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
        0x0c2080006fc0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
        0x0c2080006fd0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
        0x0c2080006fe0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
        Heap right redzone:      fb
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack partial redzone:   f4
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
      ==16830==ABORTING
      190327  9:40:07 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.24-MariaDB-debug-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=1
      max_threads=153
      thread_count=8
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63102 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x62a00008a270
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fe6d52c3c70 thread_stack 0x5b000
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077)[0x7fe6ee1c6077]
      /home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld(my_print_stacktrace+0xab)[0x558f0880a61f]
      /home/buildbot/buildbot/build/mariadb-10.2.24/sql/mysqld(handle_fatal_signal+0x8da)[0x558f077fa7b8]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fe6eca40390]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fe6ebdfa428]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fe6ebdfc02a]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xaddc9)[0x7fe6ee229dc9]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa0769)[0x7fe6ee21c769]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa55d2)[0x7fe6ee2215d2]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(__asan_report_error+0x416)[0x7fe6ee21b6e6]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(__asan_report_load1+0x33)[0x7fe6ee21cb13]
      strings/strend.c:45(strend)[0x558f088eb90e]
      sql/sql_show.cc:3430(get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*))[0x558f073abacc]
      sql/sql_show.cc:3573(show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*))[0x558f073ac938]
      sql/sql_show.cc:7496(fill_variables(THD*, TABLE_LIST*, Item*))[0x558f073da7f1]
      sql/sql_show.cc:8425(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x558f073e276c]
      sql/sql_select.cc:3586(JOIN::exec_inner())[0x558f072e4e39]
      sql/sql_select.cc:3419(JOIN::exec())[0x558f072e316c]
      sql/sql_select.cc:3820(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x558f072e64b9]
      sql/sql_select.cc:376(handle_select(THD*, LEX*, select_result*, unsigned long))[0x558f072c5c7f]
      sql/sql_parse.cc:6479(execute_sqlcom_select(THD*, TABLE_LIST*))[0x558f0724d5ef]
      sql/sql_parse.cc:3537(mysql_execute_command(THD*))[0x558f0723aab5]
      sql/sp_head.cc:3246(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x558f07b98bc6]
      sql/sp_head.cc:3009(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x558f07b97849]
      sql/sp_head.cc:3162(sp_instr_stmt::execute(THD*, unsigned int*))[0x558f07b984a3]
      sql/sp_head.cc:1327(sp_head::execute(THD*, bool))[0x558f07b8de44]
      sql/sp_head.cc:2116(sp_head::execute_procedure(THD*, List<Item>*))[0x558f07b91c9b]
      sql/sql_parse.cc:2956(do_execute_sp(THD*, sp_head*))[0x558f07236f59]
      sql/sql_parse.cc:5827(mysql_execute_command(THD*))[0x558f07248a8a]
      sql/sql_parse.cc:8013(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x558f07255d53]
      sql/sql_parse.cc:1834(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x558f07231408]
      sql/sql_parse.cc:1386(do_command(THD*))[0x558f0722e587]
      sql/sql_connect.cc:1335(do_handle_one_connection(CONNECT*))[0x558f07556368]
      sql/sql_connect.cc:1242(handle_one_connection)[0x558f07555d70]
      perfschema/pfs.cc:1864(pfs_spawn_thread)[0x558f07d06a94]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fe6eca366ba]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe6ebecb82d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x62500027b188): SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES     WHERE variable_name NOT IN ('timestamp', 'innodb_file_format_max')      AND variable_name not like "Last_IO_Err*"      AND variable_name != 'INNODB_IBUF_MAX_SIZE'      AND variable_name != 'INNODB_USE_NATIVE_AIO'      AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP'      AND variable_name not like 'GTID%POS'      AND variable_name != 'GTID_BINLOG_STATE'    ORDER BY variable_name
      Connection ID (thread ID): 18
      Status: NOT_KILLED
      

      Attachments

        Issue Links

          Activity

            alice Alice Sherepa added a comment -

            Similar failure with multi_source.info_logs :
            ./mtr --repeat=200 --parallel=auto multi_source.info_logs --force-restart

            10.3 cba9ed12790727c703

            multi_source.info_logs                   w1 [ 28 fail ]
                    Test ended at 2020-01-10 12:21:05
             
            CURRENT_TEST: multi_source.info_logs
             
            The server [mysqld.3 - pid: 12813, winpid: 12813, exit: 256] crashed while running 'check testcase after test'
            Server log from this test:
            ----------SERVER LOG START-----------
            2020-01-10 12:21:02 0 [Note] /10.3/sql/mysqld (mysqld 10.3.22-MariaDB-debug-log) starting as process 12610 ...
            2020-01-10 12:21:02 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186)
            2020-01-10 12:21:02 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
            2020-01-10 12:21:02 0 [Warning] setrlimit could not change the size of core files to 'infinity';  We may not be able to generate a core file on signals
            2020-01-10 12:21:02 0 [Note] Plugin 'partition' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'InnoDB' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'SEQUENCE' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_LOCKS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMPMEM' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'FEEDBACK' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'user_variables' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_TRX' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_METRICS' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
            2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
            2020-01-10 12:21:02 0 [Warning] /10.3/sql/mysqld: unknown option '--loose-pam-debug'
            2020-01-10 12:21:02 0 [Note] Server socket created on IP: '127.0.0.1'.
            2020-01-10 12:21:02 0 [Note] Reading of all Master_info entries succeeded
            2020-01-10 12:21:02 0 [Note] Added new Master_info '' to hash table
            2020-01-10 12:21:02 0 [Note] /10.3/sql/mysqld: ready for connections.
            Version: '10.3.22-MariaDB-debug-log'  socket: '/10.3/mysql-test/var/tmp/1/mysqld.3.sock'  port: 16002  Source distribution
            2020-01-10 12:21:03 4 [Note] Added new Master_info 'master1' to hash table
            2020-01-10 12:21:03 4 [Note] Master connection name: 'master1'  Master_info_file: 'master-master1.info'  Relay_info_file: 'relay.bin-master1.info'
            2020-01-10 12:21:03 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'.
            2020-01-10 12:21:03 5 [Note] Master 'master1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log '' at position 4
            2020-01-10 12:21:03 6 [Note] Master 'master1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './relay-master1.000001' position: 4
            2020-01-10 12:21:03 5 [Note] Master 'master1': Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4
            2020-01-10 12:21:03 4 [Note] Added new Master_info 'MASTER 2.2' to hash table
            2020-01-10 12:21:03 4 [Note] Master connection name: 'MASTER 2.2'  Master_info_file: 'master-master@00202@002e2.info'  Relay_info_file: 'relay.bin-master@00202@002e2.info'
            2020-01-10 12:21:03 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16001', master_log_file='', master_log_pos='4'.
            2020-01-10 12:21:03 7 [Note] Master 'MASTER 2.2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16001' in log '' at position 4
            2020-01-10 12:21:03 7 [Note] Master 'MASTER 2.2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'FIRST' at position 4
            2020-01-10 12:21:03 8 [Note] Master 'MASTER 2.2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './relay-master@00202@002e2.000001' position: 4
            2020-01-10 12:21:03 6 [Note] Master 'master1': Error reading relay log event: slave SQL thread was killed
            2020-01-10 12:21:03 6 [Note] Master 'master1': Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
            2020-01-10 12:21:03 5 [Note] Master 'master1': Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
            2020-01-10 12:21:03 4 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/master-master1.info'.
            2020-01-10 12:21:03 4 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/relay.bin-master1.info'.
            2020-01-10 12:21:03 4 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay.bin.info'
            2020-01-10 12:21:03 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'.
            2020-01-10 12:21:03 9 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log '' at position 4
            2020-01-10 12:21:03 9 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4
            2020-01-10 12:21:03 10 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './relay.000001' position: 4
            2020-01-10 12:21:03 0 [Note] /10.3/sql/mysqld (initiated by: root[root] @ localhost [127.0.0.1]): Normal shutdown
            2020-01-10 12:21:03 0 [Note] Event Scheduler: Purging the queue. 0 events
            2020-01-10 12:21:03 8 [Note] Master 'MASTER 2.2': Error reading relay log event: slave SQL thread was killed
            2020-01-10 12:21:03 8 [Note] Master 'MASTER 2.2': Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
            2020-01-10 12:21:03 7 [Note] Master 'MASTER 2.2': Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
            2020-01-10 12:21:03 10 [Note] Error reading relay log event: slave SQL thread was killed
            2020-01-10 12:21:03 10 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
            2020-01-10 12:21:03 9 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
            2020-01-10 12:21:03 0 [Note] Debug sync points hit:                   2770
            2020-01-10 12:21:03 0 [Note] Debug sync points executed:              0
            2020-01-10 12:21:03 0 [Note] Debug sync points max active per thread: 0
            2020-01-10 12:21:03 0 [Note] /10.3/sql/mysqld: Shutdown complete
             
            2020-01-10 12:21:03 0 [Note] /10.3/sql/mysqld (mysqld 10.3.22-MariaDB-debug-log) starting as process 12814 ...
            2020-01-10 12:21:03 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186)
            2020-01-10 12:21:03 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
            2020-01-10 12:21:03 0 [Warning] setrlimit could not change the size of core files to 'infinity';  We may not be able to generate a core file on signals
            2020-01-10 12:21:03 0 [Note] Plugin 'partition' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'InnoDB' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'SEQUENCE' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_LOCKS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMPMEM' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'FEEDBACK' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'user_variables' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_TRX' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_METRICS' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
            2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
            2020-01-10 12:21:04 0 [Warning] /10.3/sql/mysqld: unknown option '--loose-pam-debug'
            2020-01-10 12:21:04 0 [Note] Server socket created on IP: '127.0.0.1'.
            2020-01-10 12:21:04 2 [Note] Reading Master_info: 'master-master@00202@002e2.info'  Relay_info:'relay.bin-master@00202@002e2.info'
            2020-01-10 12:21:04 2 [Note] Initialized Master_info from 'master-master@00202@002e2.info'
            2020-01-10 12:21:04 2 [Note] Added new Master_info 'MASTER 2.2' to hash table
            2020-01-10 12:21:04 3 [Note] Master 'MASTER 2.2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16001' in log 'master-bin.000001' at position 329
            2020-01-10 12:21:04 2 [Note] Started replication for 'MASTER 2.2'
            2020-01-10 12:21:04 0 [Note] Reading of all Master_info entries succeeded
            2020-01-10 12:21:04 4 [Note] Master 'MASTER 2.2': Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './relay-master@00202@002e2.000002' position: 629
            2020-01-10 12:21:04 0 [Note] Added new Master_info '' to hash table
            2020-01-10 12:21:04 6 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log 'master-bin.000001' at position 329
            2020-01-10 12:21:04 0 [Note] /10.3/sql/mysqld: ready for connections.
            Version: '10.3.22-MariaDB-debug-log'  socket: '/10.3/mysql-test/var/tmp/1/mysqld.3.sock'  port: 16002  Source distribution
            2020-01-10 12:21:04 7 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './relay.000002' position: 629
            2020-01-10 12:21:04 3 [Note] Master 'MASTER 2.2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'master-bin.000001' at position 329
            2020-01-10 12:21:04 6 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'master-bin.000001' at position 329
            2020-01-10 12:21:05 7 [Note] Error reading relay log event: slave SQL thread was killed
            2020-01-10 12:21:05 7 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
            2020-01-10 12:21:05 6 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
            2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/master.info'.
            2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/relay.bin.info'.
            2020-01-10 12:21:05 4 [Note] Master 'MASTER 2.2': Error reading relay log event: slave SQL thread was killed
            2020-01-10 12:21:05 4 [Note] Master 'MASTER 2.2': Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
            2020-01-10 12:21:05 3 [Note] Master 'MASTER 2.2': Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
            2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/master-master@00202@002e2.info'.
            2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/relay.bin-master@00202@002e2.info'.
            =================================================================
            ==12814==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000166d0 at pc 0x00000234aec1 bp 0x7f319c304180 sp 0x7f319c304170
            READ of size 1 at 0x60f0000166d0 thread T9
                #0 0x234aec0 in strend /10.3/strings/strend.c:45
                #1 0xa75b1f in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /10.3/sql/sql_show.cc:3652
                #2 0xa76a0e in show_status_array /10.3/sql/sql_show.cc:3795
                #3 0xaa6233 in fill_variables(THD*, TABLE_LIST*, Item*) /10.3/sql/sql_show.cc:7852
                #4 0xaaef5e in get_schema_tables_result(JOIN*, enum_schema_table_state) /10.3/sql/sql_show.cc:8873
                #5 0x9a2cb3 in JOIN::exec_inner() /10.3/sql/sql_select.cc:4061
                #6 0x9a0f8b in JOIN::exec() /10.3/sql/sql_select.cc:3892
                #7 0x9a45ff in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /10.3/sql/sql_select.cc:4297
                #8 0x97e43b in handle_select(THD*, LEX*, select_result*, unsigned long) /10.3/sql/sql_select.cc:382
                #9 0x8fee8c in execute_sqlcom_select /10.3/sql/sql_parse.cc:6294
                #10 0x8ee689 in mysql_execute_command(THD*) /10.3/sql/sql_parse.cc:3821
                #11 0x6fb55b in sp_instr_stmt::exec_core(THD*, unsigned int*) /10.3/sql/sp_head.cc:3594
                #12 0x6f9eb9 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /10.3/sql/sp_head.cc:3326
                #13 0x6fad2e in sp_instr_stmt::execute(THD*, unsigned int*) /10.3/sql/sp_head.cc:3500
                #14 0x6edb14 in sp_head::execute(THD*, bool) /10.3/sql/sp_head.cc:1356
                #15 0x6f2eeb in sp_head::execute_procedure(THD*, List<Item>*) /10.3/sql/sp_head.cc:2296
                #16 0x8e8b4d in do_execute_sp /10.3/sql/sql_parse.cc:2992
                #17 0x8ea67b in Sql_cmd_call::execute(THD*) /10.3/sql/sql_parse.cc:3232
                #18 0x8fcb92 in mysql_execute_command(THD*) /10.3/sql/sql_parse.cc:6031
                #19 0x907f0b in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /10.3/sql/sql_parse.cc:7818
                #20 0x8e28ca in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /10.3/sql/sql_parse.cc:1856
                #21 0x8df8ba in do_command(THD*) /10.3/sql/sql_parse.cc:1402
                #22 0xc54c87 in do_handle_one_connection(CONNECT*) /10.3/sql/sql_connect.cc:1403
                #23 0xc54640 in handle_one_connection /10.3/sql/sql_connect.cc:1308
                #24 0x20978f2 in pfs_spawn_thread /10.3/storage/perfschema/pfs.cc:1862
                #25 0x7f31a7cb76b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
                #26 0x7f31a714c41c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
             
            0x60f0000166d0 is located 112 bytes inside of 164-byte region [0x60f000016660,0x60f000016704)
            freed by thread T5 here:
                #0 0x7f31a8ded2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
                #1 0x228950b in free_memory /10.3/mysys/safemalloc.c:279
                #2 0x2288ba1 in sf_free /10.3/mysys/safemalloc.c:197
                #3 0x2259bb1 in my_free /10.3/mysys/my_malloc.c:223
                #4 0xf4cf97 in mysql_close_free_options /10.3/sql-common/client.c:3585
                #5 0xf4db1e in mysql_close /10.3/sql-common/client.c:3748
                #6 0x6cda50 in handle_slave_io /10.3/sql/slave.cc:5010
                #7 0x20978f2 in pfs_spawn_thread /10.3/storage/perfschema/pfs.cc:1862
                #8 0x7f31a7cb76b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
             
            previously allocated by thread T5 here:
                #0 0x7f31a8ded602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
                #1 0x228859f in sf_malloc /10.3/mysys/safemalloc.c:118
                #2 0x2259242 in my_malloc /10.3/mysys/my_malloc.c:101
                #3 0x2259eef in my_strdup /10.3/mysys/my_malloc.c:246
                #4 0xf50fd1 in mysql_options /10.3/sql-common/client.c:4094
                #5 0x6da8fb in connect_to_master /10.3/sql/slave.cc:7183
                #6 0x6da11d in safe_connect /10.3/sql/slave.cc:7113
                #7 0x6cc1c8 in handle_slave_io /10.3/sql/slave.cc:4683
                #8 0x20978f2 in pfs_spawn_thread /10.3/storage/perfschema/pfs.cc:1862
                #9 0x7f31a7cb76b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
             
            Thread T9 created by T0 here:
                #0 0x7f31a8d8b253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
                #1 0x2097cdf in spawn_thread_v1 /10.3/storage/perfschema/pfs.cc:1912
                #2 0x647226 in inline_mysql_thread_create /10.3/include/mysql/psi/mysql_thread.h:1268
                #3 0x65ca0f in create_thread_to_handle_connection(CONNECT*) /10.3/sql/mysqld.cc:6600
                #4 0x65d133 in create_new_thread /10.3/sql/mysqld.cc:6670
                #5 0x65e193 in handle_connections_sockets() /10.3/sql/mysqld.cc:6945
                #6 0x65be69 in mysqld_main(int, char**) /10.3/sql/mysqld.cc:6222
                #7 0x6458c5 in main /10.3/sql/main.cc:25
                #8 0x7f31a706582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
             
            Thread T5 created by T0 here:
                #0 0x7f31a8d8b253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
                #1 0x2097cdf in spawn_thread_v1 /10.3/storage/perfschema/pfs.cc:1912
                #2 0x6b18f6 in inline_mysql_thread_create /10.3/include/mysql/psi/mysql_thread.h:1268
                #3 0x6b8b38 in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /10.3/sql/slave.cc:1246
                #4 0x6b949a in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /10.3/sql/slave.cc:1369
                #5 0xc3cb10 in Master_info_index::init_all_master_info() /10.3/sql/rpl_mi.cc:1200
                #6 0x6b5465 in init_slave() /10.3/sql/slave.cc:698
                #7 0x65b9a8 in mysqld_main(int, char**) /10.3/sql/mysqld.cc:6162
                #8 0x6458c5 in main /10.3/sql/main.cc:25
                #9 0x7f31a706582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
             
            SUMMARY: AddressSanitizer: heap-use-after-free /10.3/strings/strend.c:45 strend
            Shadow bytes around the buggy address:
              0x0c1e7fffac80: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
              0x0c1e7fffac90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
              0x0c1e7fffaca0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fd fd
              0x0c1e7fffacb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
              0x0c1e7fffacc0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd
            =>0x0c1e7fffacd0: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
              0x0c1e7ffface0: fd fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
              0x0c1e7fffacf0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
              0x0c1e7fffad00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c1e7fffad10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c1e7fffad20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable:           00
              Partially addressable: 01 02 03 04 05 06 07 
              Heap left redzone:       fa
              Heap right redzone:      fb
              Freed heap region:       fd
              Stack left redzone:      f1
              Stack mid redzone:       f2
              Stack right redzone:     f3
              Stack partial redzone:   f4
              Stack after return:      f5
              Stack use after scope:   f8
              Global redzone:          f9
              Global init order:       f6
              Poisoned by user:        f7
              Container overflow:      fc
              Array cookie:            ac
              Intra object redzone:    bb
              ASan internal:           fe
            ==12814==ABORTING
            ----------SERVER LOG END-------------
            mysqltest failed but provided no output
            The result from queries just before the failure was:
            < snip >
            #   while 'MASTER 2.2' and '' are running
            master-master@00202@002e2.info
            master.info
            multi-master.info
            relay.bin-master@00202@002e2.info
            relay.bin.info
            # End of list
            #
            # Contents of multi-master.info
            MASTER 2.2
            # EOF
            #
            include/reset_master_slave.inc
            disconnect slave;
            connection master1;
            include/reset_master_slave.inc
            disconnect master1;
            connection master2;
            include/reset_master_slave.inc
            disconnect master2;
            

            alice Alice Sherepa added a comment - Similar failure with multi_source.info_logs : ./mtr --repeat=200 --parallel=auto multi_source.info_logs --force-restart 10.3 cba9ed12790727c703 multi_source.info_logs w1 [ 28 fail ] Test ended at 2020-01-10 12:21:05   CURRENT_TEST: multi_source.info_logs   The server [mysqld.3 - pid: 12813, winpid: 12813, exit: 256] crashed while running 'check testcase after test' Server log from this test: ----------SERVER LOG START----------- 2020-01-10 12:21:02 0 [Note] /10.3/sql/mysqld (mysqld 10.3.22-MariaDB-debug-log) starting as process 12610 ... 2020-01-10 12:21:02 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186) 2020-01-10 12:21:02 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2020-01-10 12:21:02 0 [Warning] setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals 2020-01-10 12:21:02 0 [Note] Plugin 'partition' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'InnoDB' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'SEQUENCE' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_LOCKS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMPMEM' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'user_variables' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_TRX' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_METRICS' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2020-01-10 12:21:02 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2020-01-10 12:21:02 0 [Warning] /10.3/sql/mysqld: unknown option '--loose-pam-debug' 2020-01-10 12:21:02 0 [Note] Server socket created on IP: '127.0.0.1'. 2020-01-10 12:21:02 0 [Note] Reading of all Master_info entries succeeded 2020-01-10 12:21:02 0 [Note] Added new Master_info '' to hash table 2020-01-10 12:21:02 0 [Note] /10.3/sql/mysqld: ready for connections. Version: '10.3.22-MariaDB-debug-log' socket: '/10.3/mysql-test/var/tmp/1/mysqld.3.sock' port: 16002 Source distribution 2020-01-10 12:21:03 4 [Note] Added new Master_info 'master1' to hash table 2020-01-10 12:21:03 4 [Note] Master connection name: 'master1' Master_info_file: 'master-master1.info' Relay_info_file: 'relay.bin-master1.info' 2020-01-10 12:21:03 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'. 2020-01-10 12:21:03 5 [Note] Master 'master1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log '' at position 4 2020-01-10 12:21:03 6 [Note] Master 'master1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './relay-master1.000001' position: 4 2020-01-10 12:21:03 5 [Note] Master 'master1': Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4 2020-01-10 12:21:03 4 [Note] Added new Master_info 'MASTER 2.2' to hash table 2020-01-10 12:21:03 4 [Note] Master connection name: 'MASTER 2.2' Master_info_file: 'master-master@00202@002e2.info' Relay_info_file: 'relay.bin-master@00202@002e2.info' 2020-01-10 12:21:03 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16001', master_log_file='', master_log_pos='4'. 2020-01-10 12:21:03 7 [Note] Master 'MASTER 2.2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16001' in log '' at position 4 2020-01-10 12:21:03 7 [Note] Master 'MASTER 2.2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'FIRST' at position 4 2020-01-10 12:21:03 8 [Note] Master 'MASTER 2.2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './relay-master@00202@002e2.000001' position: 4 2020-01-10 12:21:03 6 [Note] Master 'master1': Error reading relay log event: slave SQL thread was killed 2020-01-10 12:21:03 6 [Note] Master 'master1': Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-01-10 12:21:03 5 [Note] Master 'master1': Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-01-10 12:21:03 4 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/master-master1.info'. 2020-01-10 12:21:03 4 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/relay.bin-master1.info'. 2020-01-10 12:21:03 4 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay.bin.info' 2020-01-10 12:21:03 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'. 2020-01-10 12:21:03 9 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log '' at position 4 2020-01-10 12:21:03 9 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4 2020-01-10 12:21:03 10 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './relay.000001' position: 4 2020-01-10 12:21:03 0 [Note] /10.3/sql/mysqld (initiated by: root[root] @ localhost [127.0.0.1]): Normal shutdown 2020-01-10 12:21:03 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-01-10 12:21:03 8 [Note] Master 'MASTER 2.2': Error reading relay log event: slave SQL thread was killed 2020-01-10 12:21:03 8 [Note] Master 'MASTER 2.2': Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-01-10 12:21:03 7 [Note] Master 'MASTER 2.2': Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-01-10 12:21:03 10 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-10 12:21:03 10 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-01-10 12:21:03 9 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-01-10 12:21:03 0 [Note] Debug sync points hit: 2770 2020-01-10 12:21:03 0 [Note] Debug sync points executed: 0 2020-01-10 12:21:03 0 [Note] Debug sync points max active per thread: 0 2020-01-10 12:21:03 0 [Note] /10.3/sql/mysqld: Shutdown complete   2020-01-10 12:21:03 0 [Note] /10.3/sql/mysqld (mysqld 10.3.22-MariaDB-debug-log) starting as process 12814 ... 2020-01-10 12:21:03 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186) 2020-01-10 12:21:03 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2020-01-10 12:21:03 0 [Warning] setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals 2020-01-10 12:21:03 0 [Note] Plugin 'partition' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'InnoDB' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'SEQUENCE' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_LOCKS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMPMEM' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'user_variables' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_TRX' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_METRICS' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2020-01-10 12:21:04 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2020-01-10 12:21:04 0 [Warning] /10.3/sql/mysqld: unknown option '--loose-pam-debug' 2020-01-10 12:21:04 0 [Note] Server socket created on IP: '127.0.0.1'. 2020-01-10 12:21:04 2 [Note] Reading Master_info: 'master-master@00202@002e2.info' Relay_info:'relay.bin-master@00202@002e2.info' 2020-01-10 12:21:04 2 [Note] Initialized Master_info from 'master-master@00202@002e2.info' 2020-01-10 12:21:04 2 [Note] Added new Master_info 'MASTER 2.2' to hash table 2020-01-10 12:21:04 3 [Note] Master 'MASTER 2.2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16001' in log 'master-bin.000001' at position 329 2020-01-10 12:21:04 2 [Note] Started replication for 'MASTER 2.2' 2020-01-10 12:21:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-10 12:21:04 4 [Note] Master 'MASTER 2.2': Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './relay-master@00202@002e2.000002' position: 629 2020-01-10 12:21:04 0 [Note] Added new Master_info '' to hash table 2020-01-10 12:21:04 6 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log 'master-bin.000001' at position 329 2020-01-10 12:21:04 0 [Note] /10.3/sql/mysqld: ready for connections. Version: '10.3.22-MariaDB-debug-log' socket: '/10.3/mysql-test/var/tmp/1/mysqld.3.sock' port: 16002 Source distribution 2020-01-10 12:21:04 7 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './relay.000002' position: 629 2020-01-10 12:21:04 3 [Note] Master 'MASTER 2.2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'master-bin.000001' at position 329 2020-01-10 12:21:04 6 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'master-bin.000001' at position 329 2020-01-10 12:21:05 7 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-10 12:21:05 7 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-01-10 12:21:05 6 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/master.info'. 2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/relay.bin.info'. 2020-01-10 12:21:05 4 [Note] Master 'MASTER 2.2': Error reading relay log event: slave SQL thread was killed 2020-01-10 12:21:05 4 [Note] Master 'MASTER 2.2': Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-01-10 12:21:05 3 [Note] Master 'MASTER 2.2': Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/master-master@00202@002e2.info'. 2020-01-10 12:21:05 8 [Note] Deleted Master_info file '/dev/shm/var_auto_Z_GE/1/mysqld.3/data/relay.bin-master@00202@002e2.info'. ================================================================= ==12814==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000166d0 at pc 0x00000234aec1 bp 0x7f319c304180 sp 0x7f319c304170 READ of size 1 at 0x60f0000166d0 thread T9 #0 0x234aec0 in strend /10.3/strings/strend.c:45 #1 0xa75b1f in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /10.3/sql/sql_show.cc:3652 #2 0xa76a0e in show_status_array /10.3/sql/sql_show.cc:3795 #3 0xaa6233 in fill_variables(THD*, TABLE_LIST*, Item*) /10.3/sql/sql_show.cc:7852 #4 0xaaef5e in get_schema_tables_result(JOIN*, enum_schema_table_state) /10.3/sql/sql_show.cc:8873 #5 0x9a2cb3 in JOIN::exec_inner() /10.3/sql/sql_select.cc:4061 #6 0x9a0f8b in JOIN::exec() /10.3/sql/sql_select.cc:3892 #7 0x9a45ff in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /10.3/sql/sql_select.cc:4297 #8 0x97e43b in handle_select(THD*, LEX*, select_result*, unsigned long) /10.3/sql/sql_select.cc:382 #9 0x8fee8c in execute_sqlcom_select /10.3/sql/sql_parse.cc:6294 #10 0x8ee689 in mysql_execute_command(THD*) /10.3/sql/sql_parse.cc:3821 #11 0x6fb55b in sp_instr_stmt::exec_core(THD*, unsigned int*) /10.3/sql/sp_head.cc:3594 #12 0x6f9eb9 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /10.3/sql/sp_head.cc:3326 #13 0x6fad2e in sp_instr_stmt::execute(THD*, unsigned int*) /10.3/sql/sp_head.cc:3500 #14 0x6edb14 in sp_head::execute(THD*, bool) /10.3/sql/sp_head.cc:1356 #15 0x6f2eeb in sp_head::execute_procedure(THD*, List<Item>*) /10.3/sql/sp_head.cc:2296 #16 0x8e8b4d in do_execute_sp /10.3/sql/sql_parse.cc:2992 #17 0x8ea67b in Sql_cmd_call::execute(THD*) /10.3/sql/sql_parse.cc:3232 #18 0x8fcb92 in mysql_execute_command(THD*) /10.3/sql/sql_parse.cc:6031 #19 0x907f0b in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /10.3/sql/sql_parse.cc:7818 #20 0x8e28ca in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /10.3/sql/sql_parse.cc:1856 #21 0x8df8ba in do_command(THD*) /10.3/sql/sql_parse.cc:1402 #22 0xc54c87 in do_handle_one_connection(CONNECT*) /10.3/sql/sql_connect.cc:1403 #23 0xc54640 in handle_one_connection /10.3/sql/sql_connect.cc:1308 #24 0x20978f2 in pfs_spawn_thread /10.3/storage/perfschema/pfs.cc:1862 #25 0x7f31a7cb76b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) #26 0x7f31a714c41c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)   0x60f0000166d0 is located 112 bytes inside of 164-byte region [0x60f000016660,0x60f000016704) freed by thread T5 here: #0 0x7f31a8ded2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca) #1 0x228950b in free_memory /10.3/mysys/safemalloc.c:279 #2 0x2288ba1 in sf_free /10.3/mysys/safemalloc.c:197 #3 0x2259bb1 in my_free /10.3/mysys/my_malloc.c:223 #4 0xf4cf97 in mysql_close_free_options /10.3/sql-common/client.c:3585 #5 0xf4db1e in mysql_close /10.3/sql-common/client.c:3748 #6 0x6cda50 in handle_slave_io /10.3/sql/slave.cc:5010 #7 0x20978f2 in pfs_spawn_thread /10.3/storage/perfschema/pfs.cc:1862 #8 0x7f31a7cb76b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)   previously allocated by thread T5 here: #0 0x7f31a8ded602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602) #1 0x228859f in sf_malloc /10.3/mysys/safemalloc.c:118 #2 0x2259242 in my_malloc /10.3/mysys/my_malloc.c:101 #3 0x2259eef in my_strdup /10.3/mysys/my_malloc.c:246 #4 0xf50fd1 in mysql_options /10.3/sql-common/client.c:4094 #5 0x6da8fb in connect_to_master /10.3/sql/slave.cc:7183 #6 0x6da11d in safe_connect /10.3/sql/slave.cc:7113 #7 0x6cc1c8 in handle_slave_io /10.3/sql/slave.cc:4683 #8 0x20978f2 in pfs_spawn_thread /10.3/storage/perfschema/pfs.cc:1862 #9 0x7f31a7cb76b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)   Thread T9 created by T0 here: #0 0x7f31a8d8b253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253) #1 0x2097cdf in spawn_thread_v1 /10.3/storage/perfschema/pfs.cc:1912 #2 0x647226 in inline_mysql_thread_create /10.3/include/mysql/psi/mysql_thread.h:1268 #3 0x65ca0f in create_thread_to_handle_connection(CONNECT*) /10.3/sql/mysqld.cc:6600 #4 0x65d133 in create_new_thread /10.3/sql/mysqld.cc:6670 #5 0x65e193 in handle_connections_sockets() /10.3/sql/mysqld.cc:6945 #6 0x65be69 in mysqld_main(int, char**) /10.3/sql/mysqld.cc:6222 #7 0x6458c5 in main /10.3/sql/main.cc:25 #8 0x7f31a706582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)   Thread T5 created by T0 here: #0 0x7f31a8d8b253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253) #1 0x2097cdf in spawn_thread_v1 /10.3/storage/perfschema/pfs.cc:1912 #2 0x6b18f6 in inline_mysql_thread_create /10.3/include/mysql/psi/mysql_thread.h:1268 #3 0x6b8b38 in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /10.3/sql/slave.cc:1246 #4 0x6b949a in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /10.3/sql/slave.cc:1369 #5 0xc3cb10 in Master_info_index::init_all_master_info() /10.3/sql/rpl_mi.cc:1200 #6 0x6b5465 in init_slave() /10.3/sql/slave.cc:698 #7 0x65b9a8 in mysqld_main(int, char**) /10.3/sql/mysqld.cc:6162 #8 0x6458c5 in main /10.3/sql/main.cc:25 #9 0x7f31a706582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)   SUMMARY: AddressSanitizer: heap-use-after-free /10.3/strings/strend.c:45 strend Shadow bytes around the buggy address: 0x0c1e7fffac80: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa 0x0c1e7fffac90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c1e7fffaca0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fd fd 0x0c1e7fffacb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c1e7fffacc0: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd =>0x0c1e7fffacd0: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd 0x0c1e7ffface0: fd fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd 0x0c1e7fffacf0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c1e7fffad00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c1e7fffad10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c1e7fffad20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe ==12814==ABORTING ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > # while 'MASTER 2.2' and '' are running master-master@00202@002e2.info master.info multi-master.info relay.bin-master@00202@002e2.info relay.bin.info # End of list # # Contents of multi-master.info MASTER 2.2 # EOF # include/reset_master_slave.inc disconnect slave; connection master1; include/reset_master_slave.inc disconnect master1; connection master2; include/reset_master_slave.inc disconnect master2;
            alice Alice Sherepa added a comment -

            After the test there is
            call mtr.check_testcase() and

            SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES
            WHERE variable_name NOT IN ('timestamp')
              AND variable_name NOT LIKE "Last_IO_Err*"
              AND variable_name != 'INNODB_IBUF_MAX_SIZE'
              AND variable_name != 'INNODB_USE_NATIVE_AIO'
              AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP'
              AND variable_name NOT LIKE 'GTID%POS'
              AND variable_name != 'GTID_BINLOG_STATE'
            ORDER BY variable_name;
            

            inside of the procedure:

            10.5 98a67eccc57774a9322ed006ab3

            200113 11:22:53 [ERROR] mysqld got signal 6 ;
             
            /10.5/sql/mysqld(strend+0x47)[0x1d5dbf0]
            /10.5/sql/mysqld(_Z16get_one_variableP3THDPK17st_mysql_show_var13enum_var_type20enum_mysql_show_typeP17system_status_varPPK15charset_info_stPcPm+0x57c)[0xb99dc8]
            /10.5/sql/mysqld[0xb9acc7]
            /10.5/sql/mysqld(_Z14fill_variablesP3THDP10TABLE_LISTP4Item+0x4a3)[0xbca5c1]
            strings/strend.c:45(strend)[0xbd1c78]
            sql/sql_show.cc:3570(get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*))[0xabbdfd]
            sql/sql_show.cc:3713(show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*))[0xab9e8c]
            sql/sql_show.cc:7723(fill_variables(THD*, TABLE_LIST*, Item*))[0xabdb0b]
            sql/sql_show.cc:8597(get_schema_tables_result(JOIN*, enum_schema_table_state))[0xa92fc7]
            sql/sql_select.cc:4413(JOIN::exec_inner())[0xa10802]
            sql/sql_select.cc:4239(JOIN::exec())[0x9ff408]
            sql/sql_select.cc:4664(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x7e2bb2]
            sql/sql_select.cc:428(handle_select(THD*, LEX*, select_result*, unsigned long))[0x7e1430]
            sql/sql_parse.cc:6219(execute_sqlcom_select(THD*, TABLE_LIST*))[0x7e2369]
            sql/sql_parse.cc:3904(mysql_execute_command(THD*))[0x7d499c]
            sql/sp_head.cc:3666(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x7da351]
            sql/sp_head.cc:3398(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x9f9311]
            sql/sp_head.cc:3572(sp_instr_stmt::execute(THD*, unsigned int*))[0x9faed8]
            sql/sp_head.cc:1345(sp_head::execute(THD*, bool))[0xa0e457]
            sql/sp_head.cc:2350(sp_head::execute_procedure(THD*, List<Item>*))[0xa1a8c6]
            sql/sql_parse.cc:3019(do_execute_sp(THD*, sp_head*))[0x9f2c20]
            sql/sql_parse.cc:3262(Sql_cmd_call::execute(THD*))[0x9efada]
            sql/sql_parse.cc:5959(mysql_execute_command(THD*))[0xdb2649]
            sql/sql_parse.cc:7988(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0xdb202c]
            sql/sql_parse.cc:1848(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x1a21793]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fc3433b46ba]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc34264541d]
             
            Query (0x62500011d190): SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES     WHERE variable_name NOT IN ('timestamp')      AND variable_name not like "Last_IO_Err*"      AND variable_name != 'INNODB_IBUF_MAX_SIZE'      AND variable_name != 'INNODB_USE_NATIVE_AIO'      AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP'      AND variable_name not like 'GTID%POS'      AND variable_name != 'GTID_BINLOG_STATE'    ORDER BY variable_name
            

            alice Alice Sherepa added a comment - After the test there is call mtr.check_testcase() and SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ( 'timestamp' ) AND variable_name NOT LIKE "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name NOT LIKE 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' ORDER BY variable_name; inside of the procedure: 10.5 98a67eccc57774a9322ed006ab3 200113 11:22:53 [ERROR] mysqld got signal 6 ;   /10.5/sql/mysqld(strend+0x47)[0x1d5dbf0] /10.5/sql/mysqld(_Z16get_one_variableP3THDPK17st_mysql_show_var13enum_var_type20enum_mysql_show_typeP17system_status_varPPK15charset_info_stPcPm+0x57c)[0xb99dc8] /10.5/sql/mysqld[0xb9acc7] /10.5/sql/mysqld(_Z14fill_variablesP3THDP10TABLE_LISTP4Item+0x4a3)[0xbca5c1] strings/strend.c:45(strend)[0xbd1c78] sql/sql_show.cc:3570(get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*))[0xabbdfd] sql/sql_show.cc:3713(show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*))[0xab9e8c] sql/sql_show.cc:7723(fill_variables(THD*, TABLE_LIST*, Item*))[0xabdb0b] sql/sql_show.cc:8597(get_schema_tables_result(JOIN*, enum_schema_table_state))[0xa92fc7] sql/sql_select.cc:4413(JOIN::exec_inner())[0xa10802] sql/sql_select.cc:4239(JOIN::exec())[0x9ff408] sql/sql_select.cc:4664(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x7e2bb2] sql/sql_select.cc:428(handle_select(THD*, LEX*, select_result*, unsigned long))[0x7e1430] sql/sql_parse.cc:6219(execute_sqlcom_select(THD*, TABLE_LIST*))[0x7e2369] sql/sql_parse.cc:3904(mysql_execute_command(THD*))[0x7d499c] sql/sp_head.cc:3666(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x7da351] sql/sp_head.cc:3398(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x9f9311] sql/sp_head.cc:3572(sp_instr_stmt::execute(THD*, unsigned int*))[0x9faed8] sql/sp_head.cc:1345(sp_head::execute(THD*, bool))[0xa0e457] sql/sp_head.cc:2350(sp_head::execute_procedure(THD*, List<Item>*))[0xa1a8c6] sql/sql_parse.cc:3019(do_execute_sp(THD*, sp_head*))[0x9f2c20] sql/sql_parse.cc:3262(Sql_cmd_call::execute(THD*))[0x9efada] sql/sql_parse.cc:5959(mysql_execute_command(THD*))[0xdb2649] sql/sql_parse.cc:7988(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0xdb202c] sql/sql_parse.cc:1848(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x1a21793] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fc3433b46ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc34264541d]   Query (0x62500011d190): SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' ORDER BY variable_name
            alice Alice Sherepa added a comment -

            failed on buildbot, 10.2, multi_source.mdev-8874 test http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/2688/steps/mtr_nm/logs/stdio

            multi_source.mdev-8874 'innodb'          w1 [ fail ]
                    Test ended at 2020-01-16 06:58:54
             
            CURRENT_TEST: multi_source.mdev-8874
             
             
            The server [mysqld.4 - pid: 30154, winpid: 30154, exit: 256] crashed while running 'check testcase after test'
            Server log from this test:
            ----------SERVER LOG START-----------
            2020-01-16  6:58:47 140464650402048 [Note] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld (mysqld 10.2.31-MariaDB-debug-log) starting as process 30159 ...
            2020-01-16  6:58:47 140464650402048 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186)
            2020-01-16  6:58:47 140464650402048 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
            2020-01-16  6:58:47 140464650402048 [Warning] setrlimit could not change the size of core files to 'infinity';  We may not be able to generate a core file on signals
            2020-01-16  6:58:47 140464650402048 [Note] Plugin 'SEQUENCE' is disabled.
            2020-01-16  6:58:47 140464650402048 [Note] Plugin 'partition' is disabled.
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: Uses event mutexes
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: Compressed tables use zlib 1.2.8
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: Using Linux native AIO
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: Number of pools: 1
            2020-01-16  6:58:47 140464650402048 [Note] InnoDB: Using generic crc32 instructions
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: Completed initialization of buffer pool
            2020-01-16  6:58:48 140464367277824 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: Highest supported file format is Barracuda.
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: 128 out of 128 rollback segments are active.
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: Creating shared tablespace for temporary tables
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: Waiting for purge to start
            2020-01-16  6:58:48 140464650402048 [Note] InnoDB: 5.7.28 started; log sequence number 1628840
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_MUTEXES' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_CMP' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'FEEDBACK' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'user_variables' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
            2020-01-16  6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
            2020-01-16  6:58:48 140464650402048 [Warning] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld: unknown option '--loose-pam-debug'
            2020-01-16  6:58:48 140464270223104 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var/1/mysqld.4/data/ib_buffer_pool
            2020-01-16  6:58:48 140464270223104 [Note] InnoDB: Buffer pool(s) load completed at 200116  6:58:48
            2020-01-16  6:58:49 140464650402048 [Note] Server socket created on IP: '127.0.0.1'.
            2020-01-16  6:58:49 140464650402048 [Note] Reading of all Master_info entries succeeded
            2020-01-16  6:58:49 140464650402048 [Note] Added new Master_info '' to hash table
            2020-01-16  6:58:49 140464650402048 [Note] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld: ready for connections.
            Version: '10.2.31-MariaDB-debug-log'  socket: '/dev/shm/var/tmp/1/mysqld.4.sock'  port: 16003  Source distribution
            2020-01-16  6:58:50 140464214296320 [Note] Added new Master_info 'm1' to hash table
            2020-01-16  6:58:50 140464214296320 [Note] Master connection name: 'm1'  Master_info_file: 'master-m1.info'  Relay_info_file: 'relay-log-m1.info'
            2020-01-16  6:58:50 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'.
            2020-01-16  6:58:50 140464214296320 [Note] Added new Master_info 'm2' to hash table
            2020-01-16  6:58:50 140464214296320 [Note] Master connection name: 'm2'  Master_info_file: 'master-m2.info'  Relay_info_file: 'relay-log-m2.info'
            2020-01-16  6:58:50 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16001', master_log_file='', master_log_pos='4'.
            2020-01-16  6:58:50 140464214296320 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
            2020-01-16  6:58:50 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16002', master_log_file='', master_log_pos='4'.
            2020-01-16  6:58:50 140464210700032 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'FIRST' at position 4
            2020-01-16  6:58:50 140464210167552 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4
            2020-01-16  6:58:50 140464209102592 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4
            2020-01-16  6:58:50 140464209635072 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16002',replication started in log 'FIRST' at position 4
            2020-01-16  6:58:50 140464208570112 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4
            2020-01-16  6:58:50 140464208037632 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4
            2020-01-16  6:58:50 140464210167552 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed
            2020-01-16  6:58:50 140464210167552 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108
            2020-01-16  6:58:50 140464210700032 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108
            2020-01-16  6:58:50 140464209102592 [Note] Error reading relay log event: slave SQL thread was killed
            2020-01-16  6:58:50 140464209102592 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108
            2020-01-16  6:58:50 140464209635072 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108
            2020-01-16  6:58:50 140464208037632 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed
            2020-01-16  6:58:50 140464208037632 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108
            2020-01-16  6:58:50 140464208570112 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108
            2020-01-16  6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/master-m1.info'.
            2020-01-16  6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/relay-log-m1.info'.
            2020-01-16  6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/master-m2.info'.
            2020-01-16  6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/relay-log-m2.info'.
            2020-01-16  6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/master.info'.
            2020-01-16  6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/relay-log.info'.
            2020-01-16  6:58:50 140464214296320 [Note] Added new Master_info 'm1' to hash table
            2020-01-16  6:58:50 140464214296320 [Note] Master connection name: 'm1'  Master_info_file: 'master-m1.info'  Relay_info_file: 'relay-log-m1.info'
            2020-01-16  6:58:51 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'.
            2020-01-16  6:58:51 140464214296320 [Note] Added new Master_info 'm2' to hash table
            2020-01-16  6:58:51 140464214296320 [Note] Master connection name: 'm2'  Master_info_file: 'master-m2.info'  Relay_info_file: 'relay-log-m2.info'
            2020-01-16  6:58:51 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16001', master_log_file='', master_log_pos='4'.
            2020-01-16  6:58:51 140464214296320 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
            2020-01-16  6:58:51 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16002', master_log_file='', master_log_pos='4'.
            2020-01-16  6:58:51 140464208037632 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4
            2020-01-16  6:58:51 140464209102592 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4
            2020-01-16  6:58:51 140464208570112 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'FIRST' at position 4
            2020-01-16  6:58:51 140464209635072 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16002',replication started in log 'FIRST' at position 4
            2020-01-16  6:58:51 140464210167552 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4
            2020-01-16  6:58:51 140464210700032 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4
            2020-01-16  6:58:51 140464208037632 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed
            2020-01-16  6:58:51 140464208037632 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229
            2020-01-16  6:58:51 140464208570112 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229
            2020-01-16  6:58:51 140464209102592 [Note] Error reading relay log event: slave SQL thread was killed
            2020-01-16  6:58:51 140464209102592 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229
            2020-01-16  6:58:51 140464209635072 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229
            2020-01-16  6:58:51 140464210167552 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed
            2020-01-16  6:58:51 140464210167552 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229
            2020-01-16  6:58:51 140464210700032 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229
            =================================================================
            ==30159==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000030c38 at pc 0x000001f0cc69 bp 0x7fc05f985770 sp 0x7fc05f985760
            READ of size 1 at 0x607000030c38 thread T28
                #0 0x1f0cc68 in strend /home/buildbot/buildbot/build/mariadb-10.2.31/strings/strend.c:45
                #1 0x98276d in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:3437
                #2 0x9835a9 in show_status_array /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:3580
                #3 0x9b1a1b in fill_variables(THD*, TABLE_LIST*, Item*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:7521
                #4 0x9b9bd8 in get_schema_tables_result(JOIN*, enum_schema_table_state) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:8448
                #5 0x8b72a4 in JOIN::exec_inner() /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:3593
                #6 0x8b566d in JOIN::exec() /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:3424
                #7 0x8b8a82 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:3824
                #8 0x897bdf in handle_select(THD*, LEX*, select_result*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:373
                #9 0x81c0b4 in execute_sqlcom_select /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:6225
                #10 0x80a8c7 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:3532
                #11 0x1196db9 in sp_instr_stmt::exec_core(THD*, unsigned int*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:3248
                #12 0x11959e2 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:3011
                #13 0x1196678 in sp_instr_stmt::execute(THD*, unsigned int*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:3164
                #14 0x118bd67 in sp_head::execute(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:1329
                #15 0x118fbdd in sp_head::execute_procedure(THD*, List<Item>*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:2118
                #16 0x806e11 in do_execute_sp /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:2954
                #17 0x8175aa in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:5572
                #18 0x824b6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:7740
                #19 0x8011d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1831
                #20 0x7fe2c7 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1385
                #21 0xb37eec in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1336
                #22 0xb378d0 in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1241
                #23 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862
                #24 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
                #25 0x7fc0765d182c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)
             
            0x607000030c38 is located 8 bytes inside of 80-byte region [0x607000030c30,0x607000030c80)
            freed by thread T35 here:
                #0 0x7fc07891a2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
                #1 0x1e1e478 in my_free /home/buildbot/buildbot/build/mariadb-10.2.31/mysys/my_malloc.c:218
                #2 0xd5268e in mysql_close_free_options /home/buildbot/buildbot/build/mariadb-10.2.31/sql-common/client.c:3590
                #3 0xd53215 in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.31/sql-common/client.c:3753
                #4 0x67d45a in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:4650
                #5 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862
                #6 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
             
            previously allocated by thread T35 here:
                #0 0x7fc07891a602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
                #1 0x1e1da93 in my_malloc /home/buildbot/buildbot/build/mariadb-10.2.31/mysys/my_malloc.c:101
                #2 0x1e1e7b6 in my_strdup /home/buildbot/buildbot/build/mariadb-10.2.31/mysys/my_malloc.c:241
                #3 0xd56688 in mysql_options /home/buildbot/buildbot/build/mariadb-10.2.31/sql-common/client.c:4099
                #4 0x689044 in connect_to_master /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:6727
                #5 0x688866 in safe_connect /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:6657
                #6 0x67bc7f in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:4335
                #7 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862
                #8 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
             
            Thread T28 created by T0 here:
                #0 0x7fc0788b8253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
                #1 0x130d2aa in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1912
                #2 0x601906 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.2.31/include/mysql/psi/mysql_thread.h:1239
                #3 0x616685 in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6506
                #4 0x616da9 in create_new_thread /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6576
                #5 0x617e09 in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6851
                #6 0x615b63 in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6125
                #7 0x5ffc45 in main /home/buildbot/buildbot/build/mariadb-10.2.31/sql/main.cc:25
                #8 0x7fc0764eb82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
             
            Thread T35 created by T28 here:
                #0 0x7fc0788b8253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
                #1 0x130d2aa in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1912
                #2 0x6648a2 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.2.31/include/mysql/psi/mysql_thread.h:1239
                #3 0x66968f in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:934
                #4 0x669fce in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:1056
                #5 0x88d1e4 in start_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_repl.cc:3146
                #6 0xb2367b in Master_info_index::start_all_slaves(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/rpl_mi.cc:1635
                #7 0x80c090 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:3895
                #8 0x824b6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:7740
                #9 0x8011d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1831
                #10 0x7fe2c7 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1385
                #11 0xb37eec in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1336
                #12 0xb378d0 in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1241
                #13 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862
                #14 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
             
            SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.31/strings/strend.c:45 strend
            Shadow bytes around the buggy address:
              0x0c0e7fffe130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0e7fffe140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0e7fffe150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0e7fffe160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0e7fffe170: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
            =>0x0c0e7fffe180: fd fa fa fa fa fa fd[fd]fd fd fd fd fd fd fd fd
              0x0c0e7fffe190: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
              0x0c0e7fffe1a0: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
              0x0c0e7fffe1b0: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
              0x0c0e7fffe1c0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
              0x0c0e7fffe1d0: fd fd fd fd fd fd fa fa fa fa fd fd fd fd fd fd
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable:           00
              Partially addressable: 01 02 03 04 05 06 07 
              Heap left redzone:       fa
              Heap right redzone:      fb
              Freed heap region:       fd
              Stack left redzone:      f1
              Stack mid redzone:       f2
              Stack right redzone:     f3
              Stack partial redzone:   f4
              Stack after return:      f5
              Stack use after scope:   f8
              Global redzone:          f9
              Global init order:       f6
              Poisoned by user:        f7
              Container overflow:      fc
              Array cookie:            ac
              Intra object redzone:    bb
              ASan internal:           fe
            ==30159==ABORTING
            200116  6:58:53 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.2.31-MariaDB-debug-log
            key_buffer_size=1048576
            read_buffer_size=131072
            max_used_connections=1
            max_threads=153
            thread_count=8
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63103 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x62a00008a208
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7fc05f98bc70 thread_stack 0x5b000
            /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077)[0x7fc0788cc077]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(my_print_stacktrace+0xc6)[0x1e2e32e]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(handle_fatal_signal+0x8a9)[0xde7f7b]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fc077146390]
            /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fc076500428]
            /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fc07650202a]
            /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xaddc9)[0x7fc07892fdc9]
            /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa0769)[0x7fc078922769]
            /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa55d2)[0x7fc0789275d2]
            /usr/lib/x86_64-linux-gnu/libasan.so.2(__asan_report_error+0x416)[0x7fc0789216e6]
            /usr/lib/x86_64-linux-gnu/libasan.so.2(__asan_report_load1+0x33)[0x7fc078922b13]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(strend+0x47)[0x1f0cc69]
            fil/fil0fil.cc:1745(fil_space_close(char const*))[0x98276e]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld[0x9835aa]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z14fill_variablesP3THDP10TABLE_LISTP4Item+0x497)[0x9b1a1c]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0xbc0)[0x9b9bd9]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_ZN4JOIN10exec_innerEv+0x1a87)[0x8b72a5]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_ZN4JOIN4execEv+0x204)[0x8b566e]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x779)[0x8b8a83]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x4c5)[0x897be0]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld[0x81c0b5]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z21mysql_execute_commandP3THD+0x394b)[0x80a8c8]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x20)[0x1196dba]
            sql/field.cc:6166(Field_time::cmp(unsigned char const*, unsigned char const*))[0x11959e3]
            sql/field.cc:6087(Field_time::val_int())[0x1196679]
            sql/field.cc:6146(Field_time::get_date(st_mysql_time*, unsigned long long))[0x118bd68]
            sql/field.cc:5219(Field_timestamp::store_time_dec(st_mysql_time*, unsigned int))[0x118fbde]
            sql/field.cc:5563(read_bigendian(unsigned char const*, unsigned int))[0x806e12]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z21mysql_execute_commandP3THD+0x1062e)[0x8175ab]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x6a2)[0x824b6b]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ed3)[0x8011d2]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z10do_commandP3THD+0x115d)[0x7fe2c8]
            /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x48c)[0xb37eed]
            sql/sql_cache.cc:3173(Query_cache::write_result_data(Query_cache_block**, unsigned long, unsigned char*, Query_cache_block*, Query_cache_block::block_type))[0xb378d1]
            sql/sql_cache.cc:3099(Query_cache::append_result_data(Query_cache_block**, unsigned long, unsigned char*, Query_cache_block*))[0x130cebe]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fc07713c6ba]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc0765d182d]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x62500033b928): SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES     WHERE variable_name NOT IN ('timestamp', 'innodb_file_format_max')      AND variable_name not like "Last_IO_Err*"      AND variable_name != 'INNODB_IBUF_MAX_SIZE'      AND variable_name != 'INNODB_USE_NATIVE_AIO'      AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP'      AND variable_name not like 'GTID%POS'      AND variable_name != 'GTID_BINLOG_STATE'    ORDER BY variable_name
            Connection ID (thread ID): 24
            Status: NOT_KILLED
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
             
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/var/1/mysqld.4/data
            Resource Limits:
            Limit                     Soft Limit           Hard Limit           Units     
            Max cpu time              unlimited            unlimited            seconds   
            Max file size             unlimited            unlimited            bytes     
            Max data size             unlimited            unlimited            bytes     
            Max stack size            8388608              unlimited            bytes     
            Max core file size        0                    0                    bytes     
            Max resident set          unlimited            unlimited            bytes     
            Max processes             23715                23715                processes 
            Max open files            1024                 1024                 files     
            Max locked memory         65536                65536                bytes     
            Max address space         unlimited            unlimited            bytes     
            Max file locks            unlimited            unlimited            locks     
            Max pending signals       23715                23715                signals   
            Max msgqueue size         819200               819200               bytes     
            Max nice priority         0                    0                    
            Max realtime priority     0                    0                    
            Max realtime timeout      unlimited            unlimited            us        
            Core pattern: |/usr/share/apport/apport %p %s %c %P
             
            ----------SERVER LOG END-------------
            mysqltest failed but provided no output
            The result from queries just before the failure was:
            < snip >
            use c;
            #No t1 table
            show tables;
            Tables_in_c
            t2
            #CleanUp
            connection server_1;
            drop database a;
            connection server_2;
            drop database b;
            connection server_3;
            drop database c;
            connection server_4;
            stop all slaves;
            SET default_master_connection = "m1";
            include/wait_for_slave_to_stop.inc
            SET default_master_connection = "m2";
            include/wait_for_slave_to_stop.inc
            SET default_master_connection = "";
            include/wait_for_slave_to_stop.inc
             
             
             
             - saving '/dev/shm/var/1/log/multi_source.mdev-8874-innodb/' to '/dev/shm/var/log/multi_source.mdev-8874-innodb/'
            worker[1] > Restart  - using different config file
            worker[1] > Restart  - using different config file
            worker[1] > Restart  - using different config file
            worker[1] > Restart  - using different config file
            ***Warnings generated in error logs during shutdown after running tests: multi_source.mdev-8874
             
            ==30159==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000030c38 at pc 0x000001f0cc69 bp 0x7fc05f985770 sp 0x7fc05f985760
            SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.31/strings/strend.c:45 strend
            Attempting backtrace. You can use the following information to find out
            

            alice Alice Sherepa added a comment - failed on buildbot, 10.2, multi_source.mdev-8874 test http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/2688/steps/mtr_nm/logs/stdio multi_source.mdev-8874 'innodb' w1 [ fail ] Test ended at 2020-01-16 06:58:54   CURRENT_TEST: multi_source.mdev-8874     The server [mysqld.4 - pid: 30154, winpid: 30154, exit: 256] crashed while running 'check testcase after test' Server log from this test: ----------SERVER LOG START----------- 2020-01-16 6:58:47 140464650402048 [Note] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld (mysqld 10.2.31-MariaDB-debug-log) starting as process 30159 ... 2020-01-16 6:58:47 140464650402048 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186) 2020-01-16 6:58:47 140464650402048 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2020-01-16 6:58:47 140464650402048 [Warning] setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals 2020-01-16 6:58:47 140464650402048 [Note] Plugin 'SEQUENCE' is disabled. 2020-01-16 6:58:47 140464650402048 [Note] Plugin 'partition' is disabled. 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: Uses event mutexes 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: Compressed tables use zlib 1.2.8 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: Using Linux native AIO 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: Number of pools: 1 2020-01-16 6:58:47 140464650402048 [Note] InnoDB: Using generic crc32 instructions 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: Completed initialization of buffer pool 2020-01-16 6:58:48 140464367277824 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: Highest supported file format is Barracuda. 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: Waiting for purge to start 2020-01-16 6:58:48 140464650402048 [Note] InnoDB: 5.7.28 started; log sequence number 1628840 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_CMP' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'user_variables' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2020-01-16 6:58:48 140464650402048 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2020-01-16 6:58:48 140464650402048 [Warning] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld: unknown option '--loose-pam-debug' 2020-01-16 6:58:48 140464270223104 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var/1/mysqld.4/data/ib_buffer_pool 2020-01-16 6:58:48 140464270223104 [Note] InnoDB: Buffer pool(s) load completed at 200116 6:58:48 2020-01-16 6:58:49 140464650402048 [Note] Server socket created on IP: '127.0.0.1'. 2020-01-16 6:58:49 140464650402048 [Note] Reading of all Master_info entries succeeded 2020-01-16 6:58:49 140464650402048 [Note] Added new Master_info '' to hash table 2020-01-16 6:58:49 140464650402048 [Note] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld: ready for connections. Version: '10.2.31-MariaDB-debug-log' socket: '/dev/shm/var/tmp/1/mysqld.4.sock' port: 16003 Source distribution 2020-01-16 6:58:50 140464214296320 [Note] Added new Master_info 'm1' to hash table 2020-01-16 6:58:50 140464214296320 [Note] Master connection name: 'm1' Master_info_file: 'master-m1.info' Relay_info_file: 'relay-log-m1.info' 2020-01-16 6:58:50 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'. 2020-01-16 6:58:50 140464214296320 [Note] Added new Master_info 'm2' to hash table 2020-01-16 6:58:50 140464214296320 [Note] Master connection name: 'm2' Master_info_file: 'master-m2.info' Relay_info_file: 'relay-log-m2.info' 2020-01-16 6:58:50 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16001', master_log_file='', master_log_pos='4'. 2020-01-16 6:58:50 140464214296320 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-01-16 6:58:50 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16002', master_log_file='', master_log_pos='4'. 2020-01-16 6:58:50 140464210700032 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'FIRST' at position 4 2020-01-16 6:58:50 140464210167552 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4 2020-01-16 6:58:50 140464209102592 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4 2020-01-16 6:58:50 140464209635072 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16002',replication started in log 'FIRST' at position 4 2020-01-16 6:58:50 140464208570112 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4 2020-01-16 6:58:50 140464208037632 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4 2020-01-16 6:58:50 140464210167552 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed 2020-01-16 6:58:50 140464210167552 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108 2020-01-16 6:58:50 140464210700032 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108 2020-01-16 6:58:50 140464209102592 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-16 6:58:50 140464209102592 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108 2020-01-16 6:58:50 140464209635072 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108 2020-01-16 6:58:50 140464208037632 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed 2020-01-16 6:58:50 140464208037632 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108 2020-01-16 6:58:50 140464208570112 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108 2020-01-16 6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/master-m1.info'. 2020-01-16 6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/relay-log-m1.info'. 2020-01-16 6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/master-m2.info'. 2020-01-16 6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/relay-log-m2.info'. 2020-01-16 6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/master.info'. 2020-01-16 6:58:50 140464214296320 [Note] Deleted Master_info file '/dev/shm/var/1/mysqld.4/data/relay-log.info'. 2020-01-16 6:58:50 140464214296320 [Note] Added new Master_info 'm1' to hash table 2020-01-16 6:58:50 140464214296320 [Note] Master connection name: 'm1' Master_info_file: 'master-m1.info' Relay_info_file: 'relay-log-m1.info' 2020-01-16 6:58:51 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16000', master_log_file='', master_log_pos='4'. 2020-01-16 6:58:51 140464214296320 [Note] Added new Master_info 'm2' to hash table 2020-01-16 6:58:51 140464214296320 [Note] Master connection name: 'm2' Master_info_file: 'master-m2.info' Relay_info_file: 'relay-log-m2.info' 2020-01-16 6:58:51 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16001', master_log_file='', master_log_pos='4'. 2020-01-16 6:58:51 140464214296320 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-01-16 6:58:51 140464214296320 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16002', master_log_file='', master_log_pos='4'. 2020-01-16 6:58:51 140464208037632 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4 2020-01-16 6:58:51 140464209102592 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4 2020-01-16 6:58:51 140464208570112 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16001',replication started in log 'FIRST' at position 4 2020-01-16 6:58:51 140464209635072 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16002',replication started in log 'FIRST' at position 4 2020-01-16 6:58:51 140464210167552 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4 2020-01-16 6:58:51 140464210700032 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication started in log 'FIRST' at position 4 2020-01-16 6:58:51 140464208037632 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed 2020-01-16 6:58:51 140464208037632 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229 2020-01-16 6:58:51 140464208570112 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229 2020-01-16 6:58:51 140464209102592 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-16 6:58:51 140464209102592 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229 2020-01-16 6:58:51 140464209635072 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229 2020-01-16 6:58:51 140464210167552 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed 2020-01-16 6:58:51 140464210167552 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229 2020-01-16 6:58:51 140464210700032 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229 ================================================================= ==30159==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000030c38 at pc 0x000001f0cc69 bp 0x7fc05f985770 sp 0x7fc05f985760 READ of size 1 at 0x607000030c38 thread T28 #0 0x1f0cc68 in strend /home/buildbot/buildbot/build/mariadb-10.2.31/strings/strend.c:45 #1 0x98276d in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:3437 #2 0x9835a9 in show_status_array /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:3580 #3 0x9b1a1b in fill_variables(THD*, TABLE_LIST*, Item*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:7521 #4 0x9b9bd8 in get_schema_tables_result(JOIN*, enum_schema_table_state) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_show.cc:8448 #5 0x8b72a4 in JOIN::exec_inner() /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:3593 #6 0x8b566d in JOIN::exec() /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:3424 #7 0x8b8a82 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:3824 #8 0x897bdf in handle_select(THD*, LEX*, select_result*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_select.cc:373 #9 0x81c0b4 in execute_sqlcom_select /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:6225 #10 0x80a8c7 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:3532 #11 0x1196db9 in sp_instr_stmt::exec_core(THD*, unsigned int*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:3248 #12 0x11959e2 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:3011 #13 0x1196678 in sp_instr_stmt::execute(THD*, unsigned int*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:3164 #14 0x118bd67 in sp_head::execute(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:1329 #15 0x118fbdd in sp_head::execute_procedure(THD*, List<Item>*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sp_head.cc:2118 #16 0x806e11 in do_execute_sp /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:2954 #17 0x8175aa in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:5572 #18 0x824b6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:7740 #19 0x8011d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1831 #20 0x7fe2c7 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1385 #21 0xb37eec in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1336 #22 0xb378d0 in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1241 #23 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862 #24 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) #25 0x7fc0765d182c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)   0x607000030c38 is located 8 bytes inside of 80-byte region [0x607000030c30,0x607000030c80) freed by thread T35 here: #0 0x7fc07891a2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca) #1 0x1e1e478 in my_free /home/buildbot/buildbot/build/mariadb-10.2.31/mysys/my_malloc.c:218 #2 0xd5268e in mysql_close_free_options /home/buildbot/buildbot/build/mariadb-10.2.31/sql-common/client.c:3590 #3 0xd53215 in mysql_close /home/buildbot/buildbot/build/mariadb-10.2.31/sql-common/client.c:3753 #4 0x67d45a in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:4650 #5 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862 #6 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)   previously allocated by thread T35 here: #0 0x7fc07891a602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602) #1 0x1e1da93 in my_malloc /home/buildbot/buildbot/build/mariadb-10.2.31/mysys/my_malloc.c:101 #2 0x1e1e7b6 in my_strdup /home/buildbot/buildbot/build/mariadb-10.2.31/mysys/my_malloc.c:241 #3 0xd56688 in mysql_options /home/buildbot/buildbot/build/mariadb-10.2.31/sql-common/client.c:4099 #4 0x689044 in connect_to_master /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:6727 #5 0x688866 in safe_connect /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:6657 #6 0x67bc7f in handle_slave_io /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:4335 #7 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862 #8 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)   Thread T28 created by T0 here: #0 0x7fc0788b8253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253) #1 0x130d2aa in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1912 #2 0x601906 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.2.31/include/mysql/psi/mysql_thread.h:1239 #3 0x616685 in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6506 #4 0x616da9 in create_new_thread /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6576 #5 0x617e09 in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6851 #6 0x615b63 in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld.cc:6125 #7 0x5ffc45 in main /home/buildbot/buildbot/build/mariadb-10.2.31/sql/main.cc:25 #8 0x7fc0764eb82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)   Thread T35 created by T28 here: #0 0x7fc0788b8253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253) #1 0x130d2aa in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1912 #2 0x6648a2 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.2.31/include/mysql/psi/mysql_thread.h:1239 #3 0x66968f in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:934 #4 0x669fce in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/slave.cc:1056 #5 0x88d1e4 in start_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_repl.cc:3146 #6 0xb2367b in Master_info_index::start_all_slaves(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/rpl_mi.cc:1635 #7 0x80c090 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:3895 #8 0x824b6a in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:7740 #9 0x8011d1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1831 #10 0x7fe2c7 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_parse.cc:1385 #11 0xb37eec in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1336 #12 0xb378d0 in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.31/sql/sql_connect.cc:1241 #13 0x130cebd in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.31/storage/perfschema/pfs.cc:1862 #14 0x7fc07713c6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)   SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.31/strings/strend.c:45 strend Shadow bytes around the buggy address: 0x0c0e7fffe130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fffe140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fffe150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fffe160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fffe170: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd =>0x0c0e7fffe180: fd fa fa fa fa fa fd[fd]fd fd fd fd fd fd fd fd 0x0c0e7fffe190: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa 0x0c0e7fffe1a0: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa 0x0c0e7fffe1b0: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd 0x0c0e7fffe1c0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd 0x0c0e7fffe1d0: fd fd fd fd fd fd fa fa fa fa fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe ==30159==ABORTING 200116 6:58:53 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.2.31-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=1 max_threads=153 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63103 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x62a00008a208 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc05f98bc70 thread_stack 0x5b000 /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077)[0x7fc0788cc077] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(my_print_stacktrace+0xc6)[0x1e2e32e] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(handle_fatal_signal+0x8a9)[0xde7f7b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fc077146390] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fc076500428] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fc07650202a] /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xaddc9)[0x7fc07892fdc9] /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa0769)[0x7fc078922769] /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa55d2)[0x7fc0789275d2] /usr/lib/x86_64-linux-gnu/libasan.so.2(__asan_report_error+0x416)[0x7fc0789216e6] /usr/lib/x86_64-linux-gnu/libasan.so.2(__asan_report_load1+0x33)[0x7fc078922b13] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(strend+0x47)[0x1f0cc69] fil/fil0fil.cc:1745(fil_space_close(char const*))[0x98276e] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld[0x9835aa] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z14fill_variablesP3THDP10TABLE_LISTP4Item+0x497)[0x9b1a1c] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0xbc0)[0x9b9bd9] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_ZN4JOIN10exec_innerEv+0x1a87)[0x8b72a5] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_ZN4JOIN4execEv+0x204)[0x8b566e] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x779)[0x8b8a83] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x4c5)[0x897be0] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld[0x81c0b5] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z21mysql_execute_commandP3THD+0x394b)[0x80a8c8] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x20)[0x1196dba] sql/field.cc:6166(Field_time::cmp(unsigned char const*, unsigned char const*))[0x11959e3] sql/field.cc:6087(Field_time::val_int())[0x1196679] sql/field.cc:6146(Field_time::get_date(st_mysql_time*, unsigned long long))[0x118bd68] sql/field.cc:5219(Field_timestamp::store_time_dec(st_mysql_time*, unsigned int))[0x118fbde] sql/field.cc:5563(read_bigendian(unsigned char const*, unsigned int))[0x806e12] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z21mysql_execute_commandP3THD+0x1062e)[0x8175ab] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x6a2)[0x824b6b] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ed3)[0x8011d2] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z10do_commandP3THD+0x115d)[0x7fe2c8] /home/buildbot/buildbot/build/mariadb-10.2.31/sql/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x48c)[0xb37eed] sql/sql_cache.cc:3173(Query_cache::write_result_data(Query_cache_block**, unsigned long, unsigned char*, Query_cache_block*, Query_cache_block::block_type))[0xb378d1] sql/sql_cache.cc:3099(Query_cache::append_result_data(Query_cache_block**, unsigned long, unsigned char*, Query_cache_block*))[0x130cebe] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fc07713c6ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc0765d182d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x62500033b928): SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp', 'innodb_file_format_max') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' ORDER BY variable_name Connection ID (thread ID): 24 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on   The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /dev/shm/var/1/mysqld.4/data Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 0 bytes Max resident set unlimited unlimited bytes Max processes 23715 23715 processes Max open files 1024 1024 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 23715 23715 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: |/usr/share/apport/apport %p %s %c %P   ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > use c; #No t1 table show tables; Tables_in_c t2 #CleanUp connection server_1; drop database a; connection server_2; drop database b; connection server_3; drop database c; connection server_4; stop all slaves; SET default_master_connection = "m1"; include/wait_for_slave_to_stop.inc SET default_master_connection = "m2"; include/wait_for_slave_to_stop.inc SET default_master_connection = ""; include/wait_for_slave_to_stop.inc       - saving '/dev/shm/var/1/log/multi_source.mdev-8874-innodb/' to '/dev/shm/var/log/multi_source.mdev-8874-innodb/' worker[1] > Restart - using different config file worker[1] > Restart - using different config file worker[1] > Restart - using different config file worker[1] > Restart - using different config file ***Warnings generated in error logs during shutdown after running tests: multi_source.mdev-8874   ==30159==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000030c38 at pc 0x000001f0cc69 bp 0x7fc05f985770 sp 0x7fc05f985760 SUMMARY: AddressSanitizer: heap-use-after-free /home/buildbot/buildbot/build/mariadb-10.2.31/strings/strend.c:45 strend Attempting backtrace. You can use the following information to find out
            alice Alice Sherepa added a comment -

            10.4 7effcb8ed6a9fd754525

            multi_source.mdev-8874 'innodb'          w7 [ fail ]
                    Test ended at 2020-11-27 11:22:04
             
            CURRENT_TEST: multi_source.mdev-8874
             
             
            The server [mysqld.4 - pid: 28690, winpid: 28690, exit: 256] crashed while running 'check testcase after test'
            Server log from this test:
            ----------SERVER LOG START-----------
            2020-11-27 11:21:56 0 [Note] /git/10.4/sql/mysqld (mysqld 10.4.18-MariaDB-debug-log) starting as process 28738 ...
            2020-11-27 11:21:56 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32246)
            2020-11-27 11:21:56 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
            2020-11-27 11:21:56 0 [Warning] setrlimit could not change the size of core files to 'infinity';  We may not be able to generate a core file on signals
            2020-11-27 11:21:56 0 [Note] Plugin 'partition' is disabled.
            2020-11-27 11:21:56 0 [Note] Plugin 'SEQUENCE' is disabled.
            2020-11-27 11:21:56 0 [Note] InnoDB: Using Linux native AIO
            2020-11-27 11:21:56 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
            2020-11-27 11:21:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2020-11-27 11:21:56 0 [Note] InnoDB: Uses event mutexes
            2020-11-27 11:21:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            2020-11-27 11:21:56 0 [Note] InnoDB: Number of pools: 1
            2020-11-27 11:21:56 0 [Note] InnoDB: Using SSE2 crc32 instructions
            2020-11-27 11:21:56 0 [Note] mysqld: O_TMPFILE is not supported on /git/10.4/mysql-test/var/tmp/7/mysqld.4 (disabling future attempts)
            2020-11-27 11:21:56 0 [Warning] InnoDB: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
            2020-11-27 11:21:56 0 [Warning] InnoDB: io_setup() attempt 1.
            2020-11-27 11:21:57 0 [Warning] InnoDB: io_setup() attempt 2.
            2020-11-27 11:21:57 0 [Warning] InnoDB: io_setup() attempt 3.
            2020-11-27 11:21:58 0 [Warning] InnoDB: io_setup() attempt 4.
            2020-11-27 11:21:58 0 [Warning] InnoDB: io_setup() attempt 5.
            2020-11-27 11:21:59 0 [Warning] InnoDB: io_setup() failed with EAGAIN after 5 attempts.
            2020-11-27 11:21:59 0 [Note] InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
            2020-11-27 11:21:59 0 [Warning] InnoDB: Warning: Linux Native AIO disabled because _linux_create_io_ctx() failed. To get rid of this warning you can try increasing system fs.aio-max-nr to 1048576 or larger or setting innodb_use_native_aio = 0 in my.cnf
            2020-11-27 11:21:59 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
            2020-11-27 11:21:59 0 [Note] InnoDB: Completed initialization of buffer pool
            2020-11-27 11:21:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            2020-11-27 11:21:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
            2020-11-27 11:21:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            2020-11-27 11:21:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            2020-11-27 11:21:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            2020-11-27 11:21:59 0 [Note] InnoDB: 10.4.18 started; log sequence number 68850; transaction id 33
            2020-11-27 11:21:59 0 [Note] InnoDB: Loading buffer pool(s) from /git/10.4/mysql-test/var/7/mysqld.4/data/ib_buffer_pool
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMP' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
            2020-11-27 11:21:59 0 [Note] InnoDB: Buffer pool(s) load completed at 201127 11:21:59
            2020-11-27 11:21:59 0 [Note] Plugin 'FEEDBACK' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'user_variables' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
            2020-11-27 11:21:59 0 [Note] Plugin 'unix_socket' is disabled.
            2020-11-27 11:21:59 0 [Warning] /git/10.4/sql/mysqld: unknown option '--loose-pam-debug'
            2020-11-27 11:21:59 0 [Warning] /git/10.4/sql/mysqld: unknown option '--loose-aria'
            2020-11-27 11:21:59 0 [Note] Server socket created on IP: '127.0.0.1'.
            2020-11-27 11:22:00 0 [Note] Reading of all Master_info entries succeeded
            2020-11-27 11:22:00 0 [Note] Added new Master_info '' to hash table
            2020-11-27 11:22:00 0 [Note] /git/10.4/sql/mysqld: ready for connections.
            Version: '10.4.18-MariaDB-debug-log'  socket: '/git/10.4/mysql-test/var/tmp/7/mysqld.4.sock'  port: 16343  Source distribution
            2020-11-27 11:22:01 10 [Note] Added new Master_info 'm1' to hash table
            2020-11-27 11:22:01 10 [Note] Master connection name: 'm1'  Master_info_file: 'master-m1.info'  Relay_info_file: 'relay-log-m1.info'
            2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16340', master_log_file='', master_log_pos='4'.
            2020-11-27 11:22:01 10 [Note] Added new Master_info 'm2' to hash table
            2020-11-27 11:22:01 10 [Note] Master connection name: 'm2'  Master_info_file: 'master-m2.info'  Relay_info_file: 'relay-log-m2.info'
            2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16341', master_log_file='', master_log_pos='4'.
            2020-11-27 11:22:01 10 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
            2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16342', master_log_file='', master_log_pos='4'.
            2020-11-27 11:22:01 11 [Note] Master 'm2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16341' in log '' at position 4
            2020-11-27 11:22:01 12 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4
            2020-11-27 11:22:01 13 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16342' in log '' at position 4
            2020-11-27 11:22:01 14 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4
            2020-11-27 11:22:01 15 [Note] Master 'm1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16340' in log '' at position 4
            2020-11-27 11:22:01 16 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4
            2020-11-27 11:22:01 11 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16341',replication started in log 'FIRST' at position 4
            2020-11-27 11:22:01 15 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16340',replication started in log 'FIRST' at position 4
            2020-11-27 11:22:01 13 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16342',replication started in log 'FIRST' at position 4
            2020-11-27 11:22:01 12 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed
            2020-11-27 11:22:01 12 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108
            2020-11-27 11:22:01 11 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108
            2020-11-27 11:22:01 14 [Note] Error reading relay log event: slave SQL thread was killed
            2020-11-27 11:22:01 14 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108
            2020-11-27 11:22:01 13 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108
            2020-11-27 11:22:01 16 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed
            2020-11-27 11:22:01 16 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108
            2020-11-27 11:22:01 15 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108
            2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/master-m1.info'.
            2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/relay-log-m1.info'.
            2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/master-m2.info'.
            2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/relay-log-m2.info'.
            2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/master.info'.
            2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/relay-log.info'.
            2020-11-27 11:22:01 10 [Note] Added new Master_info 'm1' to hash table
            2020-11-27 11:22:01 10 [Note] Master connection name: 'm1'  Master_info_file: 'master-m1.info'  Relay_info_file: 'relay-log-m1.info'
            2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16340', master_log_file='', master_log_pos='4'.
            2020-11-27 11:22:01 10 [Note] Added new Master_info 'm2' to hash table
            2020-11-27 11:22:01 10 [Note] Master connection name: 'm2'  Master_info_file: 'master-m2.info'  Relay_info_file: 'relay-log-m2.info'
            2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16341', master_log_file='', master_log_pos='4'.
            2020-11-27 11:22:01 10 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
            2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16342', master_log_file='', master_log_pos='4'.
            2020-11-27 11:22:01 17 [Note] Master 'm2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16341' in log '' at position 4
            2020-11-27 11:22:01 18 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4
            2020-11-27 11:22:01 19 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16342' in log '' at position 4
            2020-11-27 11:22:01 20 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4
            2020-11-27 11:22:01 21 [Note] Master 'm1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16340' in log '' at position 4
            2020-11-27 11:22:01 19 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16342',replication started in log 'FIRST' at position 4
            2020-11-27 11:22:01 17 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16341',replication started in log 'FIRST' at position 4
            2020-11-27 11:22:01 22 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4
            2020-11-27 11:22:01 21 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16340',replication started in log 'FIRST' at position 4
            2020-11-27 11:22:02 18 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed
            2020-11-27 11:22:02 18 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229
            2020-11-27 11:22:02 17 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229
            2020-11-27 11:22:02 20 [Note] Error reading relay log event: slave SQL thread was killed
            2020-11-27 11:22:02 20 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229
            2020-11-27 11:22:02 19 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229
            2020-11-27 11:22:02 22 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed
            2020-11-27 11:22:02 22 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229
            2020-11-27 11:22:02 21 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229
            =================================================================
            ==28738==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000bf810 at pc 0x55f06ea6a7f2 bp 0x7f0dfe2744e0 sp 0x7f0dfe2744d8
            READ of size 1 at 0x60e0000bf810 thread T28
                #0 0x55f06ea6a7f1 in strend /git/10.4/strings/strend.c:45
                #1 0x55f06cfe6351 in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /git/10.4/sql/sql_show.cc:3691
                #2 0x55f06cfe7326 in show_status_array /git/10.4/sql/sql_show.cc:3836
                #3 0x55f06d019aee in fill_variables(THD*, TABLE_LIST*, Item*) /git/10.4/sql/sql_show.cc:7934
                #4 0x55f06d024114 in get_schema_tables_result(JOIN*, enum_schema_table_state) /git/10.4/sql/sql_show.cc:8927
                #5 0x55f06cef955b in JOIN::exec_inner() /git/10.4/sql/sql_select.cc:4439
                #6 0x55f06cef7463 in JOIN::exec() /git/10.4/sql/sql_select.cc:4264
                #7 0x55f06cefb40d in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /git/10.4/sql/sql_select.cc:4699
                #8 0x55f06cecd143 in handle_select(THD*, LEX*, select_result*, unsigned long) /git/10.4/sql/sql_select.cc:410
                #9 0x55f06ce404d6 in execute_sqlcom_select /git/10.4/sql/sql_parse.cc:6400
                #10 0x55f06ce2dd5d in mysql_execute_command(THD*) /git/10.4/sql/sql_parse.cc:3925
                #11 0x55f06cbfadc7 in sp_instr_stmt::exec_core(THD*, unsigned int*) /git/10.4/sql/sp_head.cc:3687
                #12 0x55f06cbf95db in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /git/10.4/sql/sp_head.cc:3419
                #13 0x55f06cbfa576 in sp_instr_stmt::execute(THD*, unsigned int*) /git/10.4/sql/sp_head.cc:3593
                #14 0x55f06cbec0e0 in sp_head::execute(THD*, bool) /git/10.4/sql/sp_head.cc:1364
                #15 0x55f06cbf2083 in sp_head::execute_procedure(THD*, List<Item>*) /git/10.4/sql/sp_head.cc:2370
                #16 0x55f06ce2785f in do_execute_sp /git/10.4/sql/sql_parse.cc:3040
                #17 0x55f06ce2936c in Sql_cmd_call::execute(THD*) /git/10.4/sql/sql_parse.cc:3282
                #18 0x55f06ce3dc71 in mysql_execute_command(THD*) /git/10.4/sql/sql_parse.cc:6143
                #19 0x55f06ce495a0 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /git/10.4/sql/sql_parse.cc:7938
                #20 0x55f06ce20812 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /git/10.4/sql/sql_parse.cc:1839
                #21 0x55f06ce1d291 in do_command(THD*) /git/10.4/sql/sql_parse.cc:1357
                #22 0x55f06d1fc32a in do_handle_one_connection(CONNECT*) /git/10.4/sql/sql_connect.cc:1412
                #23 0x55f06d1fbbcc in handle_one_connection /git/10.4/sql/sql_connect.cc:1316
                #24 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869
                #25 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
                #26 0x7f0e141d64ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)
             
            0x60e0000bf810 is located 112 bytes inside of 156-byte region [0x60e0000bf7a0,0x60e0000bf83c)
            freed by thread T37 here:
                #0 0x7f0e14cd1fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0)
                #1 0x55f06e97fab0 in free_memory /git/10.4/mysys/safemalloc.c:279
                #2 0x55f06e97f0de in sf_free /git/10.4/mysys/safemalloc.c:197
                #3 0x55f06e94ea99 in my_free /git/10.4/mysys/my_malloc.c:222
                #4 0x55f06d551e15 in mysql_close_free_options /git/10.4/sql-common/client.c:3312
                #5 0x55f06d55299b in mysql_close /git/10.4/sql-common/client.c:3471
                #6 0x55f06cbc7f25 in handle_slave_io /git/10.4/sql/slave.cc:5090
                #7 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869
                #8 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
             
            previously allocated by thread T37 here:
                #0 0x7f0e14cd2330 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9330)
                #1 0x55f06e97eac4 in sf_malloc /git/10.4/mysys/safemalloc.c:118
                #2 0x55f06e94dfd6 in my_malloc /git/10.4/mysys/my_malloc.c:101
                #3 0x55f06e94edf0 in my_strdup /git/10.4/mysys/my_malloc.c:245
                #4 0x55f06d556320 in mysql_options /git/10.4/sql-common/client.c:3817
                #5 0x55f06cbd73cf in connect_to_master /git/10.4/sql/slave.cc:7293
                #6 0x55f06cbd6bf3 in safe_connect /git/10.4/sql/slave.cc:7223
                #7 0x55f06cbc5e43 in handle_slave_io /git/10.4/sql/slave.cc:4762
                #8 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869
                #9 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
             
            Thread T28 created by T0 here:
                #0 0x7f0e14c39db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
                #1 0x55f06e8388bd in spawn_thread_v1 /git/10.4/storage/perfschema/pfs.cc:1919
                #2 0x55f06cb3a7b5 in inline_mysql_thread_create /git/10.4/include/mysql/psi/mysql_thread.h:1275
                #3 0x55f06cb51b10 in create_thread_to_handle_connection(CONNECT*) /git/10.4/sql/mysqld.cc:6259
                #4 0x55f06cb52265 in create_new_thread(CONNECT*) /git/10.4/sql/mysqld.cc:6329
                #5 0x55f06cb5273f in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /git/10.4/sql/mysqld.cc:6427
                #6 0x55f06cb535ec in handle_connections_sockets() /git/10.4/sql/mysqld.cc:6585
                #7 0x55f06cb51274 in mysqld_main(int, char**) /git/10.4/sql/mysqld.cc:5917
                #8 0x55f06cb38684 in main /git/10.4/sql/main.cc:25
                #9 0x7f0e1410109a in __libc_start_main ../csu/libc-start.c:308
             
            Thread T37 created by T28 here:
                #0 0x7f0e14c39db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
                #1 0x55f06e8388bd in spawn_thread_v1 /git/10.4/storage/perfschema/pfs.cc:1919
                #2 0x55f06cba8afe in inline_mysql_thread_create /git/10.4/include/mysql/psi/mysql_thread.h:1275
                #3 0x55f06cbb105a in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /git/10.4/sql/slave.cc:1296
                #4 0x55f06cbb1acb in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /git/10.4/sql/slave.cc:1412
                #5 0x55f06cebf929 in start_slave(THD*, Master_info*, bool) /git/10.4/sql/sql_repl.cc:3240
                #6 0x55f06d1e5569 in Master_info_index::start_all_slaves(THD*) /git/10.4/sql/rpl_mi.cc:1640
                #7 0x55f06ce2f8cd in mysql_execute_command(THD*) /git/10.4/sql/sql_parse.cc:4287
                #8 0x55f06ce495a0 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /git/10.4/sql/sql_parse.cc:7938
                #9 0x55f06ce20812 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /git/10.4/sql/sql_parse.cc:1839
                #10 0x55f06ce1d291 in do_command(THD*) /git/10.4/sql/sql_parse.cc:1357
                #11 0x55f06d1fc32a in do_handle_one_connection(CONNECT*) /git/10.4/sql/sql_connect.cc:1412
                #12 0x55f06d1fbbcc in handle_one_connection /git/10.4/sql/sql_connect.cc:1316
                #13 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869
                #14 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
             
            SUMMARY: AddressSanitizer: heap-use-after-free /git/10.4/strings/strend.c:45 in strend
            Shadow bytes around the buggy address:
              0x0c1c8000feb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c1c8000fec0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c1c8000fed0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c1c8000fee0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c1c8000fef0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
            =>0x0c1c8000ff00: fd fd[fd]fd fd fd fd fd fa fa fa fa fa fa fa fa
              0x0c1c8000ff10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
              0x0c1c8000ff20: fd fd fd fd fa fa fa fa fa fa fa fa 00 00 00 00
              0x0c1c8000ff30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 fa
              0x0c1c8000ff40: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
              0x0c1c8000ff50: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable:           00
              Partially addressable: 01 02 03 04 05 06 07 
              Heap left redzone:       fa
              Freed heap region:       fd
              Stack left redzone:      f1
              Stack mid redzone:       f2
              Stack right redzone:     f3
              Stack after return:      f5
              Stack use after scope:   f8
              Global redzone:          f9
              Global init order:       f6
              Poisoned by user:        f7
              Container overflow:      fc
              Array cookie:            ac
              Intra object redzone:    bb
              ASan internal:           fe
              Left alloca redzone:     ca
              Right alloca redzone:    cb
            ==28738==ABORTING
            ----------SERVER LOG END-------------
            mysqltest failed but provided no output
            The result from queries just before the failure was:
            < snip >
            use c;
            #No t1 table
            show tables;
            Tables_in_c
            t2
            #CleanUp
            connection server_1;
            drop database a;
            connection server_2;
            drop database b;
            connection server_3;
            drop database c;
            connection server_4;
            stop all slaves;
            SET default_master_connection = "m1";
            include/wait_for_slave_to_stop.inc
            SET default_master_connection = "m2";
            include/wait_for_slave_to_stop.inc
            SET default_master_connection = "";
            include/wait_for_slave_to_stop.inc
             
             
             
             - saving '/git/10.4/mysql-test/var/7/log/multi_source.mdev-8874-innodb/' to '/git/10.4/mysql-test/var/log/multi_source.mdev-8874-innodb/'
            ***Warnings generated in error logs during shutdown after running tests: multi_source.mdev-8874
             
            ==28738==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000bf810 at pc 0x55f06ea6a7f2 bp 0x7f0dfe2744e0 sp 0x7f0dfe2744d8
            SUMMARY: AddressSanitizer: heap-use-after-free /git/10.4/strings/strend.c:45 in strend
            

            alice Alice Sherepa added a comment - 10.4 7effcb8ed6a9fd754525 multi_source.mdev-8874 'innodb' w7 [ fail ] Test ended at 2020-11-27 11:22:04   CURRENT_TEST: multi_source.mdev-8874     The server [mysqld.4 - pid: 28690, winpid: 28690, exit: 256] crashed while running 'check testcase after test' Server log from this test: ----------SERVER LOG START----------- 2020-11-27 11:21:56 0 [Note] /git/10.4/sql/mysqld (mysqld 10.4.18-MariaDB-debug-log) starting as process 28738 ... 2020-11-27 11:21:56 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32246) 2020-11-27 11:21:56 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2020-11-27 11:21:56 0 [Warning] setrlimit could not change the size of core files to 'infinity'; We may not be able to generate a core file on signals 2020-11-27 11:21:56 0 [Note] Plugin 'partition' is disabled. 2020-11-27 11:21:56 0 [Note] Plugin 'SEQUENCE' is disabled. 2020-11-27 11:21:56 0 [Note] InnoDB: Using Linux native AIO 2020-11-27 11:21:56 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-11-27 11:21:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-11-27 11:21:56 0 [Note] InnoDB: Uses event mutexes 2020-11-27 11:21:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2020-11-27 11:21:56 0 [Note] InnoDB: Number of pools: 1 2020-11-27 11:21:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-11-27 11:21:56 0 [Note] mysqld: O_TMPFILE is not supported on /git/10.4/mysql-test/var/tmp/7/mysqld.4 (disabling future attempts) 2020-11-27 11:21:56 0 [Warning] InnoDB: io_setup() failed with EAGAIN. Will make 5 attempts before giving up. 2020-11-27 11:21:56 0 [Warning] InnoDB: io_setup() attempt 1. 2020-11-27 11:21:57 0 [Warning] InnoDB: io_setup() attempt 2. 2020-11-27 11:21:57 0 [Warning] InnoDB: io_setup() attempt 3. 2020-11-27 11:21:58 0 [Warning] InnoDB: io_setup() attempt 4. 2020-11-27 11:21:58 0 [Warning] InnoDB: io_setup() attempt 5. 2020-11-27 11:21:59 0 [Warning] InnoDB: io_setup() failed with EAGAIN after 5 attempts. 2020-11-27 11:21:59 0 [Note] InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf 2020-11-27 11:21:59 0 [Warning] InnoDB: Warning: Linux Native AIO disabled because _linux_create_io_ctx() failed. To get rid of this warning you can try increasing system fs.aio-max-nr to 1048576 or larger or setting innodb_use_native_aio = 0 in my.cnf 2020-11-27 11:21:59 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2020-11-27 11:21:59 0 [Note] InnoDB: Completed initialization of buffer pool 2020-11-27 11:21:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-11-27 11:21:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-11-27 11:21:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-11-27 11:21:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-11-27 11:21:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-11-27 11:21:59 0 [Note] InnoDB: 10.4.18 started; log sequence number 68850; transaction id 33 2020-11-27 11:21:59 0 [Note] InnoDB: Loading buffer pool(s) from /git/10.4/mysql-test/var/7/mysqld.4/data/ib_buffer_pool 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMP' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2020-11-27 11:21:59 0 [Note] InnoDB: Buffer pool(s) load completed at 201127 11:21:59 2020-11-27 11:21:59 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'user_variables' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2020-11-27 11:21:59 0 [Note] Plugin 'unix_socket' is disabled. 2020-11-27 11:21:59 0 [Warning] /git/10.4/sql/mysqld: unknown option '--loose-pam-debug' 2020-11-27 11:21:59 0 [Warning] /git/10.4/sql/mysqld: unknown option '--loose-aria' 2020-11-27 11:21:59 0 [Note] Server socket created on IP: '127.0.0.1'. 2020-11-27 11:22:00 0 [Note] Reading of all Master_info entries succeeded 2020-11-27 11:22:00 0 [Note] Added new Master_info '' to hash table 2020-11-27 11:22:00 0 [Note] /git/10.4/sql/mysqld: ready for connections. Version: '10.4.18-MariaDB-debug-log' socket: '/git/10.4/mysql-test/var/tmp/7/mysqld.4.sock' port: 16343 Source distribution 2020-11-27 11:22:01 10 [Note] Added new Master_info 'm1' to hash table 2020-11-27 11:22:01 10 [Note] Master connection name: 'm1' Master_info_file: 'master-m1.info' Relay_info_file: 'relay-log-m1.info' 2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16340', master_log_file='', master_log_pos='4'. 2020-11-27 11:22:01 10 [Note] Added new Master_info 'm2' to hash table 2020-11-27 11:22:01 10 [Note] Master connection name: 'm2' Master_info_file: 'master-m2.info' Relay_info_file: 'relay-log-m2.info' 2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16341', master_log_file='', master_log_pos='4'. 2020-11-27 11:22:01 10 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16342', master_log_file='', master_log_pos='4'. 2020-11-27 11:22:01 11 [Note] Master 'm2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16341' in log '' at position 4 2020-11-27 11:22:01 12 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4 2020-11-27 11:22:01 13 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16342' in log '' at position 4 2020-11-27 11:22:01 14 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4 2020-11-27 11:22:01 15 [Note] Master 'm1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16340' in log '' at position 4 2020-11-27 11:22:01 16 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4 2020-11-27 11:22:01 11 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16341',replication started in log 'FIRST' at position 4 2020-11-27 11:22:01 15 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16340',replication started in log 'FIRST' at position 4 2020-11-27 11:22:01 13 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16342',replication started in log 'FIRST' at position 4 2020-11-27 11:22:01 12 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed 2020-11-27 11:22:01 12 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108 2020-11-27 11:22:01 11 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108 2020-11-27 11:22:01 14 [Note] Error reading relay log event: slave SQL thread was killed 2020-11-27 11:22:01 14 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108 2020-11-27 11:22:01 13 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108 2020-11-27 11:22:01 16 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed 2020-11-27 11:22:01 16 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1108 2020-11-27 11:22:01 15 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1108 2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/master-m1.info'. 2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/relay-log-m1.info'. 2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/master-m2.info'. 2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/relay-log-m2.info'. 2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/master.info'. 2020-11-27 11:22:01 10 [Note] Deleted Master_info file '/git/10.4/mysql-test/var/7/mysqld.4/data/relay-log.info'. 2020-11-27 11:22:01 10 [Note] Added new Master_info 'm1' to hash table 2020-11-27 11:22:01 10 [Note] Master connection name: 'm1' Master_info_file: 'master-m1.info' Relay_info_file: 'relay-log-m1.info' 2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16340', master_log_file='', master_log_pos='4'. 2020-11-27 11:22:01 10 [Note] Added new Master_info 'm2' to hash table 2020-11-27 11:22:01 10 [Note] Master connection name: 'm2' Master_info_file: 'master-m2.info' Relay_info_file: 'relay-log-m2.info' 2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16341', master_log_file='', master_log_pos='4'. 2020-11-27 11:22:01 10 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-11-27 11:22:01 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16342', master_log_file='', master_log_pos='4'. 2020-11-27 11:22:01 17 [Note] Master 'm2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16341' in log '' at position 4 2020-11-27 11:22:01 18 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4 2020-11-27 11:22:01 19 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16342' in log '' at position 4 2020-11-27 11:22:01 20 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4 2020-11-27 11:22:01 21 [Note] Master 'm1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16340' in log '' at position 4 2020-11-27 11:22:01 19 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16342',replication started in log 'FIRST' at position 4 2020-11-27 11:22:01 17 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16341',replication started in log 'FIRST' at position 4 2020-11-27 11:22:01 22 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4 2020-11-27 11:22:01 21 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16340',replication started in log 'FIRST' at position 4 2020-11-27 11:22:02 18 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed 2020-11-27 11:22:02 18 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229 2020-11-27 11:22:02 17 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229 2020-11-27 11:22:02 20 [Note] Error reading relay log event: slave SQL thread was killed 2020-11-27 11:22:02 20 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229 2020-11-27 11:22:02 19 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229 2020-11-27 11:22:02 22 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed 2020-11-27 11:22:02 22 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1229 2020-11-27 11:22:02 21 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1229 ================================================================= ==28738==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000bf810 at pc 0x55f06ea6a7f2 bp 0x7f0dfe2744e0 sp 0x7f0dfe2744d8 READ of size 1 at 0x60e0000bf810 thread T28 #0 0x55f06ea6a7f1 in strend /git/10.4/strings/strend.c:45 #1 0x55f06cfe6351 in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /git/10.4/sql/sql_show.cc:3691 #2 0x55f06cfe7326 in show_status_array /git/10.4/sql/sql_show.cc:3836 #3 0x55f06d019aee in fill_variables(THD*, TABLE_LIST*, Item*) /git/10.4/sql/sql_show.cc:7934 #4 0x55f06d024114 in get_schema_tables_result(JOIN*, enum_schema_table_state) /git/10.4/sql/sql_show.cc:8927 #5 0x55f06cef955b in JOIN::exec_inner() /git/10.4/sql/sql_select.cc:4439 #6 0x55f06cef7463 in JOIN::exec() /git/10.4/sql/sql_select.cc:4264 #7 0x55f06cefb40d in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /git/10.4/sql/sql_select.cc:4699 #8 0x55f06cecd143 in handle_select(THD*, LEX*, select_result*, unsigned long) /git/10.4/sql/sql_select.cc:410 #9 0x55f06ce404d6 in execute_sqlcom_select /git/10.4/sql/sql_parse.cc:6400 #10 0x55f06ce2dd5d in mysql_execute_command(THD*) /git/10.4/sql/sql_parse.cc:3925 #11 0x55f06cbfadc7 in sp_instr_stmt::exec_core(THD*, unsigned int*) /git/10.4/sql/sp_head.cc:3687 #12 0x55f06cbf95db in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /git/10.4/sql/sp_head.cc:3419 #13 0x55f06cbfa576 in sp_instr_stmt::execute(THD*, unsigned int*) /git/10.4/sql/sp_head.cc:3593 #14 0x55f06cbec0e0 in sp_head::execute(THD*, bool) /git/10.4/sql/sp_head.cc:1364 #15 0x55f06cbf2083 in sp_head::execute_procedure(THD*, List<Item>*) /git/10.4/sql/sp_head.cc:2370 #16 0x55f06ce2785f in do_execute_sp /git/10.4/sql/sql_parse.cc:3040 #17 0x55f06ce2936c in Sql_cmd_call::execute(THD*) /git/10.4/sql/sql_parse.cc:3282 #18 0x55f06ce3dc71 in mysql_execute_command(THD*) /git/10.4/sql/sql_parse.cc:6143 #19 0x55f06ce495a0 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /git/10.4/sql/sql_parse.cc:7938 #20 0x55f06ce20812 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /git/10.4/sql/sql_parse.cc:1839 #21 0x55f06ce1d291 in do_command(THD*) /git/10.4/sql/sql_parse.cc:1357 #22 0x55f06d1fc32a in do_handle_one_connection(CONNECT*) /git/10.4/sql/sql_connect.cc:1412 #23 0x55f06d1fbbcc in handle_one_connection /git/10.4/sql/sql_connect.cc:1316 #24 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869 #25 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486 #26 0x7f0e141d64ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)   0x60e0000bf810 is located 112 bytes inside of 156-byte region [0x60e0000bf7a0,0x60e0000bf83c) freed by thread T37 here: #0 0x7f0e14cd1fb0 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe8fb0) #1 0x55f06e97fab0 in free_memory /git/10.4/mysys/safemalloc.c:279 #2 0x55f06e97f0de in sf_free /git/10.4/mysys/safemalloc.c:197 #3 0x55f06e94ea99 in my_free /git/10.4/mysys/my_malloc.c:222 #4 0x55f06d551e15 in mysql_close_free_options /git/10.4/sql-common/client.c:3312 #5 0x55f06d55299b in mysql_close /git/10.4/sql-common/client.c:3471 #6 0x55f06cbc7f25 in handle_slave_io /git/10.4/sql/slave.cc:5090 #7 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869 #8 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486   previously allocated by thread T37 here: #0 0x7f0e14cd2330 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9330) #1 0x55f06e97eac4 in sf_malloc /git/10.4/mysys/safemalloc.c:118 #2 0x55f06e94dfd6 in my_malloc /git/10.4/mysys/my_malloc.c:101 #3 0x55f06e94edf0 in my_strdup /git/10.4/mysys/my_malloc.c:245 #4 0x55f06d556320 in mysql_options /git/10.4/sql-common/client.c:3817 #5 0x55f06cbd73cf in connect_to_master /git/10.4/sql/slave.cc:7293 #6 0x55f06cbd6bf3 in safe_connect /git/10.4/sql/slave.cc:7223 #7 0x55f06cbc5e43 in handle_slave_io /git/10.4/sql/slave.cc:4762 #8 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869 #9 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486   Thread T28 created by T0 here: #0 0x7f0e14c39db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0) #1 0x55f06e8388bd in spawn_thread_v1 /git/10.4/storage/perfschema/pfs.cc:1919 #2 0x55f06cb3a7b5 in inline_mysql_thread_create /git/10.4/include/mysql/psi/mysql_thread.h:1275 #3 0x55f06cb51b10 in create_thread_to_handle_connection(CONNECT*) /git/10.4/sql/mysqld.cc:6259 #4 0x55f06cb52265 in create_new_thread(CONNECT*) /git/10.4/sql/mysqld.cc:6329 #5 0x55f06cb5273f in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /git/10.4/sql/mysqld.cc:6427 #6 0x55f06cb535ec in handle_connections_sockets() /git/10.4/sql/mysqld.cc:6585 #7 0x55f06cb51274 in mysqld_main(int, char**) /git/10.4/sql/mysqld.cc:5917 #8 0x55f06cb38684 in main /git/10.4/sql/main.cc:25 #9 0x7f0e1410109a in __libc_start_main ../csu/libc-start.c:308   Thread T37 created by T28 here: #0 0x7f0e14c39db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0) #1 0x55f06e8388bd in spawn_thread_v1 /git/10.4/storage/perfschema/pfs.cc:1919 #2 0x55f06cba8afe in inline_mysql_thread_create /git/10.4/include/mysql/psi/mysql_thread.h:1275 #3 0x55f06cbb105a in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /git/10.4/sql/slave.cc:1296 #4 0x55f06cbb1acb in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /git/10.4/sql/slave.cc:1412 #5 0x55f06cebf929 in start_slave(THD*, Master_info*, bool) /git/10.4/sql/sql_repl.cc:3240 #6 0x55f06d1e5569 in Master_info_index::start_all_slaves(THD*) /git/10.4/sql/rpl_mi.cc:1640 #7 0x55f06ce2f8cd in mysql_execute_command(THD*) /git/10.4/sql/sql_parse.cc:4287 #8 0x55f06ce495a0 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /git/10.4/sql/sql_parse.cc:7938 #9 0x55f06ce20812 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /git/10.4/sql/sql_parse.cc:1839 #10 0x55f06ce1d291 in do_command(THD*) /git/10.4/sql/sql_parse.cc:1357 #11 0x55f06d1fc32a in do_handle_one_connection(CONNECT*) /git/10.4/sql/sql_connect.cc:1412 #12 0x55f06d1fbbcc in handle_one_connection /git/10.4/sql/sql_connect.cc:1316 #13 0x55f06e8384d0 in pfs_spawn_thread /git/10.4/storage/perfschema/pfs.cc:1869 #14 0x7f0e14bcffa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486   SUMMARY: AddressSanitizer: heap-use-after-free /git/10.4/strings/strend.c:45 in strend Shadow bytes around the buggy address: 0x0c1c8000feb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c1c8000fec0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c1c8000fed0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c1c8000fee0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c1c8000fef0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd =>0x0c1c8000ff00: fd fd[fd]fd fd fd fd fd fa fa fa fa fa fa fa fa 0x0c1c8000ff10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c1c8000ff20: fd fd fd fd fa fa fa fa fa fa fa fa 00 00 00 00 0x0c1c8000ff30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 fa 0x0c1c8000ff40: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd 0x0c1c8000ff50: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==28738==ABORTING ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > use c; #No t1 table show tables; Tables_in_c t2 #CleanUp connection server_1; drop database a; connection server_2; drop database b; connection server_3; drop database c; connection server_4; stop all slaves; SET default_master_connection = "m1"; include/wait_for_slave_to_stop.inc SET default_master_connection = "m2"; include/wait_for_slave_to_stop.inc SET default_master_connection = ""; include/wait_for_slave_to_stop.inc       - saving '/git/10.4/mysql-test/var/7/log/multi_source.mdev-8874-innodb/' to '/git/10.4/mysql-test/var/log/multi_source.mdev-8874-innodb/' ***Warnings generated in error logs during shutdown after running tests: multi_source.mdev-8874   ==28738==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000bf810 at pc 0x55f06ea6a7f2 bp 0x7f0dfe2744e0 sp 0x7f0dfe2744d8 SUMMARY: AddressSanitizer: heap-use-after-free /git/10.4/strings/strend.c:45 in strend
            alice Alice Sherepa added a comment - http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/5271/steps/mtr_nm/logs/stdio

            Issue is related to charset_dir var. There are two charset_dir variables one on client side and the other on server side.

            Client variable: mysql->options.charset_dir
            Server variable: 'charsets_dir'

            On 'mysql_close_free_options' the client side variable gets freed and initialized with zero. On some cases the server variable points to the client variable and when the server variable tries to access the freed memory, ASAN issue gets reported. Need to identify the exact race scenario and fix the issue.

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Issue is related to charset_dir var. There are two charset_dir variables one on client side and the other on server side. Client variable: mysql->options.charset_dir Server variable: 'charsets_dir' On 'mysql_close_free_options' the client side variable gets freed and initialized with zero. On some cases the server variable points to the client variable and when the server variable tries to access the freed memory, ASAN issue gets reported. Need to identify the exact race scenario and fix the issue.

            The issue happens due to the race between 'save' and 'restore' of 'charsets_dir' global variable

            In multi-source replication the slave server tries to establish two `mysql` connections with two master servers.

            #0  mysql_set_character_set_with_default_collation (mysql=0x61a000030c98) at /home/sujatha/bug_repo/test-10.5/sql-common/client.c:1747
            #1  0x000055d6c9f7ea7b in mysql_init_character_set (mysql=0x61a000030c98) at /home/sujatha/bug_repo/test-10.5/sql-common/client.c:1799
            #2  0x000055d6c9f8624a in mysql_real_connect (mysql=0x61a000030c98, host=0x62a00005a8ce "localhost", user=0x62a00005a983 "root", 
                passwd=0x62a00005ab04 "", db=0x0, port=16000, unix_socket=0x0, client_flag=2147483648)
                at /home/sujatha/bug_repo/test-10.5/sql-common/client.c:3067
            #3  0x000055d6c95e1b31 in connect_to_master (thd=0x62b000046218, mysql=0x61a000030c98, mi=0x62a00005a200, reconnect=false, 
                suppress_warnings=false) at /home/sujatha/bug_repo/test-10.5/sql/slave.cc:7210
            #4  0x000055d6c95e106a in safe_connect (thd=0x62b000046218, mysql=0x61a000030c98, mi=0x62a00005a200)
                at /home/sujatha/bug_repo/test-10.5/sql/slave.cc:7122
            #5  0x000055d6c95d0081 in handle_slave_io (arg=0x62a00005a200) at /home/sujatha/bug_repo/test-10.5/sql/slave.cc:4652
            #6  0x000055d6ca951277 in pfs_spawn_thread (arg=0x61500001fe98) at /home/sujatha/bug_repo/test-10.5/storage/perfschema/pfs.cc:2201
            #7  0x00007fa8b81c76db in start_thread (arg=0x7fa8a86f3700) at pthread_create.c:463
            #8  0x00007fa8b736271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
            

            They both reach 'mysql_set_character_set_with_default_collation'. Within this function two variables are of our
            interest.

            1. charsets_dir : It is a global variable. It points to static array i.e static char mysql_charsets_dir[FN_REFLEN+1]
            2. mysql->options.charset_dir : It is dynamically allocated on heap.

            File Name: sql/slave.cc
            Function Name: `connect_to_master` provides 'charsets_dir' as argument to `mysql_options` to create 'mysql->options.charset_dir'. Then memory is dynamically allocated and it holds copy of 'charsets_dir'.

            /* This one is not strictly needed but we have it here for completeness */
            mysql_options(mysql, MYSQL_SET_CHARSET_DIR, (char *) charsets_dir);

            File Name: sql-common/client.c
            case MYSQL_SET_CHARSET_DIR:
            my_free(mysql->options.charset_dir);
            mysql->options.charset_dir= opt_strdup(arg,MYF(MY_WME)); (Memory gets allocated on heap)

            3. Within 'mysql_set_character_set_with_default_collation '

              static void mysql_set_character_set_with_default_collation(MYSQL *mysql)
              {
                const char *save= charsets_dir;
                if (mysql->options.charset_dir)
                  charsets_dir=mysql->options.charset_dir;
                  ...
                  charsets_dir= save;
              }
             

            For example:
            charsets_dir=mysql_charsets_dir=0x56164f9747e0
            mysql->options.charset_dir:0x607000080098 (mysql_1)
            mysql->options.charset_dir:0x607000087098 (mysql_2)

            SHOW VARIABLES 'character_sets_dir' execution at this time will report 'AddressSanitizer: heap-use-after-free'

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - The issue happens due to the race between 'save' and 'restore' of 'charsets_dir' global variable In multi-source replication the slave server tries to establish two `mysql` connections with two master servers. #0 mysql_set_character_set_with_default_collation (mysql=0x61a000030c98) at /home/sujatha/bug_repo/test-10.5/sql-common/client.c:1747 #1 0x000055d6c9f7ea7b in mysql_init_character_set (mysql=0x61a000030c98) at /home/sujatha/bug_repo/test-10.5/sql-common/client.c:1799 #2 0x000055d6c9f8624a in mysql_real_connect (mysql=0x61a000030c98, host=0x62a00005a8ce "localhost", user=0x62a00005a983 "root", passwd=0x62a00005ab04 "", db=0x0, port=16000, unix_socket=0x0, client_flag=2147483648) at /home/sujatha/bug_repo/test-10.5/sql-common/client.c:3067 #3 0x000055d6c95e1b31 in connect_to_master (thd=0x62b000046218, mysql=0x61a000030c98, mi=0x62a00005a200, reconnect=false, suppress_warnings=false) at /home/sujatha/bug_repo/test-10.5/sql/slave.cc:7210 #4 0x000055d6c95e106a in safe_connect (thd=0x62b000046218, mysql=0x61a000030c98, mi=0x62a00005a200) at /home/sujatha/bug_repo/test-10.5/sql/slave.cc:7122 #5 0x000055d6c95d0081 in handle_slave_io (arg=0x62a00005a200) at /home/sujatha/bug_repo/test-10.5/sql/slave.cc:4652 #6 0x000055d6ca951277 in pfs_spawn_thread (arg=0x61500001fe98) at /home/sujatha/bug_repo/test-10.5/storage/perfschema/pfs.cc:2201 #7 0x00007fa8b81c76db in start_thread (arg=0x7fa8a86f3700) at pthread_create.c:463 #8 0x00007fa8b736271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 They both reach 'mysql_set_character_set_with_default_collation'. Within this function two variables are of our interest. 1. charsets_dir : It is a global variable. It points to static array i.e static char mysql_charsets_dir [FN_REFLEN+1] 2. mysql->options.charset_dir : It is dynamically allocated on heap. File Name: sql/slave.cc Function Name: `connect_to_master` provides 'charsets_dir' as argument to `mysql_options` to create 'mysql->options.charset_dir'. Then memory is dynamically allocated and it holds copy of 'charsets_dir'. /* This one is not strictly needed but we have it here for completeness */ mysql_options(mysql, MYSQL_SET_CHARSET_DIR, (char *) charsets_dir); File Name: sql-common/client.c case MYSQL_SET_CHARSET_DIR: my_free(mysql->options.charset_dir); mysql->options.charset_dir= opt_strdup(arg,MYF(MY_WME)); (Memory gets allocated on heap) 3. Within 'mysql_set_character_set_with_default_collation ' static void mysql_set_character_set_with_default_collation(MYSQL *mysql) { const char *save= charsets_dir; if (mysql->options.charset_dir) charsets_dir=mysql->options.charset_dir; ... charsets_dir= save; } For example: charsets_dir=mysql_charsets_dir=0x56164f9747e0 mysql->options.charset_dir:0x607000080098 (mysql_1) mysql->options.charset_dir:0x607000087098 (mysql_2) SHOW VARIABLES 'character_sets_dir' execution at this time will report 'AddressSanitizer: heap-use-after-free'

            Recent occurrence:

            10.6 e441c32a0be631c4f39c7337718d1d91ad721b79

            ***Warnings generated in error logs during shutdown after running tests: multi_source.mdev-9544
             
            ==455386==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000070058 at pc 0x556ea7839ea4 bp 0x7fd694c16f60 sp 0x7fd694c16f58
            SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.6/strings/strend.c:45 in strend
            Attempting backtrace. You can use the following information to find out
             
            multi_source.mdev-9544 'innodb'          w5 [ pass ]   2551
             
            MTR's internal check of the test case 'multi_source.mdev-9544' failed.
            This means that the test case does not preserve the state that existed
            before the test case was executed.  Most likely the test case did not
            do a proper clean-up. It could also be caused by the previous test run
            by this thread, if the server wasn't restarted.
            This is the diff of the states of the servers before and after the
            test case was executed:
            mysqltest: Logging to '/dev/shm/10.6/mysql-test/var/5/tmp/check-mysqld_4.log'.
            mysqltest: Results saved in '/dev/shm/10.6/mysql-test/var/5/tmp/check-mysqld_4.result'.
            mysqltest: Connecting to server localhost:16103 (socket /dev/shm/10.6/mysql-test/var/tmp/5/mysqld.4.sock) as 'root', connection 'default', attempt 0 ...
            mysqltest: ... Connected.
            mysqltest: Start processing test commands from './include/check-testcase.test' ...
            mysqltest: At line 96: query 'call mtr.check_testcase()' failed: <Unknown> (2013): Lost connection to server during query
            not ok
            

            In the mysqld.4.err I found the following:

            10.6 e441c32a0be631c4f39c7337718d1d91ad721b79

            CURRENT_TEST: multi_source.mdev-9544
            $ /dev/shm/10.6/sql/mariadbd --defaults-group-suffix=.4 --defaults-file=/dev/shm/10.6/mysql-test/var/5/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
            2023-01-05 10:16:15 0 [Note] /dev/shm/10.6/sql/mariadbd (server 10.6.12-MariaDB-debug-log) starting as process 455386 ...
            2023-01-05 10:16:15 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32222)
            2023-01-05 10:16:15 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
            2023-01-05 10:16:15 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
            2023-01-05 10:16:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
            2023-01-05 10:16:15 0 [Note] InnoDB: Using transactional memory
            2023-01-05 10:16:15 0 [Note] InnoDB: Number of pools: 1
            2023-01-05 10:16:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
            2023-01-05 10:16:15 0 [Note] InnoDB: Using liburing
            2023-01-05 10:16:15 0 [Note] InnoDB: Initializing buffer pool, total size = 8388608, chunk size = 8388608
            2023-01-05 10:16:15 0 [Note] InnoDB: Completed initialization of buffer pool
            2023-01-05 10:16:15 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed
            2023-01-05 10:16:15 0 [Note] InnoDB: 128 rollback segments are active.
            2023-01-05 10:16:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables
            2023-01-05 10:16:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            2023-01-05 10:16:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            2023-01-05 10:16:15 0 [Note] InnoDB: 10.6.12 started; log sequence number 52307; transaction id 16
            2023-01-05 10:16:15 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/10.6/mysql-test/var/5/mysqld.4/data/ib_buffer_pool
            2023-01-05 10:16:15 0 [Note] InnoDB: Buffer pool(s) load completed at 230105 10:16:15
            2023-01-05 10:16:15 0 [Note] Plugin 'partition' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'SEQUENCE' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMP' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_WAITS' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_QUEUES' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'FEEDBACK' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_GROUPS' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'user_variables' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_STATS' is disabled.
            2023-01-05 10:16:15 0 [Note] Plugin 'unix_socket' is disabled.
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown option '--loose-pam-debug'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown option '--loose-enable-performance-schema'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-accounts-size=100'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-digests-size=200'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-hosts-size=100'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-users-size=100'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-mutex-instances=5000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-rwlock-instances=5000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-cond-instances=1000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-file-instances=10000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-socket-instances=1000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-table-instances=500'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-table-lock-stat=500'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-index-stat=5000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-table-handles=1000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-prepared-statements-instances=100'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-program-instances=5000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-setup-actors-size=100'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-setup-objects-size=100'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-waits-history-size=10'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-waits-history-long-size=10000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-stages-history-size=10'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-stages-history-long-size=1000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-statements-history-size=10'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-statements-history-long-size=1000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-transactions-history-size=10'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-transactions-history-long-size=1000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-thread-instances=200'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-session-connect-attrs-size=2048'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-metadata-locks=10000'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-instrument=%=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-stages-current=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-stages-history=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-stages-history-long=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-statements-current=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-statements-history=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-statements-history-long=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-transactions-current=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-transactions-history=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-transactions-history-long=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-waits-current=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-waits-history=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-waits-history-long=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-global-instrumentation=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-thread-instrumentation=ON'
            2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown option '--loose-aria'
            2023-01-05 10:16:15 0 [Note] Server socket created on IP: '127.0.0.1'.
            2023-01-05 10:16:15 0 [Note] /dev/shm/10.6/sql/mariadbd: ready for connections.
            Version: '10.6.12-MariaDB-debug-log'  socket: '/dev/shm/10.6/mysql-test/var/tmp/5/mysqld.4.sock'  port: 16103  Source distribution
            2023-01-05 10:16:15 5 [Note] Master connection name: 'm1'  Master_info_file: 'master-m1.info'  Relay_info_file: 'relay-log-m1.info'
            2023-01-05 10:16:15 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16100', master_log_file='', master_log_pos='4'.
            2023-01-05 10:16:15 5 [Note] Master connection name: 'm2'  Master_info_file: 'master-m2.info'  Relay_info_file: 'relay-log-m2.info'
            2023-01-05 10:16:15 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16101', master_log_file='', master_log_pos='4'.
            2023-01-05 10:16:15 5 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
            2023-01-05 10:16:15 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16102', master_log_file='', master_log_pos='4'.
            2023-01-05 10:16:15 6 [Note] Master 'm2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16101' in log '' at position 4
            2023-01-05 10:16:15 6 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16101',replication started in log 'FIRST' at position 4
            2023-01-05 10:16:15 7 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4
            2023-01-05 10:16:15 8 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16102' in log '' at position 4
            2023-01-05 10:16:15 9 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4
            2023-01-05 10:16:15 10 [Note] Master 'm1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16100' in log '' at position 4
            2023-01-05 10:16:15 11 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4
            2023-01-05 10:16:15 10 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16100',replication started in log 'FIRST' at position 4
            2023-01-05 10:16:15 8 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16102',replication started in log 'FIRST' at position 4
            2023-01-05 10:16:18 7 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed
            2023-01-05 10:16:18 7 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16101
            2023-01-05 10:16:18 6 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16101
            2023-01-05 10:16:18 9 [Note] Error reading relay log event: slave SQL thread was killed
            2023-01-05 10:16:18 9 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16102
            2023-01-05 10:16:18 8 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16102
            2023-01-05 10:16:18 11 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed
            2023-01-05 10:16:18 11 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16100
            2023-01-05 10:16:18 10 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16100
            =================================================================
            ==455386==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000070058 at pc 0x556ea7839ea4 bp 0x7fd694c16f60 sp 0x7fd694c16f58
            READ of size 1 at 0x606000070058 thread T25
                #0 0x556ea7839ea3 in strend /mariadb/10.6/strings/strend.c:45
                #1 0x556ea6a08427 in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /mariadb/10.6/sql/sql_show.cc:3693
                #2 0x556ea6a3410f in show_status_array /mariadb/10.6/sql/sql_show.cc:3837
                #3 0x556ea6a34abe in fill_variables(THD*, TABLE_LIST*, Item*) /mariadb/10.6/sql/sql_show.cc:7942
                #4 0x556ea6a2b376 in get_schema_tables_result(JOIN*, enum_schema_table_state) /mariadb/10.6/sql/sql_show.cc:8881
                #5 0x556ea69a8d28 in JOIN::exec_inner() /mariadb/10.6/sql/sql_select.cc:4751
                #6 0x556ea69a939f in JOIN::exec() /mariadb/10.6/sql/sql_select.cc:4572
                #7 0x556ea69a56b0 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /mariadb/10.6/sql/sql_select.cc:5051
                #8 0x556ea69a5f46 in handle_select(THD*, LEX*, select_result*, unsigned long) /mariadb/10.6/sql/sql_select.cc:554
                #9 0x556ea6850caa in execute_sqlcom_select /mariadb/10.6/sql/sql_parse.cc:6259
                #10 0x556ea6869e1d in mysql_execute_command(THD*, bool) /mariadb/10.6/sql/sql_parse.cc:3949
                #11 0x556ea65bc4be in sp_instr_stmt::exec_core(THD*, unsigned int*) /mariadb/10.6/sql/sp_head.cc:3779
                #12 0x556ea65cc731 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /mariadb/10.6/sql/sp_head.cc:3504
                #13 0x556ea65cd55d in sp_instr_stmt::execute(THD*, unsigned int*) /mariadb/10.6/sql/sp_head.cc:3685
                #14 0x556ea65d3e51 in sp_head::execute(THD*, bool) /mariadb/10.6/sql/sp_head.cc:1438
                #15 0x556ea65d6f78 in sp_head::execute_procedure(THD*, List<Item>*) /mariadb/10.6/sql/sp_head.cc:2452
                #16 0x556ea684f7c3 in do_execute_sp /mariadb/10.6/sql/sql_parse.cc:3026
                #17 0x556ea6859275 in Sql_cmd_call::execute(THD*) /mariadb/10.6/sql/sql_parse.cc:3272
                #18 0x556ea6876048 in mysql_execute_command(THD*, bool) /mariadb/10.6/sql/sql_parse.cc:6000
                #19 0x556ea68779a0 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /mariadb/10.6/sql/sql_parse.cc:8019
                #20 0x556ea687b786 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /mariadb/10.6/sql/sql_parse.cc:1896
                #21 0x556ea687e6ef in do_command(THD*, bool) /mariadb/10.6/sql/sql_parse.cc:1409
                #22 0x556ea6be9a20 in do_handle_one_connection(CONNECT*, bool) /mariadb/10.6/sql/sql_connect.cc:1416
                #23 0x556ea6be9fb2 in handle_one_connection /mariadb/10.6/sql/sql_connect.cc:1318
                #24 0x7fd6a40a7fd3 in start_thread nptl/pthread_create.c:442
                #25 0x7fd6a412866b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
             
            0x606000070058 is located 24 bytes inside of 64-byte region [0x606000070040,0x606000070080)
            freed by thread T30 here:
                #0 0x7fd6a4eb76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
                #1 0x556ea776ebb2 in my_free /mariadb/10.6/mysys/my_malloc.c:211
             
            previously allocated by thread T30 here:
                #0 0x7fd6a4eb89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
                #1 0x556ea776e99a in my_malloc /mariadb/10.6/mysys/my_malloc.c:90
             
            Thread T25 created by T0 here:
                #0 0x7fd6a4e49726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
                #1 0x556ea612bd70 in create_thread_to_handle_connection(CONNECT*) /mariadb/10.6/sql/mysqld.cc:5992
             
            Thread T30 created by T25 here:
                #0 0x7fd6a4e49726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
                #1 0x556ea6585596 in start_slave_thread(void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /mariadb/10.6/sql/slave.cc:1150
             
            SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.6/strings/strend.c:45 in strend
            Shadow bytes around the buggy address:
              0x0c0c80005fb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c80005fc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c80005fd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c80005fe0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
              0x0c0c80005ff0: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd
            =>0x0c0c80006000: fd fd fd fd fa fa fa fa fd fd fd[fd]fd fd fd fd
              0x0c0c80006010: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
              0x0c0c80006020: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
              0x0c0c80006030: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
              0x0c0c80006040: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
              0x0c0c80006050: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
            Shadow byte legend (one shadow byte represents 8 application bytes):
              Addressable:           00
              Partially addressable: 01 02 03 04 05 06 07 
              Heap left redzone:       fa
              Freed heap region:       fd
              Stack left redzone:      f1
              Stack mid redzone:       f2
              Stack right redzone:     f3
              Stack after return:      f5
              Stack use after scope:   f8
              Global redzone:          f9
              Global init order:       f6
              Poisoned by user:        f7
              Container overflow:      fc
              Array cookie:            ac
              Intra object redzone:    bb
              ASan internal:           fe
              Left alloca redzone:     ca
              Right alloca redzone:    cb
            ==455386==ABORTING
            

            marko Marko Mäkelä added a comment - Recent occurrence: 10.6 e441c32a0be631c4f39c7337718d1d91ad721b79 ***Warnings generated in error logs during shutdown after running tests: multi_source.mdev-9544   ==455386==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000070058 at pc 0x556ea7839ea4 bp 0x7fd694c16f60 sp 0x7fd694c16f58 SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.6/strings/strend.c:45 in strend Attempting backtrace. You can use the following information to find out   multi_source.mdev-9544 'innodb' w5 [ pass ] 2551   MTR's internal check of the test case 'multi_source.mdev-9544' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/dev/shm/10.6/mysql-test/var/5/tmp/check-mysqld_4.log'. mysqltest: Results saved in '/dev/shm/10.6/mysql-test/var/5/tmp/check-mysqld_4.result'. mysqltest: Connecting to server localhost:16103 (socket /dev/shm/10.6/mysql-test/var/tmp/5/mysqld.4.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: At line 96: query 'call mtr.check_testcase()' failed: <Unknown> (2013): Lost connection to server during query not ok In the mysqld.4.err I found the following: 10.6 e441c32a0be631c4f39c7337718d1d91ad721b79 CURRENT_TEST: multi_source.mdev-9544 $ /dev/shm/10.6/sql/mariadbd --defaults-group-suffix=.4 --defaults-file=/dev/shm/10.6/mysql-test/var/5/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300 2023-01-05 10:16:15 0 [Note] /dev/shm/10.6/sql/mariadbd (server 10.6.12-MariaDB-debug-log) starting as process 455386 ... 2023-01-05 10:16:15 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32222) 2023-01-05 10:16:15 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2023-01-05 10:16:15 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2023-01-05 10:16:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.13 2023-01-05 10:16:15 0 [Note] InnoDB: Using transactional memory 2023-01-05 10:16:15 0 [Note] InnoDB: Number of pools: 1 2023-01-05 10:16:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2023-01-05 10:16:15 0 [Note] InnoDB: Using liburing 2023-01-05 10:16:15 0 [Note] InnoDB: Initializing buffer pool, total size = 8388608, chunk size = 8388608 2023-01-05 10:16:15 0 [Note] InnoDB: Completed initialization of buffer pool 2023-01-05 10:16:15 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed 2023-01-05 10:16:15 0 [Note] InnoDB: 128 rollback segments are active. 2023-01-05 10:16:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2023-01-05 10:16:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2023-01-05 10:16:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2023-01-05 10:16:15 0 [Note] InnoDB: 10.6.12 started; log sequence number 52307; transaction id 16 2023-01-05 10:16:15 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/10.6/mysql-test/var/5/mysqld.4/data/ib_buffer_pool 2023-01-05 10:16:15 0 [Note] InnoDB: Buffer pool(s) load completed at 230105 10:16:15 2023-01-05 10:16:15 0 [Note] Plugin 'partition' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'SEQUENCE' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMP' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_WAITS' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_QUEUES' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_GROUPS' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'user_variables' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'THREAD_POOL_STATS' is disabled. 2023-01-05 10:16:15 0 [Note] Plugin 'unix_socket' is disabled. 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown option '--loose-pam-debug' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown option '--loose-enable-performance-schema' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-accounts-size=100' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-digests-size=200' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-hosts-size=100' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-users-size=100' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-mutex-instances=5000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-rwlock-instances=5000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-cond-instances=1000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-file-instances=10000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-socket-instances=1000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-table-instances=500' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-table-lock-stat=500' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-index-stat=5000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-table-handles=1000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-prepared-statements-instances=100' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-program-instances=5000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-setup-actors-size=100' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-setup-objects-size=100' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-waits-history-size=10' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-waits-history-long-size=10000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-stages-history-size=10' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-stages-history-long-size=1000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-statements-history-size=10' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-statements-history-long-size=1000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-transactions-history-size=10' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-events-transactions-history-long-size=1000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-thread-instances=200' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-session-connect-attrs-size=2048' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-max-metadata-locks=10000' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-instrument=%=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-stages-current=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-stages-history=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-stages-history-long=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-statements-current=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-statements-history=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-statements-history-long=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-transactions-current=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-transactions-history=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-transactions-history-long=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-waits-current=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-waits-history=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-events-waits-history-long=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-global-instrumentation=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown variable 'loose-performance-schema-consumer-thread-instrumentation=ON' 2023-01-05 10:16:15 0 [Warning] /dev/shm/10.6/sql/mariadbd: unknown option '--loose-aria' 2023-01-05 10:16:15 0 [Note] Server socket created on IP: '127.0.0.1'. 2023-01-05 10:16:15 0 [Note] /dev/shm/10.6/sql/mariadbd: ready for connections. Version: '10.6.12-MariaDB-debug-log' socket: '/dev/shm/10.6/mysql-test/var/tmp/5/mysqld.4.sock' port: 16103 Source distribution 2023-01-05 10:16:15 5 [Note] Master connection name: 'm1' Master_info_file: 'master-m1.info' Relay_info_file: 'relay-log-m1.info' 2023-01-05 10:16:15 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16100', master_log_file='', master_log_pos='4'. 2023-01-05 10:16:15 5 [Note] Master connection name: 'm2' Master_info_file: 'master-m2.info' Relay_info_file: 'relay-log-m2.info' 2023-01-05 10:16:15 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16101', master_log_file='', master_log_pos='4'. 2023-01-05 10:16:15 5 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2023-01-05 10:16:15 5 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16102', master_log_file='', master_log_pos='4'. 2023-01-05 10:16:15 6 [Note] Master 'm2': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16101' in log '' at position 4 2023-01-05 10:16:15 6 [Note] Master 'm2': Slave I/O thread: connected to master 'root@127.0.0.1:16101',replication started in log 'FIRST' at position 4 2023-01-05 10:16:15 7 [Note] Master 'm2': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m2.000001' position: 4 2023-01-05 10:16:15 8 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16102' in log '' at position 4 2023-01-05 10:16:15 9 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin.000001' position: 4 2023-01-05 10:16:15 10 [Note] Master 'm1': Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16100' in log '' at position 4 2023-01-05 10:16:15 11 [Note] Master 'm1': Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './mysqld-relay-bin-m1.000001' position: 4 2023-01-05 10:16:15 10 [Note] Master 'm1': Slave I/O thread: connected to master 'root@127.0.0.1:16100',replication started in log 'FIRST' at position 4 2023-01-05 10:16:15 8 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16102',replication started in log 'FIRST' at position 4 2023-01-05 10:16:18 7 [Note] Master 'm2': Error reading relay log event: slave SQL thread was killed 2023-01-05 10:16:18 7 [Note] Master 'm2': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16101 2023-01-05 10:16:18 6 [Note] Master 'm2': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16101 2023-01-05 10:16:18 9 [Note] Error reading relay log event: slave SQL thread was killed 2023-01-05 10:16:18 9 [Note] Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16102 2023-01-05 10:16:18 8 [Note] Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16102 2023-01-05 10:16:18 11 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed 2023-01-05 10:16:18 11 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16100 2023-01-05 10:16:18 10 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16100 ================================================================= ==455386==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000070058 at pc 0x556ea7839ea4 bp 0x7fd694c16f60 sp 0x7fd694c16f58 READ of size 1 at 0x606000070058 thread T25 #0 0x556ea7839ea3 in strend /mariadb/10.6/strings/strend.c:45 #1 0x556ea6a08427 in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /mariadb/10.6/sql/sql_show.cc:3693 #2 0x556ea6a3410f in show_status_array /mariadb/10.6/sql/sql_show.cc:3837 #3 0x556ea6a34abe in fill_variables(THD*, TABLE_LIST*, Item*) /mariadb/10.6/sql/sql_show.cc:7942 #4 0x556ea6a2b376 in get_schema_tables_result(JOIN*, enum_schema_table_state) /mariadb/10.6/sql/sql_show.cc:8881 #5 0x556ea69a8d28 in JOIN::exec_inner() /mariadb/10.6/sql/sql_select.cc:4751 #6 0x556ea69a939f in JOIN::exec() /mariadb/10.6/sql/sql_select.cc:4572 #7 0x556ea69a56b0 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /mariadb/10.6/sql/sql_select.cc:5051 #8 0x556ea69a5f46 in handle_select(THD*, LEX*, select_result*, unsigned long) /mariadb/10.6/sql/sql_select.cc:554 #9 0x556ea6850caa in execute_sqlcom_select /mariadb/10.6/sql/sql_parse.cc:6259 #10 0x556ea6869e1d in mysql_execute_command(THD*, bool) /mariadb/10.6/sql/sql_parse.cc:3949 #11 0x556ea65bc4be in sp_instr_stmt::exec_core(THD*, unsigned int*) /mariadb/10.6/sql/sp_head.cc:3779 #12 0x556ea65cc731 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /mariadb/10.6/sql/sp_head.cc:3504 #13 0x556ea65cd55d in sp_instr_stmt::execute(THD*, unsigned int*) /mariadb/10.6/sql/sp_head.cc:3685 #14 0x556ea65d3e51 in sp_head::execute(THD*, bool) /mariadb/10.6/sql/sp_head.cc:1438 #15 0x556ea65d6f78 in sp_head::execute_procedure(THD*, List<Item>*) /mariadb/10.6/sql/sp_head.cc:2452 #16 0x556ea684f7c3 in do_execute_sp /mariadb/10.6/sql/sql_parse.cc:3026 #17 0x556ea6859275 in Sql_cmd_call::execute(THD*) /mariadb/10.6/sql/sql_parse.cc:3272 #18 0x556ea6876048 in mysql_execute_command(THD*, bool) /mariadb/10.6/sql/sql_parse.cc:6000 #19 0x556ea68779a0 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /mariadb/10.6/sql/sql_parse.cc:8019 #20 0x556ea687b786 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /mariadb/10.6/sql/sql_parse.cc:1896 #21 0x556ea687e6ef in do_command(THD*, bool) /mariadb/10.6/sql/sql_parse.cc:1409 #22 0x556ea6be9a20 in do_handle_one_connection(CONNECT*, bool) /mariadb/10.6/sql/sql_connect.cc:1416 #23 0x556ea6be9fb2 in handle_one_connection /mariadb/10.6/sql/sql_connect.cc:1318 #24 0x7fd6a40a7fd3 in start_thread nptl/pthread_create.c:442 #25 0x7fd6a412866b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81   0x606000070058 is located 24 bytes inside of 64-byte region [0x606000070040,0x606000070080) freed by thread T30 here: #0 0x7fd6a4eb76a8 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 #1 0x556ea776ebb2 in my_free /mariadb/10.6/mysys/my_malloc.c:211   previously allocated by thread T30 here: #0 0x7fd6a4eb89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 #1 0x556ea776e99a in my_malloc /mariadb/10.6/mysys/my_malloc.c:90   Thread T25 created by T0 here: #0 0x7fd6a4e49726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207 #1 0x556ea612bd70 in create_thread_to_handle_connection(CONNECT*) /mariadb/10.6/sql/mysqld.cc:5992   Thread T30 created by T25 here: #0 0x7fd6a4e49726 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207 #1 0x556ea6585596 in start_slave_thread(void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /mariadb/10.6/sql/slave.cc:1150   SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.6/strings/strend.c:45 in strend Shadow bytes around the buggy address: 0x0c0c80005fb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0c80005fc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0c80005fd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0c80005fe0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0c80005ff0: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fd =>0x0c0c80006000: fd fd fd fd fa fa fa fa fd fd fd[fd]fd fd fd fd 0x0c0c80006010: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa 0x0c0c80006020: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd 0x0c0c80006030: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa 0x0c0c80006040: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa 0x0c0c80006050: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==455386==ABORTING

            Yes, I think that the MemorySanitizer in recent clang can catch heap-use-after-free, by declaring freed memory as uninitialized. There is also MSAN_OPTIONS=poison_in_dtor=1 which was recently enabled by default; see MDEV-30942.

            Here is one more recent failure:

            bb-10.9-release 6f91631bae0bc6dfc527ec3c172e9fa9f83a1161

            multi_source.mdev-9544 'innodb'          w20 [ fail ]
                    Test ended at 2023-08-02 18:56:47
             
            CURRENT_TEST: multi_source.mdev-9544
             
             
            The server [mysqld.4 - pid: 35689, winpid: 35689, exit: 256] crashed while running 'check testcase after test'
            …
            2023-08-02 18:56:43 11 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed
            2023-08-02 18:56:43 11 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16300
            2023-08-02 18:56:43 10 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16300
            ==35746==WARNING: MemorySanitizer: use-of-uninitialized-value
                #0 0x55e36abca6b6 in strend /home/buildbot/amd64-debian-11-msan/build/strings/strend.c:45:3
                #1 0x55e368204382 in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc
                #2 0x55e36822abfc in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc:3897:14
                #3 0x55e3682296ff in fill_variables(THD*, TABLE_LIST*, Item*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc:7996:8
                #4 0x55e368234bec in get_schema_tables_result(JOIN*, enum_schema_table_state) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc:8941:11
                #5 0x55e3680e48e6 in JOIN::exec_inner() /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:4806:7
                #6 0x55e3680e208a in JOIN::exec() /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:4627:3
                #7 0x55e368055fd2 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:5107:9
                #8 0x55e368054b75 in handle_select(THD*, LEX*, select_result*, unsigned long) /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:584:10
                #9 0x55e367f3d91f in execute_sqlcom_select(THD*, TABLE_LIST*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:6288:12
                #10 0x55e367f1deff in mysql_execute_command(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:3958:12
                #11 0x55e367bc941d in sp_instr_stmt::exec_core(THD*, unsigned int*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:3854:12
                #12 0x55e367bc47d7 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:3579:17
                #13 0x55e367bc742d in sp_instr_stmt::execute(THD*, unsigned int*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:3760:25
                #14 0x55e367ba47ad in sp_head::execute(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:1456:20
                #15 0x55e367bad0de in sp_head::execute_procedure(THD*, List<Item>*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:2443:5
                #16 0x55e367f17f0c in do_execute_sp(THD*, sp_head*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:3034:16
                #17 0x55e367f17077 in Sql_cmd_call::execute(THD*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:3280:9
                #18 0x55e367f1dc01 in mysql_execute_command(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:6024:26
                #19 0x55e367f06db9 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:8060:18
                #20 0x55e367efd570 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:1894:7
                #21 0x55e367f084e2 in do_command(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:1407:17
                #22 0x55e368542e86 in do_handle_one_connection(CONNECT*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_connect.cc:1416:11
                #23 0x55e368542489 in handle_one_connection /home/buildbot/amd64-debian-11-msan/build/sql/sql_connect.cc:1318:5
                #24 0x55e36979fede in pfs_spawn_thread /home/buildbot/amd64-debian-11-msan/build/storage/perfschema/pfs.cc:2201:3
                #25 0x7fcb764eeea6 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7ea6) (BuildId: 255e355c207aba91a59ae1f808e3b4da443abf0c)
                #26 0x7fcb75ef6a2e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xfba2e) (BuildId: e15ec78d51a522023f9cfc58dc284f379d81860b)
             
              Uninitialized value was created by a heap deallocation
                #0 0x55e36795672c in free (/home/buildbot/amd64-debian-11-msan/build/sql/mariadbd+0x86872c) (BuildId: d58deefb908e1a6bd640a02927dc493177d22787)
                #1 0x55e368a6fba6 in mysql_close_free_options /home/buildbot/amd64-debian-11-msan/build/sql-common/client.c:3290:3
            

            The reason why I got interested in this test was that I got this local test failure yesterday for a 10.6 based branch:

            multi_source.mdev-9544 'innodb'          w30 [ pass ]   2264
             
            MTR's internal check of the test case 'multi_source.mdev-9544' failed.
            …
            -CHARACTER_SETS_DIR	/mariadb/10.6/sql/share/charsets/
            +CHARACTER_SETS_DIR	????????????????????????????????????????
            

            This would not be flagged as a failure by mtr, so I would guess it to be rather common on buildbot as well.

            marko Marko Mäkelä added a comment - Yes, I think that the MemorySanitizer in recent clang can catch heap-use-after-free, by declaring freed memory as uninitialized. There is also MSAN_OPTIONS=poison_in_dtor=1 which was recently enabled by default; see MDEV-30942 . Here is one more recent failure : bb-10.9-release 6f91631bae0bc6dfc527ec3c172e9fa9f83a1161 multi_source.mdev-9544 'innodb' w20 [ fail ] Test ended at 2023-08-02 18:56:47   CURRENT_TEST: multi_source.mdev-9544     The server [mysqld.4 - pid: 35689, winpid: 35689, exit: 256] crashed while running 'check testcase after test' … 2023-08-02 18:56:43 11 [Note] Master 'm1': Error reading relay log event: slave SQL thread was killed 2023-08-02 18:56:43 11 [Note] Master 'm1': Slave SQL thread exiting, replication stopped in log 'mysqld-bin.000001' at position 1256, master: 127.0.0.1:16300 2023-08-02 18:56:43 10 [Note] Master 'm1': Slave I/O thread exiting, read up to log 'mysqld-bin.000001', position 1256, master 127.0.0.1:16300 ==35746==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x55e36abca6b6 in strend /home/buildbot/amd64-debian-11-msan/build/strings/strend.c:45:3 #1 0x55e368204382 in get_one_variable(THD*, st_mysql_show_var const*, enum_var_type, enum_mysql_show_type, system_status_var*, charset_info_st const**, char*, unsigned long*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc #2 0x55e36822abfc in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc:3897:14 #3 0x55e3682296ff in fill_variables(THD*, TABLE_LIST*, Item*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc:7996:8 #4 0x55e368234bec in get_schema_tables_result(JOIN*, enum_schema_table_state) /home/buildbot/amd64-debian-11-msan/build/sql/sql_show.cc:8941:11 #5 0x55e3680e48e6 in JOIN::exec_inner() /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:4806:7 #6 0x55e3680e208a in JOIN::exec() /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:4627:3 #7 0x55e368055fd2 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:5107:9 #8 0x55e368054b75 in handle_select(THD*, LEX*, select_result*, unsigned long) /home/buildbot/amd64-debian-11-msan/build/sql/sql_select.cc:584:10 #9 0x55e367f3d91f in execute_sqlcom_select(THD*, TABLE_LIST*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:6288:12 #10 0x55e367f1deff in mysql_execute_command(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:3958:12 #11 0x55e367bc941d in sp_instr_stmt::exec_core(THD*, unsigned int*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:3854:12 #12 0x55e367bc47d7 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:3579:17 #13 0x55e367bc742d in sp_instr_stmt::execute(THD*, unsigned int*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:3760:25 #14 0x55e367ba47ad in sp_head::execute(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:1456:20 #15 0x55e367bad0de in sp_head::execute_procedure(THD*, List<Item>*) /home/buildbot/amd64-debian-11-msan/build/sql/sp_head.cc:2443:5 #16 0x55e367f17f0c in do_execute_sp(THD*, sp_head*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:3034:16 #17 0x55e367f17077 in Sql_cmd_call::execute(THD*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:3280:9 #18 0x55e367f1dc01 in mysql_execute_command(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:6024:26 #19 0x55e367f06db9 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:8060:18 #20 0x55e367efd570 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:1894:7 #21 0x55e367f084e2 in do_command(THD*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_parse.cc:1407:17 #22 0x55e368542e86 in do_handle_one_connection(CONNECT*, bool) /home/buildbot/amd64-debian-11-msan/build/sql/sql_connect.cc:1416:11 #23 0x55e368542489 in handle_one_connection /home/buildbot/amd64-debian-11-msan/build/sql/sql_connect.cc:1318:5 #24 0x55e36979fede in pfs_spawn_thread /home/buildbot/amd64-debian-11-msan/build/storage/perfschema/pfs.cc:2201:3 #25 0x7fcb764eeea6 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7ea6) (BuildId: 255e355c207aba91a59ae1f808e3b4da443abf0c) #26 0x7fcb75ef6a2e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xfba2e) (BuildId: e15ec78d51a522023f9cfc58dc284f379d81860b)   Uninitialized value was created by a heap deallocation #0 0x55e36795672c in free (/home/buildbot/amd64-debian-11-msan/build/sql/mariadbd+0x86872c) (BuildId: d58deefb908e1a6bd640a02927dc493177d22787) #1 0x55e368a6fba6 in mysql_close_free_options /home/buildbot/amd64-debian-11-msan/build/sql-common/client.c:3290:3 The reason why I got interested in this test was that I got this local test failure yesterday for a 10.6 based branch: multi_source.mdev-9544 'innodb' w30 [ pass ] 2264   MTR's internal check of the test case 'multi_source.mdev-9544' failed. … -CHARACTER_SETS_DIR /mariadb/10.6/sql/share/charsets/ +CHARACTER_SETS_DIR ???????????????????????????????????????? This would not be flagged as a failure by mtr , so I would guess it to be rather common on buildbot as well.

            I encountered this yesterday again:

            10.6 625a150a86ce6da15fb5c3c39924723c9933bc51

            multi_source.mdev-9544 'innodb'          w20 [ pass ]   2275
             
            MTR's internal check of the test case 'multi_source.mdev-9544' failed.
            …
            -CHARACTER_SETS_DIR	/mariadb/10.6/sql/share/charsets/
            +CHARACTER_SETS_DIR	????????????????????????????????????????
            

            marko Marko Mäkelä added a comment - I encountered this yesterday again: 10.6 625a150a86ce6da15fb5c3c39924723c9933bc51 multi_source.mdev-9544 'innodb' w20 [ pass ] 2275   MTR's internal check of the test case 'multi_source.mdev-9544' failed. … -CHARACTER_SETS_DIR /mariadb/10.6/sql/share/charsets/ +CHARACTER_SETS_DIR ????????????????????????????????????????

            This is a real serious issue in the server code that needs to be fixed, and there's already a perfect analysis by Sujatha from 3 years ago.
            I was able to reproduce easily with valgrind by injecting a small sleep and repeat-running the test:

            diff --git a/sql-common/client.c b/sql-common/client.c
            index 687f28cea28..ad40b5ce680 100644
            --- a/sql-common/client.c
            +++ b/sql-common/client.c
            @@ -1748,6 +1748,7 @@ mysql_set_character_set_with_default_collation(MYSQL *mysql)
               if (mysql->options.charset_dir)
                 charsets_dir=mysql->options.charset_dir;
             
            +  my_sleep(rand() % 250000);
               if ((mysql->charset= get_charset_by_csname(mysql->options.charset_name,
                                                          MY_CS_PRIMARY, MYF(MY_WME))))
               {
            

            knielsen Kristian Nielsen added a comment - This is a real serious issue in the server code that needs to be fixed, and there's already a perfect analysis by Sujatha from 3 years ago. I was able to reproduce easily with valgrind by injecting a small sleep and repeat-running the test: diff --git a/sql-common/client.c b/sql-common/client.c index 687f28cea28..ad40b5ce680 100644 --- a/sql-common/client.c +++ b/sql-common/client.c @@ -1748,6 +1748,7 @@ mysql_set_character_set_with_default_collation(MYSQL *mysql) if (mysql->options.charset_dir) charsets_dir=mysql->options.charset_dir; + my_sleep(rand() % 250000); if ((mysql->charset= get_charset_by_csname(mysql->options.charset_name, MY_CS_PRIMARY, MYF(MY_WME)))) {

            I have a fix for this in st-10.5-knielsen, but I'm blocked from pushing by broken branch protection on github. Feel free to pull the branch if desired.

            knielsen Kristian Nielsen added a comment - I have a fix for this in st-10.5-knielsen, but I'm blocked from pushing by broken branch protection on github. Feel free to pull the branch if desired.

            Pushed to 10.5.

            knielsen Kristian Nielsen added a comment - Pushed to 10.5.

            People

              knielsen Kristian Nielsen
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.