Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23472

ASAN use-after-poison in LatchCounter::Count::reset

    XMLWordPrintable

    Details

      Description

      10.5 6a7e646d ASAN

      ==2626==ERROR: AddressSanitizer: use-after-poison on address 0x7fc80df95c24 at pc 0x55d264b167d3 bp 0x7fc7f6587210 sp 0x7fc7f6587200
      WRITE of size 4 at 0x7fc80df95c24 thread T15
          #0 0x55d264b167d2 in LatchCounter::Count::reset() /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/sync0types.h:533
          #1 0x55d264b169e6 in LatchCounter::reset() /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/sync0types.h:591
          #2 0x55d264b16449 in MutexMonitor::reset() /home/mdbe/enterprise-tests/src/10.5/storage/innobase/sync/sync0sync.cc:251
          #3 0x55d264693c4c in innodb_monitor_set_option /home/mdbe/enterprise-tests/src/10.5/storage/innobase/handler/ha_innodb.cc:17641
          #4 0x55d264693d71 in innodb_monitor_update_wildcard /home/mdbe/enterprise-tests/src/10.5/storage/innobase/handler/ha_innodb.cc:17680
          #5 0x55d264694544 in innodb_monitor_update /home/mdbe/enterprise-tests/src/10.5/storage/innobase/handler/ha_innodb.cc:17936
          #6 0x55d264695054 in innodb_reset_all_monitor_update /home/mdbe/enterprise-tests/src/10.5/storage/innobase/handler/ha_innodb.cc:18154
          #7 0x55d2633cf9b5 in sys_var_pluginvar::global_update(THD*, set_var*) /home/mdbe/enterprise-tests/src/10.5/sql/sql_plugin.cc:3619
          #8 0x55d2630daeff in sys_var::update(THD*, set_var*) /home/mdbe/enterprise-tests/src/10.5/sql/set_var.cc:207
          #9 0x55d2630df6e5 in set_var::update(THD*) /home/mdbe/enterprise-tests/src/10.5/sql/set_var.cc:859
          #10 0x55d2630dec0c in sql_set_variables(THD*, List<set_var_base>*, bool) /home/mdbe/enterprise-tests/src/10.5/sql/set_var.cc:746
          #11 0x55d26338d916 in mysql_execute_command(THD*) /home/mdbe/enterprise-tests/src/10.5/sql/sql_parse.cc:5009
          #12 0x55d2633a16c2 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/mdbe/enterprise-tests/src/10.5/sql/sql_parse.cc:7994
          #13 0x55d26337815c in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/mdbe/enterprise-tests/src/10.5/sql/sql_parse.cc:1867
          #14 0x55d2633749da in do_command(THD*) /home/mdbe/enterprise-tests/src/10.5/sql/sql_parse.cc:1348
          #15 0x55d2637a3991 in do_handle_one_connection(CONNECT*, bool) /home/mdbe/enterprise-tests/src/10.5/sql/sql_connect.cc:1410
          #16 0x55d2637a32ea in handle_one_connection /home/mdbe/enterprise-tests/src/10.5/sql/sql_connect.cc:1312
          #17 0x55d26446d2fa in pfs_spawn_thread /home/mdbe/enterprise-tests/src/10.5/storage/perfschema/pfs.cc:2201
          #18 0x7fc81bdb56da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
          #19 0x7fc81af9ba3e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x121a3e)
       
      0x7fc80df95c24 is located 9252 bytes inside of 4194328-byte region [0x7fc80df93800,0x7fc80e393818)
      allocated by thread T0 here:
          #0 0x7fc81d089d28 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded28)
          #1 0x55d2646b4cba in ut_allocator<unsigned char, true>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/ut0new.h:372
          #2 0x55d264b93e5b in Pool<trx_t, TrxFactory, TrxPoolLock>::Pool(unsigned long) /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/ut0pool.h:66
          #3 0x55d264b92496 in PoolManager<Pool<trx_t, TrxFactory, TrxPoolLock>, TrxPoolManagerLock>::add_pool(unsigned long) /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/ut0pool.h:335
          #4 0x55d264b91ca6 in PoolManager<Pool<trx_t, TrxFactory, TrxPoolLock>, TrxPoolManagerLock>::create() /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/ut0pool.h:363
          #5 0x55d264b90618 in PoolManager<Pool<trx_t, TrxFactory, TrxPoolLock>, TrxPoolManagerLock>::PoolManager(unsigned long) /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/ut0pool.h:247
          #6 0x55d264b7b7ea in trx_pool_init() /home/mdbe/enterprise-tests/src/10.5/storage/innobase/trx/trx0trx.cc:333
          #7 0x55d264abcc34 in srv_boot() /home/mdbe/enterprise-tests/src/10.5/storage/innobase/srv/srv0srv.cc:801
          #8 0x55d264ad12e9 in srv_start(bool) /home/mdbe/enterprise-tests/src/10.5/storage/innobase/srv/srv0start.cc:1225
          #9 0x55d264648884 in innodb_init /home/mdbe/enterprise-tests/src/10.5/storage/innobase/handler/ha_innodb.cc:3930
          #10 0x55d263b56e79 in ha_initialize_handlerton(st_plugin_int*) /home/mdbe/enterprise-tests/src/10.5/sql/handler.cc:645
          #11 0x55d2633bf903 in plugin_initialize /home/mdbe/enterprise-tests/src/10.5/sql/sql_plugin.cc:1459
          #12 0x55d2633c17b7 in plugin_init(int*, char**, int) /home/mdbe/enterprise-tests/src/10.5/sql/sql_plugin.cc:1751
          #13 0x55d26308cc7b in init_server_components /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:4916
          #14 0x55d26308ebed in mysqld_main(int, char**) /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:5499
          #15 0x55d263078da9 in main /home/mdbe/enterprise-tests/src/10.5/sql/main.cc:25
          #16 0x7fc81ae9bb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
       
      Thread T15 created by T0 here:
          #0 0x7fc81cfe2d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
          #1 0x55d26446804c in my_thread_create /home/mdbe/enterprise-tests/src/10.5/storage/perfschema/my_thread.h:38
          #2 0x55d26446d6e9 in pfs_spawn_thread_v1 /home/mdbe/enterprise-tests/src/10.5/storage/perfschema/pfs.cc:2252
          #3 0x55d26307a67c in inline_mysql_thread_create /home/mdbe/enterprise-tests/src/10.5/include/mysql/psi/mysql_thread.h:1321
          #4 0x55d26308fce5 in create_thread_to_handle_connection(CONNECT*) /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:6025
          #5 0x55d263090346 in create_new_thread(CONNECT*) /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:6084
          #6 0x55d26309066c in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:6149
          #7 0x55d263091189 in handle_connections_sockets() /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:6276
          #8 0x55d26308f53e in mysqld_main(int, char**) /home/mdbe/enterprise-tests/src/10.5/sql/mysqld.cc:5671
          #9 0x55d263078da9 in main /home/mdbe/enterprise-tests/src/10.5/sql/main.cc:25
          #10 0x7fc81ae9bb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
       
      SUMMARY: AddressSanitizer: use-after-poison /home/mdbe/enterprise-tests/src/10.5/storage/innobase/include/sync0types.h:533 in LatchCounter::Count::reset()
      Shadow bytes around the buggy address:
        0x0ff981beab30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        0x0ff981beab40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        0x0ff981beab50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        0x0ff981beab60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        0x0ff981beab70: 00 00 f7 f7 f7 00 00 00 00 00 00 00 00 00 00 00
      =>0x0ff981beab80: 00 00 00 00[00]00 00 00 f7 05 f7 f7 f7 f7 f7 f7
        0x0ff981beab90: f7 f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        0x0ff981beaba0: 00 00 00 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
        0x0ff981beabb0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
        0x0ff981beabc0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
        0x0ff981beabd0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==2626==ABORTING
      200813 14:07:20 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.6-MariaDB-debug-log
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=5
      max_threads=153
      thread_count=7
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 193732 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x62b0000e0288
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fc7f6589990 thread_stack 0x5fc00
      /usr/lib/x86_64-linux-gnu/libasan.so.4(+0x558c0)[0x7fc81d0008c0]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(my_print_stacktrace+0xc3)[0x55d2650b2f47]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(handle_fatal_signal+0x9e2)[0x55d263b4d6a7]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0)[0x7fc81bdc08a0]
       
      =================================================================
      ==2753==ERROR: LeakSanitizer: detected memory leaks
       
      Direct leak of 151 byte(s) in 1 object(s) allocated from:
          #0 0x7fbc4ea2ab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
          #1 0x7fbc4e424201 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x4f201)
       
      SUMMARY: AddressSanitizer: 151 byte(s) leaked in 1 allocation(s).
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fc81aeb8f47]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fc81aeba8b1]
       
      =================================================================
      ==2762==ERROR: LeakSanitizer: detected memory leaks
       
      Direct leak of 106 byte(s) in 1 object(s) allocated from:
          #0 0x7f5345d12b40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
          #1 0x7f534570c201 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x4f201)
       
      SUMMARY: AddressSanitizer: 106 byte(s) leaked in 1 allocation(s).
      /usr/lib/x86_64-linux-gnu/libasan.so.4(+0x10072e)[0x7fc81d0ab72e]
      ??:0(__sanitizer_cov_trace_pc_guard_init)[0x7fc81d0b3518]
      ??:0(__asan_unpoison_intra_object_redzone)[0x7fc81d094515]
      ??:0(__asan_report_store4)[0x7fc81d0956fa]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x32707d3)[0x55d264b167d3]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x32709e7)[0x55d264b169e7]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x327044a)[0x55d264b1644a]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x2dedc4d)[0x55d264693c4d]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x2dedd72)[0x55d264693d72]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x2dee545)[0x55d264694545]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(+0x2def055)[0x55d264695055]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(_ZN17sys_var_pluginvar13global_updateEP3THDP7set_var+0x32e)[0x55d2633cf9b6]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(_ZN7sys_var6updateEP3THDP7set_var+0x246)[0x55d2630daf00]
      include/sync0types.h:533(LatchCounter::Count::reset())[0x55d2630df6e6]
      /home/mdbe/enterprise-tests/bld/10.5-6a7e646d-asan/bin/mariadbd(_Z17sql_set_variablesP3THDP4ListI12set_var_baseEb+0x2d1)[0x55d2630dec0d]
      handler/ha_innodb.cc:17642(innodb_monitor_set_option(monitor_info_t const*, mon_option_t))[0x55d26338d917]
      handler/ha_innodb.cc:17688(innodb_monitor_update_wildcard(char const*, mon_option_t))[0x55d2633a16c3]
      handler/ha_innodb.cc:17936(innodb_monitor_update(THD*, void*, void const*, mon_option_t, unsigned long))[0x55d26337815d]
      handler/ha_innodb.cc:18156(innodb_reset_all_monitor_update(THD*, st_mysql_sys_var*, void*, void const*))[0x55d2633749db]
      sql/sql_plugin.cc:3620(sys_var_pluginvar::global_update(THD*, set_var*))[0x55d2637a3992]
      sql/set_var.cc:207(sys_var::update(THD*, set_var*))[0x55d2637a32eb]
      sql/set_var.cc:746(sql_set_variables(THD*, List<set_var_base>*, bool))[0x55d26446d2fb]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fc81bdb56db]
       
      =================================================================
      ==2771==ERROR: LeakSanitizer: detected memory leaks
       
      Direct leak of 151 byte(s) in 1 object(s) allocated from:
          #0 0x7f86bc995b40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
          #1 0x7f86bc38f201 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x4f201)
       
      SUMMARY: AddressSanitizer: 151 byte(s) leaked in 1 allocation(s).
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc81af9ba3f]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x62b0000e72a8): SET /* QNO 353 CON_ID 10 */ GLOBAL rpl_semi_sync_slave_kill_conn_timeout= 100, table_open_cache= 10, key_cache_division_limit= 27, innodb_stats_persistent_sample_pages= 2, key_cache_file_hash_size= 15755, innodb_monitor_reset_all= '%'
      Connection ID (thread ID): 8
      Status: KILL_TIMEOUT
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /dev/shm/var_latch/mysqld.1/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             1288900              1288900              processes 
      Max open files            1024                 1024                 files     
      Max locked memory         16777216             16777216             bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       1288900              1288900              signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
       
      ----------SERVER LOG END-------------
      

      I have a test case, it is small but it's extremely unreliable and not suitable for debugging. It has to be run with a very high --repeat=N value, and still there is no guarantee, the problem can be environment- or compiler-specific. However it was sufficient for creating an rr profile which will be provided.
      The test case is attached for a record. Command-line options for running it are in a comment in the test body.

      The failure has only been observed so far on 10.5. I have no information regarding earlier versions.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: