Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21237

rpl.rpl_row_end_of_statement_loss failed in buildbot, server crash on ASAN builder

    XMLWordPrintable

Details

    • Bug
    • Status: Confirmed (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.3, 10.4, 10.5
    • 10.4, 10.5
    • Replication, Tests
    • None

    Description

      http://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/2522

      10.3 c4ed1bee

      rpl.rpl_row_end_of_statement_loss 'row'  w2 [ fail ]
              Test ended at 2019-12-05 04:11:28
       
      CURRENT_TEST: rpl.rpl_row_end_of_statement_loss
      mysqltest: At line 64: query 'SET GLOBAL debug_dbug= @save_debug' failed: 2013: Lost connection to MySQL server during query
       
      The result from queries just before the failure was:
      < snip >
      connection master;
      SET @save_log_bin_compress= @@GLOBAL.log_bin_compress;
      SET @save_log_bin_compress_min_len= @@GLOBAL.log_bin_compress_min_len;
      SET @@GLOBAL.log_bin_compress=ON;
      SET @@GLOBAL.log_bin_compress_min_len=10;
      INSERT INTO t values (2, repeat('b', 8192)), (2, repeat('b', 8192));
      connection slave;
      SET GLOBAL debug_dbug="+d,simulate_stmt_end_rows_event_loss";
      START SLAVE IO_THREAD;
      include/wait_for_slave_io_error.inc [errno=1595]
      SET GLOBAL debug_dbug="-d,simulate_stmt_end_rows_event_loss";
      include/start_slave.inc
      connection master;
      connection slave;
      connection master;
      SET @@GLOBAL.log_bin_compress= @save_log_bin_compress;
      SET @@GLOBAL.log_bin_compress_min_len= @save_log_bin_compress_min_len;
      DROP TABLE t;
      connection slave;
      SET GLOBAL debug_dbug= @save_debug;
      

      ==11877==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00000221ed86 bp 0x7f6ed8d3a3d0 sp 0x7f6ed8d3a280 T39)
          #0 0x221ed85 in _db_doprnt_ /home/buildbot/buildbot/build/mariadb-10.3.21/dbug/dbug.c:1314
          #1 0x21d3aa3 in safe_mutex_lock /home/buildbot/buildbot/build/mariadb-10.3.21/mysys/thr_mutex.c:238
          #2 0x216fd28 in inline_mysql_mutex_lock /home/buildbot/buildbot/build/mariadb-10.3.21/include/mysql/psi/mysql_thread.h:695
          #3 0x2180858 in simple_key_cache_write /home/buildbot/buildbot/build/mariadb-10.3.21/mysys/mf_keycache.c:3339
          #4 0x218b161 in key_cache_write /home/buildbot/buildbot/build/mariadb-10.3.21/mysys/mf_keycache.c:6347
          #5 0x2038ebf in _mi_write_keypage /home/buildbot/buildbot/build/mariadb-10.3.21/storage/myisam/mi_page.c:90
          #6 0x1fe816c in d_search /home/buildbot/buildbot/build/mariadb-10.3.21/storage/myisam/mi_delete.c:349
          #7 0x1fe6cd3 in _mi_ck_real_delete /home/buildbot/buildbot/build/mariadb-10.3.21/storage/myisam/mi_delete.c:172
          #8 0x1fe6976 in _mi_ck_delete /home/buildbot/buildbot/build/mariadb-10.3.21/storage/myisam/mi_delete.c:141
          #9 0x1fe5a63 in mi_delete /home/buildbot/buildbot/build/mariadb-10.3.21/storage/myisam/mi_delete.c:86
          #10 0x1fac1b0 in ha_myisam::delete_row(unsigned char const*) /home/buildbot/buildbot/build/mariadb-10.3.21/storage/myisam/ha_myisam.cc:1848
          #11 0x103fa06 in handler::ha_delete_row(unsigned char const*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/handler.cc:6534
          #12 0xd4c96f in rpl_slave_state::record_gtid(THD*, rpl_gtid const*, unsigned long long, rpl_group_info*, bool, void**) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/rpl_gtid.cc:812
          #13 0xd48c55 in rpl_slave_state::record_and_update_gtid(THD*, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/rpl_gtid.cc:82
          #14 0xc30204 in Relay_log_info::stmt_done(unsigned long long, THD*, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/rpl_rli.cc:1407
          #15 0x12fe50c in Log_event::do_update_pos(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/log_event.cc:1428
          #16 0x6e5676 in Log_event::update_pos(rpl_group_info*) (/home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld+0x6e5676)
          #17 0x6cbe34 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.3.21/sql/slave.cc:3952
          #18 0x6cc4b9 in apply_event_and_update_pos(Log_event*, THD*, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/slave.cc:4062
          #19 0x6cd9da in exec_relay_log_event /home/buildbot/buildbot/build/mariadb-10.3.21/sql/slave.cc:4362
          #20 0x6d3f36 in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.3.21/sql/slave.cc:5529
          #21 0x156403a in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.3.21/storage/perfschema/pfs.cc:1862
          #22 0x7f6ef06f96b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
          #23 0x7f6eefb8e82c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)
       
      AddressSanitizer can not provide additional info.
      SUMMARY: AddressSanitizer: SEGV /home/buildbot/buildbot/build/mariadb-10.3.21/dbug/dbug.c:1314 _db_doprnt_
      Thread T39 created by T32 here:
          #0 0x7f6ef1e75253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
          #1 0x1564427 in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.3.21/storage/perfschema/pfs.cc:1912
          #2 0x6b4f7e in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.3.21/include/mysql/psi/mysql_thread.h:1268
          #3 0x6bc1c0 in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/slave.cc:1246
          #4 0x6bcbb3 in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/slave.cc:1379
          #5 0x9769b4 in start_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_repl.cc:3182
          #6 0x8f34f3 in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_parse.cc:4134
          #7 0x90b89f in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_parse.cc:7818
          #8 0x8e625e in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_parse.cc:1856
          #9 0x8e324e in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_parse.cc:1402
          #10 0xc57d53 in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_connect.cc:1403
          #11 0xc5770c in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.3.21/sql/sql_connect.cc:1308
          #12 0x156403a in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.3.21/storage/perfschema/pfs.cc:1862
          #13 0x7f6ef06f96b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
       
      Thread T32 created by T0 here:
          #0 0x7f6ef1e75253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
          #1 0x1564427 in spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.3.21/storage/perfschema/pfs.cc:1912
          #2 0x64a440 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.3.21/include/mysql/psi/mysql_thread.h:1268
          #3 0x65fe2d in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld.cc:6600
          #4 0x660551 in create_new_thread /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld.cc:6670
          #5 0x6615b1 in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld.cc:6945
          #6 0x65f27d in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld.cc:6222
          #7 0x648465 in main /home/buildbot/buildbot/build/mariadb-10.3.21/sql/main.cc:25
          #8 0x7f6eefaa882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
       
      ==11877==ABORTING
      191205  4:11:27 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.3.21-MariaDB-debug-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=4
      max_threads=153
      thread_count=13
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63286 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x62a0000c6208
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f6ed8d3d050 thread_stack 0x5fc00
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077)[0x7f6ef1e89077]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld(my_print_stacktrace+0xce)[0x21c77be]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld(handle_fatal_signal+0x8ae)[0x100ef26]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6ef0703390]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f6eefabd428]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f6eefabf02a]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xaddc9)[0x7f6ef1eecdc9]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa0769)[0x7f6ef1edf769]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa55d2)[0x7f6ef1ee45d2]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0xa0019)[0x7f6ef1edf019]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x9e061)[0x7f6ef1edd061]
      /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x9c24d)[0x7f6ef1edb24d]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6ef0703390]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld(_db_doprnt_+0x34b)[0x221ed86]
      dict/dict0mem.cc:988(dict_mem_table_fill_foreign_vcol_set(dict_table_t*))[0x21d3aa4]
      dict/dict0dict.cc:622(dict_col_t::name(dict_table_t const&) const)[0x216fd29]
      include/buf0buf.ic:320(buf_page_set_state(buf_page_t*, buf_page_state))[0x2180859]
      buf/buf0flu.cc:2650(page_cleaner_flush_pages_recommendation(unsigned long*, unsigned long))[0x218b162]
      include/buf0buf.ic:313(buf_page_set_state(buf_page_t*, buf_page_state))[0x2038ec0]
      trx/trx0roll.cc:985(trx_roll_pop_top_rec_of_trx(trx_t*, unsigned long*, mem_block_info_t*))[0x1fe816d]
      include/ut0ut.h:346(ib::logger& ib::logger::operator<< <latch_level_t>(latch_level_t const&))[0x1fe6cd4]
      sync/sync0debug.cc:1584(CreateTracker::register_latch(void const*, char const*, unsigned short))[0x1fe6977]
      sync/sync0debug.cc:1557(CreateTracker::CreateTracker())[0x1fe5a64]
      sync/sync0debug.cc:215(LatchDebug::lock_validate(latch_t const*, latch_level_t))[0x1fac1b1]
      srv/srv0mon.cc:2077(srv_mon_process_existing_counter(monitor_id_t, mon_option_t))[0x103fa07]
      sql/rpl_mi.cc:1184(Master_info_index::init_all_master_info())[0xd4c970]
      sql/sql_prepare.cc:3738(Execute_sql_statement::execute_server_code(THD*))[0xd48c56]
      sql/sql_prepare.cc:2985(reinit_stmt_before_use(THD*, LEX*))[0xc30205]
      psi/mysql_file.h:1082(inline_mysql_file_open(unsigned int, char const*, unsigned int, char const*, int, unsigned long))[0x12fe50d]
      sql/sql_yacc_ora.yy:11322(ORAparse(THD*))[0x6e5677]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld[0x6cbe35]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_info+0x38b)[0x6cc4ba]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld[0x6cd9db]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld(handle_slave_sql+0x1e10)[0x6d3f37]
      /home/buildbot/buildbot/build/mariadb-10.3.21/sql/mysqld[0x156403b]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f6ef06f96ba]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6eefb8e82d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): 
      Connection ID (thread ID): 21
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /dev/shm/var/2/mysqld.2/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    0                    bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             23715                23715                processes 
      Max open files            1024                 1024                 files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       23715                23715                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: |/usr/share/apport/apport %p %s %c %P
      
      

      More in the attached file.

      Attachments

        Activity

          People

            Elkin Andrei Elkin
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.