Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19346

Assertion `recv_sys->mlog_checkpoint_lsn <= recv_sys->recovered_lsn' failed in recv_parse_log_recs during mariabackup --prepare

    Details

      Description

      I found a problem (different assert but server setup differs somehow too) during
      RQG testing.
      The attempt to derive some MTR based test shows the assert reported here.
      1. Load some data
      2. mariabackup --backup (passes)
      3. mariabackup --prepare (fails with core)
      mariabackup: storage/innobase/log/log0recv.cc:2857: bool recv_parse_log_recs(lsn_t, store_t, bool): Assertion `recv_sys->mlog_checkpoint_lsn <= recv_sys->recovered_lsn' failed.
      190426 12:52:08 [ERROR] mysqld got signal 6 ;
      ...
      #2  <signal handler called>
      #3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #4  0x00007fae1b888f5d in __GI_abort () at abort.c:90
      #5  0x00007fae1b87ef17 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x561ff08fe288 "recv_sys->mlog_checkpoint_lsn <= recv_sys->recovered_lsn", file=file@entry=0x561ff08fc698 "storage/innobase/log/log0recv.cc", line=line@entry=2857, function=function@entry=0x561ff0901360 <recv_parse_log_recs(unsigned long, store_t, bool)::__PRETTY_FUNCTION__> "bool recv_parse_log_recs(lsn_t, store_t, bool)") at assert.c:92
      #6  0x00007fae1b87efc2 in __GI___assert_fail (assertion=0x561ff08fe288 "recv_sys->mlog_checkpoint_lsn <= recv_sys->recovered_lsn", file=0x561ff08fc698 "storage/innobase/log/log0recv.cc", line=2857, function=0x561ff0901360 <recv_parse_log_recs(unsigned long, store_t, bool)::__PRETTY_FUNCTION__> "bool recv_parse_log_recs(lsn_t, store_t, bool)") at assert.c:101
      #7  0x0000561ff02b0491 in recv_parse_log_recs (checkpoint_lsn=21762728, store=STORE_YES, apply=true) at storage/innobase/log/log0recv.cc:2857
      #8  0x0000561ff02b14df in recv_scan_log_recs (available_memory=96468992, store_to_hash=0x7ffd2e245c54, log_block=0x7fae146f6200 '\245' <repeats 200 times>..., checkpoint_lsn=21762728, start_lsn=22614528, end_lsn=22680064, contiguous_lsn=0x7ffd2e245ce0, group_scanned_lsn=0x561ff30a7530) at storage/innobase/log/log0recv.cc:3329
      #9  0x0000561ff02b19f0 in recv_group_scan_log_recs (group=0x561ff30a7500, checkpoint_lsn=21762728, contiguous_lsn=0x7ffd2e245ce0, last_phase=false) at storage/innobase/log/log0recv.cc:3427
      #10 0x0000561ff02b2d2b in recv_recovery_from_checkpoint_start (flush_lsn=1633870) at storage/innobase/log/log0recv.cc:3751
      #11 0x0000561ff03bfce7 in innobase_start_or_create_for_mysql () at storage/innobase/srv/srv0start.cc:2188
      #12 0x0000561fefb88878 in innodb_init () at extra/mariabackup/xtrabackup.cc:1990
      #13 0x0000561fefb92a1b in xtrabackup_prepare_func (argv=0x561ff2bcca90) at extra/mariabackup/xtrabackup.cc:5571
      #14 0x0000561fefb94d40 in main_low (argv=0x561ff2bcca90) at extra/mariabackup/xtrabackup.cc:6398
      #15 0x0000561fefb94552 in main (argc=7, argv=0x7ffd2e24a9f8) at extra/mariabackup/xtrabackup.cc:6191
       
      10.2 commit 3dffdee667666df9ade9f2c458bf1ea495ffba02 2019-04-25
       
      The assert seen during RQG testing (also during mariabackup --prepare) was
      mariabackup: storage/innobase/fil/fil0fil.cc:582: bool fil_node_t::read_page0(bool): Assertion `space->free_limit == 0 || space->free_limit == free_limit' failed.
      ...
      # 2019-04-25T17:37:52 [228049] | mysys/stacktrace.c:267(my_print_stacktrace)[0x563bac73a344]
      # 2019-04-25T17:37:52 [228049] | sql/signal_handler.cc:168(handle_fatal_signal)[0x563bac07401d]
      # 2019-04-25T17:37:52 [228049] | /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7fc6eb261670]
      [228050] SELECT 13 /* E_R Thread1 QNO 19501 CON_ID 16 */ ;
      # 2019-04-25T17:37:52 [228049] | linux/raise.c:58(__GI_raise)[0x7fc6e934477f]
      # 2019-04-25T17:37:52 [228049] | stdlib/abort.c:91(__GI_abort)[0x7fc6e934637a]
      # 2019-04-25T17:37:52 [228049] | assert/assert.c:92(__assert_fail_base)[0x7fc6e933cb47]
      # 2019-04-25T17:37:52 [228049] | /lib/x86_64-linux-gnu/libc.so.6(+0x2dbf2)[0x7fc6e933cbf2]
      # 2019-04-25T17:37:52 [228049] | fil/fil0fil.cc:583(fil_node_t::read_page0(bool))[0x563bac265965]
      # 2019-04-25T17:37:52 [228049] | fil/fil0fil.cc:659(fil_node_open_file(fil_node_t*))[0x563bac265e7e]
      # 2019-04-25T17:37:52 [228049] | fil/fil0fil.cc:4701(fil_node_prepare_for_io(fil_node_t*, fil_system_t*, fil_space_t*))[0x563bac272397]
      # 2019-04-25T17:37:52 [228049] | fil/fil0fil.cc:4959(fil_io(IORequest const&, bool, page_id_t, page_size_t const&, unsigned long, unsigned long, void*, void*, bool))[0x563bac27303a]
      [228050] SELECT 13 /* E_R Thread1 QNO 19502 CON_ID 16 */ ;
      # 2019-04-25T17:37:52 [228049] | buf/buf0rea.cc:181(buf_read_page_low(dberr_t*, bool, unsigned long, unsigned long, page_id_t, page_size_t const&, bool, bool))[0x563bac2279c2]
      # 2019-04-25T17:37:52 [228049] | buf/buf0rea.cc:942(buf_read_recv_pages(bool, unsigned long, unsigned long const*, unsigned long))[0x563bac229870]
      # 2019-04-25T17:37:52 [228049] | log/log0recv.cc:2282(recv_read_in_area(page_id_t))[0x563bac36d8ff]
      # 2019-04-25T17:37:52 [228049] | log/log0recv.cc:2388(recv_apply_hashed_log_recs(bool))[0x563bac36dee1]
      # 2019-04-25T17:37:52 [228049] | srv/srv0start.cc:2226(innobase_start_or_create_for_mysql())[0x563bac47efda]
      # 2019-04-25T17:37:52 [228049] | mariabackup/xtrabackup.cc:1990(innodb_init())[0x563babc491e9]
      [228050] SELECT 13 /* E_R Thread1 QNO 19503 CON_ID 16 */ ;
      # 2019-04-25T17:37:52 [228049] | mariabackup/xtrabackup.cc:5571(xtrabackup_prepare_func(char**))[0x563babc5340d]
      # 2019-04-25T17:37:52 [228049] | mariabackup/xtrabackup.cc:6398(main_low(char**))[0x563babc55649]
      # 2019-04-25T17:37:52 [228049] | mariabackup/xtrabackup.cc:6191(main)[0x563babc54e5b]
      # 2019-04-25T17:37:52 [228049] | csu/libc-start.c:325(__libc_start_main)[0x7fc6e932f3f1]
      # 2019-04-25T17:37:52 [228049] | /work/10.2_clean/bld_debug/extra/mariabackup/mariabackup(_start+0x2a)[0x563babc448fa]
       
      
      

        Attachments

        1. backup.tar.bz2
          1.69 MB
        2. mtr.prt
          26 kB
        3. PRT
          218 kB
        4. table_stress1.test
          272 kB

          Issue Links

            Activity

              People

              • Assignee:
                marko Marko Mäkelä
                Reporter:
                mleich Matthias Leich
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: