Mariabackup: Assertion `cursor->is_system() || srv_operation == SRV_OPERATION_RESTORE_DELTA || xb_close_files' failed

origin/10.6, 10.6 3f871b339429441ad907ecf7dfabdc414797e664 2021-01-25T19:48:09+02:00  build with debug+asan and run under "rr"

Workflow:

1. Start server and load some initial data

2. Several sessions run concurrent DDL/DML

3. During 2. is ongoing start Mariabackup --backup

Around begin of the backup operation some errors like

[00] 2021-01-25 19:08:01 mariabackup: Generating a list of tablespaces

# 2021-01-25T19:08:23 [323579] | 2021-01-25 19:08:03 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.

# 2021-01-25T19:08:23 [323579] | 2021-01-25 19:08:03 0 [ERROR] InnoDB: The error means the system cannot find the path specified.

# 2021-01-25T19:08:23 [323579] | 2021-01-25 19:08:03 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.

# 2021-01-25T19:08:23 [323579] | 2021-01-25 19:08:03 0 [ERROR] InnoDB: File ./test/t5.ibd: 'open' returned OS error 71.

2021-01-25 19:08:03 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.

2021-01-25 19:08:03 0 [ERROR] InnoDB: The error means the system cannot find the path specified.

2021-01-25 19:08:03 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.

2021-01-25 19:08:03 0 [Warning] InnoDB: Cannot open './test/t5.ibd'.

show up but the operation goes on end harvests finally

mariabackup: /Server/10.6F/extra/mariabackup/fil_cur.cc:155: xb_fil_cur_result_t xb_fil_cur_open(xb_fil_cur_t*, xb_read_filt_t*, fil_node_t*, uint, ulonglong): Assertion `cursor->is_system() || srv_operation == SRV_OPERATION_RESTORE_DELTA || xb_close_files' failed.

(rr) bt

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

#1  0x00001c4d1c31e859 in __GI_abort () at abort.c:79

#2  0x00001c4d1c31e729 in __assert_fail_base (fmt=0x1c4d1c4b4588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x563c9287f340 "cursor->is_system() || srv_operation == SRV_OPERATION_RESTORE_DELTA || xb_close_files", 

    file=0x563c9287f160 "/Server/10.6F/extra/mariabackup/fil_cur.cc", line=155, function=<optimized out>) at assert.c:92

#3  0x00001c4d1c32ff36 in __GI___assert_fail (assertion=0x563c9287f340 "cursor->is_system() || srv_operation == SRV_OPERATION_RESTORE_DELTA || xb_close_files", file=0x563c9287f160 "/Server/10.6F/extra/mariabackup/fil_cur.cc", line=155, 

    function=0x563c9287f3c0 "xb_fil_cur_result_t xb_fil_cur_open(xb_fil_cur_t*, xb_read_filt_t*, fil_node_t*, uint, ulonglong)") at assert.c:101

#4  0x0000563c903cd5b6 in xb_fil_cur_open (cursor=0x404449f525b0, read_filter=0x563c93ede500 <rf_pass_through>, node=0x60c000001cd8, thread_n=1, max_file_size=18446744073709551615) at /Server/10.6F/extra/mariabackup/fil_cur.cc:155

#5  0x0000563c9037f5c3 in xtrabackup_copy_datafile (node=0x60c000001cd8, thread_n=1, dest_name=0x0, write_filter=..., corrupted_pages=...) at /Server/10.6F/extra/mariabackup/xtrabackup.cc:2749

#6  0x0000563c903817bb in data_copy_thread_func (arg=0x604000005610) at /Server/10.6F/extra/mariabackup/xtrabackup.cc:3116

#7  0x00007fc5a3cf1609 in start_thread (arg=<optimized out>) at pthread_create.c:477

#8  0x00001c4d1c41b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

pluto:/data/Results/1611598278/TBR-832/dev/shm/vardir/1611598278/230/1/rr

_RR_TRACE_DIR="." rr replay --mark-stdio

RQG

====

git clone https://github.com/mleich1/rqg --branch experimental RQG

perl rqg.pl \

--grammar=conf/mariadb/table_stress_innodb_nocopy1.yy \

--gendata=conf/mariadb/table_stress.zz \

--gendata_sql=conf/mariadb/table_stress.sql \

--reporters=Mariabackup_linux \

--mysqld=--innodb_use_native_aio=1 \

--mysqld=--innodb_lock_schedule_algorithm=fcfs \

--mysqld=--loose-idle_write_transaction_timeout=0 \

--mysqld=--loose-idle_transaction_timeout=0 \

--mysqld=--loose-idle_readonly_transaction_timeout=0 \

--mysqld=--connect_timeout=60 \

--mysqld=--interactive_timeout=28800 \

--mysqld=--slave_net_timeout=60 \

--mysqld=--net_read_timeout=30 \

--mysqld=--net_write_timeout=60 \

--mysqld=--loose-table_lock_wait_timeout=50 \

--mysqld=--wait_timeout=28800 \

--mysqld=--lock-wait-timeout=86400 \

--mysqld=--innodb-lock-wait-timeout=50 \

--no-mask \

--queries=10000000 \

--seed=random \

--reporters=Backtrace \

--reporters=ErrorLog \

--reporters=Deadlock1 \

--validators=None \

--mysqld=--log_output=none \

--mysqld=--log-bin \

--mysqld=--log_bin_trust_function_creators=1 \

--mysqld=--loose-debug_assert_on_not_freed_memory=0 \

--engine=InnoDB \

--restart_timeout=120 \

--mysqld=--plugin-load-add=file_key_management.so \

--mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \

--duration=300 \

--mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \

--mysqld=--loose-innodb-sync-debug \

--mysqld=--innodb_stats_persistent=on \

--mysqld=--innodb_adaptive_hash_index=off \

--mysqld=--loose-max-statement-time=30 \

--threads=1 \

--mysqld=--innodb_page_size=16K \

--mysqld=--innodb-buffer-pool-size=8M \

--duration=300 \

--no_mask \

--workdir=<local settings> \

--vardir=<local settings> \

--mtr-build-thread=<local settings> \

--basedir1=<local settings> \

--script_debug=_nix_ \

--rr=Extended \

--rr_options=--chaos

(rr)