Stacktrace CURRENT_TEST: mariabackup.absolute_ibdata_paths mysqltest: At line 33: exec of '/home/jenkins/workspace/10.5e-MTR-NORMAL/BuildType/RelWithDebInfo/label/debian-10/bin/mariabackup --defaults-file=/var/tmp/mtr/8/my.cnf --backup --parallel=10 --target-dir=/var/tmp/mtr/8/tmp/backup 2>&1' failed, error: 34304, status: 134, errno: 11 Output from before failure: [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Using server version 10.5.8-5-MariaDB-enterprise-log /home/jenkins/workspace/10.5e-MTR-NORMAL/BuildType/RelWithDebInfo/label/debian-10/bin/mariabackup based on MariaDB server 10.5.8-5-MariaDB Linux (x86_64) [00] 2020-12-01 10:17:56 uses posix_fadvise(). [00] 2020-12-01 10:17:56 cd to /var/tmp/mtr/8/mysqld.1/data/ [00] 2020-12-01 10:17:56 open files limit requested 1024, set to 1024 [00] 2020-12-01 10:17:56 mariabackup: using the following InnoDB configuration: [00] 2020-12-01 10:17:56 innodb_data_home_dir = [00] 2020-12-01 10:17:56 innodb_data_file_path = /var/tmp/mtr/8/tmp/absolute_path_ibdata1:6M;ibdata_second:1M:autoextend [00] 2020-12-01 10:17:56 innodb_log_group_home_dir = ./ [00] 2020-12-01 10:17:56 InnoDB: Using Linux native AIO 2020-12-01 10:17:56 0 [Note] InnoDB: Number of pools: 1 2020-12-01 10:17:56 0 [Note] InnoDB: Opened 2 undo tablespaces [00] 2020-12-01 10:17:56 mariabackup: Generating a list of tablespaces [00] 2020-12-01 10:17:56 >> log scanned up to (48794) [00] 2020-12-01 10:17:56 mariabackup: Starting 10 threads for parallel data files transfer [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 Connecting to MySQL server host: localhost, user: root, password: set, port: 16080, socket: /var/tmp/mtr/tmp/8/mysqld.1.sock [00] 2020-12-01 10:17:56 BACKUP STAGE START [00] 2020-12-01 10:17:56 Acquiring BACKUP LOCKS... [00] 2020-12-01 10:17:56 Copying ./aria_log_control to /var/tmp/mtr/8/tmp/backup/aria_log_control [05] 2020-12-01 10:17:56 Copying .//undo002 to /var/tmp/mtr/8/tmp/backup/undo002 [03] 2020-12-01 10:17:56 Copying ibdata_second to /var/tmp/mtr/8/tmp/backup/ibdata_second [04] 2020-12-01 10:17:56 Copying .//undo001 to /var/tmp/mtr/8/tmp/backup/undo001 [07] 2020-12-01 10:17:56 Copying ./mysql/transaction_registry.ibd to /var/tmp/mtr/8/tmp/backup/mysql/transaction_registry.ibd [00] 2020-12-01 10:17:56 Copying ./test/t.ibd to /var/tmp/mtr/8/tmp/backup/test/t.ibd [00] 2020-12-01 10:17:56 ...done [06] 2020-12-01 10:17:56 Copying ./mysql/innodb_index_stats.ibd to /var/tmp/mtr/8/tmp/backup/mysql/innodb_index_stats.ibd [08] 2020-12-01 10:17:56 Copying ./mysql/innodb_table_stats.ibd to /var/tmp/mtr/8/tmp/backup/mysql/innodb_table_stats.ibd [07] 2020-12-01 10:17:56 ...done [08] 2020-12-01 10:17:56 ...done [02] 2020-12-01 10:17:56 Copying /var/tmp/mtr/8/tmp/absolute_path_ibdata1 to /var/tmp/mtr/8/tmp/backup/absolute_path_ibdata1 [06] 2020-12-01 10:17:56 ...done [02] 2020-12-01 10:17:56 ...done [00] 2020-12-01 10:17:56 ...done [01] 2020-12-01 10:17:56 Start scanning aria tables. [05] 2020-12-01 10:17:56 ...done [01] 2020-12-01 10:17:56 Start scanning aria log files. [00] 2020-12-01 10:17:56 Found aria log file: aria_log.00000001, current: 1, min: 1, max: 1 [01] 2020-12-01 10:17:56 Found 1 aria log files, minimum log number 1, maximum log number 1, last log number 1 [01] 2020-12-01 10:17:56 Stop scanning aria tables. [04] 2020-12-01 10:17:56 ...done [00] 2020-12-01 10:17:56 aria table file ./mysql/server_audit_filters.MAI is copied successfully. [07] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_leap_second.MAI is copied successfully. [08] 2020-12-01 10:17:56 aria table file ./mysql/db.MAI is copied successfully. [09] 2020-12-01 10:17:56 aria table file ./mysql/procs_priv.MAI is copied successfully. [02] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_transition_type.MAI is copied successfully. [06] 2020-12-01 10:17:56 aria table file ./mysql/plugin.MAI is copied successfully. [00] 2020-12-01 10:17:56 aria table file ./mysql/server_audit_filters.MAD is copied successfully. [09] 2020-12-01 10:17:56 aria table file ./mysql/procs_priv.MAD is copied successfully. [08] 2020-12-01 10:17:56 aria table file ./mysql/db.MAD is copied successfully. [01] 2020-12-01 10:17:56 aria table file ./mysql/tables_priv.MAI is copied successfully. [07] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_leap_second.MAD is copied successfully. [06] 2020-12-01 10:17:56 aria table file ./mysql/plugin.MAD is copied successfully. [02] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_transition_type.MAD is copied successfully. [01] 2020-12-01 10:17:56 aria table file ./mysql/tables_priv.MAD is copied successfully. [00] 2020-12-01 10:17:56 aria table file ./mysql/roles_mapping.MAI is copied successfully. [08] 2020-12-01 10:17:56 aria table file ./mysql/columns_priv.MAI is copied successfully. [07] 2020-12-01 10:17:56 aria table file ./mysql/event.MAI is copied successfully. [00] 2020-12-01 10:17:56 aria table file ./mysql/roles_mapping.MAD is copied successfully. [09] 2020-12-01 10:17:56 aria table file ./mysql/proc.MAI is copied successfully. [07] 2020-12-01 10:17:56 aria table file ./mysql/event.MAD is copied successfully. [08] 2020-12-01 10:17:56 aria table file ./mysql/columns_priv.MAD is copied successfully. [06] 2020-12-01 10:17:56 aria table file ./mysql/server_audit_users.MAI is copied successfully. [01] 2020-12-01 10:17:56 aria table file ./mysql/servers.MAI is copied successfully. [06] 2020-12-01 10:17:56 aria table file ./mysql/server_audit_users.MAD is copied successfully. [01] 2020-12-01 10:17:56 aria table file ./mysql/servers.MAD is copied successfully. [09] 2020-12-01 10:17:56 aria table file ./mysql/proc.MAD is copied successfully. [05] 2020-12-01 10:17:56 aria table file ./mysql/func.MAI is copied successfully. [05] 2020-12-01 10:17:56 aria table file ./mysql/func.MAD is copied successfully. [00] 2020-12-01 10:17:56 Start copying aria log file tail: ./aria_log.00000001 [07] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_transition.MAI is copied successfully. [02] 2020-12-01 10:17:56 aria table file ./mysql/proxies_priv.MAI is copied successfully. [04] 2020-12-01 10:17:56 aria table file ./mysql/time_zone.MAI is copied successfully. [00] 2020-12-01 10:17:56 Stop copying aria log file tail: ./aria_log.00000001, copied 32768 bytes [08] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_name.MAI is copied successfully. [02] 2020-12-01 10:17:56 aria table file ./mysql/proxies_priv.MAD is copied successfully. [04] 2020-12-01 10:17:56 aria table file ./mysql/time_zone.MAD is copied successfully. [07] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_transition.MAD is copied successfully. [08] 2020-12-01 10:17:56 aria table file ./mysql/time_zone_name.MAD is copied successfully. [01] 2020-12-01 10:17:56 aria table file ./mysql/global_priv.MAI is copied successfully. [01] 2020-12-01 10:17:56 aria table file ./mysql/global_priv.MAD is copied successfully. [03] 2020-12-01 10:17:56 ...done 2020-12-01 10:17:56 0x7fcb890f9700 InnoDB: Assertion failure in file /home/jenkins/workspace/10.5e-bintars-NIGHTLY/BuildType/RelWithDebInfo/label/debian-10/extra/mariabackup/fil_cur.cc line 111 InnoDB: Failing assertion: fil_system.n_open > 0 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ InnoDB: about forcing recovery. 201201 10:17:56 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.8-5-MariaDB-enterprise key_buffer_size=0 read_buffer_size=131072 max_used_connections=0 max_threads=1 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6047 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 mysys/stacktrace.c:213(my_print_stacktrace)[0x560c4b2e632e] sql/signal_handler.cc:211(handle_fatal_signal)[0x560c4add9555] ??:0(__restore_rt)[0x7fcb90d8f730] linux/raise.c:51(__GI_raise)[0x7fcb903da7bb] stdlib/abort.c:81(__GI_abort)[0x7fcb903c5535] include/ut0ut.h:338(ib::logger& ib::logger::operator<< (char* const&) [clone .isra.6])[0x560c4aa1c1cc] include/sync0arr.ic:82(sync_array_get_and_reserve_cell)[0x560c4a9cd21d] mariabackup/xtrabackup.cc:2776(xtrabackup_copy_datafile(fil_node_t*, unsigned int, char const*, xb_write_filt_t const&))[0x560c4aa72ddd] mariabackup/xtrabackup.cc:4119(InnodbDataCopier::copy_job(fil_node_t*, unsigned int))[0x560c4aa7a07c] mariabackup/thread_pool.cc:44(ThreadPool::thread_func(unsigned int))[0x560c4aa96c30] ??:0(std::error_code::default_error_condition() const)[0x7fcb907a4b2f] nptl/pthread_create.c:487(start_thread)[0x7fcb90d84fa3] x86_64/clone.S:97(clone)[0x7fcb9049c4cf] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/tmp/mtr/8/mysqld.1/data Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 128448 128448 processes Max open files 1024 1024 files Max locked memory 67108864 67108864 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 128448 128448 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core Aborted The result from queries just before the failure was: CREATE TABLE t(i INT) ENGINE INNODB; INSERT INTO t VALUES(1); # restart # xtrabackup backup