[MDEV-13890] mariabackup.xb_compressed_encrypted failed in buildbot, InnoDB: assertion failure Created: 2017-09-22  Updated: 2017-11-09  Resolved: 2017-10-27

Status: Closed
Project: MariaDB Server
Component/s: Backup, Storage Engine - InnoDB, Storage Engine - XtraDB, Tests
Affects Version/s: 10.0, 10.1, 10.2, 10.3
Fix Version/s: 10.0.34, 10.1.29, 10.2.10, 10.3.3

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-13807 mariabackup --apply-log-only does gen... Closed
Relates
relates to MDEV-14333 Mariabackup --apply-log-only crashes ... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-x86/builds/4497

mariabackup.xb_compressed_encrypted '8k,innodb' w1 [ fail ]
        Test ended at 2017-09-12 18:24:56
 
CURRENT_TEST: mariabackup.xb_compressed_encrypted
mysqltest: At line 25: exec of '/usr/bin/mariabackup --innobackupex --apply-log /run/shm/var/1/tmp/backup 2>&1' failed, error: 34304, status: 134, errno: 11
Output from before failure:
170912 18:24:49 innobackupex: Starting the apply-log operation
 
IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".
 
--innobackupex based on MariaDB server 10.2.9-MariaDB debian-linux-gnu (i686) 
xtrabackup: cd to /run/shm/var/1/tmp/backup/
Loading encryption plugin
	 Encryption plugin parameter :  '--file_key_management_encryption_algorithm=aes_cbc'
	 Encryption plugin parameter :  '--file_key_management_filekey='
	 Encryption plugin parameter :  '--file_key_management_filename=/usr/share/mysql/mysql-test/std_data/logkey.txt'
	 Encryption plugin parameter :  '--apply-log'
	 Encryption plugin parameter :  '/run/shm/var/1/tmp/backup'
xtrabackup: This target seems to be not prepared yet.
InnoDB: The universal page size of the database is set to 8192.
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = .
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Uses event mutexes
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Number of pools: 1
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Using generic crc32 instructions
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Completed initialization of buffer pool
2017-09-12 18:24:49 2851584832 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Highest supported file format is Barracuda.
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Starting crash recovery from checkpoint LSN=938243
2017-09-12 18:24:49 3055650560 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Trx id counter is 1792
2017-09-12 18:24:49 3055650560 [Note] InnoDB: Starting final batch to recover 265 pages from redo log.

2017-09-12 18:24:50 0xb6218700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.2.9/storage/innobase/trx/trx0trx.cc line 624
InnoDB: Failing assertion: trx_state_eq(trx, TRX_STATE_PREPARED) || (trx_state_eq(trx, TRX_STATE_ACTIVE) && trx->is_recovered && (!srv_was_started || srv_operation == SRV_OPERATION_RESTORE || srv_read_only_mode || srv_force_recovery >= SRV_FORCE_NO_TRX_UNDO))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170912 18:24:50 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.9-MariaDB-10.2.9+maria~trusty
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4225 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
addr2line: '--innobackupex': No such file
/usr/bin/mariabackup(my_print_stacktrace+0x2d)[0xb72759ad]
/usr/bin/mariabackup(handle_fatal_signal+0x344)[0xb6d070e4]
[0xb67ad400]
[0xb67ad424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x47)[0xb626f687]
/lib/i386-linux-gnu/libc.so.6(abort+0x143)[0xb6272ab3]
/usr/bin/mariabackup(+0x2322f9)[0xb6a022f9]
/usr/bin/mariabackup(+0x8a0752)[0xb7070752]
/usr/bin/mariabackup(+0x898ea8)[0xb7068ea8]
/usr/bin/mariabackup(+0x861d9c)[0xb7031d9c]
/usr/bin/mariabackup(+0x259d0b)[0xb6a29d0b]
/usr/bin/mariabackup(main+0x1f2)[0xb6a04812]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb625aaf3]
/usr/bin/mariabackup(+0x25101d)[0xb6a2101d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Aborted (core dumped)
 
 
 
The result from queries just before the failure was:
CREATE TABLE t1(c1 INT, b VARCHAR(2400), index(b(100),c1))
ENGINE=INNODB ROW_FORMAT=compressed ENCRYPTED=YES;
BEGIN;
COMMIT;
# xtrabackup backup
drop table t1;



 Comments   
Comment by Marko Mäkelä [ 2017-10-27 ]

This actually still fails in 10.2.10. It is an issue on its own.

Comment by Marko Mäkelä [ 2017-10-27 ]

The assertions in trx_free_prepared() and trx_undo_set_state_at_finish() were too strict.
This should also affect MariaDB server shutdown in innodb_read_only mode or when setting innodb_force_recovery to at least 3.

Comment by Marko Mäkelä [ 2017-10-27 ]

Because the assertion failures can cause InnoDB or XtraDB shutdown failure, I think that they should be fixed in 10.0 and 10.1 as well.
A backport of the patch to 10.0 will likely miss the 10.0.33 release.

Generated at Thu Feb 08 08:09:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.