[MDEV-14812] mariabackup.xb_compressed_encrypted fails in buildbot with segmentation fault Created: 2017-12-29  Updated: 2023-11-16

Status: Open
Project: MariaDB Server
Component/s: Backup, Encryption, Tests
Affects Version/s: 10.1, 10.2, 10.3
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Vladislav Vaintroub
Resolution: Unresolved Votes: 0
Labels: None

Sprint: 10.1.32

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles12-amd64/builds/4054/steps/mtr/logs/stdio

mariabackup.xb_compressed_encrypted      w4 [ fail ]
        Test ended at 2017-12-28 10:33:40
 
CURRENT_TEST: mariabackup.xb_compressed_encrypted
sh: line 1: 22918 Segmentation fault      /usr/bin/mariabackup --innobackupex --apply-log /dev/shm/var/4/tmp/backup 2>&1
mysqltest: At line 28: exec of '/usr/bin/mariabackup --innobackupex --apply-log /dev/shm/var/4/tmp/backup 2>&1' failed, error: 35584, status: 139, errno: 0
Output from before failure:
171228 10:33:40 innobackupex: Starting the apply-log operation
 
IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".
 
--innobackupex based on MariaDB server 10.1.30-MariaDB Linux (x86_64) 
mariabackup: cd to /dev/shm/var/4/tmp/backup/
Loading encryption plugin
	 Encryption plugin parameter :  '--file_key_management_encryption_algorithm=aes_cbc'
	 Encryption plugin parameter :  '--file_key_management_filekey='
	 Encryption plugin parameter :  '--file_key_management_filename=/usr/share/mysql-test/std_data/logkey.txt'
	 Encryption plugin parameter :  '--apply-log'
	 Encryption plugin parameter :  '/dev/shm/var/4/tmp/backup'
mariabackup: This target seems to be not prepared yet.
mariabackup: xtrabackup_logfile detected: size=3145728, start_lsn=(22784939)
mariabackup: using the following InnoDB configuration for recovery:
mariabackup:   innodb_data_home_dir = ./
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
mariabackup:   innodb_log_group_home_dir = ./
mariabackup:   innodb_log_files_in_group = 1
mariabackup:   innodb_log_file_size = 3145728
mariabackup: using the following InnoDB configuration for recovery:
mariabackup:   innodb_data_home_dir = ./
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
mariabackup:   innodb_log_group_home_dir = ./
mariabackup:   innodb_log_files_in_group = 1
mariabackup:   innodb_log_file_size = 3145728
mariabackup: Starting InnoDB instance for recovery.
mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: The InnoDB memory heap is disabled
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Using generic crc32 instructions
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Initializing buffer pool, size = 100.0M
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Completed initialization of buffer pool
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Highest supported file format is Barracuda.
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Starting crash recovery from checkpoint LSN=22784939
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
2017-12-28 10:33:40 140258601293696 [Note] InnoDB: Starting final batch to recover 138 pages from redo log
171228 10:33:40 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.30-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5338 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/usr/bin/mariabackup(my_print_stacktrace+0x2b)[0x7f9080c6a46b]
addr2line: '--innobackupex': No such file
/usr/bin/mariabackup(handle_fatal_signal+0x4d5)[0x7f908082add5]
/lib64/libpthread.so.0(+0xf890)[0x7f907fd9f890]
/lib64/libcrypto.so.1.0.0(+0xed6c5)[0x7f907efab6c5]
/lib64/libcrypto.so.1.0.0(lh_delete+0x21)[0x7f907efabb51]
/lib64/libcrypto.so.1.0.0(+0xeffdf)[0x7f907efadfdf]
/lib64/libcrypto.so.1.0.0(ERR_remove_thread_state+0x2d)[0x7f907efaea2d]
/usr/bin/mariabackup(my_aes_crypt_finish+0x24)[0x7f9080c7f694]
/usr/bin/mariabackup(my_aes_crypt+0x95)[0x7f9080c7f745]
/usr/bin/mariabackup(_ZN11MyCTX_nopad6finishEPhPj+0x93)[0x7f9080c7fd13]
/usr/bin/mariabackup(my_aes_crypt_finish+0x19)[0x7f9080c7f689]
/usr/bin/mariabackup(+0x6144ed)[0x7f90807e44ed]
/usr/bin/mariabackup(_Z8do_cryptPKhjPhPjP20st_encryption_schemejjjyi+0x1ad)[0x7f90807e488d]
/usr/bin/mariabackup(encryption_scheme_decrypt+0x2b)[0x7f90807e49ab]
/usr/bin/mariabackup(+0x87a732)[0x7f9080a4a732]
/usr/bin/mariabackup(+0x87a8eb)[0x7f9080a4a8eb]
/usr/bin/mariabackup(+0x820685)[0x7f90809f0685]
/usr/bin/mariabackup(+0x86b6ac)[0x7f9080a3b6ac]
/usr/bin/mariabackup(+0x999860)[0x7f9080b69860]
/lib64/libpthread.so.0(+0x80a4)[0x7f907fd980a4]
/lib64/libc.so.6(clone+0x6d)[0x7f907e1c57fd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
 
 
 
The result from queries just before the failure was:
< snip >
BEGIN
DECLARE CURRENT_NUM INT;
SET CURRENT_NUM = 0;
WHILE CURRENT_NUM < REPEAT_COUNT DO
INSERT INTO t1 VALUES(CURRENT_NUM, concat(uuid(), CURRENT_NUM, repeat('ab', floor(rand()*100) ), uuid()));
SET CURRENT_NUM = CURRENT_NUM + 1;
END WHILE;
END//
COMMIT;
SET AUTOCOMMIT=0;
CALL innodb_insert_proc(50000);
COMMIT;
# xtrabackup backup
drop table t1;



 Comments   
Comment by Elena Stepanova [ 2018-05-23 ]

SIGABRT on 10.3: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-xenial-x86/builds/6704

Comment by Alice Sherepa [ 2023-11-16 ]

https://buildbot.mariadb.net/buildbot/builders/kvm-fulltest2/builds/43532/steps/mtr_nm/logs/stdio

[Revision hash: eb8053b37756fe99e8ca12bbd966726d9f21c91a]
mariabackup.xb_compressed_encrypted '8k,innodb' w4 [ fail ]
        Test ended at 2023-10-30 05:22:29
 
CURRENT_TEST: mariabackup.xb_compressed_encrypted
mysqltest: At line 25: exec of '/mnt/buildbot/build/mariadb-10.4.32/extra/mariabackup/mariabackup --innobackupex --apply-log /mnt/buildbot/build/mariadb-10.4.32/mysql-test/var/4/tmp/backup 2>&1' failed, error: 35584, status: 139, errno: 11
Output from before failure:
231030 05:22:21 innobackupex: Starting the apply-log operation
 
IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".
 
--innobackupex based on MariaDB server 10.4.32-MariaDB Linux (i686)
[00] 2023-10-30 05:22:21 cd to /mnt/buildbot/build/mariadb-10.4.32/mysql-test/var/4/tmp/backup/
[00] 2023-10-30 05:22:21 open files limit requested 0, set to 1024
[00] 2023-10-30 05:22:21 Loading encryption plugin from file_key_management=file_key_management
[00] 2023-10-30 05:22:21 Loading encryption plugin
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '--plugin_load=file_key_management=file_key_management'
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '--file_key_management_encryption_algorithm=aes_cbc'
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '--file_key_management_filekey='
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '--file_key_management_filename=/mnt/buildbot/build/mariadb-10.4.32/mysql-test/std_data/logkey.txt'
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '--innodb_encrypt_log=0'
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '--apply-log'
[00] 2023-10-30 05:22:21 	 Encryption plugin parameter :  '/mnt/buildbot/build/mariadb-10.4.32/mysql-test/var/4/tmp/backup'
[00] 2023-10-30 05:22:21 This target seems to be not prepared yet.
[00] 2023-10-30 05:22:21 InnoDB: The universal page size of the database is set to 8192.
[00] 2023-10-30 05:22:21 mariabackup: using the following InnoDB configuration for recovery:
[00] 2023-10-30 05:22:21 innodb_data_home_dir = .
[00] 2023-10-30 05:22:21 innodb_data_file_path = ibdata1:12M:autoextend
[00] 2023-10-30 05:22:21 innodb_log_group_home_dir = .
[00] 2023-10-30 05:22:21 InnoDB: Using Linux native AIO
[00] 2023-10-30 05:22:21 Starting InnoDB instance for recovery.
[00] 2023-10-30 05:22:21 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
2023-10-30  5:22:21 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2023-10-30  5:22:21 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-10-30  5:22:21 0 [Note] InnoDB: Uses event mutexes
2023-10-30  5:22:21 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2023-10-30  5:22:21 0 [Note] InnoDB: Number of pools: 1
2023-10-30  5:22:21 0 [Note] InnoDB: Using generic crc32 instructions
2023-10-30  5:22:21 0 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
2023-10-30  5:22:21 0 [Note] InnoDB: Completed initialization of buffer pool
2023-10-30  5:22:21 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-10-30  5:22:21 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1930264
2023-10-30  5:22:22 0 [Note] InnoDB: Starting final batch to recover 234 pages from redo log.
231030  5:22:22 [ERROR] mysqld got signal 11 ;
Sorry, we probably made a mistake, and this is a bug.
 
Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.32-MariaDB-debug source revision: eb8053b37756fe99e8ca12bbd966726d9f21c91a
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4636 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/mnt/buildbot/build/mariadb-10.4.32/extra/mariabackup/mariabackup(my_print_stacktrace+0x3c)[0x8115eb83]
mysys/stacktrace.c:174(my_print_stacktrace)[0x8096270f]
addr2line: '': No such file
[0xb77a2c14]
/lib/i386-linux-gnu/libcrypto.so.1.0.0(+0xbc466)[0xb7591466]
/lib/i386-linux-gnu/libcrypto.so.1.0.0(lh_delete+0x38)[0xb75918d8]
/lib/i386-linux-gnu/libcrypto.so.1.0.0(+0xbf183)[0xb7594183]
/lib/i386-linux-gnu/libcrypto.so.1.0.0(ERR_remove_thread_state+0x50)[0xb7594d40]
mysys_ssl/my_crypt.cc:57(MyCTX::~MyCTX())[0x80bc788f]
mysys_ssl/my_crypt.cc:107(MyCTX_nopad::~MyCTX_nopad())[0x80bc81f2]
mysys_ssl/my_crypt.cc:302(my_aes_crypt_finish)[0x80bc764c]
file_key_management/file_key_management_plugin.cc:151(ctx_finish(void*, unsigned char*, unsigned int*))[0xb77969a2]
mysql/service_encryption.h:116(encryption_crypt)[0x80897667]
sql/encryption.cc:226(do_crypt(unsigned char const*, unsigned int, unsigned char*, unsigned int*, st_encryption_scheme*, unsigned int, unsigned int, unsigned int, unsigned long long, int))[0x80897d01]
sql/encryption.cc:247(encryption_scheme_decrypt)[0x80897dad]
fil/fil0crypt.cc:873(fil_space_decrypt_for_non_full_checksum(fil_space_crypt_t*, unsigned char*, unsigned int, unsigned char*))[0x80d37774]
fil/fil0crypt.cc:934(fil_space_decrypt(unsigned int, fil_space_crypt_t*, unsigned char*, unsigned int, unsigned int, unsigned char*))[0x80d379c0]
fil/fil0crypt.cc:956(fil_space_decrypt(fil_space_t const*, unsigned char*, unsigned char*))[0x80d37b3d]
buf/buf0buf.cc:575(buf_page_decrypt_after_read(buf_page_t*, fil_space_t*))[0x80c93755]
buf/buf0buf.cc:5810(buf_page_io_complete(buf_page_t*, bool, bool))[0x80ca625d]
buf/buf0rea.cc:201(buf_read_page_low(dberr_t*, bool, unsigned int, unsigned int, page_id_t, unsigned int, bool, bool))[0x80cda5cd]
buf/buf0rea.cc:934(buf_read_recv_pages(bool, unsigned int, unsigned int const*, unsigned int))[0x80cdc276]
log/log0recv.cc:2196(recv_read_in_area(page_id_t))[0x80e644ec]
log/log0recv.cc:2391(recv_apply_hashed_log_recs(bool))[0x80e65110]
srv/srv0start.cc:1926(srv_start(bool))[0x80fb431c]
mariabackup/xtrabackup.cc:2316(innodb_init())[0x8044f526]
mariabackup/xtrabackup.cc:6204(xtrabackup_prepare_func(char**))[0x8045b7b7]
mariabackup/xtrabackup.cc:7158(main_low(char**))[0x8045e0d2]
mariabackup/xtrabackup.cc:6954(main)[0x8045d921]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf7)[0xb70fb637]
/mnt/buildbot/build/mariadb-10.4.32/extra/mariabackup/mariabackup(+0x41d351)[0x80449351]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /dev/shm/var_auto_i3mT/4/tmp/backup
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             64511                64511                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       64511                64511                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/share/apport/apport %p %s %c %P
 
Kernel version: Linux version 4.4.0-21-generic (buildd@lgw01-06) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:34:49 UTC 2016
 
Segmentation fault (core dumped)
 
 
 
The result from queries just before the failure was:
CREATE TABLE t1(c1 INT, b VARCHAR(2400), index(b(100),c1))
ENGINE=INNODB ROW_FORMAT=compressed ENCRYPTED=YES;
BEGIN;
COMMIT;
# xtrabackup backup
drop table t1;

https://buildbot.mariadb.net/buildbot/builders/winx64-packages/builds/40789/steps/test/logs/stdio

[Revision hash: 728bca44e892d70f2cbc2746b89389d561c0e710]
 
mariabackup.xb_compressed_encrypted '8k,innodb' w6 [ fail ]
        Test ended at 2023-10-27 13:20:03
 
CURRENT_TEST: mariabackup.xb_compressed_encrypted
D:\winx64-packages\build\extra\mariabackup\RelWithDebInfo\mariabackup.exe based on MariaDB server 10.4.32-MariaDB Win64 (AMD64)
[01] 2023-10-27 13:19:57 Copying .\aria_log.00000001 to D:\winx64-packages\build\mysql-test\var\6\mysqld.1\data\aria_log.00000001
[01] 2023-10-27 13:19:57         ...done
[01] 2023-10-27 13:19:57 Copying .\aria_log_control to D:\winx64-packages\build\mysql-test\var\6\mysqld.1\data\aria_log_control
[01] 2023-10-27 13:19:57         ...done
[01] 2023-10-27 13:19:57 Copying ib_logfile0 to D:\winx64-packages\build\mysql-test\var\6\mysqld.1\data\ib_logfile0
[01] 2023-10-27 13:19:57         ...done
231027 13:19:57 [ERROR] mysqld got exception 0xc0000005 ;
Sorry, we probably made a mistake, and this is a bug.
 
Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.32-MariaDB source revision: 728bca44e892d70f2cbc2746b89389d561c0e710
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5328 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
ntdll.dll!memset()
ntdll.dll!RtlFreeHeap()
ntdll.dll!RtlSetDaclSecurityDescriptor()
ntdll.dll!RtlNewSecurityObjectEx()
KERNELBASE.dll!CreatePrivateObjectSecurityEx()
ntmarta.dll!AccRewriteGetNamedRights()
ntmarta.dll!AccRewriteSetHandleRights()
ntmarta.dll!SetSecurityInfo()
mariabackup.exe!fix_win_file_permissions()[backup_copy.cc:1042]
mariabackup.exe!ds_ctxt::copy_file()[backup_copy.cc:1106]
mariabackup.exe!copy_or_move_file()[backup_copy.cc:1297]
mariabackup.exe!copy_back()[backup_copy.cc:1989]
mariabackup.exe!main_low()[xtrabackup.cc:7166]
mariabackup.exe!main()[xtrabackup.cc:6954]
mariabackup.exe!__scrt_common_main_seh()[exe_common.inl:288]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
information that should help you find out what is causing the crash.
Writing a core file at D:\winx64-packages\build\mysql-test\var\6\tmp\backup
Minidump written to D:\winx64-packages\build\mysql-test\var\6\tmp\backup\mariabackup.dmp
mysqltest: In included file "D:/winx64-packages/build/src/mysql-test/suite/mariabackup/include/restart_and_restore.inc": 
included from D:/winx64-packages/build/src/mysql-test/suite/mariabackup/xb_compressed_encrypted.test at line 28:
At line 7: exec of 'D:\winx64-packages\build\extra\mariabackup\RelWithDebInfo\mariabackup.exe --defaults-file=D:/winx64-packages/build/mysql-test/var/6/my.cnf --copy-back --datadir=D:/winx64-packages/build/mysql-test/var/6/mysqld.1/data/ --target-dir=D:/winx64-packages/build/mysql-test/var/6/tmp/backup --parallel=2 --throttle=1 ' failed, error: -1073741819, status: -1073741819, errno: 2
Output from before failure:
# xtrabackup move back
 
 
 
The result from queries just before the failure was:
CREATE TABLE t1(c1 INT, b VARCHAR(2400), index(b(100),c1))
ENGINE=INNODB ROW_FORMAT=compressed ENCRYPTED=YES;
BEGIN;
COMMIT;
# xtrabackup backup
drop table t1;
# shutdown server
# remove datadir
# xtrabackup move back
 
 - found 'mariabackup.dmp' (0/5)
 
Trying 'cdb' to get a backtrace
Output from cdb follows. Faulting thread is printed twice,with and without function parameters
Search for STACK_TEXT to see the stack trace of 
the faulting thread. Callstacks of other threads are printed after it.
 
Microsoft (R) Windows Debugger Version 10.0.19041.685 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
 
 
Loading Dump File [D:\winx64-packages\build\mysql-test\var\6\log\mariabackup.xb_compressed_encrypted-8k,innodb\tmp\backup\mariabackup.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
 
 
Response                         Time (ms)     Location
OK                                             .
 
Response                         Time (ms)     Location
OK                                             .
Deferred                                       srv*C:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: .;srv*C:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: .
Windows 10 Version 20348 MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
20348.1.amd64fre.fe_release.210507-1500
Machine Name:
Debug session time: Fri Oct 27 13:19:58.000 2023 (UTC + 3:00)
System Uptime: not available
Process Uptime: 0 days 0:00:01.000
.................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(2f4.1560): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtGetContextThread:
             ret
0:000> cdb: Reading initial command '!sym prompts off; !analyze -v; .ecxr; !for_each_frame dv /t;!uniqstack -p;q'
quiet mode - symbol prompts off
 
 
 
KEY_VALUES_STRING: 1
 
    Key  : Analysis.CPU.Sec
    Value: 0
 
    Key  : Analysis.DebugAnalysisProvider.CPP
    Value: Create: 8007007e on P1-WINBUILD2
 
    Key  : Analysis.DebugData
    Value: CreateObject
 
    Key  : Analysis.DebugModel
    Value: CreateObject
 
    Key  : Analysis.Elapsed.Sec
    Value: 0
 
    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 91
 
    Key  : Analysis.System
    Value: CreateObject
 
    Key  : Timeline.Process.Start.DeltaSec
    Value: 1
 
 
CONTEXT:  (.ecxr)
rax=0000013e801c1100 rbx=ffff0000ffff0000 rcx=00000000000731a0
rdx=0000013e802342b0 rsi=0000013e802342b0 rdi=0000000000000000
rip=00007ff926aa9e6e rsp=0000003532b6e000 rbp=0000013e80210000
 r8=0000000000000000  r9=0000000047755301 r10=0000000000000003
r11=0000003532b6e090 r12=0000000000000000 r13=0000013e802342a0
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
ntdll!memset:
       mov     r8,qword ptr [rbx+8] ds:ffff0000`ffff0008=????????????????
Resetting default scope
 
EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff926aa9e6e (ntdll!memset)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: ffffffffffffffff
Attempt to read from address ffffffffffffffff
 
WRONG_SYMBOLS_TIMESTAMP: 7077fc45
 
WRONG_SYMBOLS_SIZE: 201000
 
FAULTING_MODULE: 00007ff926a00000 ntdll
 
ADDITIONAL_DEBUG_TEXT:  
You can run '.symfix; .reload' to try to fix the symbol path and load symbols. ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
 
STACK_TEXT:  
ntdll!memset
ntdll!RtlFreeHeap
ntdll!RtlSetDaclSecurityDescriptor
ntdll!RtlNewSecurityObjectEx
KERNELBASE!CreatePrivateObjectSecurityEx
ntmarta!AccRewriteGetNamedRights
ntmarta!AccRewriteSetHandleRights
ntmarta!SetSecurityInfo
mariabackup!fix_win_file_permissions
mariabackup!ds_ctxt::copy_file
mariabackup!copy_or_move_file
mariabackup!copy_back
mariabackup!main_low
mariabackup!main
mariabackup!__scrt_common_main_seh
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
 
 
STACK_COMMAND:  .ecxr ; kb ; ** Pseudo Context ** Pseudo ** Value: 198691ded90 ** ; kb
 
BUGCHECK_CODE:  7077fc45
 
EXCEPTION_CODE_STR:  7077FC45
 
EXCEPTION_STR:  WRONG_SYMBOLS
 
PROCESS_NAME:  ntdll.wrong.symbols.dll
 
IMAGE_NAME:  ntdll.wrong.symbols.dll
 
MODULE_NAME: ntdll_wrong_symbols
 
SYMBOL_NAME:  ntdll_wrong_symbols!7077FC45201000
 
FAILURE_BUCKET_ID:  WRONG_SYMBOLS_X64_20348.1.amd64fre.fe_release.210507-1500_TIMESTAMP_291017-060557_7077FC45_ntdll.wrong.symbols.dll!7077FC45201000
 
OS_VERSION:  10.0.20348.1
 
BUILDLAB_STR:  fe_release
 
OSPLATFORM_TYPE:  x64
 
OSNAME:  Windows 10
 
FAILURE_ID_HASH:  {aba3fecc-312f-6283-f42f-300a4db12055}
 
Followup:     MachineOwner
---------
 
rax=0000013e801c1100 rbx=ffff0000ffff0000 rcx=00000000000731a0
rdx=0000013e802342b0 rsi=0000013e802342b0 rdi=0000000000000000
rip=00007ff926aa9e6e rsp=0000003532b6e000 rbp=0000013e80210000
 r8=0000000000000000  r9=0000000047755301 r10=0000000000000003
r11=0000003532b6e090 r12=0000000000000000 r13=0000013e802342a0
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
ntdll!memset:
       mov     r8,qword ptr [rbx+8] ds:ffff0000`ffff0008=????????????????
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!memset
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!RtlFreeHeap
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!RtlSetDaclSecurityDescriptor
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!RtlNewSecurityObjectEx
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
KERNELBASE!CreatePrivateObjectSecurityEx
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntmarta!AccRewriteGetNamedRights
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntmarta!AccRewriteSetHandleRights
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntmarta!SetSecurityInfo
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!fix_win_file_permissions [D:\winx64-packages\build\src\extra\mariabackup\backup_copy.cc @ 1042]
char * file = <value unavailable>
struct _EXPLICIT_ACCESS_A ea = struct _EXPLICIT_ACCESS_A
struct _ACL * pNewDACL = 0x0000013e`8023e770
unsigned long size = 0xc
struct fix_win_file_permissions::__l2::<unnamed-type-tokenInfoBuffer> tokenInfoBuffer = struct fix_win_file_permissions::__l2::<unnamed-type-tokenInfoBuffer>
void * hFile = 0x00000000`0000004c
struct _SECURITY_DESCRIPTOR * pSD = 0x0000013e`8021d0e0
struct _ACL * pOldDACL = 0x0000013e`8021d120
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!ds_ctxt::copy_file [D:\winx64-packages\build\src\extra\mariabackup\backup_copy.cc @ 1106]
struct ds_ctxt * this = 0x0000013e`8023c458
char * src_file_path = 0x00007ff7`180c5d00 "ib_logfile0"
char * dst_file_path = <value unavailable>
unsigned int thread_n = 1
struct datafile_cur_t cursor = struct datafile_cur_t
char * dst_path = <value unavailable>
xb_fil_cur_result_t res = <value unavailable>
char [512] dst_name = char [512] "ib_logfile0"
struct ds_file_t * dstfile = 0x0000013e`802126c8
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!copy_or_move_file [D:\winx64-packages\build\src\extra\mariabackup\backup_copy.cc @ 1297]
struct ds_ctxt * datasink0 = 0x0000013e`8023c458
char * src_file_path = 0x00007ff7`180c5d00 "ib_logfile0"
char * dst_file_path = 0x00007ff7`180c5d00 "ib_logfile0"
char * dst_dir = 0x00007ff7`187bb4c0 "--- memory read error at address 0x00007ff7`187bb4c0 ---"
unsigned int thread_n = 1
bool copy = true
bool ret = false
char [512] filedir = char [512] ""
unsigned int64 filedir_len = 0
struct ds_ctxt * datasink = 0x0000013e`8023c458
char * ibd_filepath = 0x00000000`00000066 "--- memory read error at address 0x00000000`00000066 ---"
char * link_filepath = 0x00000000`00000066 "--- memory read error at address 0x00000000`00000066 ---"
char * filepath = 0xffffffff`ffffffff "--- memory read error at address 0xffffffff`ffffffff ---"
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!copy_back [D:\winx64-packages\build\src\extra\mariabackup\backup_copy.cc @ 1989]
struct datadir_iter_t * it = <value unavailable>
bool ret = false
class Copy_back_dst_dir aria_log_dir_path_dst = class Copy_back_dst_dir
struct _stat64 stat_arg = struct _stat64
struct datadir_node_t node = struct datadir_node_t
char * aria_log_dir_path_abs = <value unavailable>
struct ds_ctxt * ds_tmp = 0x0000013e`8023c458
class Copy_back_dst_dir dst_dir_buf = class Copy_back_dst_dir
char * dst_dir = 0x00007ff7`187bb4c0 "--- memory read error at address 0x00007ff7`187bb4c0 ---"
unsigned int i = 0x66
char [20] filename = char [20] "undo001"
char [512] filename = char [512] "D:/winx64-packages/build/mysql-test/var/6/mysqld.1/data//ib_logfile101"
class std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<Datafile> > > iter = <value unavailable>
char * filename = <value unavailable>
int i_tmp = 0n-2145004704
char * filename = <value unavailable>
char c_tmp = 0n1 ''
char *[8] ext_list = <value unavailable>
char [512] path = <value unavailable>
char [128] errbuf = <value unavailable>
class std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<Datafile> > > iter = <value unavailable>
char * ibfile = <value unavailable>
char * innobase_buffer_pool_filename = <value unavailable>
char * srv_log_group_home_dir = <value unavailable>
char * innobase_data_home_dir = <value unavailable>
char * srv_undo_dir = <value unavailable>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!main_low [D:\winx64-packages\build\src\extra\mariabackup\xtrabackup.cc @ 7166]
char ** argv = 0x0000013e`8025cc50
char [512] cwd = char [512] "D:\winx64-packages\build\src\mysql-test\"
char * endchar = 0x00007ff7`17fa8a30 "H???"
char [512] filename = char [512] ""
int num = <value unavailable>
char * opt_incremental_history_name = <value unavailable>
char * xtrabackup_incremental = <value unavailable>
char xtrabackup_backup = <value unavailable>
char * opt_mysql_tmpdir = <value unavailable>
char * xtrabackup_extra_lsndir = <value unavailable>
char * xtrabackup_incremental_dir = <value unavailable>
char * xtrabackup_incremental_basedir = <value unavailable>
char xtrabackup_prepare = <value unavailable>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!main [D:\winx64-packages\build\src\extra\mariabackup\xtrabackup.cc @ 6954]
int argc = 0n7
char ** argv = 0x0000013e`80226c40
char ** client_defaults = 0x0000013e`8025ca78
char ** backup_defaults = 0x0000013e`8025d2c8
char ** server_defaults = 0x0000013e`8025cc50
int status = <value unavailable>
struct st_mysql_mutex LOCK_error_log = <value unavailable>
unsigned long THR_THD = <value unavailable>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
(Inline Function) --------`-------- mariabackup!invoke_main [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mariabackup!__scrt_common_main_seh [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
bool has_cctor = false
int main_result = <value unavailable>
<function> ** tls_init_callback = <value unavailable>
bool is_nested = <value unavailable>
<function> ** tls_dtor_callback = <value unavailable>
int main_result = <value unavailable>
__scrt_native_startup_state __scrt_current_native_startup_state = <value unavailable>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
kernel32!BaseThreadInitThunk
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!RtlUserThreadStart
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type ".hh dbgerr005" for details.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ntdll!memset
Processing 4 threads, please wait
 
.  0  Id: 2f4.1560 Suspend: 0 Teb: 00000035`32d19000 Unfrozen
      Priority: 0  Priority class: 32
 
ntdll!NtGetContextThread
0x0000003e`0000003e
0x2
 
.  1  Id: 2f4.1dc4 Suspend: 0 Teb: 00000035`32d1b000 Unfrozen
      Priority: 0  Priority class: 32
 
ntdll!ZwWaitForWorkViaWorkerFactory
ntdll!RtlInitializeResource
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
 
Total threads: 4
Duplicate callstacks: 2 (windbg thread #s follow):
2, 3
quit:
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\atlmfc.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\concurrency.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\cpp_rest.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\stl.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\Windows.Data.Json.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\Windows.Devices.Geolocation.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\Windows.Devices.Sensors.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\Windows.Media.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\windows.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\Visualizers\winrt.natvis'

Comment by Marko Mäkelä [ 2023-11-16 ]

The Windows crash in copy_or_move_file() of ib_logfile0, specifically in RtlFreeHeap() hints to memory corruption. I do not think that it is in any way related to the GNU/Linux stack trace, which occurs when applying log, somewhere in encryption related code:

  virtual ~MyCTX()
  {
    EVP_CIPHER_CTX_reset(ctx);
    ERR_remove_state(0);
  }

I hope that wlad will have some ideas regarding the failure on Windows, and perhaps also regarding the encryption failure. Could it be a race condition?

Comment by Marko Mäkelä [ 2023-11-16 ]

For what it is worth, in the cross-reference I did not find any failures of this test on 10.5, so these particular problems could be specific to the 10.4 branch. Starting with 10.8 there are more failures, possibly related to some changed timing, and MDEV-24821.

Generated at Thu Feb 08 08:16:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.