Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.2.8
-
None
-
Centos 7
-
10.2.10
Description
I have a backup script, which takes a full backup 1 day of the week followed by incremental backups for the following days.
Command ran was:
mariabackup \
--backup \
--parallel=8 \
--user=$mysql_user \
--password=$mysql_pass \
--no-timestamp \
--compress \
--compress-threads=8 \
--incremental-basedir=$last_backup \
--socket=$socket \
--defaults-group="mysqld$instance" \
--target-dir=$inc_backup_path 2> $logfile_i
Previously we had this working using the command:
innobackupex \
--parallel=8 \
--user=$mysql_user \
--password=$mysql_pass \
--no-timestamp \
--compress \
--compress-threads=8 \
--incremental \
--incremental-basedir=$last_backup \
--socket=$socket \
--defaults-group="mysqld$instance" \
$inc_backup_path 2> $logfile_i
Any ideas?
We previously were unable to perform backups at all until the 10.2.8 version of mariabackup.
Our tables range in size from 1-2GB to upwards of a 1T.
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
I am sorry that you observe the problem - I need to compile a test case to verify that.
Does it fail every time or sometimes? (how often)? Is the error shown at start of backup / in the middle / after completion ?
Do you have chance to provide configuration file(s) and full backup logs ?
Hi Andrii,
Thanks for your response, I can't pinpoint exactly when it occurs but it appears to be relatively soon within the backup start process.
I would say it's our 1T table that is causing an issue.
Config files, and log from the crash dump attached.
Thanks.
It looks that I was able to reproduce the problem with few 150G+ files and following stack trace, now trying to compile smaller test case:
+ /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup --defaults-file=m1-10.2/my.cnf --backup --parallel=8 --compress --compress-threads=8 --target-dir=m1-10.2/bkup1 --incremental-basedir=m1-10.2/bkup
|
170918 09:38:56 Connecting to MySQL server host: localhost, user: root, password: not set, port: 3307, socket: /home/a/mariadb-environs/m1-10.2/dt/my.sock
|
Using server version 10.2.9-MariaDB
|
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup based on MariaDB server 10.2.9-MariaDB Linux (x86_64)
|
incremental backup from 527421805670 is enabled.
|
xtrabackup: uses posix_fadvise().
|
xtrabackup: cd to /home/a/mariadb-environs/m1-10.2/dt/
|
xtrabackup: open files limit requested 0, set to 1024
|
xtrabackup: using the following InnoDB configuration:
|
xtrabackup: innodb_data_home_dir = .
|
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
|
xtrabackup: innodb_log_group_home_dir = ./
|
2017-09-18 9:38:56 139914032555840 [Note] InnoDB: Number of pools: 1
|
170918 09:38:56 >> log scanned up to (532663833177)
|
xtrabackup: Generating a list of tablespaces
|
2017-09-18 9:38:56 139914032555840 [Warning] InnoDB: Allocated tablespace ID 1 for mysql/innodb_table_stats, old maximum was 0
|
xtrabackup: using the full scan for incremental backup
|
xtrabackup: Starting 8 threads for parallel data files transfer
|
170918 09:38:56 [07] Compressing ./test/t1.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/test/t1.ibd.delta.qp
|
170918 09:38:56 [06] Compressing ./test/t2.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/test/t2.ibd.delta.qp
|
170918 09:38:56 [01] Compressing ./mysql/innodb_index_stats.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/mysql/innodb_index_stats.ibd.delta.qp
|
170918 09:38:56 [05] Compressing ./test/t3.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/test/t3.ibd.delta.qp
|
170918 09:38:56 [02] Compressing ./ibdata1 to /home/a/mariadb-environs/m1-10.2/bkup1/ibdata1.delta.qp
|
170918 09:38:56 [03] Compressing ./mysql/innodb_table_stats.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/mysql/innodb_table_stats.ibd.delta.qp
|
170918 09:38:56 [04] Compressing ./mysql/gtid_slave_pos.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/mysql/gtid_slave_pos.ibd.delta.qp
|
170918 09:38:56 [04] ...done
|
170918 09:38:56 [01] ...done
|
170918 09:38:56 [03] ...done
|
170918 09:38:57 >> log scanned up to (532663833177)
|
170918 9:39:17 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.2.9-MariaDB
|
key_buffer_size=0
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=1
|
thread_count=0
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5392 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(my_print_stacktrace+0x29)[0x56362ef98499]
|
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(handle_fatal_signal+0x3bd)[0x56362eb3130d]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f4045f54670]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x1507bf)[0x7f404451d7bf]
|
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(+0x438a7c)[0x56362e8b9a7c]
|
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(+0x42e3d1)[0x56362e8af3d1]
|
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(+0x42e62d)[0x56362e8af62d]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f4045f4a6da]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f40444d5d7f]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
MDEV-13798-2.sh: line 11: 6822 Segmentation fault (core dumped)
|
a@UBINTI:~/mariadb-environs$ addr2line -fie /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup 0x438a7c
|
_ZL22wf_incremental_processP20xb_write_filt_ctxt_tP9ds_file_t
|
/home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/write_filt.cc:147
|
a@UBINTI:~/mariadb-environs$ addr2line -fie /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup 0x42e3d1
|
_ZL24xtrabackup_copy_datafileP10fil_node_tj
|
/home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2271
|
a@UBINTI:~/mariadb-environs$ addr2line -fie /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup 0x42e62d
|
_ZL21data_copy_thread_funcPv
|
/home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2519
|
Hi wayne, is there chance for you to try the same incremental backup, just without both --parallel=8 and --compress-threads=8 options and confirm if it is feasible workaround for your environment?
wayne could you also confirm approximate number of .idb files in this system ? Is it around 70-80 or much more?
marko Could you confirm if my analysis is correct here:
Code has structure xb_wf_incremental_ctxt_t with members delta_buf_base and delta_buf, which are initialized as below for object cp :
https://github.com/MariaDB/server/blob/10.2/extra/mariabackup/write_filt.cc#L77
/* allocate buffer for incremental backup (4096 pages) */ |
buf_size = (cursor->page_size.physical() / 4 + 1)
|
* cursor->page_size.physical();
|
cp->delta_buf_base = static_cast<byte *>(malloc(buf_size)); |
memset(cp->delta_buf_base, 0, buf_size); |
cp->delta_buf = static_cast<byte *> |
(ut_align(cp->delta_buf_base, UNIV_PAGE_SIZE_MAX));
|
And now mentioned crash with gdb at https://github.com/MariaDB/server/blob/mariadb-10.2.8/extra/mariabackup/write_filt.cc#L144 :
Thread 3 "mariabackup" received signal SIGSEGV, Segmentation fault. |
[Switching to Thread 0x7fffd4d52700 (LWP 31473)]
|
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:242 |
242 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory. |
(gdb) bt
|
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:242
|
#1 0x000055555598cadc in memcpy (__len=<optimized out>, __src=0x7fffcf998000, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:53
|
#2 wf_incremental_process (ctxt=0x7fffd4d51730, dstfile=0x7fffc8000a58) at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/write_filt.cc:145
|
#3 0x0000555555982371 in xtrabackup_copy_datafile (node=<optimized out>, thread_n=1)
|
at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2282
|
#4 0x00005555559825fe in data_copy_thread_func (arg=0x555557264ce0) at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2530
|
#5 0x00007ffff7bc06da in start_thread (arg=0x7fffd4d52700) at pthread_create.c:456
|
#6 0x00007ffff614bd7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
|
(gdb) f 2
|
#2 wf_incremental_process (ctxt=0x7fffd4d51730, dstfile=0x7fffc8000a58) at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/write_filt.cc:145
|
145 page_size);
|
(gdb) l
|
140 }
|
141
|
142 mach_write_to_4(cp->delta_buf + cp->npages * 4,
|
143 cursor->buf_page_no + i);
|
144 memcpy(cp->delta_buf + cp->npages * page_size, page, |
145 page_size);
|
146
|
147 cp->npages++;
|
148 }
|
149
|
(gdb) p cp->delta_buf
|
$1 = (byte *) 0x7fffc3800000 "xtra" |
(gdb) p cp->npages
|
$2 = 4095
|
(gdb) p cursor
|
$4 = (xb_fil_cur_t *) 0x7fffd4d51950
|
(gdb) p cursor->buf_page_no
|
$5 = 44160
|
(gdb) p i
|
$6 = 231
|
The crash happened with default page size and it according to gdb it was writing page 4095 .
Remembering code above - allocated memory fragment was 4097 x 16k bytes, which was further aligned to UNIV_PAGE_SIZE_MAX boundary (64k), which in worst case could shrink allocated memory below 4095 pages ?
I am not sure if it is feasible to create a reliable test case for this, so going to verify looking at source code with developers approval.
wayne thank you for all the help - please ignore my earlier requests for now, as I hope that developers will confirm the findings above.
Attached heavy mtr test case reliably shows problem in 10.2, which disappears after attached patch :
CREATE TABLE t1 select uuid() a, uuid() b, uuid() c from seq_1_to_1000;
|
CREATE TABLE t2 select * from t1;
|
CREATE TABLE t3 select * from t1;
|
# Create full backup , modify table, then create incremental/differential backup
|
start transaction;
|
INSERT INTO t1 select uuid(), uuid(), uuid() from seq_1_to_1000000;
|
INSERT INTO t2 select uuid(), uuid(), uuid() from seq_1_to_1000000;
|
INSERT INTO t3 select uuid(), uuid(), uuid() from seq_1_to_1000000;
|
*** Error in `/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup': free(): invalid pointer: 0x00007f9effffb010 ***
|
======= Backtrace: =========
|
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f9f4e05108b]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x82c3a)[0x7f9f4e05ac3a]
|
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f9f4e05ed2c]
|
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup[0x5521b2]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f9f4f27d6da]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f9f4e0e0d7f]
|
======= Memory map: ========
|
00400000-014a0000 r-xp 00000000 08:13 221033 /home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup
|
0169f000-01766000 rw-p 0109f000 08:13 221033 /home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup
|
01766000-01ff6000 rw-p 00000000 00:00 0
|
03498000-03586000 rw-p 00000000 00:00 0 [heap]
|
7f9efbff6000-7f9f04000000 rw-p 00000000 00:00 0
|
7f9f04000000-7f9f04021000 rw-p 00000000 00:00 0
|
7f9f04021000-7f9f08000000 ---p 00000000 00:00 0
|
7f9f0bfec000-7f9f0fff1000 rw-p 00000000 00:00 0
|
7f9f10000000-7f9f10021000 rw-p 00000000 00:00 0
|
7f9f10021000-7f9f14000000 ---p 00000000 00:00 0
|
7f9f1c000000-7f9f1c021000 rw-p 00000000 00:00 0
|
7f9f1c021000-7f9f20000000 ---p 00000000 00:00 0
|
7f9f2c000000-7f9f2c021000 rw-p 00000000 00:00 0
|
7f9f2c021000-7f9f30000000 ---p 00000000 00:00 0
|
7f9f30000000-7f9f30021000 rw-p 00000000 00:00 0
|
7f9f30021000-7f9f34000000 ---p 00000000 00:00 0
|
7f9f34000000-7f9f34021000 rw-p 00000000 00:00 0
|
7f9f34021000-7f9f38000000 ---p 00000000 00:00 0
|
7f9f38000000-7f9f38021000 rw-p 00000000 00:00 0
|
7f9f38021000-7f9f3c000000 ---p 00000000 00:00 0
|
7f9f3e5f9000-7f9f3e5fa000 ---p 00000000 00:00 0
|
7f9f3e5fa000-7f9f3edfa000 rw-p 00000000 00:00 0
|
7f9f3edfa000-7f9f3edfb000 ---p 00000000 00:00 0
|
7f9f3edfb000-7f9f3f5fb000 rw-p 00000000 00:00 0
|
7f9f40000000-7f9f40021000 rw-p 00000000 00:00 0
|
7f9f40021000-7f9f44000000 ---p 00000000 00:00 0
|
7f9f457fb000-7f9f457fc000 ---p 00000000 00:00 0
|
7f9f457fc000-7f9f45ffc000 rw-p 00000000 00:00 0
|
7f9f45ffc000-7f9f45ffd000 ---p 00000000 00:00 0
|
7f9f45ffd000-7f9f467fd000 rw-p 00000000 00:00 0
|
7f9f467fd000-7f9f467fe000 ---p 00000000 00:00 0
|
7f9f467fe000-7f9f46ffe000 rw-p 00000000 00:00 0
|
7f9f46ffe000-7f9f46fff000 ---p 00000000 00:00 0
|
7f9f46fff000-7f9f477ff000 rw-p 00000000 00:00 0
|
7f9f477ff000-7f9f47800000 ---p 00000000 00:00 0
|
7f9f47800000-7f9f48000000 rw-p 00000000 00:00 0
|
7f9f48000000-7f9f48021000 rw-p 00000000 00:00 0
|
7f9f48021000-7f9f4c000000 ---p 00000000 00:00 0
|
7f9f4c15e000-7f9f4c15f000 ---p 00000000 00:00 0
|
7f9f4c15f000-7f9f4cb64000 rw-p 00000000 00:00 0
|
7f9f4cb64000-7f9f4cb65000 ---p 00000000 00:00 0
|
7f9f4cb65000-7f9f4dbaf000 rw-p 00000000 00:00 0
|
7f9f4dbaf000-7f9f4dbba000 r-xp 00000000 08:13 536929667 /lib/x86_64-linux-gnu/libnss_files-2.24.so
|
7f9f4dbba000-7f9f4ddb9000 ---p 0000b000 08:13 536929667 /lib/x86_64-linux-gnu/libnss_files-2.24.so
|
7f9f4ddb9000-7f9f4ddba000 r--p 0000a000 08:13 536929667 /lib/x86_64-linux-gnu/libnss_files-2.24.so
|
7f9f4ddba000-7f9f4ddbb000 rw-p 0000b000 08:13 536929667 /lib/x86_64-linux-gnu/libnss_files-2.24.so
|
7f9f4ddbb000-7f9f4ddc1000 rw-p 00000000 00:00 0
|
7f9f4ddc1000-7f9f4ddd7000 r-xp 00000000 08:13 536924403 /lib/x86_64-linux-gnu/libgcc_s.so.1
|
7f9f4ddd7000-7f9f4dfd6000 ---p 00016000 08:13 536924403 /lib/x86_64-linux-gnu/libgcc_s.so.1
|
7f9f4dfd6000-7f9f4dfd7000 r--p 00015000 08:13 536924403 /lib/x86_64-linux-gnu/libgcc_s.so.1
|
7f9f4dfd7000-7f9f4dfd8000 rw-p 00016000 08:13 536924403 /lib/x86_64-linux-gnu/libgcc_s.so.1
|
7f9f4dfd8000-7f9f4e196000 r-xp 00000000 08:13 536923520 /lib/x86_64-linux-gnu/libc-2.24.so
|
7f9f4e196000-7f9f4e395000 ---p 001be000 08:13 536923520 /lib/x86_64-linux-gnu/libc-2.24.so
|
7f9f4e395000-7f9f4e399000 r--p 001bd000 08:13 536923520 /lib/x86_64-linux-gnu/libc-2.24.so
|
7f9f4e399000-7f9f4e39b000 rw-p 001c1000 08:13 536923520 /lib/x86_64-linux-gnu/libc-2.24.so
|
7f9f4e39b000-7f9f4e39f000 rw-p 00000000 00:00 0
|
7f9f4e39f000-7f9f4e4a7000 r-xp 00000000 08:13 536923558 /lib/x86_64-linux-gnu/libm-2.24.so
|
7f9f4e4a7000-7f9f4e6a6000 ---p 00108000 08:13 536923558 /lib/x86_64-linux-gnu/libm-2.24.so
|
7f9f4e6a6000-7f9f4e6a7000 r--p 00107000 08:13 536923558 /lib/x86_64-linux-gnu/libm-2.24.so
|
7f9f4e6a7000-7f9f4e6a8000 rw-p 00108000 08:13 536923558 /lib/x86_64-linux-gnu/libm-2.24.so
|
7f9f4e6a8000-7f9f4e821000 r-xp 00000000 08:13 1074794301 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
|
7f9f4e821000-7f9f4ea20000 ---p 00179000 08:13 1074794301 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
|
7f9f4ea20000-7f9f4ea2a000 r--p 00178000 08:13 1074794301 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
|
7f9f4ea2a000-7f9f4ea2c000 rw-p 00182000 08:13 1074794301 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
|
7f9f4ea2c000-7f9f4ea30000 rw-p 00000000 00:00 0
|
7f9f4ea30000-7f9f4ea33000 r-xp 00000000 08:13 536923538 /lib/x86_64-linux-gnu/libdl-2.24.so
|
7f9f4ea33000-7f9f4ec32000 ---p 00003000 08:13 536923538 /lib/x86_64-linux-gnu/libdl-2.24.so
|
7f9f4ec32000-7f9f4ec33000 r--p 00002000 08:13 536923538 /lib/x86_64-linux-gnu/libdl-2.24.so
|
7f9f4ec33000-7f9f4ec34000 rw-p 00003000 08:13 536923538 /lib/x86_64-linux-gnu/libdl-2.24.so
|
7f9f4ec34000-7f9f4ec3d000 r-xp 00000000 08:13 536923537 /lib/x86_64-linux-gnu/libcrypt-2.24.so
|
7f9f4ec3d000-7f9f4ee3c000 ---p 00009000 08:13 536923537 /lib/x86_64-linux-gnu/libcrypt-2.24.so
|
7f9f4ee3c000-7f9f4ee3d000 r--p 00008000 08:13 536923537 /lib/x86_64-linux-gnu/libcrypt-2.24.so
|
7f9f4ee3d000-7f9f4ee3e000 rw-p 00009000 08:13 536923537 /lib/x86_64-linux-gnu/libcrypt-2.24.so
|
7f9f4ee3e000-7f9f4ee6c000 rw-p 00000000 00:00 0
|
7f9f4ee6c000-7f9f4ee73000 r-xp 00000000 08:13 536953566 /lib/x86_64-linux-gnu/librt-2.24.so
|
7f9f4ee73000-7f9f4f072000 ---p 00007000 08:13 536953566 /lib/x86_64-linux-gnu/librt-2.24.so
|
7f9f4f072000-7f9f4f073000 r--p 00006000 08:13 536953566 /lib/x86_64-linux-gnu/librt-2.24.so
|
7f9f4f073000-7f9f4f074000 rw-p 00007000 08:13 536953566 /lib/x86_64-linux-gnu/librt-2.24.so
|
7f9f4f074000-7f9f4f075000 r-xp 00000000 08:13 539008302 /lib/x86_64-linux-gnu/libaio.so.1.0.1
|
7f9f4f075000-7f9f4f274000 ---p 00001000 08:13 539008302 /lib/x86_64-linux-gnu/libaio.so.1.0.1
|
7f9f4f274000-7f9f4f275000 r--p 00000000 08:13 539008302 /lib/x86_64-linux-gnu/libaio.so.1.0.1
|
7f9f4f275000-7f9f4f276000 rw-p 00000000 00:00 0
|
7f9f4f276000-7f9f4f28e000 r-xp 00000000 08:13 536953509 /lib/x86_64-linux-gnu/libpthread-2.24.so
|
7f9f4f28e000-7f9f4f48e000 ---p 00018000 08:13 536953509 /lib/x86_64-linux-gnu/libpthread-2.24.so
|
7f9f4f48e000-7f9f4f48f000 r--p 00018000 08:13 536953509 /lib/x86_64-linux-gnu/libpthread-2.24.so
|
7f9f4f48f000-7f9f4f490000 rw-p 00019000 08:13 536953509 /lib/x86_64-linux-gnu/libpthread-2.24.so
|
7f9f4f490000-7f9f4f494000 rw-p 00000000 00:00 0
|
7f9f4f494000-7f9f4f4ba000 r-xp 00000000 08:13 536922919 /lib/x86_64-linux-gnu/ld-2.24.so
|
7f9f4f4da000-7f9f4f695000 rw-p 00000000 00:00 0
|
7f9f4f695000-7f9f4f6b9000 rw-p 00000000 00:00 0
|
7f9f4f6b9000-7f9f4f6ba000 r--p 00025000 08:13 536922919 /lib/x86_64-linux-gnu/ld-2.24.so
|
7f9f4f6ba000-7f9f4f6bb000 rw-p 00026000 08:13 536922919 /lib/x86_64-linux-gnu/ld-2.24.so
|
7f9f4f6bb000-7f9f4f6bc000 rw-p 00000000 00:00 0
|
7ffcebaf9000-7ffcebb1d000 rw-p 00000000 00:00 0 [stack]
|
7ffcebb9f000-7ffcebba1000 r--p 00000000 00:00 0 [vvar]
|
7ffcebba1000-7ffcebba3000 r-xp 00000000 00:00 0 [vdso]
|
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
|
mariabackup.MDEV-13798 '16k,innodb' [ fail ]
|
Test ended at 2017-09-19 13:37:04
|
|
|
CURRENT_TEST: mariabackup.MDEV-13798
|
170919 13:36:54 Connecting to MySQL server host: localhost, user: root, password: set, port: 16020, socket: /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/mysqld.1.sock
|
Using server version 10.2.8-MariaDB-log
|
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup based on MariaDB server 10.2.8-MariaDB Linux (x86_64)
|
incremental backup from 2213423 is enabled.
|
xtrabackup: uses posix_fadvise().
|
xtrabackup: cd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/mysqld.1/data/
|
xtrabackup: open files limit requested 1024, set to 1024
|
xtrabackup: using the following InnoDB configuration:
|
xtrabackup: innodb_data_home_dir = .
|
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
|
xtrabackup: innodb_log_group_home_dir = ./
|
2017-09-19 13:36:54 140322208814592 [Note] InnoDB: Number of pools: 1
|
170919 13:36:54 >> log scanned up to (514417320)
|
xtrabackup: Generating a list of tablespaces
|
2017-09-19 13:36:54 140322208814592 [Warning] InnoDB: Allocated tablespace ID 1 for mysql/innodb_table_stats, old maximum was 0
|
xtrabackup: using the full scan for incremental backup
|
xtrabackup: Starting 8 threads for parallel data files transfer
|
170919 13:36:54 [06] Copying ./test/t3.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/test/t3.ibd.delta
|
170919 13:36:54 [02] Copying ./mysql/innodb_index_stats.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/mysql/innodb_index_stats.ibd.delta
|
170919 13:36:54 [03] Copying ./test/t1.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/test/t1.ibd.delta
|
170919 13:36:54 [05] Copying ./test/t2.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/test/t2.ibd.delta
|
170919 13:36:54 [04] Copying ./mysql/innodb_table_stats.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/mysql/innodb_table_stats.ibd.delta
|
170919 13:36:54 [01] Copying ./ibdata1 to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/ibdata1.delta
|
170919 13:36:54 [02] ...done
|
170919 13:36:54 [04] ...done
|
170919 13:36:57 [01] ...done
|
170919 13:36:57 >> log scanned up to (514772389)
|
170919 13:36:58 [05] ...done
|
170919 13:36:58 [03] ...done
|
170919 13:36:59 [06] ...done
|
170919 13:37:00 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.2.8-MariaDB
|
key_buffer_size=0
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=1
|
thread_count=0
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5391 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup(my_print_stacktrace+0x2e)[0xe0e4fe]
|
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup(handle_fatal_signal+0x444)[0x883264]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f9f4f287670]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x9f)[0x7f9f4e00d77f]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f9f4e00f37a]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x79090)[0x7f9f4e051090]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x82c3a)[0x7f9f4e05ac3a]
|
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f9f4e05ed2c]
|
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup[0x5521b2]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f9f4f27d6da]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f9f4e0e0d7f]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Aborted (core dumped)
|
mysqltest: At line 25: exec of '/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup --defaults-file=/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/my.cnf --backup --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wait-query-type=all --target-dir=/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1 --incremental-basedir=/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup --parallel=8' failed, error: 34304, status: 134, errno: 11
|
Output from before failure:
|
INSERT INTO t3 select uuid(), uuid(), uuid() from seq_1_to_1000000;
|
|
|
- saving '/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/log/mariabackup.MDEV-13798-16k,innodb/' to '/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/log/mariabackup.MDEV-13798-16k,innodb/'
|
I don't understand why the problem is not repeatable in 10.1 (with this and even heavier test, using 6 tables the same way), at least I couldn't reproduce it. But I think similar patch to wf_incremental_init() is needed in 10.1 as well.
jplindst, please have a look at MDEV-13822 as well. I believe that --incremental is broken in 10.1 already. I also believe that Percona XtraBackup is working around that issue at a low level, while MariaDB implemented a more accurate solution in MDEV-11556.
I would strongly suggest compiling -DWITH_ASAN and running all tests with
ASAN_OPTIONS=abort_on_error=1,detect_leaks=0 ./mtr --suite=mariabackup
|
Unfortunately, last time I tried, there were some memory leaks somewhere, possibly only in the non-debug build. Those should of course be fixed too.
What does the AddressSanitizer report for this test case?
creating incremental backup======= Backtrace: ================ Memory map: ========