[MDEV-13798] Error in MariaBackup incremental backup Created: 2017-09-13  Updated: 2017-10-10  Resolved: 2017-10-10

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.2.8
Fix Version/s: 10.1.29, 10.2.10

Type: Bug Priority: Critical
Reporter: Wayne Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None
Environment:

Centos 7

Attachments: File 10.2.diff     File MDEV-13798.test     Text File crash_data.txt     Text File log-from-backup.txt     File mysql.cnf.upload    
Issue Links:
Relates
relates to MDEV-13822 Mariabackup --prepare --incremental s... Closed
Sprint: 10.2.10

 Description   

I have a backup script, which takes a full backup 1 day of the week followed by incremental backups for the following days.

Command ran was:

mariabackup \
--backup \
--parallel=8 \
--user=$mysql_user \
--password=$mysql_pass \
--no-timestamp \
--compress \
--compress-threads=8 \
--incremental-basedir=$last_backup \
--socket=$socket \
--defaults-group="mysqld$instance" \
--target-dir=$inc_backup_path 2> $logfile_i

Previously we had this working using the command:

innobackupex \
--parallel=8 \
--user=$mysql_user \
--password=$mysql_pass \
--no-timestamp \
--compress \
--compress-threads=8 \
--incremental \
--incremental-basedir=$last_backup \
--socket=$socket \
--defaults-group="mysqld$instance" \
$inc_backup_path 2> $logfile_i

Any ideas?

We previously were unable to perform backups at all until the 10.2.8 version of mariabackup.

Our tables range in size from 1-2GB to upwards of a 1T.

 Comments   
Comment by Wayne [ 2017-09-13 ] 

creating incremental backup
 
*** Error in `mariabackup': free(): invalid pointer: 0x00007f611fffb010 ***
 
======= Backtrace: =========
 
/lib64/libc.so.6(+0x7c503)[0x7f615889a503]
 
mariabackup(+0x4504df)[0x7f615ab7e4df]
 
mariabackup(+0x45063d)[0x7f615ab7e63d]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7f615a2eddc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f615891576d]
 
======= Memory map: ========
 
7f61017c4000-7f61157dd000 rw-p 00000000 00:00 0
 
7f6117ff1000-7f6124000000 rw-p 00000000 00:00 0
 
7f6124000000-7f6124b05000 rw-p 00000000 00:00 0
 
7f6124b05000-7f6128000000 ---p 00000000 00:00 0
 
7f6128000000-7f6128b05000 rw-p 00000000 00:00 0
 
7f6128b05000-7f612c000000 ---p 00000000 00:00 0
 
7f612c000000-7f612cb05000 rw-p 00000000 00:00 0
 
7f612cb05000-7f6130000000 ---p 00000000 00:00 0
 
7f6130000000-7f6130b05000 rw-p 00000000 00:00 0
 
7f6130b05000-7f6134000000 ---p 00000000 00:00 0
 
7f6134000000-7f6134b05000 rw-p 00000000 00:00 0
 
7f6134b05000-7f6138000000 ---p 00000000 00:00 0
 
7f6138000000-7f6138b05000 rw-p 00000000 00:00 0
 
7f6138b05000-7f613c000000 ---p 00000000 00:00 0
 
7f613c000000-7f613cb05000 rw-p 00000000 00:00 0
 
7f613cb05000-7f6140000000 ---p 00000000 00:00 0
 
7f61427fd000-7f61427fe000 ---p 00000000 00:00 0
 
7f61427fe000-7f6142ffe000 rw-p 00000000 00:00 0                          [stack:27325]
 
7f6144000000-7f6144b05000 rw-p 00000000 00:00 0
 
7f6144b05000-7f6148000000 ---p 00000000 00:00 0
 
7f61487f9000-7f61487fa000 ---p 00000000 00:00 0
 
7f61487fa000-7f6148ffa000 rw-p 00000000 00:00 0                          [stack:27330]
 
7f6148ffa000-7f6148ffb000 ---p 00000000 00:00 0
 
7f6148ffb000-7f61497fb000 rw-p 00000000 00:00 0                          [stack:27329]
 
7f61497fb000-7f61497fc000 ---p 00000000 00:00 0
 
7f61497fc000-7f6149ffc000 rw-p 00000000 00:00 0                          [stack:27328]
 
7f6149ffc000-7f6149ffd000 ---p 00000000 00:00 0
 
7f6149ffd000-7f614a7fd000 rw-p 00000000 00:00 0                          [stack:27327]
 
7f614a7fd000-7f614a7fe000 ---p 00000000 00:00 0
 
7f614a7fe000-7f614affe000 rw-p 00000000 00:00 0                          [stack:27326]
 
7f614affe000-7f614afff000 ---p 00000000 00:00 0
 
7f614afff000-7f614b7ff000 rw-p 00000000 00:00 0                          [stack:27324]
 
7f614b7ff000-7f614b800000 ---p 00000000 00:00 0
 
7f614b800000-7f614c000000 rw-p 00000000 00:00 0                          [stack:27323]
 
7f614c000000-7f614c021000 rw-p 00000000 00:00 0
 
7f614c021000-7f6150000000 ---p 00000000 00:00 0
 
7f6150764000-7f6150969000 rw-p 00000000 00:00 0
 
7f6150969000-7f615096a000 ---p 00000000 00:00 0
 
7f615096a000-7f615126b000 rw-p 00000000 00:00 0
 
7f615126b000-7f615126c000 ---p 00000000 00:00 0
 
7f615126c000-7f6151a6c000 rw-p 00000000 00:00 0                          [stack:27320]
 
7f6151a6c000-7f6151a6d000 ---p 00000000 00:00 0
 
7f6151a6d000-7f615226d000 rw-p 00000000 00:00 0                          [stack:27319]
 
7f615226d000-7f615226e000 ---p 00000000 00:00 0
 
7f615226e000-7f6152a6e000 rw-p 00000000 00:00 0                          [stack:27318]
 
7f6152a6e000-7f6152a6f000 ---p 00000000 00:00 0
 
7f6152a6f000-7f615326f000 rw-p 00000000 00:00 0                          [stack:27317]
 
7f615326f000-7f6153270000 ---p 00000000 00:00 0
 
7f6153270000-7f6153a70000 rw-p 00000000 00:00 0                          [stack:27316]
 
7f6153a70000-7f6153a71000 ---p 00000000 00:00 0
 
7f6153a71000-7f6154271000 rw-p 00000000 00:00 0                          [stack:27315]
 
7f6154271000-7f6154272000 ---p 00000000 00:00 0
 
7f6154272000-7f6154a72000 rw-p 00000000 00:00 0                          [stack:27314]
 
7f6154a72000-7f6154a73000 ---p 00000000 00:00 0
 
7f6154a73000-7f6155c8e000 rw-p 00000000 00:00 0                          [stack:27313]
 
7f6155c8e000-7f6155c9a000 r-xp 00000000 fd:01 537284001                  /usr/lib64/libnss_files-2.17.so
 
7f6155c9a000-7f6155e99000 ---p 0000c000 fd:01 537284001                  /usr/lib64/libnss_files-2.17.so
 
7f6155e99000-7f6155e9a000 r--p 0000b000 fd:01 537284001                  /usr/lib64/libnss_files-2.17.so
 
7f6155e9a000-7f6155e9b000 rw-p 0000c000 fd:01 537284001                  /usr/lib64/libnss_files-2.17.so
 
7f6155e9b000-7f6155ea1000 rw-p 00000000 00:00 0
 
7f6155ea6000-7f6155ea9000 r-xp 00000000 fd:01 537284659                  /usr/lib64/libkeyutils.so.1.5
 
7f6155ea9000-7f61560a8000 ---p 00003000 fd:01 537284659                  /usr/lib64/libkeyutils.so.1.5
 
7f61560a8000-7f61560a9000 r--p 00002000 fd:01 537284659                  /usr/lib64/libkeyutils.so.1.5
 
7f61560a9000-7f61560aa000 rw-p 00003000 fd:01 537284659                  /usr/lib64/libkeyutils.so.1.5
 
7f61560ae000-7f61560bb000 r-xp 00000000 fd:01 538197595                  /usr/lib64/libkrb5support.so.0.1
 
7f61560bb000-7f61562bb000 ---p 0000d000 fd:01 538197595                  /usr/lib64/libkrb5support.so.0.1
 
7f61562bb000-7f61562bc000 r--p 0000d000 fd:01 538197595                  /usr/lib64/libkrb5support.so.0.1
 
7f61562bc000-7f61562bd000 rw-p 0000e000 fd:01 538197595                  /usr/lib64/libkrb5support.so.0.1
 
7f61562be000-7f61562cd000 r-xp 00000000 fd:01 537284435                  /usr/lib64/libbz2.so.1.0.6
 
7f61562cd000-7f61564cc000 ---p 0000f000 fd:01 537284435                  /usr/lib64/libbz2.so.1.0.6
 
7f61564cc000-7f61564cd000 r--p 0000e000 fd:01 537284435                  /usr/lib64/libbz2.so.1.0.6
 
7f61564cd000-7f61564ce000 rw-p 0000f000 fd:01 537284435                  /usr/lib64/libbz2.so.1.0.6
 
7f61564ce000-7f61564e5000 r-xp 00000000 fd:01 538185298                  /usr/lib64/libelf-0.166.so
 
7f61564e5000-7f61566e4000 ---p 00017000 fd:01 538185298                  /usr/lib64/libelf-0.166.so
 
7f61566e4000-7f61566e5000 r--p 00016000 fd:01 538185298                  /usr/lib64/libelf-0.166.so
 
7f61566e5000-7f61566e6000 rw-p 00017000 fd:01 538185298                  /usr/lib64/libelf-0.166.so
 
7f61566e6000-7f6156746000 r-xp 00000000 fd:01 537285624                  /usr/lib64/libpcre.so.1.2.0
 
7f6156746000-7f6156945000 ---p 00060000 fd:01 537285624                  /usr/lib64/libpcre.so.1.2.0
 
7f6156945000-7f6156946000 r--p 0005f000 fd:01 537285624                  /usr/lib64/libpcre.so.1.2.0
 
7f6156946000-7f6156947000 rw-p 00060000 fd:01 537285624                  /usr/lib64/libpcre.so.1.2.0
 
7f615694e000-7f6156952000 r-xp 00000000 fd:01 537284461                  /usr/lib64/libattr.so.1.1.0
 
7f6156952000-7f6156b51000 ---p 00004000 fd:01 537284461                  /usr/lib64/libattr.so.1.1.0
 
7f6156b51000-7f6156b52000 r--p 00003000 fd:01 537284461                  /usr/lib64/libattr.so.1.1.0
 
7f6156b52000-7f6156b53000 rw-p 00004000 fd:01 537284461                  /usr/lib64/libattr.so.1.1.0
 
7f6156b56000-7f6156b85000 r-xp 00000000 fd:01 537285635                  /usr/lib64/libk5crypto.so.3.1
 
7f6156b85000-7f6156d84000 ---p 0002f000 fd:01 537285635                  /usr/lib64/libk5crypto.so.3.1
 
7f6156d84000-7f6156d86000 r--p 0002e000 fd:01 537285635                  /usr/lib64/libk5crypto.so.3.1
 
7f6156d86000-7f6156d87000 rw-p 00030000 fd:01 537285635                  /usr/lib64/libk5crypto.so.3.1
 
7f6156d87000-7f6156d88000 rw-p 00000000 00:00 0
 
7f6156d8e000-7f6156d91000 r-xp 00000000 fd:01 537284427                  /usr/lib64/libcom_err.so.2.1
 
7f6156d91000-7f6156f90000 ---p 00003000 fd:01 537284427                  /usr/lib64/libcom_err.so.2.1
 
7f6156f90000-7f6156f91000 r--p 00002000 fd:01 537284427                  /usr/lib64/libcom_err.so.2.1
 
7f6156f91000-7f6156f92000 rw-p 00003000 fd:01 537284427                  /usr/lib64/libcom_err.so.2.1
 
7f6156f96000-7f615706c000 r-xp 00000000 fd:01 538197593                  /usr/lib64/libkrb5.so.3.3
 
7f615706c000-7f615726c000 ---p 000d6000 fd:01 538197593                  /usr/lib64/libkrb5.so.3.3
 
7f615726c000-7f615727a000 r--p 000d6000 fd:01 538197593                  /usr/lib64/libkrb5.so.3.3
 
7f615727a000-7f615727d000 rw-p 000e4000 fd:01 538197593                  /usr/lib64/libkrb5.so.3.3
 
7f615727e000-7f61572c9000 r-xp 00000000 fd:01 538197590                  /usr/lib64/libgssapi_krb5.so.2.2
 
7f61572c9000-7f61574c9000 ---p 0004b000 fd:01 538197590                  /usr/lib64/libgssapi_krb5.so.2.2
 
7f61574c9000-7f61574ca000 r--p 0004b000 fd:01 538197590                  /usr/lib64/libgssapi_krb5.so.2.2
 
7f61574ca000-7f61574cc000 rw-p 0004c000 fd:01 538197590                  /usr/lib64/libgssapi_krb5.so.2.2
 
7f61574ce000-7f61574e3000 r-xp 00000000 fd:01 537284562                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f61574e3000-7f61576e2000 ---p 00015000 fd:01 537284562                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f61576e2000-7f61576e3000 r--p 00014000 fd:01 537284562                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f61576e3000-7f61576e4000 rw-p 00015000 fd:01 537284562                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 
7f61576e6000-7f615772b000 r-xp 00000000 fd:01 537419371                  /usr/lib64/libdw-0.166.so
 
7f615772b000-7f615792b000 ---p 00045000 fd:01 537419371                  /usr/lib64/libdw-0.166.so
 
7f615792b000-7f615792d000 r--p 00045000 fd:01 537419371                  /usr/lib64/libdw-0.166.so
 
7f615792d000-7f615792e000 rw-p 00047000 fd:01 537419371                  /usr/lib64/libdw-0.166.so
 
7f615792e000-7f6157944000 r-xp 00000000 fd:01 537521790                  /usr/lib64/libresolv-2.17.so
 
7f6157944000-7f6157b44000 ---p 00016000 fd:01 537521790                  /usr/lib64/libresolv-2.17.so
 
7f6157b44000-7f6157b45000 r--p 00016000 fd:01 537521790                  /usr/lib64/libresolv-2.17.so
 
7f6157b45000-7f6157b46000 rw-p 00017000 fd:01 537521790                  /usr/lib64/libresolv-2.17.so
 
7f6157b46000-7f6157b48000 rw-p 00000000 00:00 0
 
7f6157b4e000-7f6157b52000 r-xp 00000000 fd:01 537284459                  /usr/lib64/libgpg-error.so.0.10.0
 
7f6157b52000-7f6157d51000 ---p 00004000 fd:01 537284459                  /usr/lib64/libgpg-error.so.0.10.0
 
7f6157d51000-7f6157d52000 r--p 00003000 fd:01 537284459                  /usr/lib64/libgpg-error.so.0.10.0
 
7f6157d52000-7f6157d53000 rw-p 00004000 fd:01 537284459                  /usr/lib64/libgpg-error.so.0.10.0
 
7f6157d56000-7f6157dd2000 r-xp 00000000 fd:01 538063510                  /usr/lib64/libgcrypt.so.11.8.2
 
7f6157dd2000-7f6157fd2000 ---p 0007c000 fd:01 538063510                  /usr/lib64/libgcrypt.so.11.8.2
 
7f6157fd2000-7f6157fd3000 r--p 0007c000 fd:01 538063510                  /usr/lib64/libgcrypt.so.11.8.2
 
7f6157fd3000-7f6157fd6000 rw-p 0007d000 fd:01 538063510                  /usr/lib64/libgcrypt.so.11.8.2
 
7f6157fd6000-7f6157fd7000 rw-p 00000000 00:00 0
 
7f6157fde000-7f6158002000 r-xp 00000000 fd:01 537284361                  /usr/lib64/libselinux.so.1
 
7f6158002000-7f6158201000 ---p 00024000 fd:01 537284361                  /usr/lib64/libselinux.so.1
 
7f6158201000-7f6158202000 r--p 00023000 fd:01 537284361                  /usr/lib64/libselinux.so.1
 
7f6158202000-7f6158203000 rw-p 00024000 fd:01 537284361                  /usr/lib64/libselinux.so.1
 
7f6158203000-7f6158205000 rw-p 00000000 00:00 0
 
7f6158206000-7f615820d000 r-xp 00000000 fd:01 537689733                  /usr/lib64/librt-2.17.so
 
7f615820d000-7f615840c000 ---p 00007000 fd:01 537689733                  /usr/lib64/librt-2.17.so
 
7f615840c000-7f615840d000 r--p 00006000 fd:01 537689733                  /usr/lib64/librt-2.17.so
 
7f615840d000-7f615840e000 rw-p 00007000 fd:01 537689733                  /usr/lib64/librt-2.17.so
 
7f615840e000-7f6158412000 r-xp 00000000 fd:01 537284466                  /usr/lib64/libcap.so.2.22
 
7f6158412000-7f6158611000 ---p 00004000 fd:01 537284466                  /usr/lib64/libcap.so.2.22
 
7f6158611000-7f6158612000 r--p 00003000 fd:01 537284466                  /usr/lib64/libcap.so.2.22
 
7f6158612000-7f6158613000 rw-p 00004000 fd:01 537284466                  /usr/lib64/libcap.so.2.22
 
7f6158616000-7f6158618000 r-xp 00000000 fd:01 537283816                  /usr/lib64/libfreebl3.so
 
7f6158618000-7f6158817000 ---p 00002000 fd:01 537283816                  /usr/lib64/libfreebl3.so
 
7f6158817000-7f6158818000 r--p 00001000 fd:01 537283816                  /usr/lib64/libfreebl3.so
 
7f6158818000-7f6158819000 rw-p 00002000 fd:01 537283816                  /usr/lib64/libfreebl3.so
 
7f615881e000-7f61589d5000 r-xp 00000000 fd:01 537283983                  /usr/lib64/libc-2.17.so
 
7f61589d5000-7f6158bd4000 ---p 001b7000 fd:01 537283983                  /usr/lib64/libc-2.17.so
 
7f6158bd4000-7f6158bd8000 r--p 001b6000 fd:01 537283983                  /usr/lib64/libc-2.17.so
 
7f6158bd8000-7f6158bda000 rw-p 001ba000 fd:01 537283983                  /usr/lib64/libc-2.17.so
 
7f6158bda000-7f6158bdf000 rw-p 00000000 00:00 0
 
7f6158be6000-7f6158ce6000 r-xp 00000000 fd:01 537283991                  /usr/lib64/libm-2.17.so
 
7f6158ce6000-7f6158ee6000 ---p 00100000 fd:01 537283991                  /usr/lib64/libm-2.17.so
 
7f6158ee6000-7f6158ee7000 r--p 00100000 fd:01 537283991                  /usr/lib64/libm-2.17.so
 
7f6158ee7000-7f6158ee8000 rw-p 00101000 fd:01 537283991                  /usr/lib64/libm-2.17.so
 
7f6158eee000-7f6158fd7000 r-xp 00000000 fd:01 537284311                  /usr/lib64/libstdc++.so.6.0.19
 
7f6158fd7000-7f61591d7000 ---p 000e9000 fd:01 537284311                  /usr/lib64/libstdc++.so.6.0.19
 
7f61591d7000-7f61591e0000 r--p 000e9000 fd:01 537284311                  /usr/lib64/libstdc++.so.6.0.19
 
7f61591e0000-7f61591e2000 rw-p 000f2000 fd:01 537284311                  /usr/lib64/libstdc++.so.6.0.19
 
7f61591e2000-7f61591f7000 rw-p 00000000 00:00 0
 
7f61591fe000-7f6159200000 r-xp 00000000 fd:01 537283989                  /usr/lib64/libdl-2.17.so
 
7f6159200000-7f6159400000 ---p 00002000 fd:01 537283989                  /usr/lib64/libdl-2.17.so
 
7f6159400000-7f6159401000 r--p 00002000 fd:01 537283989                  /usr/lib64/libdl-2.17.so
 
7f6159401000-7f6159402000 rw-p 00003000 fd:01 537283989                  /usr/lib64/libdl-2.17.so
 
7f6159406000-7f61595c6000 r-xp 00000000 fd:01 537284925                  /usr/lib64/libcrypto.so.1.0.1e
 
7f61595c6000-7f61597c6000 ---p 001c0000 fd:01 537284925                  /usr/lib64/libcrypto.so.1.0.1e
 
7f61597c6000-7f61597e0000 r--p 001c0000 fd:01 537284925                  /usr/lib64/libcrypto.so.1.0.1e
 
7f61597e0000-7f61597ec000 rw-p 001da000 fd:01 537284925                  /usr/lib64/libcrypto.so.1.0.1e
 
7f61597ec000-7f61597f0000 rw-p 00000000 00:00 0
 
7f61597f6000-7f615985a000 r-xp 00000000 fd:01 537284927                  /usr/lib64/libssl.so.1.0.1e
 
7f615985a000-7f6159a59000 ---p 00064000 fd:01 537284927                  /usr/lib64/libssl.so.1.0.1e
 
7f6159a59000-7f6159a5d000 r--p 00063000 fd:01 537284927                  /usr/lib64/libssl.so.1.0.1e
 
7f6159a5d000-7f6159a64000 rw-p 00067000 fd:01 537284927                  /usr/lib64/libssl.so.1.0.1e
 
7f6159a66000-7f6159a6e000 r-xp 00000000 fd:01 537283987                  /usr/lib64/libcrypt-2.17.so
 
7f6159a6e000-7f6159c6d000 ---p 00008000 fd:01 537283987                  /usr/lib64/libcrypt-2.17.so
 
7f6159c6d000-7f6159c6e000 r--p 00007000 fd:01 537283987                  /usr/lib64/libcrypt-2.17.so
 
7f6159c6e000-7f6159c6f000 rw-p 00008000 fd:01 537283987                  /usr/lib64/libcrypt-2.17.so
 
7f6159c6f000-7f6159c9d000 rw-p 00000000 00:00 0
 
7f6159c9e000-7f6159cb3000 r-xp 00000000 fd:01 537284257                  /usr/lib64/libz.so.1.2.7
 
7f6159cb3000-7f6159eb2000 ---p 00015000 fd:01 537284257                  /usr/lib64/libz.so.1.2.7
 
7f6159eb2000-7f6159eb3000 r--p 00014000 fd:01 537284257                  /usr/lib64/libz.so.1.2.7
 
7f6159eb3000-7f6159eb4000 rw-p 00015000 fd:01 537284257                  /usr/lib64/libz.so.1.2.7
 
7f6159eb6000-7f6159eb7000 r-xp 00000000 fd:01 537457496                  /usr/lib64/libaio.so.1.0.1
 
7f6159eb7000-7f615a0b6000 ---p 00001000 fd:01 537457496                  /usr/lib64/libaio.so.1.0.1
 
7f615a0b6000-7f615a0b7000 r--p 00000000 fd:01 537457496                  /usr/lib64/libaio.so.1.0.1
 
7f615a0b7000-7f615a0b8000 rw-p 00001000 fd:01 537457496                  /usr/lib64/libaio.so.1.0.1
 
7f615a0be000-7f615a0e3000 r-xp 00000000 fd:01 537284303                  /usr/lib64/liblzma.so.5.2.2
 
7f615a0e3000-7f615a2e2000 ---p 00025000 fd:01 537284303                  /usr/lib64/liblzma.so.5.2.2
 
7f615a2e2000-7f615a2e3000 r--p 00024000 fd:01 537284303                  /usr/lib64/liblzma.so.5.2.2
 
7f615a2e3000-7f615a2e4000 rw-p 00025000 fd:01 537284303                  /usr/lib64/liblzma.so.5.2.2
 
7f615a2e6000-7f615a2fd000 r-xp 00000000 fd:01 537285540                  /usr/lib64/libpthread-2.17.so
 
7f615a2fd000-7f615a4fc000 ---p 00017000 fd:01 537285540                  /usr/lib64/libpthread-2.17.so
 
7f615a4fc000-7f615a4fd000 r--p 00016000 fd:01 537285540                  /usr/lib64/libpthread-2.17.so
 
7f615a4fd000-7f615a4fe000 rw-p 00017000 fd:01 537285540                  /usr/lib64/libpthread-2.17.so
 
7f615a4fe000-7f615a502000 rw-p 00000000 00:00 0
 
7f615a506000-7f615a526000 r-xp 00000000 fd:01 537283976                  /usr/lib64/ld-2.17.so
 
7f615a550000-7f615a6ee000 rw-p 00000000 00:00 0
 
7f615a6ee000-7f615a714000 r-xp 00000000 fd:01 537284248                  /usr/lib64/libsystemd.so.0.6.0
 
7f615a714000-7f615a715000 r--p 00025000 fd:01 537284248                  /usr/lib64/libsystemd.so.0.6.0
 
7f615a715000-7f615a716000 rw-p 00026000 fd:01 537284248                  /usr/lib64/libsystemd.so.0.6.0
 
7f615a71e000-7f615a726000 rw-p 00000000 00:00 0
 
7f615a726000-7f615a727000 r--p 00020000 fd:01 537283976                  /usr/lib64/ld-2.17.so
 
7f615a727000-7f615a728000 rw-p 00021000 fd:01 537283976                  /usr/lib64/ld-2.17.so
 
7f615a728000-7f615a72a000 rw-p 00000000 00:00 0
 
7f615a72a000-7f615a72e000 rw-p 00000000 00:00 0
 
7f615a72e000-7f615b8ca000 r-xp 00000000 fd:01 268439672                  /usr/bin/mariabackup
 
7f615b8ca000-7f615b8d0000 rw-p 00000000 00:00 0
 
7f615bac9000-7f615bba3000 r--p 0119b000 fd:01 268439672                  /usr/bin/mariabackup
 
7f615bba3000-7f615bc5e000 rw-p 01275000 fd:01 268439672                  /usr/bin/mariabackup
 
7f615bc5e000-7f615c4f1000 rw-p 00000000 00:00 0
 
7f615d4d5000-7f615d660000 rw-p 00000000 00:00 0                          [heap]
 
7fffe39ad000-7fffe39ce000 rw-p 00000000 00:00 0                          [stack]
 
7fffe39ee000-7fffe39f0000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
/usr/local/lib/surrogate/lib/actions/inc: line 5: 27312 Aborted                 mariabackup --backup --parallel=8 --user=$mysql_user --password=$mysql_pass --no-timestamp --compress --compress-threads=8 --incremental-basedir=$last_backup --socket=$socket --defaults-group="mysqld$instance" --target-dir=$inc_backup_path 2> $logfile_i
 
checking if the backup completed sucsessfully

Comment by Andrii Nikitin (Inactive) [ 2017-09-14 ]

I am sorry that you observe the problem - I need to compile a test case to verify that.
Does it fail every time or sometimes? (how often)? Is the error shown at start of backup / in the middle / after completion ?
Do you have chance to provide configuration file(s) and full backup logs ?

Comment by Wayne [ 2017-09-14 ]

Hi Andrii,

Thanks for your response, I can't pinpoint exactly when it occurs but it appears to be relatively soon within the backup start process.

I would say it's our 1T table that is causing an issue.

Config files, and log from the crash dump attached.

Thanks.

mysql.cnf.upload log-from-backup.txt

Comment by Andrii Nikitin (Inactive) [ 2017-09-18 ]

It looks that I was able to reproduce the problem with few 150G+ files and following stack trace, now trying to compile smaller test case:

+ /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup --defaults-file=m1-10.2/my.cnf --backup --parallel=8 --compress --compress-threads=8 --target-dir=m1-10.2/bkup1 --incremental-basedir=m1-10.2/bkup
 
170918 09:38:56 Connecting to MySQL server host: localhost, user: root, password: not set, port: 3307, socket: /home/a/mariadb-environs/m1-10.2/dt/my.sock
 
Using server version 10.2.9-MariaDB
 
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup based on MariaDB server 10.2.9-MariaDB Linux (x86_64)
 
incremental backup from 527421805670 is enabled.
 
xtrabackup: uses posix_fadvise().
 
xtrabackup: cd to /home/a/mariadb-environs/m1-10.2/dt/
 
xtrabackup: open files limit requested 0, set to 1024
 
xtrabackup: using the following InnoDB configuration:
 
xtrabackup:   innodb_data_home_dir = .
 
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
 
xtrabackup:   innodb_log_group_home_dir = ./
 
2017-09-18  9:38:56 139914032555840 [Note] InnoDB: Number of pools: 1
 
170918 09:38:56 >> log scanned up to (532663833177)
 
xtrabackup: Generating a list of tablespaces
 
2017-09-18  9:38:56 139914032555840 [Warning] InnoDB: Allocated tablespace ID 1 for mysql/innodb_table_stats, old maximum was 0
 
xtrabackup: using the full scan for incremental backup
 
xtrabackup: Starting 8 threads for parallel data files transfer
 
170918 09:38:56 [07] Compressing ./test/t1.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/test/t1.ibd.delta.qp
 
170918 09:38:56 [06] Compressing ./test/t2.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/test/t2.ibd.delta.qp
 
170918 09:38:56 [01] Compressing ./mysql/innodb_index_stats.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/mysql/innodb_index_stats.ibd.delta.qp
 
170918 09:38:56 [05] Compressing ./test/t3.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/test/t3.ibd.delta.qp
 
170918 09:38:56 [02] Compressing ./ibdata1 to /home/a/mariadb-environs/m1-10.2/bkup1/ibdata1.delta.qp
 
170918 09:38:56 [03] Compressing ./mysql/innodb_table_stats.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/mysql/innodb_table_stats.ibd.delta.qp
 
170918 09:38:56 [04] Compressing ./mysql/gtid_slave_pos.ibd to /home/a/mariadb-environs/m1-10.2/bkup1/mysql/gtid_slave_pos.ibd.delta.qp
 
170918 09:38:56 [04]        ...done
 
170918 09:38:56 [01]        ...done
 
170918 09:38:56 [03]        ...done
 
170918 09:38:57 >> log scanned up to (532663833177)
 
170918  9:39:17 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.9-MariaDB
 
key_buffer_size=0
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=1
 
thread_count=0
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5392 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(my_print_stacktrace+0x29)[0x56362ef98499]
 
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(handle_fatal_signal+0x3bd)[0x56362eb3130d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f4045f54670]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x1507bf)[0x7f404451d7bf]
 
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(+0x438a7c)[0x56362e8b9a7c]
 
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(+0x42e3d1)[0x56362e8af3d1]
 
/home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup(+0x42e62d)[0x56362e8af62d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f4045f4a6da]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f40444d5d7f]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
MDEV-13798-2.sh: line 11:  6822 Segmentation fault      (core dumped) 
a@UBINTI:~/mariadb-environs$ addr2line -fie /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup 0x438a7c
 
_ZL22wf_incremental_processP20xb_write_filt_ctxt_tP9ds_file_t
 
/home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/write_filt.cc:147
 
a@UBINTI:~/mariadb-environs$ addr2line -fie /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup 0x42e3d1
 
_ZL24xtrabackup_copy_datafileP10fil_node_tj
 
/home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2271
 
a@UBINTI:~/mariadb-environs$ addr2line -fie /home/a/mariadb-environs/m1-10.2/build/extra/mariabackup//mariabackup 0x42e62d
 
_ZL21data_copy_thread_funcPv
 
/home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2519

Comment by Andrii Nikitin (Inactive) [ 2017-09-18 ]

Hi wayne, is there chance for you to try the same incremental backup, just without both --parallel=8 and --compress-threads=8 options and confirm if it is feasible workaround for your environment?

Comment by Wayne [ 2017-09-18 ]

Hi Andrii,

I'll give that a go and report back my findings!

Thanks,
Wayne

Comment by Andrii Nikitin (Inactive) [ 2017-09-18 ]

wayne could you also confirm approximate number of .idb files in this system ? Is it around 70-80 or much more?

Comment by Andrii Nikitin (Inactive) [ 2017-09-18 ]

marko Could you confirm if my analysis is correct here:

Code has structure xb_wf_incremental_ctxt_t with members delta_buf_base and delta_buf, which are initialized as below for object cp :
https://github.com/MariaDB/server/blob/10.2/extra/mariabackup/write_filt.cc#L77

        /* allocate buffer for incremental backup (4096 pages) */
 
        buf_size = (cursor->page_size.physical() / 4 + 1)
 
                * cursor->page_size.physical();
 
        cp->delta_buf_base = static_cast<byte *>(malloc(buf_size));
 
        memset(cp->delta_buf_base, 0, buf_size);
 
        cp->delta_buf = static_cast<byte *>
 
                (ut_align(cp->delta_buf_base, UNIV_PAGE_SIZE_MAX));

And now mentioned crash with gdb at https://github.com/MariaDB/server/blob/mariadb-10.2.8/extra/mariabackup/write_filt.cc#L144 :

Thread 3 "mariabackup" received signal SIGSEGV, Segmentation fault.
 
[Switching to Thread 0x7fffd4d52700 (LWP 31473)]
 
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:242
 
242	../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
 
(gdb) bt
 
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:242
 
#1  0x000055555598cadc in memcpy (__len=<optimized out>, __src=0x7fffcf998000, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:53
 
#2  wf_incremental_process (ctxt=0x7fffd4d51730, dstfile=0x7fffc8000a58) at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/write_filt.cc:145
 
#3  0x0000555555982371 in xtrabackup_copy_datafile (node=<optimized out>, thread_n=1)
 
    at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2282
 
#4  0x00005555559825fe in data_copy_thread_func (arg=0x555557264ce0) at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/xtrabackup.cc:2530
 
#5  0x00007ffff7bc06da in start_thread (arg=0x7fffd4d52700) at pthread_create.c:456
 
#6  0x00007ffff614bd7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 
(gdb) f 2
 
#2  wf_incremental_process (ctxt=0x7fffd4d51730, dstfile=0x7fffc8000a58) at /home/a/mariadb-environs/_depot/m-branch/10.2/extra/mariabackup/write_filt.cc:145
 
145			       page_size);
 
(gdb) l
 
140			}
 
141	
142			mach_write_to_4(cp->delta_buf + cp->npages * 4,
 
143					cursor->buf_page_no + i);
 
144			memcpy(cp->delta_buf + cp->npages * page_size, page,
 
145			       page_size);
 
146	
147			cp->npages++;
 
148		}
 
149	
(gdb) p cp->delta_buf
 
$1 = (byte *) 0x7fffc3800000 "xtra"
 
(gdb) p cp->npages
 
$2 = 4095
 
(gdb) p cursor
 
$4 = (xb_fil_cur_t *) 0x7fffd4d51950
 
(gdb) p cursor->buf_page_no
 
$5 = 44160
 
(gdb) p i
 
$6 = 231

The crash happened with default page size and it according to gdb it was writing page 4095 .

Remembering code above - allocated memory fragment was 4097 x 16k bytes, which was further aligned to UNIV_PAGE_SIZE_MAX boundary (64k), which in worst case could shrink allocated memory below 4095 pages ?

I am not sure if it is feasible to create a reliable test case for this, so going to verify looking at source code with developers approval.

Comment by Andrii Nikitin (Inactive) [ 2017-09-18 ]

wayne thank you for all the help - please ignore my earlier requests for now, as I hope that developers will confirm the findings above.

Comment by Andrii Nikitin (Inactive) [ 2017-09-19 ]

Attached heavy mtr test case reliably shows problem in 10.2, which disappears after attached patch :

CREATE TABLE t1 select uuid() a, uuid() b, uuid() c from seq_1_to_1000;
 
CREATE TABLE t2 select * from t1;
 
CREATE TABLE t3 select * from t1;
 
# Create full backup , modify table, then create incremental/differential backup
 
start transaction;
 
INSERT INTO t1 select uuid(), uuid(), uuid() from seq_1_to_1000000;
 
INSERT INTO t2 select uuid(), uuid(), uuid() from seq_1_to_1000000;
 
INSERT INTO t3 select uuid(), uuid(), uuid() from seq_1_to_1000000;
 
*** Error in `/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup': free(): invalid pointer: 0x00007f9effffb010 ***
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f9f4e05108b]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x82c3a)[0x7f9f4e05ac3a]
 
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f9f4e05ed2c]
 
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup[0x5521b2]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f9f4f27d6da]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f9f4e0e0d7f]
 
======= Memory map: ========
 
00400000-014a0000 r-xp 00000000 08:13 221033                             /home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup
 
0169f000-01766000 rw-p 0109f000 08:13 221033                             /home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup
 
01766000-01ff6000 rw-p 00000000 00:00 0 
03498000-03586000 rw-p 00000000 00:00 0                                  [heap]
 
7f9efbff6000-7f9f04000000 rw-p 00000000 00:00 0 
7f9f04000000-7f9f04021000 rw-p 00000000 00:00 0 
7f9f04021000-7f9f08000000 ---p 00000000 00:00 0 
7f9f0bfec000-7f9f0fff1000 rw-p 00000000 00:00 0 
7f9f10000000-7f9f10021000 rw-p 00000000 00:00 0 
7f9f10021000-7f9f14000000 ---p 00000000 00:00 0 
7f9f1c000000-7f9f1c021000 rw-p 00000000 00:00 0 
7f9f1c021000-7f9f20000000 ---p 00000000 00:00 0 
7f9f2c000000-7f9f2c021000 rw-p 00000000 00:00 0 
7f9f2c021000-7f9f30000000 ---p 00000000 00:00 0 
7f9f30000000-7f9f30021000 rw-p 00000000 00:00 0 
7f9f30021000-7f9f34000000 ---p 00000000 00:00 0 
7f9f34000000-7f9f34021000 rw-p 00000000 00:00 0 
7f9f34021000-7f9f38000000 ---p 00000000 00:00 0 
7f9f38000000-7f9f38021000 rw-p 00000000 00:00 0 
7f9f38021000-7f9f3c000000 ---p 00000000 00:00 0 
7f9f3e5f9000-7f9f3e5fa000 ---p 00000000 00:00 0 
7f9f3e5fa000-7f9f3edfa000 rw-p 00000000 00:00 0 
7f9f3edfa000-7f9f3edfb000 ---p 00000000 00:00 0 
7f9f3edfb000-7f9f3f5fb000 rw-p 00000000 00:00 0 
7f9f40000000-7f9f40021000 rw-p 00000000 00:00 0 
7f9f40021000-7f9f44000000 ---p 00000000 00:00 0 
7f9f457fb000-7f9f457fc000 ---p 00000000 00:00 0 
7f9f457fc000-7f9f45ffc000 rw-p 00000000 00:00 0 
7f9f45ffc000-7f9f45ffd000 ---p 00000000 00:00 0 
7f9f45ffd000-7f9f467fd000 rw-p 00000000 00:00 0 
7f9f467fd000-7f9f467fe000 ---p 00000000 00:00 0 
7f9f467fe000-7f9f46ffe000 rw-p 00000000 00:00 0 
7f9f46ffe000-7f9f46fff000 ---p 00000000 00:00 0 
7f9f46fff000-7f9f477ff000 rw-p 00000000 00:00 0 
7f9f477ff000-7f9f47800000 ---p 00000000 00:00 0 
7f9f47800000-7f9f48000000 rw-p 00000000 00:00 0 
7f9f48000000-7f9f48021000 rw-p 00000000 00:00 0 
7f9f48021000-7f9f4c000000 ---p 00000000 00:00 0 
7f9f4c15e000-7f9f4c15f000 ---p 00000000 00:00 0 
7f9f4c15f000-7f9f4cb64000 rw-p 00000000 00:00 0 
7f9f4cb64000-7f9f4cb65000 ---p 00000000 00:00 0 
7f9f4cb65000-7f9f4dbaf000 rw-p 00000000 00:00 0 
7f9f4dbaf000-7f9f4dbba000 r-xp 00000000 08:13 536929667                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f9f4dbba000-7f9f4ddb9000 ---p 0000b000 08:13 536929667                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f9f4ddb9000-7f9f4ddba000 r--p 0000a000 08:13 536929667                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f9f4ddba000-7f9f4ddbb000 rw-p 0000b000 08:13 536929667                  /lib/x86_64-linux-gnu/libnss_files-2.24.so
 
7f9f4ddbb000-7f9f4ddc1000 rw-p 00000000 00:00 0 
7f9f4ddc1000-7f9f4ddd7000 r-xp 00000000 08:13 536924403                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f9f4ddd7000-7f9f4dfd6000 ---p 00016000 08:13 536924403                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f9f4dfd6000-7f9f4dfd7000 r--p 00015000 08:13 536924403                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f9f4dfd7000-7f9f4dfd8000 rw-p 00016000 08:13 536924403                  /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f9f4dfd8000-7f9f4e196000 r-xp 00000000 08:13 536923520                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7f9f4e196000-7f9f4e395000 ---p 001be000 08:13 536923520                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7f9f4e395000-7f9f4e399000 r--p 001bd000 08:13 536923520                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7f9f4e399000-7f9f4e39b000 rw-p 001c1000 08:13 536923520                  /lib/x86_64-linux-gnu/libc-2.24.so
 
7f9f4e39b000-7f9f4e39f000 rw-p 00000000 00:00 0 
7f9f4e39f000-7f9f4e4a7000 r-xp 00000000 08:13 536923558                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7f9f4e4a7000-7f9f4e6a6000 ---p 00108000 08:13 536923558                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7f9f4e6a6000-7f9f4e6a7000 r--p 00107000 08:13 536923558                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7f9f4e6a7000-7f9f4e6a8000 rw-p 00108000 08:13 536923558                  /lib/x86_64-linux-gnu/libm-2.24.so
 
7f9f4e6a8000-7f9f4e821000 r-xp 00000000 08:13 1074794301                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f9f4e821000-7f9f4ea20000 ---p 00179000 08:13 1074794301                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f9f4ea20000-7f9f4ea2a000 r--p 00178000 08:13 1074794301                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f9f4ea2a000-7f9f4ea2c000 rw-p 00182000 08:13 1074794301                 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
 
7f9f4ea2c000-7f9f4ea30000 rw-p 00000000 00:00 0 
7f9f4ea30000-7f9f4ea33000 r-xp 00000000 08:13 536923538                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f9f4ea33000-7f9f4ec32000 ---p 00003000 08:13 536923538                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f9f4ec32000-7f9f4ec33000 r--p 00002000 08:13 536923538                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f9f4ec33000-7f9f4ec34000 rw-p 00003000 08:13 536923538                  /lib/x86_64-linux-gnu/libdl-2.24.so
 
7f9f4ec34000-7f9f4ec3d000 r-xp 00000000 08:13 536923537                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f9f4ec3d000-7f9f4ee3c000 ---p 00009000 08:13 536923537                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f9f4ee3c000-7f9f4ee3d000 r--p 00008000 08:13 536923537                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f9f4ee3d000-7f9f4ee3e000 rw-p 00009000 08:13 536923537                  /lib/x86_64-linux-gnu/libcrypt-2.24.so
 
7f9f4ee3e000-7f9f4ee6c000 rw-p 00000000 00:00 0 
7f9f4ee6c000-7f9f4ee73000 r-xp 00000000 08:13 536953566                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7f9f4ee73000-7f9f4f072000 ---p 00007000 08:13 536953566                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7f9f4f072000-7f9f4f073000 r--p 00006000 08:13 536953566                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7f9f4f073000-7f9f4f074000 rw-p 00007000 08:13 536953566                  /lib/x86_64-linux-gnu/librt-2.24.so
 
7f9f4f074000-7f9f4f075000 r-xp 00000000 08:13 539008302                  /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f9f4f075000-7f9f4f274000 ---p 00001000 08:13 539008302                  /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f9f4f274000-7f9f4f275000 r--p 00000000 08:13 539008302                  /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f9f4f275000-7f9f4f276000 rw-p 00000000 00:00 0 
7f9f4f276000-7f9f4f28e000 r-xp 00000000 08:13 536953509                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f9f4f28e000-7f9f4f48e000 ---p 00018000 08:13 536953509                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f9f4f48e000-7f9f4f48f000 r--p 00018000 08:13 536953509                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f9f4f48f000-7f9f4f490000 rw-p 00019000 08:13 536953509                  /lib/x86_64-linux-gnu/libpthread-2.24.so
 
7f9f4f490000-7f9f4f494000 rw-p 00000000 00:00 0 
7f9f4f494000-7f9f4f4ba000 r-xp 00000000 08:13 536922919                  /lib/x86_64-linux-gnu/ld-2.24.so
 
7f9f4f4da000-7f9f4f695000 rw-p 00000000 00:00 0 
7f9f4f695000-7f9f4f6b9000 rw-p 00000000 00:00 0 
7f9f4f6b9000-7f9f4f6ba000 r--p 00025000 08:13 536922919                  /lib/x86_64-linux-gnu/ld-2.24.so
 
7f9f4f6ba000-7f9f4f6bb000 rw-p 00026000 08:13 536922919                  /lib/x86_64-linux-gnu/ld-2.24.so
 
7f9f4f6bb000-7f9f4f6bc000 rw-p 00000000 00:00 0 
7ffcebaf9000-7ffcebb1d000 rw-p 00000000 00:00 0                          [stack]
 
7ffcebb9f000-7ffcebba1000 r--p 00000000 00:00 0                          [vvar]
 
7ffcebba1000-7ffcebba3000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
mariabackup.MDEV-13798 '16k,innodb'      [ fail ]
 
        Test ended at 2017-09-19 13:37:04
 
 
 
CURRENT_TEST: mariabackup.MDEV-13798
 
170919 13:36:54 Connecting to MySQL server host: localhost, user: root, password: set, port: 16020, socket: /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/mysqld.1.sock
 
Using server version 10.2.8-MariaDB-log
 
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup based on MariaDB server 10.2.8-MariaDB Linux (x86_64) 
incremental backup from 2213423 is enabled.
 
xtrabackup: uses posix_fadvise().
 
xtrabackup: cd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/mysqld.1/data/
 
xtrabackup: open files limit requested 1024, set to 1024
 
xtrabackup: using the following InnoDB configuration:
 
xtrabackup:   innodb_data_home_dir = .
 
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
 
xtrabackup:   innodb_log_group_home_dir = ./
 
2017-09-19 13:36:54 140322208814592 [Note] InnoDB: Number of pools: 1
 
170919 13:36:54 >> log scanned up to (514417320)
 
xtrabackup: Generating a list of tablespaces
 
2017-09-19 13:36:54 140322208814592 [Warning] InnoDB: Allocated tablespace ID 1 for mysql/innodb_table_stats, old maximum was 0
 
xtrabackup: using the full scan for incremental backup
 
xtrabackup: Starting 8 threads for parallel data files transfer
 
170919 13:36:54 [06] Copying ./test/t3.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/test/t3.ibd.delta
 
170919 13:36:54 [02] Copying ./mysql/innodb_index_stats.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/mysql/innodb_index_stats.ibd.delta
 
170919 13:36:54 [03] Copying ./test/t1.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/test/t1.ibd.delta
 
170919 13:36:54 [05] Copying ./test/t2.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/test/t2.ibd.delta
 
170919 13:36:54 [04] Copying ./mysql/innodb_table_stats.ibd to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/mysql/innodb_table_stats.ibd.delta
 
170919 13:36:54 [01] Copying ./ibdata1 to /home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1/ibdata1.delta
 
170919 13:36:54 [02]        ...done
 
170919 13:36:54 [04]        ...done
 
170919 13:36:57 [01]        ...done
 
170919 13:36:57 >> log scanned up to (514772389)
 
170919 13:36:58 [05]        ...done
 
170919 13:36:58 [03]        ...done
 
170919 13:36:59 [06]        ...done
 
170919 13:37:00 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.8-MariaDB
 
key_buffer_size=0
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=1
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5391 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup(my_print_stacktrace+0x2e)[0xe0e4fe]
 
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup(handle_fatal_signal+0x444)[0x883264]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f9f4f287670]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x9f)[0x7f9f4e00d77f]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f9f4e00f37a]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x79090)[0x7f9f4e051090]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x82c3a)[0x7f9f4e05ac3a]
 
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f9f4e05ed2c]
 
/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup[0x5521b2]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f9f4f27d6da]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f9f4e0e0d7f]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Aborted (core dumped)
 
mysqltest: At line 25: exec of '/home/a/mariadb-environs/_depot/m-tar/10.2.8/bin/mariabackup --defaults-file=/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/my.cnf  --backup --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wait-query-type=all --target-dir=/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup_inc1 --incremental-basedir=/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/tmp/backup  --parallel=8' failed, error: 34304, status: 134, errno: 11
 
Output from before failure:
 
INSERT INTO t3 select uuid(), uuid(), uuid() from seq_1_to_1000000;
 
 
 
 - saving '/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/log/mariabackup.MDEV-13798-16k,innodb/' to '/home/a/mariadb-environs/_depot/m-tar/10.2.8/mysql-test/var/log/mariabackup.MDEV-13798-16k,innodb/'

I don't understand why the problem is not repeatable in 10.1 (with this and even heavier test, using 6 tables the same way), at least I couldn't reproduce it. But I think similar patch to wf_incremental_init() is needed in 10.1 as well.

Comment by Marko Mäkelä [ 2017-09-29 ]

jplindst, please have a look at MDEV-13822 as well. I believe that --incremental is broken in 10.1 already. I also believe that Percona XtraBackup is working around that issue at a low level, while MariaDB implemented a more accurate solution in MDEV-11556.

Comment by Marko Mäkelä [ 2017-10-04 ]

I would strongly suggest compiling -DWITH_ASAN and running all tests with

ASAN_OPTIONS=abort_on_error=1,detect_leaks=0 ./mtr --suite=mariabackup

Unfortunately, last time I tried, there were some memory leaks somewhere, possibly only in the non-debug build. Those should of course be fixed too.
What does the AddressSanitizer report for this test case?

Generated at Thu Feb 08 08:08:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.