[MDEV-18419] mariabackup fails to backup any databases Created: 2019-01-29  Updated: 2020-07-01  Resolved: 2019-01-29

Status: Closed
Project: MariaDB Server
Component/s: Configuration, mariabackup
Affects Version/s: 10.3.11
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Steven M Castano Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

2 Ubuntu 18.04.1 virtual machines under VMware with MariaDB 10.3.11 install in a galera cluster with a 3 VM in the Amazon cloud for async replication.


Issue Links:
Relates
relates to MDEV-23059 Mariabackup crashed during backup, ma... Closed

 Description   

When attempting to do a backup with the mariabackup, which I wanted to use for it's ability to create incremental backups, the initial full backup fails to even begin. Here is the error output:

190129 13:48:32 Connecting to MySQL server host: localhost, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 10.3.11-MariaDB-1:10.3.11+maria~bionic-log
mariabackup based on MariaDB server 10.3.11-MariaDB debian-linux-gnu (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /mnt/data/mysql/data/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup:   innodb_data_home_dir =
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
mariabackup:   innodb_log_group_home_dir = ./
2019-01-29 13:48:32 0x7fa7361b97c0 InnoDB: Using Linux native AIO
2019-01-29 13:48:32 0 [Note] InnoDB: Number of pools: 1
2019-01-29 13:48:32 0 [ERROR] InnoDB: mmap(17179869184 bytes) failed; errno 12
2019-01-29 13:48:32 0x7fa7361b97c0  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.11/storage/innobase/include/ut0new.h line 254
InnoDB: Failing assertion: ptr != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
190129 13:48:32 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.11-MariaDB-1:10.3.11+maria~bionic
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5565 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x55a190a942ee]
mariabackup(handle_fatal_signal+0x5a5)[0x55a190591f65]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fa735d99890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fa733a53e97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fa733a55801]
mariabackup(+0x4a5f70)[0x55a190259f70]
mariabackup(+0x9f9338)[0x55a1907ad338]
mariabackup(+0xa0050f)[0x55a1907b450f]
mariabackup(+0x4c346b)[0x55a19027746b]
mariabackup(main+0x185)[0x55a19025c255]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fa733a36b97]
mariabackup(_start+0x2a)[0x55a19027177a]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Aborted



 Comments   
Comment by Elena Stepanova [ 2019-01-29 ]

It complains about the lack of memory. Do you have enough memory on the machine?

Comment by Steven M Castano [ 2019-01-29 ]

I do.. InnoDB buffer pool is set to 16G, memory in the VM is 32G and reports that plenty is free:

              total        used        free      shared  buff/cache   available
Mem:            31G         17G         13G        1.7M        511M         13G
Swap:            0B          0B          0B

Comment by Elena Stepanova [ 2019-01-29 ]

MariaBackup runs its own instance of InnoDB. If you just run it without providing any specific options, it will use the configuration from your config file, and thus will also want 16G for InnoDB buffer pool alone.
Try to run it with a smaller buffer pool by providing the value on the command line or in a separate config file/defaults group (see the page by the link above for more details).

Comment by Steven M Castano [ 2019-01-29 ]

I'm looking into that not and testing some settings, it does appear that this will work... I'm just letting a full backup run, then I'll test an incremental and everything should be good!

Comment by Steven M Castano [ 2019-01-29 ]

It looks like the advice given was exactly correct, specifying the memory size on the command line and/or adding more memory for another large pool works great. All databases completed both full and minor incremental backups.

Thanks for the help!

Comment by Elena Stepanova [ 2019-01-29 ]

No problem. Good luck!

Generated at Thu Feb 08 08:43:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.