MariaDB provides a backup solution in 10.1 release line and higher. To declare it being universal, we need to ensure it works (and keeps working) with MariaDB 10.0 and 5.5, until their EOL, which is still far away.
First testing for this was done in the scope of MDEV-14936, but at that point MariaBackup was still at alpha stage, and no regression testing has been done since then. Besides, it was tested with scripts obfuscating the process, so we don't know if the basic backup / restore procedure as it's documented for MariaBackup actually works.
backuped data from 10.2 is restored with the tool mariabackup based on MariaDB-backup v10.2.12-MariaDB Linux (x86_64) on MariaDB server v10.0
but mysql service fail to start — it occurs crash
80118 15:56:16 [Note] /usr/sbin/mysqld (mysqld 10.0.33-MariaDB) starting as process 1833 ...
180118 15:56:16 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
180118 15:56:16 [Note] InnoDB: Using mutexes to ref count buffer pool pages
180118 15:56:16 [Note] InnoDB: The InnoDB memory heap is disabled
180118 15:56:16 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
180118 15:56:16 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
180118 15:56:16 [Note] InnoDB: Compressed tables use zlib 1.2.7
180118 15:56:16 [Note] InnoDB: Using Linux native AIO
180118 15:56:16 [Note] InnoDB: Not using CPU crc32 instructions
180118 15:56:16 [Note] InnoDB: Initializing buffer pool, size = 128.0M
180118 15:56:17 [Note] InnoDB: Completed initialization of buffer pool
180118 15:56:17 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
180118 15:56:17 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
180118 15:56:18 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
180118 15:56:18 [Warning] InnoDB: New log files created, LSN=349671716
180118 15:56:18 [Note] InnoDB: Highest supported file format is Barracuda.
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
180118 15:56:18 [Note] InnoDB: Creating tablespace and datafile system tables.
180118 15:56:18 [ERROR] InnoDB: Creation of SYS_TABLESPACES and SYS_DATAFILES has failed with error 18. Tablespace is full. Dropping incompletely created tables.
2018-01-18 15:56:18 7f2b6be5f8c0 InnoDB: Assertion failure in thread 139824470554816 in file dict0crea.cc line 1899
InnoDB: Failing assertion: err == DB_OUT_OF_FILE_SPACE || err == DB_TOO_MANY_CONCURRENT_TRXS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
180118 15:56:18 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.0.33-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467018 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xbda88e]
/usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x73fd26]
/lib64/libpthread.so.0(+0xf5e0)[0x7f2b6ba455e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f2b6a1781f7]
/lib64/libc.so.6(abort+0x148)[0x7f2b6a1798e8]
/usr/sbin/mysqld[0xae2ac8]
/usr/sbin/mysqld[0xa5fb6a]
/usr/sbin/mysqld[0x993ffc]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5e)[0x741c3e]
/usr/sbin/mysqld[0x5e5cf6]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x494)[0x5e6f64]
/usr/sbin/mysqld[0x53a92c]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x50f)[0x54063f]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f2b6a164c05]
/usr/sbin/mysqld[0x535979]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
180118 15:56:19 mysqld_safe mysqld from pid file /var/lib/mysql/t4w3.xentio.lan.pid ended
Zdravelina Sokolovska (Inactive)
added a comment - - edited backuped data from 10.2 is restored with the tool mariabackup based on MariaDB-backup v10.2.12-MariaDB Linux (x86_64) on MariaDB server v10.0
but mysql service fail to start — it occurs crash
80118 15:56:16 [Note] /usr/sbin/mysqld (mysqld 10.0.33-MariaDB) starting as process 1833 ...
180118 15:56:16 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
180118 15:56:16 [Note] InnoDB: Using mutexes to ref count buffer pool pages
180118 15:56:16 [Note] InnoDB: The InnoDB memory heap is disabled
180118 15:56:16 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
180118 15:56:16 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
180118 15:56:16 [Note] InnoDB: Compressed tables use zlib 1.2.7
180118 15:56:16 [Note] InnoDB: Using Linux native AIO
180118 15:56:16 [Note] InnoDB: Not using CPU crc32 instructions
180118 15:56:16 [Note] InnoDB: Initializing buffer pool, size = 128.0M
180118 15:56:17 [Note] InnoDB: Completed initialization of buffer pool
180118 15:56:17 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
180118 15:56:17 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
180118 15:56:18 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
180118 15:56:18 [Warning] InnoDB: New log files created, LSN=349671716
180118 15:56:18 [Note] InnoDB: Highest supported file format is Barracuda.
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
180118 15:56:18 [Note] InnoDB: Creating tablespace and datafile system tables.
180118 15:56:18 [ERROR] InnoDB: Creation of SYS_TABLESPACES and SYS_DATAFILES has failed with error 18. Tablespace is full. Dropping incompletely created tables.
2018-01-18 15:56:18 7f2b6be5f8c0 InnoDB: Assertion failure in thread 139824470554816 in file dict0crea.cc line 1899
InnoDB: Failing assertion: err == DB_OUT_OF_FILE_SPACE || err == DB_TOO_MANY_CONCURRENT_TRXS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
180118 15:56:18 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.0.33-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467018 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0xbda88e]
/usr/sbin/mysqld(handle_fatal_signal+0x3b6) [0x73fd26]
/lib64/libpthread.so.0(+0xf5e0) [0x7f2b6ba455e0]
/lib64/libc.so.6(gsignal+0x37) [0x7f2b6a1781f7]
/lib64/libc.so.6(abort+0x148) [0x7f2b6a1798e8]
/usr/sbin/mysqld [0xae2ac8]
/usr/sbin/mysqld [0xa5fb6a]
/usr/sbin/mysqld [0x993ffc]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5e) [0x741c3e]
/usr/sbin/mysqld [0x5e5cf6]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x494) [0x5e6f64]
/usr/sbin/mysqld [0x53a92c]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x50f) [0x54063f]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2b6a164c05]
/usr/sbin/mysqld [0x535979]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
180118 15:56:19 mysqld_safe mysqld from pid file /var/lib/mysql/t4w3.xentio.lan.pid ended
– In order to get MariaDB-backup v10.1 and use on MariaDB-Server v10.0 , respectively on MariaDB-Server v5.5
should clean the repo cache and change MariaDB.repo to pint out 10.1
yum clean all
rm -rf /var/cache/yum
change /etc/yum.repos.d/MariaDB.repo to pint out 10.1
install package MariaDB-backup
Zdravelina Sokolovska (Inactive)
added a comment - – MariaDB-backup package is not found for baseurl = http://yum.mariadb.org/10.0/centos7-amd64 or 5.5
– In order to get MariaDB-backup v10.1 and use on MariaDB-Server v10.0 , respectively on MariaDB-Server v5.5
should clean the repo cache and change MariaDB.repo to pint out 10.1
yum clean all
rm -rf /var/cache/yum
change /etc/yum.repos.d/MariaDB.repo to pint out 10.1
install package MariaDB-backup
checked full mariabackup based on MariaDB-backup v10.1
on MariaDB-Server v10.0 and MariaDB-Server v5.5
with backup taken from quiet server
full backup taken from server v5.5 is restored successfully on
server v5.5 with backup-tool v10.1
server v10.0 with backup-tool v10.1
server v10.1 with backup-tool v10.1
server v10.2 with backup-tool v10.2
full backup taken from server v10.0 is restored successfully on
server v10.0 with backup-tool v10.1
server v10.2 with backup-tool v10.2
full backup taken from server v10.2 is restored successfully on
server v10.2 with backup-tool v10.2
checked also with backup taken from one Node and restored on other Node with server v10.2 with backup-tool v10.2
note : when move from one Node to another Node, the step --prepare should be executed on the New Node
before restoring backuped data
partial backup fails to be done server v5.5
note : the partial backup might be done with --export option and --tables option
but it is not automated by the mariabackup tool and it's needed to copy manually data and
also manually from mysql to perform ALTER TABLE xx.yy import TABLESPACE ; statement
note : it found that Service mysql fail to start after restoring from mariabackup ; the problem appears due to ownerchip mysql:mysql of backuped and restored files and directories is not preserved by the mariabackup toll
the Workaround is to change manually the ownerchipod datadir
chown -R mysql:mysql /var/lib/mysql
Zdravelina Sokolovska (Inactive)
added a comment - - edited checked full mariabackup based on MariaDB-backup v10.1
on MariaDB-Server v10.0 and MariaDB-Server v5.5
with backup taken from quiet server
full backup taken from server v5.5 is restored successfully on
server v5.5 with backup-tool v10.1
server v10.0 with backup-tool v10.1
server v10.1 with backup-tool v10.1
server v10.2 with backup-tool v10.2
full backup taken from server v10.0 is restored successfully on
server v10.0 with backup-tool v10.1
server v10.2 with backup-tool v10.2
full backup taken from server v10.2 is restored successfully on
server v10.2 with backup-tool v10.2
checked also with backup taken from one Node and restored on other Node with server v10.2 with backup-tool v10.2
note : when move from one Node to another Node, the step --prepare should be executed on the New Node
before restoring backuped data
partial backup fails to be done server v5.5
note : the partial backup might be done with --export option and --tables option
but it is not automated by the mariabackup tool and it's needed to copy manually data and
also manually from mysql to perform ALTER TABLE xx.yy import TABLESPACE ; statement
note : it found that Service mysql fail to start after restoring from mariabackup ; the problem appears due to ownerchip mysql:mysql of backuped and restored files and directories is not preserved by the mariabackup toll
the Workaround is to change manually the ownerchipod datadir
chown -R mysql:mysql /var/lib/mysql
permission on restore problem has been reported as MDEV-14976
I suspect a mariadb-5.5 with a configuration defined as will be ignored by mariabackup.
[mariadb-5.5]
innodb....
The server version with configuration in a version mysql configuration group matching mariabackup will be solved by https://github.com/MariaDB/server/pull/554 I suspect.
Daniel Black
added a comment - permission on restore problem has been reported as MDEV-14976
I suspect a mariadb-5.5 with a configuration defined as will be ignored by mariabackup.
[mariadb-5.5]
innodb....
The server version with configuration in a version mysql configuration group matching mariabackup will be solved by https://github.com/MariaDB/server/pull/554 I suspect.
Task
Major
MDEV-14976 Service mysql fail to start after restoring from mariabackup
backuped data from 10.2 is restored with the tool mariabackup based on MariaDB-backup v10.2.12-MariaDB Linux (x86_64) on MariaDB server v10.0
but mysql service fail to start — it occurs crash
80118 15:56:16 [Note] /usr/sbin/mysqld (mysqld 10.0.33-MariaDB) starting as process 1833 ...
180118 15:56:16 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
180118 15:56:16 [Note] InnoDB: Using mutexes to ref count buffer pool pages
180118 15:56:16 [Note] InnoDB: The InnoDB memory heap is disabled
180118 15:56:16 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
180118 15:56:16 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
180118 15:56:16 [Note] InnoDB: Compressed tables use zlib 1.2.7
180118 15:56:16 [Note] InnoDB: Using Linux native AIO
180118 15:56:16 [Note] InnoDB: Not using CPU crc32 instructions
180118 15:56:16 [Note] InnoDB: Initializing buffer pool, size = 128.0M
180118 15:56:17 [Note] InnoDB: Completed initialization of buffer pool
180118 15:56:17 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
180118 15:56:17 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
180118 15:56:18 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
180118 15:56:18 [Warning] InnoDB: New log files created, LSN=349671716
180118 15:56:18 [Note] InnoDB: Highest supported file format is Barracuda.
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
InnoDB: wrong number of columns in SYS_INDEXES record
180118 15:56:18 [Note] InnoDB: Creating tablespace and datafile system tables.
180118 15:56:18 [ERROR] InnoDB: Creation of SYS_TABLESPACES and SYS_DATAFILES has failed with error 18. Tablespace is full. Dropping incompletely created tables.
2018-01-18 15:56:18 7f2b6be5f8c0 InnoDB: Assertion failure in thread 139824470554816 in file dict0crea.cc line 1899
InnoDB: Failing assertion: err == DB_OUT_OF_FILE_SPACE || err == DB_TOO_MANY_CONCURRENT_TRXS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
180118 15:56:18 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.0.33-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467018 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xbda88e]
/usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x73fd26]
/lib64/libpthread.so.0(+0xf5e0)[0x7f2b6ba455e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f2b6a1781f7]
/lib64/libc.so.6(abort+0x148)[0x7f2b6a1798e8]
/usr/sbin/mysqld[0xae2ac8]
/usr/sbin/mysqld[0xa5fb6a]
/usr/sbin/mysqld[0x993ffc]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5e)[0x741c3e]
/usr/sbin/mysqld[0x5e5cf6]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x494)[0x5e6f64]
/usr/sbin/mysqld[0x53a92c]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x50f)[0x54063f]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f2b6a164c05]
/usr/sbin/mysqld[0x535979]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
180118 15:56:19 mysqld_safe mysqld from pid file /var/lib/mysql/t4w3.xentio.lan.pid ended