[MDEV-31630] Dumping and reimporting dump file since 10.6 upgrade is 5x slower than 10.1 Created: 2023-07-05  Updated: 2023-07-06

Status: Open
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.6.14
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Jason Pullen Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

CentOS 7
50Gb database.



 Description   

Dumping and reimporting dump file since 10.6 upgrade is 5x slower than 10.1.
Please help resolve this as this slowness is rendering MariaDB unusable in our setup.

Previous dump time 25-30 mins.
Current dump time 2-3 hours.



 Comments   
Comment by Marko Mäkelä [ 2023-07-05 ]

Which exact MariaDB Server version is this? Can you attach the configuration file? Could this be explained by MDEV-24854?

Comment by Jason Pullen [ 2023-07-05 ]

Version 10.6.14

my.cnf

[mysqld]

'## General
datadir = /var/lib/mysql/data
tmpdir = /var/lib/mysql/tmp
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
skip-name-resolve
sql-mode = NO_ENGINE_SUBSTITUTION

optimizer_search_depth = 0
'## Cache
thread-cache-size = 16
table-open-cache = 4096
table-definition-cache = 2048
query-cache-size = 32M
query-cache-limit = 1M

'## Per-thread Buffers
sort-buffer-size = 2M
read-buffer-size = 1M
read-rnd-buffer-size = 1M
join-buffer-size = 1M

'## Temp Tables
tmp-table-size = 2G
max-heap-table-size = 2G

'## Networking
back-log = 100
max-connections = 300
max-connect-errors = 10000
max-allowed-packet = 1024M
interactive-timeout = 3600
wait-timeout = 600

'### Storage Engines
default-storage-engine = InnoDB
enforce-storage-engine = InnoDB

'## MyISAM
'## No longer used
'# reduce as now using aria instead of myisam
key-buffer-size = 16M
'# reduce as now using aria instead of myisam
myisam-sort-buffer-size = 8M

'## Aria
aria_pagecache_buffer_size = 128M
aria_sort_buffer_size = 256M

'## InnoDB
innodb-buffer-pool-size = 16G
innodb-log-file-size = 1024M
innodb-file-per-table = 1
innodb-tmpdir = /var/lib/mysql/tmp
innodb-large-prefix = 1
innodb_ft_min_token_size = 2
innodb-default-row-format = DYNAMIC

server-id = 1
log_bin = /var/log/mysql/binlog/bin-log
replicate-do-db = live_internal_1
replicate-do-db = live_internal_2
replicate-do-db = live_internal_3
replicate-do-db = live_internal_4
replicate-do-db = live_internal_5
replicate-do-db = live_internal_6
replicate-do-db = live_internal_7

'# log_bin = /var/log/mysql/bin-log # turned on during first schema configuration
binlog-format = MIXED

'## Logging
log-output = FILE
'#general-log
general-log-file = /var/log/mysql/mysqld.log
log-error = /var/log/mysql/mysql_error.log

'# turn on for slow logging
slow-query-log = 1
'#slow-query-log-file = /var/log/mysql/mysql_slow_query.log
slow-query-log-file = /var/lib/mysql/data/test20-mysql_slow_query.log

'#log-slow-slave-statements
long-query-time = 2

lower_case_table_names = 1
'# stop word list
'#ft_stopword_file=/etc/stop_list.txt
'# allow 2-character word indexing
ft_min_word_len=2

'# set collations
collation-server = utf8mb4_unicode_ci
init-connect='SET NAMES utf8mb4'
character-set-server = utf8mb4

[mysqld_safe]
log-error = /var/log/mysql/mysqld_safe_error.log
open-files-limit = 65535

[mysql]
no-auto-rehash

'## Characters/Encoding
default-character-set = utf8mb4

[client]
default-character-set = utf8mb4

Comment by Jason Pullen [ 2023-07-05 ]

Sorry had to add ' before '## to get it to render correctly in your Jira

Comment by Daniel Black [ 2023-07-05 ]

Possibly also caused by MDEV-29967 (fix coming).

Due you have a spare replica/spare hardware to test if this resolves your issue?

The last 10.6 build including the MDEV-29967 fix is available with this repo - https://ci.mariadb.org/36428/amd64-centos-7-rpm-autobake/MariaDB.repo

Comment by Daniel Black [ 2023-07-06 ]

If you use the --verbose mysqldump option does this lead to a particular backup stage that is slower?

Comment by Jason Pullen [ 2023-07-06 ]

I see from many forums this issue is being reported by many users, is the slow dump time due to all the new validation? if so is there a way to switch this off during the dump/import processes?

Further, have you reviewed the supplied my.ini settings? is there anything I should change/include when using a 50Gb database?

Comment by Daniel Black [ 2023-07-06 ]

The validation of the slow dumps is pretty much there. MDEV-29967 was an unfortunate regression that broke an optimization that helped a significantly, without a control switch that can correct the regression.

Validating that its fixed in the next version properly ahead of release. If its the case that its another cause entirely, after finding out sufficient information to determine case, it will be another 3 months before the next release.

A lot of your config looks like it came as a Debian default rather than implemented with a good reason so I'd consider omitting in stages:

  • query_cache
  • General (paths), maybe keep socket if used (and not default, check mariadbd --help --verbose
  • mysqld_safe probably not using mysqld_safe right? - omit
  • Per-thread Buffers / Temp Tables - omit

Anything else depends on workload, suggest https://dba.stackexchange.com/unanswered/tagged/mariadb with SHOW GLOBAL STATUS for a community assessment.

Generated at Thu Feb 08 10:25:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.