[MDEV-27060] mariadb_upgrade runs every startup Created: 2021-11-16  Updated: 2023-07-23  Resolved: 2021-11-18

Status: Closed
Project: MariaDB Server
Component/s: Scripts & Clients
Affects Version/s: 10.5.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Thomas Gruenert Assignee: Daniel Black
Resolution: Not a Bug Votes: 1
Labels: None
Environment:

Linux docker-compose

Attachments: Text File 1_startupAfterCrash.log     Text File 2_startupNormal.log    
Issue Links:
Relates
relates to MDEV-30499 mariadb in-server upgrade Open

 Description   

Documentation says:

Documentation

mysql_upgrade also saves the MariaDB version number in a file named mysql_upgrade_info in the data directory. This is used to quickly check whether all tables have been checked for this release so that table-checking can be skipped. For this reason, mysql_upgrade needs to be run as a user with write access to the data directory.

Problem
But mariadb_upgrade does check all tables every startup. This is a problem with many databases and tables on a not so fast storage.

How to test
Test has be done with bitnami docker container. It calls mariadb_upgrade every startup. According to documentation this should be ok.

File "mysql_upgrade_info" is written and get correct timestamp and content. But this information seems to be ignored by the tool.

Expected behavior
mysql_upgrade should work like it is documented.

Why critical?
mariadb_upgrade take 2h for our installation. A container can restart for various reasons. And this means a relevant production downtime for us.

—

test setup via docker-compose.yaml

version: '2.0'
 
 
 
services:
 
  mariadb:
 
    image: docker.io/bitnami/mariadb:10.5.12
 
    ports:
 
      - '3306:3306'
 
    volumes:
 
      - '/root/fixMariadbUpgrade:/bitnami/mariadb'
 
      - /root/fixMariadbUpgrade/my_custom.cnf:/opt/bitnami/mariadb/conf/my_custom.cnf:ro
 
    environment:
 
      - MARIADB_ROOT_PASSWORD=****
 
      - BITNAMI_DEBUG=true

—
my_custom.cnf:

[mysqld]
 
innodb_page_size=64k
 
skip-name-resolve
 
explicit_defaults_for_timestamp
 
basedir=/opt/bitnami/mariadb
 
plugin_dir=/opt/bitnami/mariadb/plugin
 
port=3306
 
socket=/opt/bitnami/mariadb/tmp/mysql.sock
 
tmpdir=/opt/bitnami/mariadb/tmp
 
max_allowed_packet=16M
 
bind-address=0.0.0.0
 
pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid
 
log-error=/opt/bitnami/mariadb/logs/mysqld.log
 
character-set-server=utf8mb4
 
collation-server=utf8mb4_unicode_ci
 
max_connections = 1186
 
[client]
 
port=3306
 
socket=/opt/bitnami/mariadb/tmp/mysql.sock
 
default-character-set=utf8mb4
 
plugin_dir=/opt/bitnami/mariadb/plugin
 
[manager]
 
port=3306
 
socket=/opt/bitnami/mariadb/tmp/mysql.sock
 
pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid



 Comments   
Comment by Daniel Black [ 2021-11-17 ]

tgruenert thanks for the bug report.

The continued running of mysql_upgrade on every restart is odd. I don't have an explanation looking at version in the observations below:

from inside docker.io/bitnami/mariadb:10.5.12 container

 
1001@18331cf58781:/bitnami/mariadb/data$ more mysql_upgrade_info 
10.5.12-MariaDB
 
 
 
1001@18331cf58781:/bitnami/mariadb/data$ mysql_upgrade --version
 
mysql_upgrade  Ver 1.4 Distrib 10.5.12-MariaDB, for Linux (x86_64)
 
 
 
1001@18331cf58781:/bitnami/mariadb/data$ mysql_upgrade          
This installation of MariaDB is already upgraded to 10.5.12-MariaDB, use --force if you still need to run mysql_upgrade

Do you have the same?

I did a little digging into how bitnami script their mysql_upgrade. This is part of the mysql_initialize (in the setup.sh script. This is completed before run.sh script is run an you have a functioning system again.

The time consuming part it seems from you, (the container logs would confirm this), is the checking of all tables. Adding the following group/config option to your configuration for mariadb-upgrade would save the most time.

mariadb-upgrade restrict to system tables

 
 
 
[mariadb-upgrade]
 
upgrade-system-tables

If you wish to do a full check, even while system is operation, you can mariadb-upgrade --skip-upgrade-system-tables or mariadb-check --check-upgrade --auto-repair --all-databases --skip-database=mysql

Obviously I can't change bitnami scripts, but there's a work around for you above, and I'll learn from them for docker library's mysql_upgrade when implemented.

I need more info as to why the version check on mysql_upgrade isn't working correctly.

Comment by Thomas Gruenert [ 2021-11-17 ]

The workaround rocks for us and eliminate a real huge pain. Thank you Daniel!

I´ve give the ticket an update with two uploaded log files from our small test system. This may help.

And i´ve checked out the steps inside bitnami container as you show and indeed this is working exactly the same way on my side. What more I can do to help you?

Comment by Daniel Black [ 2021-11-18 ]

So "Not a bug" is really "Not our bug"

The bitnami scripts use --force in mysql_upgrade https://github.com/bitnami/bitnami-docker-mariadb/blob/master/10.5/debian-10/rootfs/opt/bitnami/scripts/libmariadb.sh#L735 which bypasses the version check. This is all the force does do.

I can't see a reason for it. Recommend bugging bitnami to remove it.

Adding force=0 to the config file will have no effect as the bitnami use puts force as an arguement which takes precedence.

Comment by Thomas Gruenert [ 2021-11-18 ]

To keep anybody updated about the issue please follow now https://github.com/bitnami/bitnami-docker-mariadb/issues/252

Comment by Daniel Black [ 2023-01-30 ]

And regression - 22193

Comment by Daniel Black [ 2023-04-28 ]

And regression fixed:

bitnami mariadb release notes, and mariadb-galera release notes

  • 10.3.38-debian-11-r9
  • 10.4.28-debian-11-r9
  • 10.5.19-debian-11-r9
  • 10.6.12-debian-11-r8
  • 10.8.7-debian-11-r9
  • 10.9.5-debian-11-r9
  • 10.10.3-debian-11-r10
  • 10.11.2-debian-11-r3
Generated at Thu Feb 08 09:50:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.