[MDEV-27068] running mariadb-upgrade on mariadb server with other user make it hangs forever Created: 2021-11-17  Updated: 2023-07-23  Resolved: 2022-01-24

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.6.5
Fix Version/s: 10.2.42, 10.3.33, 10.4.23, 10.5.14, 10.6.6, 10.7.2

Type: Bug Priority: Major
Reporter: Marc Gariépy Assignee: Michael Widenius
Resolution: Fixed Votes: 1
Labels: None
Environment:

Ubuntu focal 20.04

Attachments: File debug-MDEV-27068.tar.gz     File mariadb-issue-osa.tar.gz    
Issue Links:
Issue split
split to MDEV-27606 change debian packaging to not run ma... Open
PartOf
includes MDEV-27279 mariadb_upgrade add --check-if-upgrad... Closed
includes MDEV-27613 Fixing debian to only run the full my... Stalled
Problem/Incident
causes MDEV-27789 mysql_upgrade / mariadb-upgrade in 10... Closed
Relates
relates to MDEV-27167 system variable deferred-networking t... Closed
relates to MDEV-14622 Running mysql_upgrade in background u... Closed
relates to MDEV-30499 mariadb in-server upgrade Open
Sub-Tasks:
Key
 Summary
 Type
 Status
 Assignee
MDEV-27107 prevent two mariadb-upgrade running i... Technical task Closed Michael Widenius  
MDEV-27279 mariadb_upgrade add --check-if-upgrad... Technical task Closed Michael Widenius  

 Description   

When deploying mariadb via openstack-ansible in the CI we get timeout sometimes.

i did ran the test on a private cloud and i can reproduce the issue most time i run the deployment tooling.

here is the issue we saw in the CI.
https://zuul.opendev.org/t/openstack/build/fed013733f9f415bb2d1118a26287bab/log/job-output.txt#10054

I'm available to do some more testing and provide trace and other info is needed, you can reach me on IRC.

Thanks

 Comments   
Comment by Michael Widenius [ 2021-11-18 ]

In what contex do you get timeouts?

I checked your Attachement and there is a core. However to be able to fully use the core I would need to also have the corresponding mysqld or mariadbd binary.

The journal includes a trace of a crash:
Nov 16 20:18:10 aio1-galera-container-b6377aeb /etc/mysql/debian-start[3862]: sys.schema_redundant_indexes OK
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: 211116 20:20:17 [ERROR] mysqld got signal 6 ;
....
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: stack_bottom = 0x0 thread_stack 0x49000
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: Printing to addr2line failed
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /usr/sbin/mariadbd(my_print_stacktrace+0x32)[0x55e28d2e3442]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55e28cd966a5]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f8ffd0173c0]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /lib/x86_64-linux-gnu/libc.so.6(__poll+0x4f)[0x7f8ffcbecaff]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /usr/sbin/mariadbd(_Z26handle_connections_socketsv+0x1da)[0x55e28ca942ba]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x891)[0x55e28ca94df1]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f8ffcafe0b3]
Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd[3831]: /usr/sbin/mariadbd(_start+0x2e)[0x55e28ca8979e]

The crash comes from a 'signal 6', SIGABRT that is probably sent out from the outside.
There is 2 minutes of delay between the last log entry and the ABORT.
This indicates that a program or user killed the server with hard because of some unknown reason.

The server was running the following during the crash:
Running 'mysqlcheck' with connection arguments: --port='3306' --socket='/var/run/mysqld/mysqld.sock' --host='localhost' --socket='/var/run/mysqld/mysqld.sock' --host='localhost' --socket='/var/run/mysqld/mysqld.sock'

It was processing sys.views when the server was killed. There was a strange timeout of 2 minutes in
the output from mysqlcheck, but that could be a caching issue.

If the databases are huge, then the above command can take a long time. The 2 minute timeout is probably not enough to finish the above command.

Suggested fix: Increase the system timeout when running the mysql --upgrade process.

There is of course a small chance that there is a bug when checking sys schema views that causes a deadlock. It would be great if you could somehow verify if this is the case by running mysql_upgrade separately on a server and check if you get any issues.

Comment by Marc Gariépy [ 2021-11-18 ]

Hello,

The timeout happens on a greenfield deployment in our CI.

Here is a test result with the issue:
https://zuul.opendev.org/t/openstack/build/0afa668757224613b70604023c38b802

The kill was generated by me with a `kill -6`.

The process is still hung from 2 days ago.

the database is empty at this point as it's tooling that does the initial install of mariadb.

Comment by Marc Gariépy [ 2021-11-18 ]

After some debugging i found out that the issue was that on first start the mysql_upgrade is run and we did ran it another time in parallel in the playbook before it was done.

If we remove the call for mysql_upgrade that was on our playbook everything works as expected on debian based system.

On centos8 it's not done by the system on installation and i haven't seen it in the installation document either.

Comment by Marc Gariépy [ 2021-11-18 ]

Apparently i did talked a bit too fast. it seems that the issue do still happen.

here is the trace with GDB

  • etc-mysql.tar.gz > tar of /etc/mysql/
  • mysqldb.tar.gz > tar of /var/lib/mysql
  • mysqld_full_bt_all_threads.txt > gdb trace of the running process
  • pkg-list-apt > dpkg -l on the systems
  • process > ps auxf of the running processes
  • test.log > journalctl of the mariadb service

debug-MDEV-27068.tar.gz

Comment by Marc Gariépy [ 2021-11-19 ]

MariaDB [(none)]> show processlist
-> ;
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Id User Host db Command Time State Info Progress

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2 system user   NULL Sleep 75998 closing tables NULL 0.000
1 system user   NULL Sleep 75998 wsrep aborter idle NULL 0.000
6 system user   NULL Sleep 75997 wsrep applier idle NULL 0.000
17 root localhost mysql Query 75992 Waiting for table metadata lock alter table proxies_priv modify Grantor varchar(384) COLLATE utf8_bin not null default '' 0.000
21 root localhost NULL Query 75992 Waiting for table level lock CREATE USER 'monitoring'@'localhost' 0.000
31 root localhost NULL Query 0 starting show processlist 0.000

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6 rows in set (0.001 sec)

Comment by Michael Widenius [ 2021-11-19 ]

From discussion on Libera.Chat:

The main issues are on Debian installations, where this problem happens:

  • Debian system always runs mysql_upgrade/maria_upgrade for every
    MariaDB server service start.
  • If maria_upgrade has to be run, as in case of a major upgrade, then
    check and possible fix of tables can take from a few seconds to
    20-30 seconds (normally) but potentially much longer if there is was
    a chance in a character collation.
  • If during this time another maria_upgrade is run then it is very likely
    that maria_upgrade will hang because of unexpected conflicts. This
    is because maria_upgrade was not designed to be run in parallel.
  • If anyone else connects to mysqld/mariadbd during maria_upgrade, bad
    things can happen as the tables are not up to date for the system.
    The possible problems are hangs or wrong data from selects or errors from
    table updates (as the indexes may not be up to date).

Suggested solution:

  • Add an option to maria_upgrade '--check-if-upgrade-is-needed' that
    returns 0 or 1 depending on maria_upgrade should be run.
    This is trivial as there is already checks in the code if maria_upgrade
    needs to be run.
  • In Debian system files add the following logic to the place where
    maria_upgrade is now run. If needed move the code to be just before
    the MariaDB server is started.
  • run maria_upgrade --check-if-upgrade-is-needed
  • If yes
  • Start MariadB server with --skip-grants --socket=/tmp/mariadb-upgrade.sock --skip-networking
  • run maria_upgrade --socket=/tmp/mariadb-upgrade.sock
  • stop the MariaDB server

The above should ensure that no one can connect to the mariadbd server while
maria_upgrade is running. If anyone tries to run the above script twice
it will also block as the second MariaDB server will not start until the
first has completed (innodb file locks takes care of this).

Comment by Sergei Golubchik [ 2021-11-19 ]

See the linked issue. Users will complain about long-running mariadb-upgrade that keeps the server unaccessible.

It's easy to prevent two mariadb-upgrade running in parallel, if it'll connect to the server and issue something like

SELECT GET_LOCK("mariadb-upgrade");

As a bonus, it can use the connection to avoid exec'ing mariadb command-line client.

An even simpler approach would be to get an advisory lock on mysql_upgrade_info file. Or on datadir (solves cases when the file doesn't exist yet).

In either approach one can run mariadb-upgrade and when it finishes everything will be upgraded. No matter if there were concurrent upgrades or not.

Comment by Sergei Golubchik [ 2021-11-19 ]

A second issue here is that having mariadb-upgrade running in the background when many packages are being upgraded is race condition prone. It'd be better for mariadb-server postinstall script to upgrade system tables (but not user tables — see linked issue).

This can be done by running mariadb-upgrade --upgrade-system-tables from the postinst script (and because of locking above it'll be safe).

But the problem is that it'll write the info file, so a full mariadb-upgrade won't run after it. Or, if the full background mariadb-upgrade will start first, the fast version will wait till the very end.

Comment by Michael Widenius [ 2021-11-20 ]

One cannot run two maria_upgrade in parallel. This results on the server getting locked.
One may also gets lock with just one maria_upgrade running and users having access to the server at the same time (According to not verified reports at Libera.chat)
Also, there is no point whatsoever in having users access any tables when mysql_upgrade is running as this can lead to corruptions if index collations have been changed.

As this is only happening when the server is getting upgraded, which means the server was taken down anyway, it should not matter much that there is a required check of tables when the server starts. The check will only take a notable time if there is a lot of tables (10,000+) or if there is a lot of tables that need to update the table because of changed collations, in which case one should not allow anyone use of the tables anyway.

I think that if one has 10,000+ tables, one will schedule a proper downtime for any upgrade anyways, in which case that time for mysql_upgrade should be neglectable.

Another possible fix is to have maria_upgrade by default not doing anything if this is a minor upgrade. This should solve the things for most users as the longer upgrade run would only be done when they upgrade their system (in which case any mysql_upgrade time should be neglectable)

Comment by Faustin Lammler [ 2021-11-22 ]

> On centos8 it's not done by the system on installation and i haven't seen it in the installation document either.

Yes this is correct, a manual action is required (https://github.com/MariaDB/server/blob/10.6/support-files/rpm/server-prein.sh#L41-L61)

Whereas on Debian, mysql_upgrade is called (by systemd or sysV init) on every (re)start of the server and the problem is well explained in MDEV-14622, see also https://github.com/MariaDB/server/blob/10.6/debian/additions/debian-start. My understanding is that MDEV-14622 only resolved the spider problem, it does not fix the fact that the server should not be accessed while mysql_upgrade is run and having it run in the background is not an acceptable solution.

> Another possible fix is to have maria_upgrade by default not doing anything if this is a minor upgrade. This should solve the things for most users as the longer upgrade run would only be done when they upgrade their system (in which case any mysql_upgrade time should be neglectable)

This is true for users relying on official Debian/Ubuntu repositories as a major MariaDB upgrade means a dist-upgrade (downtime is expected). For users relying on MDBF repository this can be problematic as they may not be upgrading the system but only MariaDB and do not expect downtime.

I see two different issue raised by the actual MDEV:
1. mariadb_upgrade can't be run in parallel, this should be fixed directly in the program, as discussed with danblack, he will look at it.
2. we need to change the way we handle mysql_upgrade on the packaging level in Debian/Ubuntu, I'll take care of that.

For 2, I do not have a solution to propose so far.
Here are some ideas based on previous discussions and what my understanding of the problem is:

  • do the same as for RPM (I am not sure that it's desirable but it's a clean solution so it should be listed);
  • do the same as for RPM if we detect a lots of tables (is 10,000+ the good threshold?, only run --upgrade-system-tables?);
  • run mysql_upgrade at post-install, but how to block access to the server until mysql_upgrade finishes (I don't see how `--skip-grants --socket=/tmp/mariadb-upgrade.sock --skip-networking` can be implemented in a systemd/sysv init context)?

Finally, from MDEV-14622
> it should be the server itself that checks at startup that its internal data structure (the "mysql" database) is up-to-date then corrects/update it before accepting connections

This looks like a good feature and I am wondering if we should create a MDEV so it has a chance to be implemented in future release?

Comment by Elena Stepanova [ 2021-11-22 ]

I don't know whether it's better to split (2) into a separate issue or handle within this one, but I hope that the final solution won't be downgraded to dealing with (1) only, like MDEV-14622 was previously downgraded to patching Spider, which of course didn't solve anything other than one of numerous side-effects of the root cause issue.

Similarly, fixing "running mysql_upgrade in parallel" won't solve two general problems on Debian, both of which still exist, as constant sporadic failures in our upgrade tests show:

1) Debian runs mysql_upgrade in background, and anything is possible during this time. For example, currently upgrade tests on 10.6 fail for us every day when they involve Columnstore installation: server gets upgraded, mysql_upgrade kicks in, then Columnstore starts getting upgraded, clashes with mysql_upgrade, everything hangs. There is no workaround for it: the user doesn't trigger this parallelism and cannot do anything to prevent it. While an extra package can be installed separately, it cannot be upgraded separately (but please, don't solve it by just hacking Columnstore, as it was done in MDEV-14622 for Spider. Columnstore here is just one more example, it will keep happening with various packages until the problem is solved).

2) Debian upgrade runs mysql_upgrade non-deterministically. It was already described in MDEV-14622 and it is still the case. Below is a proof-log of 10.5=>10.6 upgrade (official releases) I performed today on Debian 9. As you see after the package upgrade ends, sys schema wasn't created, general log is empty, which indicates mysql_upgrade wasn't even started. I have to run it manually to finalize the upgrade. When I did exactly the same on, say, Ubuntu 20.04, sys schema was there and the general log was full of mysql_upgrade stuff. I can't say whether the same happens every time on a certain debian/ubuntu flavor or it is sporadic, but it is definitely possible to end up with not upgraded schema.

debian-9-amd64-clean-lxq1k0:~$ wget https://dlm.mariadb.com/3/mariadb/mariadb_repo_setup
 
--2021-11-22 13:12:46--  https://dlm.mariadb.com/3/mariadb/mariadb_repo_setup
 
Resolving dlm.mariadb.com (dlm.mariadb.com)... 104.18.135.24, 104.17.191.14, 2606:4700::6811:bf0e, ...
 
Connecting to dlm.mariadb.com (dlm.mariadb.com)|104.18.135.24|:443... connected.
 
HTTP request sent, awaiting response... 302 Found
 
Location: https://storage.googleapis.com/downloads-cdn.mariadb.com/aptyum/1.0/1.0.0/MariaDB/mariadb_repo_setup?GoogleAccessId=downloads-sync%40downloads-234321.iam.gserviceaccount.com&Expires=1637587366&Signature=gaS93eVKCVNVvoLQPE3xvjNyGLhIe4h8BulDeLOGKdjU4JnbHfJN530hWBe8BgjMpDeEGSwAo8RRYEYm%2BoXeFhqDwS8giA0u0lbHN0wPeJVr%2BUROBP5zPAYhwGJsUq%2BI3SG5lnT8e7kb3ubATO3sIiWrZwUyEfWNwCbLi1x2b1iDCZj7IqREEI1Eei6Wg3qcm8qpENCBt0oZ90TV5rGdRsd%2BQxSivOUE8sRSMk1KS7JV8HlhiZ%2B5IVOp3ISJm%2FJY4EkH%2B6MTt%2FsaN3ber%2Bey7%2BOOFuIssAghEsgMowg7qOk03DUgSEadxljt98o7cPIbh%2F6ctd6qW7OeiqSIGFAi0w%3D%3D [following]
 
--2021-11-22 13:12:46--  https://storage.googleapis.com/downloads-cdn.mariadb.com/aptyum/1.0/1.0.0/MariaDB/mariadb_repo_setup?GoogleAccessId=downloads-sync%40downloads-234321.iam.gserviceaccount.com&Expires=1637587366&Signature=gaS93eVKCVNVvoLQPE3xvjNyGLhIe4h8BulDeLOGKdjU4JnbHfJN530hWBe8BgjMpDeEGSwAo8RRYEYm%2BoXeFhqDwS8giA0u0lbHN0wPeJVr%2BUROBP5zPAYhwGJsUq%2BI3SG5lnT8e7kb3ubATO3sIiWrZwUyEfWNwCbLi1x2b1iDCZj7IqREEI1Eei6Wg3qcm8qpENCBt0oZ90TV5rGdRsd%2BQxSivOUE8sRSMk1KS7JV8HlhiZ%2B5IVOp3ISJm%2FJY4EkH%2B6MTt%2FsaN3ber%2Bey7%2BOOFuIssAghEsgMowg7qOk03DUgSEadxljt98o7cPIbh%2F6ctd6qW7OeiqSIGFAi0w%3D%3D
 
Resolving storage.googleapis.com (storage.googleapis.com)... 108.177.121.128, 142.250.103.128, 108.177.120.128, ...
 
Connecting to storage.googleapis.com (storage.googleapis.com)|108.177.121.128|:443... connected.
 
HTTP request sent, awaiting response... 200 OK
 
Length: 26626 (26K) [application/octet-stream]
 
Saving to: ‘mariadb_repo_setup’
 
 
 
mariadb_repo_setup                                        100%[=====================================================================================================================================>]  26.00K  --.-KB/s    in 0s      
 
 
2021-11-22 13:12:47 (99.6 MB/s) - ‘mariadb_repo_setup’ saved [26626/26626]
 
 
 
debian-9-amd64-clean-lxq1k0:~$ chmod +x mariadb_repo_setup 
debian-9-amd64-clean-lxq1k0:~$ sudo ./mariadb_repo_setup --mariadb-server-version=10.5 --skip-maxscale --skip-tools
 
[info] Checking for script prerequisites.
 
[info] Repository file successfully written to /etc/apt/sources.list.d/mariadb.list
 
[info] Adding trusted package signing keys...
 
[info] Running apt-get update...
 
[info] Done adding trusted package signing keys
 
debian-9-amd64-clean-lxq1k0:~$ sudo apt-get update
 
Hit:1 http://security.debian.org stretch/updates InRelease
 
Ign:2 http://deb.debian.org/debian stretch InRelease
 
Hit:3 http://deb.debian.org/debian stretch-updates InRelease             
Hit:4 http://deb.debian.org/debian stretch-backports InRelease           
Hit:5 http://deb.debian.org/debian stretch Release 
Hit:6 http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease
 
Get:7 http://packages.cloud.google.com/apt google-compute-engine-stretch-stable InRelease [5,525 B]
 
Hit:8 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-stretch InRelease
 
Hit:10 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch InRelease
 
Fetched 5,525 B in 0s (9,201 B/s)
 
Reading package lists... Done
 
debian-9-amd64-clean-lxq1k0:~$ sudo apt-get install -y mariadb-server mariadb-client mariadb-backup mariadb-plugin-rocksdb
 
Reading package lists... Done
 
Building dependency tree       
Reading state information... Done
 
The following additional packages will be installed:
 
  galera-4 gawk libaio1 libcgi-fast-perl libcgi-pm-perl libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl
 
  libio-html-perl liblwp-mediatypes-perl libmariadb3 libmariadbclient18 libmpfr4 libpcre2-8-0 libperl5.24 libreadline5 libsigsegv2 libsnappy1v5 libterm-readkey-perl libtimedate-perl liburi-perl lsof mariadb-client-10.5
 
  mariadb-client-core-10.5 mariadb-common mariadb-server-10.5 mariadb-server-core-10.5 mysql-common perl perl-modules-5.24 psmisc python3-mysqldb rename rsync socat
 
Suggested packages:
 
  gawk-doc libclone-perl libmldbm-perl libnet-daemon-perl libsql-statement-perl libdata-dump-perl libipc-sharedcache-perl libwww-perl mailx mariadb-test netcat-openbsd perl-doc libterm-readline-gnu-perl
 
  | libterm-readline-perl-perl make python-egenix-mxdatetime python3-mysqldb-dbg
 
The following NEW packages will be installed:
 
  galera-4 gawk libaio1 libcgi-fast-perl libcgi-pm-perl libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl
 
  libio-html-perl liblwp-mediatypes-perl libmariadb3 libmariadbclient18 libmpfr4 libpcre2-8-0 libperl5.24 libreadline5 libsigsegv2 libsnappy1v5 libterm-readkey-perl libtimedate-perl liburi-perl lsof mariadb-backup mariadb-client
 
  mariadb-client-10.5 mariadb-client-core-10.5 mariadb-common mariadb-plugin-rocksdb mariadb-server mariadb-server-10.5 mariadb-server-core-10.5 mysql-common perl perl-modules-5.24 psmisc python3-mysqldb rename rsync socat
 
0 upgraded, 45 newly installed, 0 to remove and 3 not upgraded.
 
Need to get 43.3 MB of archives.
 
After this operation, 289 MB of additional disk space will be used.
 
Get:1 http://deb.debian.org/debian stretch/main amd64 perl-modules-5.24 all 5.24.1-3+deb9u7 [2,723 kB]
 
Get:2 http://security.debian.org stretch/updates/main amd64 libdbi-perl amd64 1.636-1+deb9u1 [766 kB]
 
Get:3 http://deb.debian.org/debian stretch/main amd64 libperl5.24 amd64 5.24.1-3+deb9u7 [3,527 kB]
 
Get:4 http://deb.debian.org/debian stretch/main amd64 perl amd64 5.24.1-3+deb9u7 [218 kB]
 
Get:5 http://deb.debian.org/debian stretch/main amd64 libmpfr4 amd64 3.1.5-1 [556 kB]
 
Get:6 http://deb.debian.org/debian stretch/main amd64 libsigsegv2 amd64 2.10-5 [28.9 kB]
 
Get:7 http://deb.debian.org/debian stretch/main amd64 gawk amd64 1:4.1.4+dfsg-1 [571 kB]
 
Get:8 http://deb.debian.org/debian stretch/main amd64 lsof amd64 4.89+dfsg-0.1 [313 kB]
 
Get:9 http://deb.debian.org/debian stretch/main amd64 libreadline5 amd64 5.2+dfsg-3+b1 [119 kB]
 
Get:10 http://deb.debian.org/debian stretch/main amd64 libaio1 amd64 0.3.110-3 [9,412 B]
 
Get:11 http://deb.debian.org/debian stretch/main amd64 libpcre2-8-0 amd64 10.22-3 [182 kB]
 
Get:12 http://deb.debian.org/debian stretch/main amd64 psmisc amd64 22.21-2.1+b2 [123 kB]
 
Get:13 http://deb.debian.org/debian stretch/main amd64 rsync amd64 3.1.2-1+deb9u2 [393 kB]
 
Get:14 http://deb.debian.org/debian stretch/main amd64 socat amd64 1.7.3.1-2+deb9u1 [353 kB]
 
Get:15 http://deb.debian.org/debian stretch/main amd64 libhtml-tagset-perl all 3.20-3 [12.7 kB]
 
Get:16 http://deb.debian.org/debian stretch/main amd64 liburi-perl all 1.71-1 [88.6 kB]
 
Get:17 http://deb.debian.org/debian stretch/main amd64 libhtml-parser-perl amd64 3.72-3 [104 kB]
 
Get:18 http://deb.debian.org/debian stretch/main amd64 libcgi-pm-perl all 4.35-1 [222 kB]
 
Get:19 http://deb.debian.org/debian stretch/main amd64 libfcgi-perl amd64 0.78-2 [38.2 kB]
 
Get:20 http://deb.debian.org/debian stretch/main amd64 libcgi-fast-perl all 1:2.12-1 [11.2 kB]
 
Get:21 http://deb.debian.org/debian stretch/main amd64 libdbd-mysql-perl amd64 4.041-2 [114 kB]
 
Get:22 http://deb.debian.org/debian stretch/main amd64 libencode-locale-perl all 1.05-1 [13.7 kB]
 
Get:23 http://deb.debian.org/debian stretch/main amd64 libhtml-template-perl all 2.95-2 [67.1 kB]
 
Get:24 http://deb.debian.org/debian stretch/main amd64 libtimedate-perl all 2.3000-2+deb9u1 [37.9 kB]
 
Get:25 http://deb.debian.org/debian stretch/main amd64 libhttp-date-perl all 6.02-1 [10.7 kB]
 
Get:26 http://deb.debian.org/debian stretch/main amd64 libio-html-perl all 1.001-1 [17.6 kB]
 
Get:27 http://deb.debian.org/debian stretch/main amd64 liblwp-mediatypes-perl all 6.02-1 [22.1 kB]
 
Get:28 http://deb.debian.org/debian stretch/main amd64 libhttp-message-perl all 6.11-1 [75.9 kB]
 
Get:29 http://deb.debian.org/debian stretch/main amd64 libterm-readkey-perl amd64 2.37-1 [27.2 kB]
 
Get:30 http://deb.debian.org/debian stretch/main amd64 libsnappy1v5 amd64 1.1.3-3 [51.0 kB]
 
Get:31 http://deb.debian.org/debian stretch/main amd64 python3-mysqldb amd64 1.3.7-1.1 [48.9 kB]
 
Get:32 http://deb.debian.org/debian stretch/main amd64 rename all 0.20-4 [12.5 kB]
 
Get:33 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mysql-common all 1:10.5.13+maria~stretch [3,190 B]
 
Get:34 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-common all 1:10.5.13+maria~stretch [4,378 B]
 
Get:35 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 galera-4 amd64 26.4.9-stretch [8,984 kB]
 
Get:36 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 libmariadb3 amd64 1:10.5.13+maria~stretch [148 kB]                                                                                            
Get:37 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-client-core-10.5 amd64 1:10.5.13+maria~stretch [772 kB]                                                                               
Get:38 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-client-10.5 amd64 1:10.5.13+maria~stretch [1,541 kB]                                                                                  
Get:39 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-server-core-10.5 amd64 1:10.5.13+maria~stretch [6,908 kB]                                                                             
Get:40 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-server-10.5 amd64 1:10.5.13+maria~stretch [4,467 kB]                                                                                  
Get:41 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 libmariadbclient18 amd64 1:10.5.13+maria~stretch [3,178 B]                                                                                    
Get:42 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-backup amd64 1:10.5.13+maria~stretch [6,452 kB]                                                                                       
Get:43 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-client all 1:10.5.13+maria~stretch [3,016 B]                                                                                          
Get:44 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-plugin-rocksdb amd64 1:10.5.13+maria~stretch [3,110 kB]                                                                               
Get:45 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-server all 1:10.5.13+maria~stretch [3,144 B]                                                                                          
Fetched 43.3 MB in 42s (1,024 kB/s)                                                                                                                                                                                                    
Extracting templates from packages: 100%
 
Preconfiguring packages ...
 
Selecting previously unselected package perl-modules-5.24.
 
(Reading database ... 50132 files and directories currently installed.)
 
Preparing to unpack .../perl-modules-5.24_5.24.1-3+deb9u7_all.deb ...
 
Unpacking perl-modules-5.24 (5.24.1-3+deb9u7) ...
 
Selecting previously unselected package libperl5.24:amd64.
 
Preparing to unpack .../libperl5.24_5.24.1-3+deb9u7_amd64.deb ...
 
Unpacking libperl5.24:amd64 (5.24.1-3+deb9u7) ...
 
Selecting previously unselected package perl.
 
Preparing to unpack .../perl_5.24.1-3+deb9u7_amd64.deb ...
 
Unpacking perl (5.24.1-3+deb9u7) ...
 
Selecting previously unselected package libmpfr4:amd64.
 
Preparing to unpack .../libmpfr4_3.1.5-1_amd64.deb ...
 
Unpacking libmpfr4:amd64 (3.1.5-1) ...
 
Selecting previously unselected package libsigsegv2:amd64.
 
Preparing to unpack .../libsigsegv2_2.10-5_amd64.deb ...
 
Unpacking libsigsegv2:amd64 (2.10-5) ...
 
Setting up libmpfr4:amd64 (3.1.5-1) ...
 
Setting up libsigsegv2:amd64 (2.10-5) ...
 
Selecting previously unselected package gawk.
 
(Reading database ... 51970 files and directories currently installed.)
 
Preparing to unpack .../00-gawk_1%3a4.1.4+dfsg-1_amd64.deb ...
 
Unpacking gawk (1:4.1.4+dfsg-1) ...
 
Selecting previously unselected package mysql-common.
 
Preparing to unpack .../01-mysql-common_1%3a10.5.13+maria~stretch_all.deb ...
 
Unpacking mysql-common (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package mariadb-common.
 
Preparing to unpack .../02-mariadb-common_1%3a10.5.13+maria~stretch_all.deb ...
 
Unpacking mariadb-common (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package galera-4.
 
Preparing to unpack .../03-galera-4_26.4.9-stretch_amd64.deb ...
 
Unpacking galera-4 (26.4.9-stretch) ...
 
Selecting previously unselected package libdbi-perl.
 
Preparing to unpack .../04-libdbi-perl_1.636-1+deb9u1_amd64.deb ...
 
Unpacking libdbi-perl (1.636-1+deb9u1) ...
 
Selecting previously unselected package lsof.
 
Preparing to unpack .../05-lsof_4.89+dfsg-0.1_amd64.deb ...
 
Unpacking lsof (4.89+dfsg-0.1) ...
 
Selecting previously unselected package libmariadb3:amd64.
 
Preparing to unpack .../06-libmariadb3_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking libmariadb3:amd64 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package libreadline5:amd64.
 
Preparing to unpack .../07-libreadline5_5.2+dfsg-3+b1_amd64.deb ...
 
Unpacking libreadline5:amd64 (5.2+dfsg-3+b1) ...
 
Selecting previously unselected package mariadb-client-core-10.5.
 
Preparing to unpack .../08-mariadb-client-core-10.5_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking mariadb-client-core-10.5 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package mariadb-client-10.5.
 
Preparing to unpack .../09-mariadb-client-10.5_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking mariadb-client-10.5 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package libaio1:amd64.
 
Preparing to unpack .../10-libaio1_0.3.110-3_amd64.deb ...
 
Unpacking libaio1:amd64 (0.3.110-3) ...
 
Selecting previously unselected package libpcre2-8-0:amd64.
 
Preparing to unpack .../11-libpcre2-8-0_10.22-3_amd64.deb ...
 
Unpacking libpcre2-8-0:amd64 (10.22-3) ...
 
Selecting previously unselected package mariadb-server-core-10.5.
 
Preparing to unpack .../12-mariadb-server-core-10.5_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking mariadb-server-core-10.5 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package psmisc.
 
Preparing to unpack .../13-psmisc_22.21-2.1+b2_amd64.deb ...
 
Unpacking psmisc (22.21-2.1+b2) ...
 
Selecting previously unselected package rsync.
 
Preparing to unpack .../14-rsync_3.1.2-1+deb9u2_amd64.deb ...
 
Unpacking rsync (3.1.2-1+deb9u2) ...
 
Selecting previously unselected package socat.
 
Preparing to unpack .../15-socat_1.7.3.1-2+deb9u1_amd64.deb ...
 
Unpacking socat (1.7.3.1-2+deb9u1) ...
 
Setting up mysql-common (1:10.5.13+maria~stretch) ...
 
Setting up mariadb-common (1:10.5.13+maria~stretch) ...
 
update-alternatives: error: no alternatives for my.cnf
 
update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
 
Selecting previously unselected package mariadb-server-10.5.
 
(Reading database ... 52681 files and directories currently installed.)
 
Preparing to unpack .../00-mariadb-server-10.5_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking mariadb-server-10.5 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package libhtml-tagset-perl.
 
Preparing to unpack .../01-libhtml-tagset-perl_3.20-3_all.deb ...
 
Unpacking libhtml-tagset-perl (3.20-3) ...
 
Selecting previously unselected package liburi-perl.
 
Preparing to unpack .../02-liburi-perl_1.71-1_all.deb ...
 
Unpacking liburi-perl (1.71-1) ...
 
Selecting previously unselected package libhtml-parser-perl.
 
Preparing to unpack .../03-libhtml-parser-perl_3.72-3_amd64.deb ...
 
Unpacking libhtml-parser-perl (3.72-3) ...
 
Selecting previously unselected package libcgi-pm-perl.
 
Preparing to unpack .../04-libcgi-pm-perl_4.35-1_all.deb ...
 
Unpacking libcgi-pm-perl (4.35-1) ...
 
Selecting previously unselected package libfcgi-perl.
 
Preparing to unpack .../05-libfcgi-perl_0.78-2_amd64.deb ...
 
Unpacking libfcgi-perl (0.78-2) ...
 
Selecting previously unselected package libcgi-fast-perl.
 
Preparing to unpack .../06-libcgi-fast-perl_1%3a2.12-1_all.deb ...
 
Unpacking libcgi-fast-perl (1:2.12-1) ...
 
Selecting previously unselected package libmariadbclient18.
 
Preparing to unpack .../07-libmariadbclient18_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking libmariadbclient18 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package libdbd-mysql-perl.
 
Preparing to unpack .../08-libdbd-mysql-perl_4.041-2_amd64.deb ...
 
Unpacking libdbd-mysql-perl (4.041-2) ...
 
Selecting previously unselected package libencode-locale-perl.
 
Preparing to unpack .../09-libencode-locale-perl_1.05-1_all.deb ...
 
Unpacking libencode-locale-perl (1.05-1) ...
 
Selecting previously unselected package libhtml-template-perl.
 
Preparing to unpack .../10-libhtml-template-perl_2.95-2_all.deb ...
 
Unpacking libhtml-template-perl (2.95-2) ...
 
Selecting previously unselected package libtimedate-perl.
 
Preparing to unpack .../11-libtimedate-perl_2.3000-2+deb9u1_all.deb ...
 
Unpacking libtimedate-perl (2.3000-2+deb9u1) ...
 
Selecting previously unselected package libhttp-date-perl.
 
Preparing to unpack .../12-libhttp-date-perl_6.02-1_all.deb ...
 
Unpacking libhttp-date-perl (6.02-1) ...
 
Selecting previously unselected package libio-html-perl.
 
Preparing to unpack .../13-libio-html-perl_1.001-1_all.deb ...
 
Unpacking libio-html-perl (1.001-1) ...
 
Selecting previously unselected package liblwp-mediatypes-perl.
 
Preparing to unpack .../14-liblwp-mediatypes-perl_6.02-1_all.deb ...
 
Unpacking liblwp-mediatypes-perl (6.02-1) ...
 
Selecting previously unselected package libhttp-message-perl.
 
Preparing to unpack .../15-libhttp-message-perl_6.11-1_all.deb ...
 
Unpacking libhttp-message-perl (6.11-1) ...
 
Selecting previously unselected package libterm-readkey-perl.
 
Preparing to unpack .../16-libterm-readkey-perl_2.37-1_amd64.deb ...
 
Unpacking libterm-readkey-perl (2.37-1) ...
 
Selecting previously unselected package mariadb-backup.
 
Preparing to unpack .../17-mariadb-backup_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking mariadb-backup (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package mariadb-client.
 
Preparing to unpack .../18-mariadb-client_1%3a10.5.13+maria~stretch_all.deb ...
 
Unpacking mariadb-client (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package libsnappy1v5:amd64.
 
Preparing to unpack .../19-libsnappy1v5_1.1.3-3_amd64.deb ...
 
Unpacking libsnappy1v5:amd64 (1.1.3-3) ...
 
Selecting previously unselected package mariadb-plugin-rocksdb.
 
Preparing to unpack .../20-mariadb-plugin-rocksdb_1%3a10.5.13+maria~stretch_amd64.deb ...
 
Unpacking mariadb-plugin-rocksdb (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package mariadb-server.
 
Preparing to unpack .../21-mariadb-server_1%3a10.5.13+maria~stretch_all.deb ...
 
Unpacking mariadb-server (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package python3-mysqldb.
 
Preparing to unpack .../22-python3-mysqldb_1.3.7-1.1_amd64.deb ...
 
Unpacking python3-mysqldb (1.3.7-1.1) ...
 
Selecting previously unselected package rename.
 
Preparing to unpack .../23-rename_0.20-4_all.deb ...
 
Unpacking rename (0.20-4) ...
 
Setting up libmariadb3:amd64 (1:10.5.13+maria~stretch) ...
 
Setting up perl-modules-5.24 (5.24.1-3+deb9u7) ...
 
Setting up libperl5.24:amd64 (5.24.1-3+deb9u7) ...
 
Setting up galera-4 (26.4.9-stretch) ...
 
Setting up psmisc (22.21-2.1+b2) ...
 
Setting up socat (1.7.3.1-2+deb9u1) ...
 
Setting up lsof (4.89+dfsg-0.1) ...
 
Setting up libmariadbclient18 (1:10.5.13+maria~stretch) ...
 
Setting up gawk (1:4.1.4+dfsg-1) ...
 
Setting up rsync (3.1.2-1+deb9u2) ...
 
Created symlink /etc/systemd/system/multi-user.target.wants/rsync.service → /lib/systemd/system/rsync.service.
 
Setting up perl (5.24.1-3+deb9u7) ...
 
update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode
 
Setting up liblwp-mediatypes-perl (6.02-1) ...
 
Processing triggers for libc-bin (2.24-11+deb9u4) ...
 
Setting up libaio1:amd64 (0.3.110-3) ...
 
Setting up liburi-perl (1.71-1) ...
 
Setting up libsnappy1v5:amd64 (1.1.3-3) ...
 
Processing triggers for systemd (232-25+deb9u13) ...
 
Processing triggers for man-db (2.7.6.1-2) ...
 
Setting up libreadline5:amd64 (5.2+dfsg-3+b1) ...
 
Setting up libpcre2-8-0:amd64 (10.22-3) ...
 
Setting up libfcgi-perl (0.78-2) ...
 
Setting up libdbi-perl (1.636-1+deb9u1) ...
 
Setting up libhtml-tagset-perl (3.20-3) ...
 
Setting up mariadb-client-core-10.5 (1:10.5.13+maria~stretch) ...
 
Setting up libencode-locale-perl (1.05-1) ...
 
Setting up libtimedate-perl (2.3000-2+deb9u1) ...
 
Setting up rename (0.20-4) ...
 
update-alternatives: using /usr/bin/file-rename to provide /usr/bin/rename (rename) in auto mode
 
Setting up mariadb-client-10.5 (1:10.5.13+maria~stretch) ...
 
Setting up libio-html-perl (1.001-1) ...
 
Setting up mariadb-client (1:10.5.13+maria~stretch) ...
 
Setting up mariadb-server-core-10.5 (1:10.5.13+maria~stretch) ...
 
Setting up python3-mysqldb (1.3.7-1.1) ...
 
Setting up libterm-readkey-perl (2.37-1) ...
 
Setting up mariadb-backup (1:10.5.13+maria~stretch) ...
 
Setting up mariadb-server-10.5 (1:10.5.13+maria~stretch) ...
 
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service.
 
Setting up libhtml-parser-perl (3.72-3) ...
 
Setting up libcgi-pm-perl (4.35-1) ...
 
Setting up mariadb-server (1:10.5.13+maria~stretch) ...
 
Setting up libdbd-mysql-perl (4.041-2) ...
 
Setting up libhttp-date-perl (6.02-1) ...
 
Setting up libhtml-template-perl (2.95-2) ...
 
Setting up libcgi-fast-perl (1:2.12-1) ...
 
Setting up mariadb-plugin-rocksdb (1:10.5.13+maria~stretch) ...
 
Setting up libhttp-message-perl (6.11-1) ...
 
Processing triggers for libc-bin (2.24-11+deb9u4) ...
 
Processing triggers for systemd (232-25+deb9u13) ...
 
Processing triggers for mariadb-server-10.5 (1:10.5.13+maria~stretch) ...
 
debian-9-amd64-clean-lxq1k0:~$ sudo sh -c "echo '[mysqld]' >> /etc/mysql/mariadb.conf.d/general_log.cnf"
 
debian-9-amd64-clean-lxq1k0:~$ sudo sh -c "echo 'general_log=1' >> /etc/mysql/mariadb.conf.d/general_log.cnf"
 
debian-9-amd64-clean-lxq1k0:~$ sudo sh -c "echo 'general_log_file=/tmp/my.log' >> /etc/mysql/mariadb.conf.d/general_log.cnf"
 
debian-9-amd64-clean-lxq1k0:~$ sudo ./mariadb_repo_setup --mariadb-server-version=10.6 --skip-maxscale --skip-tools
 
[info] Checking for script prerequisites.
 
[warning] Found existing file at /etc/apt/sources.list.d/mariadb.list. Moving to /etc/apt/sources.list.d/mariadb.list.old_1
 
[info] Repository file successfully written to /etc/apt/sources.list.d/mariadb.list
 
[info] Adding trusted package signing keys...
 
[info] Running apt-get update...
 
[info] Done adding trusted package signing keys
 
debian-9-amd64-clean-lxq1k0:~$ sudo apt-get update
 
Hit:1 http://security.debian.org stretch/updates InRelease
 
Ign:2 http://deb.debian.org/debian stretch InRelease
 
Hit:3 http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease
 
Hit:4 http://deb.debian.org/debian stretch-updates InRelease             
Hit:5 http://deb.debian.org/debian stretch-backports InRelease           
Hit:6 http://deb.debian.org/debian stretch Release                       
Get:7 http://packages.cloud.google.com/apt google-compute-engine-stretch-stable InRelease [5,525 B]
 
Hit:8 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-stretch InRelease
 
Get:9 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch InRelease [4,642 B]
 
Fetched 10.2 kB in 0s (14.8 kB/s)
 
Reading package lists... Done
 
N: Ignoring file 'mariadb.list.old_1' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension
 
debian-9-amd64-clean-lxq1k0:~$ sudo apt-get install -y mariadb-server mariadb-client mariadb-backup mariadb-plugin-rocksdb
 
Reading package lists... Done
 
Building dependency tree       
Reading state information... Done
 
The following additional packages will be installed:
 
  mariadb-client-10.6 mariadb-client-core-10.6 mariadb-common mariadb-server-10.6 mariadb-server-core-10.6
 
Suggested packages:
 
  mailx mariadb-test netcat-openbsd
 
The following packages will be REMOVED:
 
  mariadb-client-10.5 mariadb-client-core-10.5 mariadb-server-10.5 mariadb-server-core-10.5
 
The following NEW packages will be installed:
 
  mariadb-client-10.6 mariadb-client-core-10.6 mariadb-server-10.6 mariadb-server-core-10.6
 
The following packages will be upgraded:
 
  mariadb-backup mariadb-client mariadb-common mariadb-plugin-rocksdb mariadb-server
 
5 upgraded, 4 newly installed, 4 to remove and 6 not upgraded.
 
Need to get 22.9 MB of archives.
 
After this operation, 1,434 kB of additional disk space will be used.
 
Get:1 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-common all 1:10.6.5+maria~stretch [3,976 B]
 
Get:2 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-client all 1:10.6.5+maria~stretch [2,622 B]
 
Get:3 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-server all 1:10.6.5+maria~stretch [2,748 B]
 
Get:4 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-plugin-rocksdb amd64 1:10.6.5+maria~stretch [3,113 kB]
 
Get:5 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-backup amd64 1:10.6.5+maria~stretch [6,407 kB]
 
Get:6 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-client-core-10.6 amd64 1:10.6.5+maria~stretch [819 kB]
 
Get:7 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-client-10.6 amd64 1:10.6.5+maria~stretch [2,314 kB]
 
Get:8 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-server-core-10.6 amd64 1:10.6.5+maria~stretch [6,928 kB]
 
Get:9 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-server-10.6 amd64 1:10.6.5+maria~stretch [3,338 kB]
 
Fetched 22.9 MB in 4s (4,772 kB/s)             
N: Ignoring file 'mariadb.list.old_1' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension
 
Preconfiguring packages ...
 
(Reading database ... 53200 files and directories currently installed.)
 
Preparing to unpack .../mariadb-common_1%3a10.6.5+maria~stretch_all.deb ...
 
Unpacking mariadb-common (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ...
 
Preparing to unpack .../mariadb-client_1%3a10.6.5+maria~stretch_all.deb ...
 
Unpacking mariadb-client (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ...
 
Preparing to unpack .../mariadb-server_1%3a10.6.5+maria~stretch_all.deb ...
 
Unpacking mariadb-server (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ...
 
Preparing to unpack .../mariadb-plugin-rocksdb_1%3a10.6.5+maria~stretch_amd64.deb ...
 
Unpacking mariadb-plugin-rocksdb (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ...
 
(Reading database ... 53200 files and directories currently installed.)
 
Removing mariadb-server-10.5 (1:10.5.13+maria~stretch) ...
 
Removing mariadb-server-core-10.5 (1:10.5.13+maria~stretch) ...
 
Removing mariadb-client-10.5 (1:10.5.13+maria~stretch) ...
 
(Reading database ... 52900 files and directories currently installed.)
 
Preparing to unpack .../mariadb-backup_1%3a10.6.5+maria~stretch_amd64.deb ...
 
Unpacking mariadb-backup (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ...
 
(Reading database ... 52900 files and directories currently installed.)
 
Removing mariadb-client-core-10.5 (1:10.5.13+maria~stretch) ...
 
Selecting previously unselected package mariadb-client-core-10.6.
 
(Reading database ... 52890 files and directories currently installed.)
 
Preparing to unpack .../mariadb-client-core-10.6_1%3a10.6.5+maria~stretch_amd64.deb ...
 
Unpacking mariadb-client-core-10.6 (1:10.6.5+maria~stretch) ...
 
Selecting previously unselected package mariadb-client-10.6.
 
Preparing to unpack .../mariadb-client-10.6_1%3a10.6.5+maria~stretch_amd64.deb ...
 
Unpacking mariadb-client-10.6 (1:10.6.5+maria~stretch) ...
 
Selecting previously unselected package mariadb-server-core-10.6.
 
Preparing to unpack .../mariadb-server-core-10.6_1%3a10.6.5+maria~stretch_amd64.deb ...
 
Unpacking mariadb-server-core-10.6 (1:10.6.5+maria~stretch) ...
 
Setting up mariadb-common (1:10.6.5+maria~stretch) ...
 
Selecting previously unselected package mariadb-server-10.6.
 
(Reading database ... 53106 files and directories currently installed.)
 
Preparing to unpack .../mariadb-server-10.6_1%3a10.6.5+maria~stretch_amd64.deb ...
 
/var/lib/mysql: found previous version 10.5
 
Unpacking mariadb-server-10.6 (1:10.6.5+maria~stretch) ...
 
Setting up mariadb-server-core-10.6 (1:10.6.5+maria~stretch) ...
 
Processing triggers for systemd (232-25+deb9u13) ...
 
Setting up mariadb-client-core-10.6 (1:10.6.5+maria~stretch) ...
 
Processing triggers for man-db (2.7.6.1-2) ...
 
Setting up mariadb-client-10.6 (1:10.6.5+maria~stretch) ...
 
Setting up mariadb-client (1:10.6.5+maria~stretch) ...
 
Setting up mariadb-server-10.6 (1:10.6.5+maria~stretch) ...
 
Installing new version of config file /etc/init.d/mariadb ...
 
Installing new version of config file /etc/mysql/mariadb.conf.d/50-server.cnf ...
 
Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.
 
mariadb-extra.socket is a disabled or a static unit, not starting it.
 
mariadb-extra.socket is a disabled or a static unit, not starting it.
 
Setting up mariadb-backup (1:10.6.5+maria~stretch) ...
 
Setting up mariadb-plugin-rocksdb (1:10.6.5+maria~stretch) ...
 
Setting up mariadb-server (1:10.6.5+maria~stretch) ...
 
Processing triggers for systemd (232-25+deb9u13) ...
 
N: Ignoring file 'mariadb.list.old_1' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension
 
debian-9-amd64-clean-lxq1k0:~$ sudo mysql -e "show databases"
 
+--------------------+
 
| Database           |
 
+--------------------+
 
| information_schema |
 
| mysql              |
 
| performance_schema |
 
+--------------------+
 
debian-9-amd64-clean-lxq1k0:~$ sudo cat /tmp/my.log 
/usr/sbin/mariadbd, Version: 10.6.5-MariaDB-1:10.6.5+maria~stretch-log (mariadb.org binary distribution). started with:
 
Tcp port: 0  Unix socket: /run/mysqld/mysqld.sock
 
Time		    Id Command	Argument
 
211122 13:14:15	     3 Connect	root@localhost on  using Socket
 
		     3 Query	show databases
 
		     3 Quit	
debian-9-amd64-clean-lxq1k0:~$ sudo cat /tmp/my.log 
/usr/sbin/mariadbd, Version: 10.6.5-MariaDB-1:10.6.5+maria~stretch-log (mariadb.org binary distribution). started with:
 
Tcp port: 0  Unix socket: /run/mysqld/mysqld.sock
 
Time		    Id Command	Argument
 
211122 13:14:15	     3 Connect	root@localhost on  using Socket
 
		     3 Query	show databases
 
		     3 Quit	
debian-9-amd64-clean-lxq1k0:~$ sudo mysql_upgrade
 
Phase 1/7: Checking and upgrading mysql database
 
Processing databases
 
mysql
 
mysql.column_stats                                 OK
 
mysql.columns_priv                                 OK
 
mysql.db                                           OK
 
mysql.event                                        OK
 
mysql.func                                         OK
 
mysql.global_priv                                  OK
 
mysql.gtid_slave_pos                               OK
 
mysql.help_category                                OK
 
mysql.help_keyword                                 OK
 
mysql.help_relation                                OK
 
mysql.help_topic                                   OK
 
mysql.index_stats                                  OK
 
mysql.innodb_index_stats                           OK
 
mysql.innodb_table_stats                           OK
 
mysql.plugin                                       OK
 
mysql.proc                                         OK
 
mysql.procs_priv                                   OK
 
mysql.proxies_priv                                 OK
 
mysql.roles_mapping                                OK
 
mysql.servers                                      OK
 
mysql.table_stats                                  OK
 
mysql.tables_priv                                  OK
 
mysql.time_zone                                    OK
 
mysql.time_zone_leap_second                        OK
 
mysql.time_zone_name                               OK
 
mysql.time_zone_transition                         OK
 
mysql.time_zone_transition_type                    OK
 
mysql.transaction_registry                         OK
 
Phase 2/7: Installing used storage engines... Skipped
 
Phase 3/7: Fixing views
 
mysql.user                                         OK
 
Phase 4/7: Running 'mysql_fix_privilege_tables'
 
Phase 5/7: Fixing table and database names
 
Phase 6/7: Checking and upgrading tables
 
Processing databases
 
information_schema
 
performance_schema
 
sys
 
sys.sys_config                                     OK
 
Phase 7/7: Running 'FLUSH PRIVILEGES'
 
OK
 
debian-9-amd64-clean-lxq1k0:~$

Comment by Sergei Golubchik [ 2021-11-22 ]

My suggested solution to (1) above allows to solve test issues by running

apt-get install mariadb-server && mysql_upgrade

this will guarantee that all tables are properly upgraded and there are no concurrently running mysql_upgrade's anymore. Buildbot scripts can be amended to do it.

Of course, one can ask, why do we auto-run mysql_upgrade in debian-start, if it has to be run manually anyway. And this is a good question.

Comment by Elena Stepanova [ 2021-11-22 ]

one can ask, why do we auto-run mysql_upgrade in debian-start

I can think of one reason. Back to the problem of having other packages to upgrade together with the server in one transaction, the extra packages can very well expect the new server schema, not the old one. That is, when engine X gets upgraded from 10.5 to 10.6, it may need the 10.6 server schema, not 10.5, and the user cannot run mysql_upgrade in between to help with this.
Of course, now when mysql_upgrade runs in the background, it doesn't achieve this goal either.

Comment by Faustin Lammler [ 2021-11-22 ]

> I don't know whether it's better to split (2) into a separate issue or handle within this one, but I hope that the final solution won't be downgraded to dealing with (1) only, like MDEV-14622 was previously downgraded to patching Spider, which of course didn't solve anything other than one of numerous side-effects of the root cause issue.

elenst as there is already lots of information here about the Debian packaging problem, I suggest we handle 2 here and that's also why I have created MDEV-27107 for 1.

> My suggested solution to (1) above allows to solve test issues by running
> apt-get install mariadb-server && mysql_upgrade
> this will guarantee that all tables are properly upgraded and there are no concurrently running mysql_upgrade's anymore. Buildbot scripts can be amended to do it.

serg ok so that means to add mysql_upgrade in post-inst.

What can we do with elenst above scenario? And do you have an idea how we can prevent access to the DB during post-inst mysql_upgrade run as I guess this is still required?

Comment by Sergei Golubchik [ 2021-11-22 ]

faust, that did not mean it at all. I said that if my suggestion from that link will be implemented then Buildbot can avoid mysql_upgrade concurrency problems by invoking mysql_upgrade explicitly. It does not say anything about post-inst.

Comment by Elena Stepanova [ 2021-11-22 ]

Invoking mysql_upgrade explicitly and hitting the deadlock in CIs is the least of our problems discussed here, because it's relatively easy to work around.
It appears regularly in new or modified tests which are initially written optimistically, assuming it would work, but then we add all sorts of "wait till there is no mysql_upgrade process running [anymore]", and that does the trick.

Our big problem is a clash with other packages, which we can't do anything about other than stop testing the affected package altogether.

Comment by Otto Kekäläinen [ 2021-11-23 ]

Running mariadb-upgrade requires a mariadbd server that is online, so that it can accept connections and run the commands. Therefore the upgrade script is not part of the distro packaging, as there is no guarantee that an upgrade would actually end up with a new running server. On some systems the server might be intentionally shut down for the duration of the system update and in those cases it should not start at a random time. In some cases users might want the server to stay running while files update, and restart at a user defined time.

Thus the mariadb-upgrade runs as part of the systemd script / SysV init, because only there we can be sure that the server is actually running.

I have suggested a couple of times that the mariadbd binary should take care of its own upgrade. Relying on an external script and external stuff to trigger that script is error prone as we see here. I think it would be a better design if every time mariadbd starts it would automatically by itself do a sanity check and upgrade if it is needed. The upgrade actions of binary data / database has absolutely zero distro specific things and there is no need that distros / service scripts need to manage when and how the binary data format upgrade is triggered.

Could mariadbd perhaps in a future release take care of its own upgrade so we would get rid of external mariadb-upgrade scripts?

Comment by Daniel Black [ 2021-12-08 ]

Looking at required and desired system server variables on what should happen on a temporary instances. May not be complete. Suggestions/edits welcome.

  • innodb_buffer_pool_load_at_startup=0 - preserve unnecessary load
  • innodb_buffer_pool_dump_at_shutdown=0 - our temporary usage isn't a standard workload
  • innodb_force_recovery=0 - ensure the default is true
  • innodb_buffer_pool_size - cap to a few G, especially if its just a mariadb-upgrade of system tables
  • innodb page flushing options to more aggressively flush (mariadb-upgrade should be only altering a innodb table once)
  • skip-networking - required
  • skip-slave-start - (I'm assuming this stops the SQL thread)
  • socket - non-standard required
  • enforce_storage_engine=none
  • skip-log-bin - disable binary logging of the mariabd-upgrade, replicas should instigate their own update
  • wsrep_on = off
  • flush-time = 30 (closing tables after seconds)
Comment by Michael Widenius [ 2021-12-14 ]

Otto, I don't think it is good to have the code in the server. This is perfectly doable in a system or user level script.

Why do you think my proposal is not doable?
(In other words, have a separate section in the systemd scripts that starts and stops the server specially for mysql_upgrade?)
That is the easiest and safest thing to do).
Wonder how PostgreSQL does it? (As PostgreSQL very often changes disk format and one has to dump and restore the database on major version upgrades)

Comment by Michael Widenius [ 2021-12-14 ]

This is my suggestion what Faustin (who is working on this) could start with.
Let's first work on getting mysql_upgrade to support all options needed for this task:

  • Add option '--check-if-upgrade-is-needed' to maria_upgrade the returns 0 or
    1 to the operating system (trough exit call) if a full maria_upgrade should
    be run.
  • Now we only check if the last run was done with the same MariaDB version
    as last time. Change the check so that we check if the major version
    has changed (10.1.xxx -> 10.2.xxx).
    Change the error message from:
    default one:
    "This installation of MariaDB is already upgraded to 10.7.2-MariaDB, use --force if you still need to run mysql_upgrade"
    to
    "This installation of MariaDB is already upgraded to 10.7, use --force if you still need to run mysql_upgrade"

When coding this, take into account that there is a small change that we
in the future may need to run mysql_upgrade for a minor version. We
should have a function that check the version against a minor version
(by default current_major_version.99, like 10.6.99) and if the internal
version is smaller, then we still need to run upgrade.

  • mysql_upgrade creates a 'mysql_upgrade_info' file in the data directory.
    When mysql_upgrade is started, take an advisory lock on the file
    (create it if it does not exists). This ensures we cannot have two
    mysql_upgrade run at the same time.

Please create a separate Jira entry for the above and link it to this one.

Future work:

  • Find out why why running two mysql_upgrade in parllel causes lockups.
    If we can identify this and fix it, things will be much better as
    there is a chance that mysql_upgrade will now lock also for other cases
    than just another mysql_upgrade running.
  • To make maria_upgrade faster, we should consider adding multiple thread
    support to mysql_check_table and run multiple check table & alter table in
    parallel. This should speed up mysql_upgrade for major version upgrades
    notable (in proportional to the number of CPU's in the system).
Comment by Daniel Black [ 2021-12-15 ]

monty can you expand on "If anyone else connects to mysqld/mariadbd during maria_upgrade, bad things can happen as the tables are not up to date for the system.
The possible problems are hangs or wrong data from selects or errors from table updates (as the indexes may not be up to date)."

I'm assuming the first sentence is referring to the system tables component of mysql_upgrade. The second sentence is related to the collation (and other?) user table updates. Why are collation updates needed? Where are they implemented? Is "the indexes may not up to date" referring to joining on a char column against another table? What is the consequence to the user if these user tables updates aren't performed?

I'm mainly looking at these questions from a documentation point of view as to what sort of error/warning/notice to give to the user if for reasons the upgrade does only system tables updates.

Other related subtasks related to minimizing unneeded mysql_upgrades:

  • should mysql_install_db create a mysql_upgrade_info file for its current version?
  • should mariabackup save/restore this mysql_upgrade_info?

With a split of mysql_upgrade for system and user tables should these independent aspects be recorded separately rather than one encompassing mysql_upgrade_info file?

Comment by Sergei Golubchik [ 2021-12-16 ]

danblack, if we find a bug in a collation (say, "b" < "a") and we decide to fix it, then after the upgrade indexes that use this collation will be, basically, corrupted. In the index "b" will be stored before "a", while in the server "b" will be larger than "a" (after the bug fix). So all these tables will have to be repaired.

Because of that, changing a collation is very expensive and we very rarely do it. May be never in MariaDB yet. But technically it can happen.

Comment by Marko Mäkelä [ 2021-12-21 ]

For what it is worth, I believe that after an incompatible change to a collation, it may become impossible to dump all records from an InnoDB table whose PRIMARY KEY would depend on the collation. This is because retrieving records from InnoDB involves storing and restoring a persistent cursor position, and that in turn will depend on key comparisons and the assumption that the index is sorted by the key.

Comment by Michael Widenius [ 2022-01-20 ]

Added --force --force option
Pushed to 10.2 tree

Comment by Daniel Black [ 2022-01-20 ]

monty, a second undocumented --force option is user unfriendly as it ties its functionality to the first --force.

Perhaps a --skip-create-upgrade-info as a more descriptive name?

Did you want to make mysql_install_db create the upgrade_info file or leave that for a separate ticket?

Few review items on https://github.com/MariaDB/server/commit/0713614c3416eaa7a9faeac4707bbc810756b496 in client/mysql_upgrade.c left there from last time. Only two typos + why the use of STDCALL in calc_server_version.

Comment by Michael Widenius [ 2022-01-24 ]

We have several other causes where you specify an option several times and it does more, like --version, --big --big (in mtr) etc.

As this is something that one should never ever do in normal circumstances (because it leads to other (documented) troubles), adding a new flag is overkill.
--force --force is also good from the point that it is backward compatible with older versions, so one can use this without having to check if the mysql_upgrade supports it or not.

The option is documented in both mysql_upgrade and the KB, as it should be.

Comment by Daniel Black [ 2022-01-25 ]

ok.

Comment by Faustin Lammler [ 2022-01-25 ]

> 2) Debian upgrade runs mysql_upgrade non-deterministically. It was already described in MDEV-14622 and it is still the case. Below is a proof-log of 10.5=>10.6 upgrade (official releases) I performed today on Debian 9. As you see after the package upgrade ends, sys schema wasn't created, general log is empty, which indicates mysql_upgrade wasn't even started. I have to run it manually to finalize the upgrade. When I did exactly the same on, say, Ubuntu 20.04, sys schema was there and the general log was full of mysql_upgrade stuff. I can't say whether the same happens every time on a certain debian/ubuntu flavor or it is sporadic, but it is definitely possible to end up with not upgraded schema.

elenst did you noticed this behavior with other Debian/Ubuntu versions? I can confirm it on Debian 9 but I am not able to reproduce it with something else than Debian 9.

Comment by Elena Stepanova [ 2022-01-25 ]

Elena Stepanova did you noticed this behavior with other Debian/Ubuntu versions?

I can't say one way or another. If I did, I wouldn't necessarily register it because I find it irrelevant. Unless somebody knows about a specific bug fixed in all other versions so that the fix has magically made this code behave deterministically, relying on better/more lucky timing of certain versions over others isn't wise.

Comment by Otto Kekäläinen [ 2023-07-23 ]

The argumentation in the comments of this Jira is basically pros and cons for implementing MDEV-30499. After thinking about this long and hard, I still stand by the arguments I listed in https://jira.mariadb.org/browse/MDEV-27068?focusedCommentId=206749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-206749 and think it would be best if mariadbd server itself took care of the upgrade without relying on external scripts or users to run mariadb-upgrade. Such a dependency is simply too brittle.

Generated at Thu Feb 08 09:50:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.