Michael Widenius added a comment - 2021-11-18 12:19 In what contex do you get timeouts? I checked your Attachement and there is a core. However to be able to fully use the core I would need to also have the corresponding mysqld or mariadbd binary. The journal includes a trace of a crash: Nov 16 20:18:10 aio1-galera-container-b6377aeb /etc/mysql/debian-start [3862] : sys.schema_redundant_indexes OK Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : 211116 20:20:17 [ERROR] mysqld got signal 6 ; .... Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : stack_bottom = 0x0 thread_stack 0x49000 Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : Printing to addr2line failed Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /usr/sbin/mariadbd(my_print_stacktrace+0x32) [0x55e28d2e3442] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /usr/sbin/mariadbd(handle_fatal_signal+0x485) [0x55e28cd966a5] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f8ffd0173c0] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /lib/x86_64-linux-gnu/libc.so.6(__poll+0x4f) [0x7f8ffcbecaff] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /usr/sbin/mariadbd(_Z26handle_connections_socketsv+0x1da) [0x55e28ca942ba] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x891) [0x55e28ca94df1] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f8ffcafe0b3] Nov 16 20:20:17 aio1-galera-container-b6377aeb mariadbd [3831] : /usr/sbin/mariadbd(_start+0x2e) [0x55e28ca8979e] The crash comes from a 'signal 6', SIGABRT that is probably sent out from the outside. There is 2 minutes of delay between the last log entry and the ABORT. This indicates that a program or user killed the server with hard because of some unknown reason. The server was running the following during the crash: Running 'mysqlcheck' with connection arguments: --port='3306' --socket='/var/run/mysqld/mysqld.sock' --host='localhost' --socket='/var/run/mysqld/mysqld.sock' --host='localhost' --socket='/var/run/mysqld/mysqld.sock' It was processing sys.views when the server was killed. There was a strange timeout of 2 minutes in the output from mysqlcheck, but that could be a caching issue. If the databases are huge, then the above command can take a long time. The 2 minute timeout is probably not enough to finish the above command. Suggested fix: Increase the system timeout when running the mysql --upgrade process. There is of course a small chance that there is a bug when checking sys schema views that causes a deadlock. It would be great if you could somehow verify if this is the case by running mysql_upgrade separately on a server and check if you get any issues.

Marc Gariépy added a comment - 2021-11-18 13:18 Hello, The timeout happens on a greenfield deployment in our CI. Here is a test result with the issue: https://zuul.opendev.org/t/openstack/build/0afa668757224613b70604023c38b802 The last task that has been run and worked was "Create galera users" https://8f934895f3c98f9beb6d-3ef9e777966cf361bd9799f22afa5b88.ssl.cf1.rackcdn.com/817385/1/check/openstack-ansible-deploy-aio_distro_metal-ubuntu-focal/0afa668/logs/ara-report/files/323.html#line-16 the following tasks do hangs sometimes in the CI (but I could reproduce the issue inside a VM on another cloud) https://8f934895f3c98f9beb6d-3ef9e777966cf361bd9799f22afa5b88.ssl.cf1.rackcdn.com/817385/1/check/openstack-ansible-deploy-aio_distro_metal-ubuntu-focal/0afa668/logs/ara-report/files/323.html#line-51 The kill was generated by me with a `kill -6`. The process is still hung from 2 days ago. the database is empty at this point as it's tooling that does the initial install of mariadb.

Marc Gariépy added a comment - 2021-11-18 15:39 After some debugging i found out that the issue was that on first start the mysql_upgrade is run and we did ran it another time in parallel in the playbook before it was done. If we remove the call for mysql_upgrade that was on our playbook everything works as expected on debian based system. On centos8 it's not done by the system on installation and i haven't seen it in the installation document either.

Marc Gariépy added a comment - 2021-11-18 20:04 Apparently i did talked a bit too fast. it seems that the issue do still happen. here is the trace with GDB etc-mysql.tar.gz > tar of /etc/mysql/ mysqldb.tar.gz > tar of /var/lib/mysql mysqld_full_bt_all_threads.txt > gdb trace of the running process pkg-list-apt > dpkg -l on the systems process > ps auxf of the running processes test.log > journalctl of the mariadb service debug-MDEV-27068.tar.gz

Marc Gariépy added a comment - 2021-11-19 13:30 MariaDB [(none)] > show processlist -> ; --- ----------- --------- ----- ------- ----- ------------------------------- ---------------------------------------------------------------------------------------------- --------- Id User Host db Command Time State Info Progress --- ----------- --------- ----- ------- ----- ------------------------------- ---------------------------------------------------------------------------------------------- --------- 2 system user   NULL Sleep 75998 closing tables NULL 0.000 1 system user   NULL Sleep 75998 wsrep aborter idle NULL 0.000 6 system user   NULL Sleep 75997 wsrep applier idle NULL 0.000 17 root localhost mysql Query 75992 Waiting for table metadata lock alter table proxies_priv modify Grantor varchar(384) COLLATE utf8_bin not null default '' 0.000 21 root localhost NULL Query 75992 Waiting for table level lock CREATE USER 'monitoring'@'localhost' 0.000 31 root localhost NULL Query 0 starting show processlist 0.000 --- ----------- --------- ----- ------- ----- ------------------------------- ---------------------------------------------------------------------------------------------- --------- 6 rows in set (0.001 sec)

Michael Widenius added a comment - 2021-11-19 14:18 - edited From discussion on Libera.Chat: The main issues are on Debian installations, where this problem happens: Debian system always runs mysql_upgrade/maria_upgrade for every MariaDB server service start. If maria_upgrade has to be run, as in case of a major upgrade, then check and possible fix of tables can take from a few seconds to 20-30 seconds (normally) but potentially much longer if there is was a chance in a character collation. If during this time another maria_upgrade is run then it is very likely that maria_upgrade will hang because of unexpected conflicts. This is because maria_upgrade was not designed to be run in parallel. If anyone else connects to mysqld/mariadbd during maria_upgrade, bad things can happen as the tables are not up to date for the system. The possible problems are hangs or wrong data from selects or errors from table updates (as the indexes may not be up to date). Suggested solution: Add an option to maria_upgrade '--check-if-upgrade-is-needed' that returns 0 or 1 depending on maria_upgrade should be run. This is trivial as there is already checks in the code if maria_upgrade needs to be run. In Debian system files add the following logic to the place where maria_upgrade is now run. If needed move the code to be just before the MariaDB server is started. run maria_upgrade --check-if-upgrade-is-needed If yes Start MariadB server with --skip-grants --socket=/tmp/mariadb-upgrade.sock --skip-networking run maria_upgrade --socket=/tmp/mariadb-upgrade.sock stop the MariaDB server The above should ensure that no one can connect to the mariadbd server while maria_upgrade is running. If anyone tries to run the above script twice it will also block as the second MariaDB server will not start until the first has completed (innodb file locks takes care of this).

Sergei Golubchik added a comment - 2021-11-19 15:22 See the linked issue. Users will complain about long-running mariadb-upgrade that keeps the server unaccessible. It's easy to prevent two mariadb-upgrade running in parallel, if it'll connect to the server and issue something like SELECT GET_LOCK( "mariadb-upgrade" ); As a bonus, it can use the connection to avoid exec'ing mariadb command-line client. An even simpler approach would be to get an advisory lock on mysql_upgrade_info file. Or on datadir (solves cases when the file doesn't exist yet). In either approach one can run mariadb-upgrade and when it finishes everything will be upgraded. No matter if there were concurrent upgrades or not.

Sergei Golubchik added a comment - 2021-11-19 15:27 A second issue here is that having mariadb-upgrade running in the background when many packages are being upgraded is race condition prone. It'd be better for mariadb-server postinstall script to upgrade system tables (but not user tables — see linked issue). This can be done by running mariadb-upgrade --upgrade-system-tables from the postinst script (and because of locking above it'll be safe). But the problem is that it'll write the info file, so a full mariadb-upgrade won't run after it. Or, if the full background mariadb-upgrade will start first, the fast version will wait till the very end.

Michael Widenius added a comment - 2021-11-20 12:57 - edited One cannot run two maria_upgrade in parallel. This results on the server getting locked. One may also gets lock with just one maria_upgrade running and users having access to the server at the same time (According to not verified reports at Libera.chat) Also, there is no point whatsoever in having users access any tables when mysql_upgrade is running as this can lead to corruptions if index collations have been changed. As this is only happening when the server is getting upgraded, which means the server was taken down anyway, it should not matter much that there is a required check of tables when the server starts. The check will only take a notable time if there is a lot of tables (10,000+) or if there is a lot of tables that need to update the table because of changed collations, in which case one should not allow anyone use of the tables anyway. I think that if one has 10,000+ tables, one will schedule a proper downtime for any upgrade anyways, in which case that time for mysql_upgrade should be neglectable. Another possible fix is to have maria_upgrade by default not doing anything if this is a minor upgrade. This should solve the things for most users as the longer upgrade run would only be done when they upgrade their system (in which case any mysql_upgrade time should be neglectable)

Faustin Lammler added a comment - 2021-11-22 12:56 > On centos8 it's not done by the system on installation and i haven't seen it in the installation document either. Yes this is correct, a manual action is required ( https://github.com/MariaDB/server/blob/10.6/support-files/rpm/server-prein.sh#L41-L61 ) Whereas on Debian, mysql_upgrade is called (by systemd or sysV init) on every (re)start of the server and the problem is well explained in MDEV-14622 , see also https://github.com/MariaDB/server/blob/10.6/debian/additions/debian-start . My understanding is that MDEV-14622 only resolved the spider problem, it does not fix the fact that the server should not be accessed while mysql_upgrade is run and having it run in the background is not an acceptable solution. > Another possible fix is to have maria_upgrade by default not doing anything if this is a minor upgrade. This should solve the things for most users as the longer upgrade run would only be done when they upgrade their system (in which case any mysql_upgrade time should be neglectable) This is true for users relying on official Debian/Ubuntu repositories as a major MariaDB upgrade means a dist-upgrade (downtime is expected). For users relying on MDBF repository this can be problematic as they may not be upgrading the system but only MariaDB and do not expect downtime. I see two different issue raised by the actual MDEV: 1. mariadb_upgrade can't be run in parallel, this should be fixed directly in the program, as discussed with danblack , he will look at it. 2. we need to change the way we handle mysql_upgrade on the packaging level in Debian/Ubuntu, I'll take care of that. For 2, I do not have a solution to propose so far. Here are some ideas based on previous discussions and what my understanding of the problem is: do the same as for RPM (I am not sure that it's desirable but it's a clean solution so it should be listed); do the same as for RPM if we detect a lots of tables (is 10,000+ the good threshold?, only run --upgrade-system-tables?); run mysql_upgrade at post-install, but how to block access to the server until mysql_upgrade finishes (I don't see how `--skip-grants --socket=/tmp/mariadb-upgrade.sock --skip-networking` can be implemented in a systemd/sysv init context)? Finally, from MDEV-14622 > it should be the server itself that checks at startup that its internal data structure (the "mysql" database) is up-to-date then corrects/update it before accepting connections This looks like a good feature and I am wondering if we should create a MDEV so it has a chance to be implemented in future release?

Elena Stepanova added a comment - 2021-11-22 13:27 I don't know whether it's better to split (2) into a separate issue or handle within this one, but I hope that the final solution won't be downgraded to dealing with (1) only, like MDEV-14622 was previously downgraded to patching Spider, which of course didn't solve anything other than one of numerous side-effects of the root cause issue. Similarly, fixing "running mysql_upgrade in parallel" won't solve two general problems on Debian, both of which still exist, as constant sporadic failures in our upgrade tests show: 1) Debian runs mysql_upgrade in background, and anything is possible during this time. For example, currently upgrade tests on 10.6 fail for us every day when they involve Columnstore installation: server gets upgraded, mysql_upgrade kicks in, then Columnstore starts getting upgraded, clashes with mysql_upgrade, everything hangs. There is no workaround for it: the user doesn't trigger this parallelism and cannot do anything to prevent it. While an extra package can be installed separately, it cannot be upgraded separately (but please, don't solve it by just hacking Columnstore, as it was done in MDEV-14622 for Spider. Columnstore here is just one more example , it will keep happening with various packages until the problem is solved). 2) Debian upgrade runs mysql_upgrade non-deterministically . It was already described in MDEV-14622 and it is still the case. Below is a proof-log of 10.5=>10.6 upgrade (official releases) I performed today on Debian 9. As you see after the package upgrade ends, sys schema wasn't created, general log is empty, which indicates mysql_upgrade wasn't even started. I have to run it manually to finalize the upgrade. When I did exactly the same on, say, Ubuntu 20.04, sys schema was there and the general log was full of mysql_upgrade stuff. I can't say whether the same happens every time on a certain debian/ubuntu flavor or it is sporadic, but it is definitely possible to end up with not upgraded schema. debian-9-amd64-clean-lxq1k0:~$ wget https://dlm.mariadb.com/3/mariadb/mariadb_repo_setup --2021-11-22 13:12:46-- https://dlm.mariadb.com/3/mariadb/mariadb_repo_setup Resolving dlm.mariadb.com (dlm.mariadb.com)... 104.18.135.24, 104.17.191.14, 2606:4700::6811:bf0e, ... Connecting to dlm.mariadb.com (dlm.mariadb.com)|104.18.135.24|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://storage.googleapis.com/downloads-cdn.mariadb.com/aptyum/1.0/1.0.0/MariaDB/mariadb_repo_setup?GoogleAccessId=downloads-sync%40downloads-234321.iam.gserviceaccount.com&Expires=1637587366&Signature=gaS93eVKCVNVvoLQPE3xvjNyGLhIe4h8BulDeLOGKdjU4JnbHfJN530hWBe8BgjMpDeEGSwAo8RRYEYm%2BoXeFhqDwS8giA0u0lbHN0wPeJVr%2BUROBP5zPAYhwGJsUq%2BI3SG5lnT8e7kb3ubATO3sIiWrZwUyEfWNwCbLi1x2b1iDCZj7IqREEI1Eei6Wg3qcm8qpENCBt0oZ90TV5rGdRsd%2BQxSivOUE8sRSMk1KS7JV8HlhiZ%2B5IVOp3ISJm%2FJY4EkH%2B6MTt%2FsaN3ber%2Bey7%2BOOFuIssAghEsgMowg7qOk03DUgSEadxljt98o7cPIbh%2F6ctd6qW7OeiqSIGFAi0w%3D%3D [following] --2021-11-22 13:12:46-- https://storage.googleapis.com/downloads-cdn.mariadb.com/aptyum/1.0/1.0.0/MariaDB/mariadb_repo_setup?GoogleAccessId=downloads-sync%40downloads-234321.iam.gserviceaccount.com&Expires=1637587366&Signature=gaS93eVKCVNVvoLQPE3xvjNyGLhIe4h8BulDeLOGKdjU4JnbHfJN530hWBe8BgjMpDeEGSwAo8RRYEYm%2BoXeFhqDwS8giA0u0lbHN0wPeJVr%2BUROBP5zPAYhwGJsUq%2BI3SG5lnT8e7kb3ubATO3sIiWrZwUyEfWNwCbLi1x2b1iDCZj7IqREEI1Eei6Wg3qcm8qpENCBt0oZ90TV5rGdRsd%2BQxSivOUE8sRSMk1KS7JV8HlhiZ%2B5IVOp3ISJm%2FJY4EkH%2B6MTt%2FsaN3ber%2Bey7%2BOOFuIssAghEsgMowg7qOk03DUgSEadxljt98o7cPIbh%2F6ctd6qW7OeiqSIGFAi0w%3D%3D Resolving storage.googleapis.com (storage.googleapis.com)... 108.177.121.128, 142.250.103.128, 108.177.120.128, ... Connecting to storage.googleapis.com (storage.googleapis.com)|108.177.121.128|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 26626 (26K) [application/octet-stream] Saving to: ‘mariadb_repo_setup’   mariadb_repo_setup 100%[=====================================================================================================================================>] 26.00K --.-KB/s in 0s   2021-11-22 13:12:47 (99.6 MB/s) - ‘mariadb_repo_setup’ saved [26626/26626]   debian-9-amd64-clean-lxq1k0:~$ chmod +x mariadb_repo_setup debian-9-amd64-clean-lxq1k0:~$ sudo ./mariadb_repo_setup --mariadb-server-version=10.5 --skip-maxscale --skip-tools [info] Checking for script prerequisites. [info] Repository file successfully written to /etc/apt/sources.list.d/mariadb.list [info] Adding trusted package signing keys... [info] Running apt-get update... [info] Done adding trusted package signing keys debian-9-amd64-clean-lxq1k0:~$ sudo apt-get update Hit:1 http://security.debian.org stretch/updates InRelease Ign:2 http://deb.debian.org/debian stretch InRelease Hit:3 http://deb.debian.org/debian stretch-updates InRelease Hit:4 http://deb.debian.org/debian stretch-backports InRelease Hit:5 http://deb.debian.org/debian stretch Release Hit:6 http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease Get:7 http://packages.cloud.google.com/apt google-compute-engine-stretch-stable InRelease [5,525 B] Hit:8 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-stretch InRelease Hit:10 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch InRelease Fetched 5,525 B in 0s (9,201 B/s) Reading package lists... Done debian-9-amd64-clean-lxq1k0:~$ sudo apt-get install -y mariadb-server mariadb-client mariadb-backup mariadb-plugin-rocksdb Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: galera-4 gawk libaio1 libcgi-fast-perl libcgi-pm-perl libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libmariadb3 libmariadbclient18 libmpfr4 libpcre2-8-0 libperl5.24 libreadline5 libsigsegv2 libsnappy1v5 libterm-readkey-perl libtimedate-perl liburi-perl lsof mariadb-client-10.5 mariadb-client-core-10.5 mariadb-common mariadb-server-10.5 mariadb-server-core-10.5 mysql-common perl perl-modules-5.24 psmisc python3-mysqldb rename rsync socat Suggested packages: gawk-doc libclone-perl libmldbm-perl libnet-daemon-perl libsql-statement-perl libdata-dump-perl libipc-sharedcache-perl libwww-perl mailx mariadb-test netcat-openbsd perl-doc libterm-readline-gnu-perl | libterm-readline-perl-perl make python-egenix-mxdatetime python3-mysqldb-dbg The following NEW packages will be installed: galera-4 gawk libaio1 libcgi-fast-perl libcgi-pm-perl libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libmariadb3 libmariadbclient18 libmpfr4 libpcre2-8-0 libperl5.24 libreadline5 libsigsegv2 libsnappy1v5 libterm-readkey-perl libtimedate-perl liburi-perl lsof mariadb-backup mariadb-client mariadb-client-10.5 mariadb-client-core-10.5 mariadb-common mariadb-plugin-rocksdb mariadb-server mariadb-server-10.5 mariadb-server-core-10.5 mysql-common perl perl-modules-5.24 psmisc python3-mysqldb rename rsync socat 0 upgraded, 45 newly installed, 0 to remove and 3 not upgraded. Need to get 43.3 MB of archives. After this operation, 289 MB of additional disk space will be used. Get:1 http://deb.debian.org/debian stretch/main amd64 perl-modules-5.24 all 5.24.1-3+deb9u7 [2,723 kB] Get:2 http://security.debian.org stretch/updates/main amd64 libdbi-perl amd64 1.636-1+deb9u1 [766 kB] Get:3 http://deb.debian.org/debian stretch/main amd64 libperl5.24 amd64 5.24.1-3+deb9u7 [3,527 kB] Get:4 http://deb.debian.org/debian stretch/main amd64 perl amd64 5.24.1-3+deb9u7 [218 kB] Get:5 http://deb.debian.org/debian stretch/main amd64 libmpfr4 amd64 3.1.5-1 [556 kB] Get:6 http://deb.debian.org/debian stretch/main amd64 libsigsegv2 amd64 2.10-5 [28.9 kB] Get:7 http://deb.debian.org/debian stretch/main amd64 gawk amd64 1:4.1.4+dfsg-1 [571 kB] Get:8 http://deb.debian.org/debian stretch/main amd64 lsof amd64 4.89+dfsg-0.1 [313 kB] Get:9 http://deb.debian.org/debian stretch/main amd64 libreadline5 amd64 5.2+dfsg-3+b1 [119 kB] Get:10 http://deb.debian.org/debian stretch/main amd64 libaio1 amd64 0.3.110-3 [9,412 B] Get:11 http://deb.debian.org/debian stretch/main amd64 libpcre2-8-0 amd64 10.22-3 [182 kB] Get:12 http://deb.debian.org/debian stretch/main amd64 psmisc amd64 22.21-2.1+b2 [123 kB] Get:13 http://deb.debian.org/debian stretch/main amd64 rsync amd64 3.1.2-1+deb9u2 [393 kB] Get:14 http://deb.debian.org/debian stretch/main amd64 socat amd64 1.7.3.1-2+deb9u1 [353 kB] Get:15 http://deb.debian.org/debian stretch/main amd64 libhtml-tagset-perl all 3.20-3 [12.7 kB] Get:16 http://deb.debian.org/debian stretch/main amd64 liburi-perl all 1.71-1 [88.6 kB] Get:17 http://deb.debian.org/debian stretch/main amd64 libhtml-parser-perl amd64 3.72-3 [104 kB] Get:18 http://deb.debian.org/debian stretch/main amd64 libcgi-pm-perl all 4.35-1 [222 kB] Get:19 http://deb.debian.org/debian stretch/main amd64 libfcgi-perl amd64 0.78-2 [38.2 kB] Get:20 http://deb.debian.org/debian stretch/main amd64 libcgi-fast-perl all 1:2.12-1 [11.2 kB] Get:21 http://deb.debian.org/debian stretch/main amd64 libdbd-mysql-perl amd64 4.041-2 [114 kB] Get:22 http://deb.debian.org/debian stretch/main amd64 libencode-locale-perl all 1.05-1 [13.7 kB] Get:23 http://deb.debian.org/debian stretch/main amd64 libhtml-template-perl all 2.95-2 [67.1 kB] Get:24 http://deb.debian.org/debian stretch/main amd64 libtimedate-perl all 2.3000-2+deb9u1 [37.9 kB] Get:25 http://deb.debian.org/debian stretch/main amd64 libhttp-date-perl all 6.02-1 [10.7 kB] Get:26 http://deb.debian.org/debian stretch/main amd64 libio-html-perl all 1.001-1 [17.6 kB] Get:27 http://deb.debian.org/debian stretch/main amd64 liblwp-mediatypes-perl all 6.02-1 [22.1 kB] Get:28 http://deb.debian.org/debian stretch/main amd64 libhttp-message-perl all 6.11-1 [75.9 kB] Get:29 http://deb.debian.org/debian stretch/main amd64 libterm-readkey-perl amd64 2.37-1 [27.2 kB] Get:30 http://deb.debian.org/debian stretch/main amd64 libsnappy1v5 amd64 1.1.3-3 [51.0 kB] Get:31 http://deb.debian.org/debian stretch/main amd64 python3-mysqldb amd64 1.3.7-1.1 [48.9 kB] Get:32 http://deb.debian.org/debian stretch/main amd64 rename all 0.20-4 [12.5 kB] Get:33 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mysql-common all 1:10.5.13+maria~stretch [3,190 B] Get:34 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-common all 1:10.5.13+maria~stretch [4,378 B] Get:35 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 galera-4 amd64 26.4.9-stretch [8,984 kB] Get:36 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 libmariadb3 amd64 1:10.5.13+maria~stretch [148 kB] Get:37 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-client-core-10.5 amd64 1:10.5.13+maria~stretch [772 kB] Get:38 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-client-10.5 amd64 1:10.5.13+maria~stretch [1,541 kB] Get:39 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-server-core-10.5 amd64 1:10.5.13+maria~stretch [6,908 kB] Get:40 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-server-10.5 amd64 1:10.5.13+maria~stretch [4,467 kB] Get:41 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 libmariadbclient18 amd64 1:10.5.13+maria~stretch [3,178 B] Get:42 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-backup amd64 1:10.5.13+maria~stretch [6,452 kB] Get:43 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-client all 1:10.5.13+maria~stretch [3,016 B] Get:44 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-plugin-rocksdb amd64 1:10.5.13+maria~stretch [3,110 kB] Get:45 https://downloads.mariadb.com/MariaDB/mariadb-10.5/repo/debian stretch/main amd64 mariadb-server all 1:10.5.13+maria~stretch [3,144 B] Fetched 43.3 MB in 42s (1,024 kB/s) Extracting templates from packages: 100% Preconfiguring packages ... Selecting previously unselected package perl-modules-5.24. (Reading database ... 50132 files and directories currently installed.) Preparing to unpack .../perl-modules-5.24_5.24.1-3+deb9u7_all.deb ... Unpacking perl-modules-5.24 (5.24.1-3+deb9u7) ... Selecting previously unselected package libperl5.24:amd64. Preparing to unpack .../libperl5.24_5.24.1-3+deb9u7_amd64.deb ... Unpacking libperl5.24:amd64 (5.24.1-3+deb9u7) ... Selecting previously unselected package perl. Preparing to unpack .../perl_5.24.1-3+deb9u7_amd64.deb ... Unpacking perl (5.24.1-3+deb9u7) ... Selecting previously unselected package libmpfr4:amd64. Preparing to unpack .../libmpfr4_3.1.5-1_amd64.deb ... Unpacking libmpfr4:amd64 (3.1.5-1) ... Selecting previously unselected package libsigsegv2:amd64. Preparing to unpack .../libsigsegv2_2.10-5_amd64.deb ... Unpacking libsigsegv2:amd64 (2.10-5) ... Setting up libmpfr4:amd64 (3.1.5-1) ... Setting up libsigsegv2:amd64 (2.10-5) ... Selecting previously unselected package gawk. (Reading database ... 51970 files and directories currently installed.) Preparing to unpack .../00-gawk_1%3a4.1.4+dfsg-1_amd64.deb ... Unpacking gawk (1:4.1.4+dfsg-1) ... Selecting previously unselected package mysql-common. Preparing to unpack .../01-mysql-common_1%3a10.5.13+maria~stretch_all.deb ... Unpacking mysql-common (1:10.5.13+maria~stretch) ... Selecting previously unselected package mariadb-common. Preparing to unpack .../02-mariadb-common_1%3a10.5.13+maria~stretch_all.deb ... Unpacking mariadb-common (1:10.5.13+maria~stretch) ... Selecting previously unselected package galera-4. Preparing to unpack .../03-galera-4_26.4.9-stretch_amd64.deb ... Unpacking galera-4 (26.4.9-stretch) ... Selecting previously unselected package libdbi-perl. Preparing to unpack .../04-libdbi-perl_1.636-1+deb9u1_amd64.deb ... Unpacking libdbi-perl (1.636-1+deb9u1) ... Selecting previously unselected package lsof. Preparing to unpack .../05-lsof_4.89+dfsg-0.1_amd64.deb ... Unpacking lsof (4.89+dfsg-0.1) ... Selecting previously unselected package libmariadb3:amd64. Preparing to unpack .../06-libmariadb3_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking libmariadb3:amd64 (1:10.5.13+maria~stretch) ... Selecting previously unselected package libreadline5:amd64. Preparing to unpack .../07-libreadline5_5.2+dfsg-3+b1_amd64.deb ... Unpacking libreadline5:amd64 (5.2+dfsg-3+b1) ... Selecting previously unselected package mariadb-client-core-10.5. Preparing to unpack .../08-mariadb-client-core-10.5_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking mariadb-client-core-10.5 (1:10.5.13+maria~stretch) ... Selecting previously unselected package mariadb-client-10.5. Preparing to unpack .../09-mariadb-client-10.5_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking mariadb-client-10.5 (1:10.5.13+maria~stretch) ... Selecting previously unselected package libaio1:amd64. Preparing to unpack .../10-libaio1_0.3.110-3_amd64.deb ... Unpacking libaio1:amd64 (0.3.110-3) ... Selecting previously unselected package libpcre2-8-0:amd64. Preparing to unpack .../11-libpcre2-8-0_10.22-3_amd64.deb ... Unpacking libpcre2-8-0:amd64 (10.22-3) ... Selecting previously unselected package mariadb-server-core-10.5. Preparing to unpack .../12-mariadb-server-core-10.5_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking mariadb-server-core-10.5 (1:10.5.13+maria~stretch) ... Selecting previously unselected package psmisc. Preparing to unpack .../13-psmisc_22.21-2.1+b2_amd64.deb ... Unpacking psmisc (22.21-2.1+b2) ... Selecting previously unselected package rsync. Preparing to unpack .../14-rsync_3.1.2-1+deb9u2_amd64.deb ... Unpacking rsync (3.1.2-1+deb9u2) ... Selecting previously unselected package socat. Preparing to unpack .../15-socat_1.7.3.1-2+deb9u1_amd64.deb ... Unpacking socat (1.7.3.1-2+deb9u1) ... Setting up mysql-common (1:10.5.13+maria~stretch) ... Setting up mariadb-common (1:10.5.13+maria~stretch) ... update-alternatives: error: no alternatives for my.cnf update-alternatives: using /etc/mysql/mariadb.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode Selecting previously unselected package mariadb-server-10.5. (Reading database ... 52681 files and directories currently installed.) Preparing to unpack .../00-mariadb-server-10.5_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking mariadb-server-10.5 (1:10.5.13+maria~stretch) ... Selecting previously unselected package libhtml-tagset-perl. Preparing to unpack .../01-libhtml-tagset-perl_3.20-3_all.deb ... Unpacking libhtml-tagset-perl (3.20-3) ... Selecting previously unselected package liburi-perl. Preparing to unpack .../02-liburi-perl_1.71-1_all.deb ... Unpacking liburi-perl (1.71-1) ... Selecting previously unselected package libhtml-parser-perl. Preparing to unpack .../03-libhtml-parser-perl_3.72-3_amd64.deb ... Unpacking libhtml-parser-perl (3.72-3) ... Selecting previously unselected package libcgi-pm-perl. Preparing to unpack .../04-libcgi-pm-perl_4.35-1_all.deb ... Unpacking libcgi-pm-perl (4.35-1) ... Selecting previously unselected package libfcgi-perl. Preparing to unpack .../05-libfcgi-perl_0.78-2_amd64.deb ... Unpacking libfcgi-perl (0.78-2) ... Selecting previously unselected package libcgi-fast-perl. Preparing to unpack .../06-libcgi-fast-perl_1%3a2.12-1_all.deb ... Unpacking libcgi-fast-perl (1:2.12-1) ... Selecting previously unselected package libmariadbclient18. Preparing to unpack .../07-libmariadbclient18_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking libmariadbclient18 (1:10.5.13+maria~stretch) ... Selecting previously unselected package libdbd-mysql-perl. Preparing to unpack .../08-libdbd-mysql-perl_4.041-2_amd64.deb ... Unpacking libdbd-mysql-perl (4.041-2) ... Selecting previously unselected package libencode-locale-perl. Preparing to unpack .../09-libencode-locale-perl_1.05-1_all.deb ... Unpacking libencode-locale-perl (1.05-1) ... Selecting previously unselected package libhtml-template-perl. Preparing to unpack .../10-libhtml-template-perl_2.95-2_all.deb ... Unpacking libhtml-template-perl (2.95-2) ... Selecting previously unselected package libtimedate-perl. Preparing to unpack .../11-libtimedate-perl_2.3000-2+deb9u1_all.deb ... Unpacking libtimedate-perl (2.3000-2+deb9u1) ... Selecting previously unselected package libhttp-date-perl. Preparing to unpack .../12-libhttp-date-perl_6.02-1_all.deb ... Unpacking libhttp-date-perl (6.02-1) ... Selecting previously unselected package libio-html-perl. Preparing to unpack .../13-libio-html-perl_1.001-1_all.deb ... Unpacking libio-html-perl (1.001-1) ... Selecting previously unselected package liblwp-mediatypes-perl. Preparing to unpack .../14-liblwp-mediatypes-perl_6.02-1_all.deb ... Unpacking liblwp-mediatypes-perl (6.02-1) ... Selecting previously unselected package libhttp-message-perl. Preparing to unpack .../15-libhttp-message-perl_6.11-1_all.deb ... Unpacking libhttp-message-perl (6.11-1) ... Selecting previously unselected package libterm-readkey-perl. Preparing to unpack .../16-libterm-readkey-perl_2.37-1_amd64.deb ... Unpacking libterm-readkey-perl (2.37-1) ... Selecting previously unselected package mariadb-backup. Preparing to unpack .../17-mariadb-backup_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking mariadb-backup (1:10.5.13+maria~stretch) ... Selecting previously unselected package mariadb-client. Preparing to unpack .../18-mariadb-client_1%3a10.5.13+maria~stretch_all.deb ... Unpacking mariadb-client (1:10.5.13+maria~stretch) ... Selecting previously unselected package libsnappy1v5:amd64. Preparing to unpack .../19-libsnappy1v5_1.1.3-3_amd64.deb ... Unpacking libsnappy1v5:amd64 (1.1.3-3) ... Selecting previously unselected package mariadb-plugin-rocksdb. Preparing to unpack .../20-mariadb-plugin-rocksdb_1%3a10.5.13+maria~stretch_amd64.deb ... Unpacking mariadb-plugin-rocksdb (1:10.5.13+maria~stretch) ... Selecting previously unselected package mariadb-server. Preparing to unpack .../21-mariadb-server_1%3a10.5.13+maria~stretch_all.deb ... Unpacking mariadb-server (1:10.5.13+maria~stretch) ... Selecting previously unselected package python3-mysqldb. Preparing to unpack .../22-python3-mysqldb_1.3.7-1.1_amd64.deb ... Unpacking python3-mysqldb (1.3.7-1.1) ... Selecting previously unselected package rename. Preparing to unpack .../23-rename_0.20-4_all.deb ... Unpacking rename (0.20-4) ... Setting up libmariadb3:amd64 (1:10.5.13+maria~stretch) ... Setting up perl-modules-5.24 (5.24.1-3+deb9u7) ... Setting up libperl5.24:amd64 (5.24.1-3+deb9u7) ... Setting up galera-4 (26.4.9-stretch) ... Setting up psmisc (22.21-2.1+b2) ... Setting up socat (1.7.3.1-2+deb9u1) ... Setting up lsof (4.89+dfsg-0.1) ... Setting up libmariadbclient18 (1:10.5.13+maria~stretch) ... Setting up gawk (1:4.1.4+dfsg-1) ... Setting up rsync (3.1.2-1+deb9u2) ... Created symlink /etc/systemd/system/multi-user.target.wants/rsync.service → /lib/systemd/system/rsync.service. Setting up perl (5.24.1-3+deb9u7) ... update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode Setting up liblwp-mediatypes-perl (6.02-1) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... Setting up libaio1:amd64 (0.3.110-3) ... Setting up liburi-perl (1.71-1) ... Setting up libsnappy1v5:amd64 (1.1.3-3) ... Processing triggers for systemd (232-25+deb9u13) ... Processing triggers for man-db (2.7.6.1-2) ... Setting up libreadline5:amd64 (5.2+dfsg-3+b1) ... Setting up libpcre2-8-0:amd64 (10.22-3) ... Setting up libfcgi-perl (0.78-2) ... Setting up libdbi-perl (1.636-1+deb9u1) ... Setting up libhtml-tagset-perl (3.20-3) ... Setting up mariadb-client-core-10.5 (1:10.5.13+maria~stretch) ... Setting up libencode-locale-perl (1.05-1) ... Setting up libtimedate-perl (2.3000-2+deb9u1) ... Setting up rename (0.20-4) ... update-alternatives: using /usr/bin/file-rename to provide /usr/bin/rename (rename) in auto mode Setting up mariadb-client-10.5 (1:10.5.13+maria~stretch) ... Setting up libio-html-perl (1.001-1) ... Setting up mariadb-client (1:10.5.13+maria~stretch) ... Setting up mariadb-server-core-10.5 (1:10.5.13+maria~stretch) ... Setting up python3-mysqldb (1.3.7-1.1) ... Setting up libterm-readkey-perl (2.37-1) ... Setting up mariadb-backup (1:10.5.13+maria~stretch) ... Setting up mariadb-server-10.5 (1:10.5.13+maria~stretch) ... Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service. Setting up libhtml-parser-perl (3.72-3) ... Setting up libcgi-pm-perl (4.35-1) ... Setting up mariadb-server (1:10.5.13+maria~stretch) ... Setting up libdbd-mysql-perl (4.041-2) ... Setting up libhttp-date-perl (6.02-1) ... Setting up libhtml-template-perl (2.95-2) ... Setting up libcgi-fast-perl (1:2.12-1) ... Setting up mariadb-plugin-rocksdb (1:10.5.13+maria~stretch) ... Setting up libhttp-message-perl (6.11-1) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... Processing triggers for systemd (232-25+deb9u13) ... Processing triggers for mariadb-server-10.5 (1:10.5.13+maria~stretch) ... debian-9-amd64-clean-lxq1k0:~$ sudo sh -c "echo '[mysqld]' >> /etc/mysql/mariadb.conf.d/general_log.cnf" debian-9-amd64-clean-lxq1k0:~$ sudo sh -c "echo 'general_log=1' >> /etc/mysql/mariadb.conf.d/general_log.cnf" debian-9-amd64-clean-lxq1k0:~$ sudo sh -c "echo 'general_log_file=/tmp/my.log' >> /etc/mysql/mariadb.conf.d/general_log.cnf" debian-9-amd64-clean-lxq1k0:~$ sudo ./mariadb_repo_setup --mariadb-server-version=10.6 --skip-maxscale --skip-tools [info] Checking for script prerequisites. [warning] Found existing file at /etc/apt/sources.list.d/mariadb.list. Moving to /etc/apt/sources.list.d/mariadb.list.old_1 [info] Repository file successfully written to /etc/apt/sources.list.d/mariadb.list [info] Adding trusted package signing keys... [info] Running apt-get update... [info] Done adding trusted package signing keys debian-9-amd64-clean-lxq1k0:~$ sudo apt-get update Hit:1 http://security.debian.org stretch/updates InRelease Ign:2 http://deb.debian.org/debian stretch InRelease Hit:3 http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease Hit:4 http://deb.debian.org/debian stretch-updates InRelease Hit:5 http://deb.debian.org/debian stretch-backports InRelease Hit:6 http://deb.debian.org/debian stretch Release Get:7 http://packages.cloud.google.com/apt google-compute-engine-stretch-stable InRelease [5,525 B] Hit:8 http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-stretch InRelease Get:9 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch InRelease [4,642 B] Fetched 10.2 kB in 0s (14.8 kB/s) Reading package lists... Done N: Ignoring file 'mariadb.list.old_1' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension debian-9-amd64-clean-lxq1k0:~$ sudo apt-get install -y mariadb-server mariadb-client mariadb-backup mariadb-plugin-rocksdb Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: mariadb-client-10.6 mariadb-client-core-10.6 mariadb-common mariadb-server-10.6 mariadb-server-core-10.6 Suggested packages: mailx mariadb-test netcat-openbsd The following packages will be REMOVED: mariadb-client-10.5 mariadb-client-core-10.5 mariadb-server-10.5 mariadb-server-core-10.5 The following NEW packages will be installed: mariadb-client-10.6 mariadb-client-core-10.6 mariadb-server-10.6 mariadb-server-core-10.6 The following packages will be upgraded: mariadb-backup mariadb-client mariadb-common mariadb-plugin-rocksdb mariadb-server 5 upgraded, 4 newly installed, 4 to remove and 6 not upgraded. Need to get 22.9 MB of archives. After this operation, 1,434 kB of additional disk space will be used. Get:1 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-common all 1:10.6.5+maria~stretch [3,976 B] Get:2 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-client all 1:10.6.5+maria~stretch [2,622 B] Get:3 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-server all 1:10.6.5+maria~stretch [2,748 B] Get:4 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-plugin-rocksdb amd64 1:10.6.5+maria~stretch [3,113 kB] Get:5 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-backup amd64 1:10.6.5+maria~stretch [6,407 kB] Get:6 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-client-core-10.6 amd64 1:10.6.5+maria~stretch [819 kB] Get:7 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-client-10.6 amd64 1:10.6.5+maria~stretch [2,314 kB] Get:8 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-server-core-10.6 amd64 1:10.6.5+maria~stretch [6,928 kB] Get:9 https://dlm.mariadb.com/repo/mariadb-server/10.6/repo/debian stretch/main amd64 mariadb-server-10.6 amd64 1:10.6.5+maria~stretch [3,338 kB] Fetched 22.9 MB in 4s (4,772 kB/s) N: Ignoring file 'mariadb.list.old_1' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension Preconfiguring packages ... (Reading database ... 53200 files and directories currently installed.) Preparing to unpack .../mariadb-common_1%3a10.6.5+maria~stretch_all.deb ... Unpacking mariadb-common (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ... Preparing to unpack .../mariadb-client_1%3a10.6.5+maria~stretch_all.deb ... Unpacking mariadb-client (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ... Preparing to unpack .../mariadb-server_1%3a10.6.5+maria~stretch_all.deb ... Unpacking mariadb-server (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ... Preparing to unpack .../mariadb-plugin-rocksdb_1%3a10.6.5+maria~stretch_amd64.deb ... Unpacking mariadb-plugin-rocksdb (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ... (Reading database ... 53200 files and directories currently installed.) Removing mariadb-server-10.5 (1:10.5.13+maria~stretch) ... Removing mariadb-server-core-10.5 (1:10.5.13+maria~stretch) ... Removing mariadb-client-10.5 (1:10.5.13+maria~stretch) ... (Reading database ... 52900 files and directories currently installed.) Preparing to unpack .../mariadb-backup_1%3a10.6.5+maria~stretch_amd64.deb ... Unpacking mariadb-backup (1:10.6.5+maria~stretch) over (1:10.5.13+maria~stretch) ... (Reading database ... 52900 files and directories currently installed.) Removing mariadb-client-core-10.5 (1:10.5.13+maria~stretch) ... Selecting previously unselected package mariadb-client-core-10.6. (Reading database ... 52890 files and directories currently installed.) Preparing to unpack .../mariadb-client-core-10.6_1%3a10.6.5+maria~stretch_amd64.deb ... Unpacking mariadb-client-core-10.6 (1:10.6.5+maria~stretch) ... Selecting previously unselected package mariadb-client-10.6. Preparing to unpack .../mariadb-client-10.6_1%3a10.6.5+maria~stretch_amd64.deb ... Unpacking mariadb-client-10.6 (1:10.6.5+maria~stretch) ... Selecting previously unselected package mariadb-server-core-10.6. Preparing to unpack .../mariadb-server-core-10.6_1%3a10.6.5+maria~stretch_amd64.deb ... Unpacking mariadb-server-core-10.6 (1:10.6.5+maria~stretch) ... Setting up mariadb-common (1:10.6.5+maria~stretch) ... Selecting previously unselected package mariadb-server-10.6. (Reading database ... 53106 files and directories currently installed.) Preparing to unpack .../mariadb-server-10.6_1%3a10.6.5+maria~stretch_amd64.deb ... /var/lib/mysql: found previous version 10.5 Unpacking mariadb-server-10.6 (1:10.6.5+maria~stretch) ... Setting up mariadb-server-core-10.6 (1:10.6.5+maria~stretch) ... Processing triggers for systemd (232-25+deb9u13) ... Setting up mariadb-client-core-10.6 (1:10.6.5+maria~stretch) ... Processing triggers for man-db (2.7.6.1-2) ... Setting up mariadb-client-10.6 (1:10.6.5+maria~stretch) ... Setting up mariadb-client (1:10.6.5+maria~stretch) ... Setting up mariadb-server-10.6 (1:10.6.5+maria~stretch) ... Installing new version of config file /etc/init.d/mariadb ... Installing new version of config file /etc/mysql/mariadb.conf.d/50-server.cnf ... Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units. mariadb-extra.socket is a disabled or a static unit, not starting it. mariadb-extra.socket is a disabled or a static unit, not starting it. Setting up mariadb-backup (1:10.6.5+maria~stretch) ... Setting up mariadb-plugin-rocksdb (1:10.6.5+maria~stretch) ... Setting up mariadb-server (1:10.6.5+maria~stretch) ... Processing triggers for systemd (232-25+deb9u13) ... N: Ignoring file 'mariadb.list.old_1' in directory '/etc/apt/sources.list.d/' as it has an invalid filename extension debian-9-amd64-clean-lxq1k0:~$ sudo mysql -e "show databases" +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ debian-9-amd64-clean-lxq1k0:~$ sudo cat /tmp/my.log /usr/sbin/mariadbd, Version: 10.6.5-MariaDB-1:10.6.5+maria~stretch-log (mariadb.org binary distribution). started with: Tcp port: 0 Unix socket: /run/mysqld/mysqld.sock Time Id Command Argument 211122 13:14:15 3 Connect root@localhost on using Socket 3 Query show databases 3 Quit debian-9-amd64-clean-lxq1k0:~$ sudo cat /tmp/my.log /usr/sbin/mariadbd, Version: 10.6.5-MariaDB-1:10.6.5+maria~stretch-log (mariadb.org binary distribution). started with: Tcp port: 0 Unix socket: /run/mysqld/mysqld.sock Time Id Command Argument 211122 13:14:15 3 Connect root@localhost on using Socket 3 Query show databases 3 Quit debian-9-amd64-clean-lxq1k0:~$ sudo mysql_upgrade Phase 1/7: Checking and upgrading mysql database Processing databases mysql mysql.column_stats OK mysql.columns_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.global_priv OK mysql.gtid_slave_pos OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.index_stats OK mysql.innodb_index_stats OK mysql.innodb_table_stats OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.roles_mapping OK mysql.servers OK mysql.table_stats OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.transaction_registry OK Phase 2/7: Installing used storage engines... Skipped Phase 3/7: Fixing views mysql.user OK Phase 4/7: Running 'mysql_fix_privilege_tables' Phase 5/7: Fixing table and database names Phase 6/7: Checking and upgrading tables Processing databases information_schema performance_schema sys sys.sys_config OK Phase 7/7: Running 'FLUSH PRIVILEGES' OK debian-9-amd64-clean-lxq1k0:~$

Sergei Golubchik added a comment - 2021-11-22 13:46 My suggested solution to (1) above allows to solve test issues by running apt-get install mariadb-server && mysql_upgrade this will guarantee that all tables are properly upgraded and there are no concurrently running mysql_upgrade's anymore. Buildbot scripts can be amended to do it. Of course, one can ask, why do we auto-run mysql_upgrade in debian-start, if it has to be run manually anyway. And this is a good question.

Elena Stepanova added a comment - 2021-11-22 13:58 one can ask, why do we auto-run mysql_upgrade in debian-start I can think of one reason. Back to the problem of having other packages to upgrade together with the server in one transaction, the extra packages can very well expect the new server schema, not the old one. That is, when engine X gets upgraded from 10.5 to 10.6, it may need the 10.6 server schema, not 10.5, and the user cannot run mysql_upgrade in between to help with this. Of course, now when mysql_upgrade runs in the background, it doesn't achieve this goal either.

Faustin Lammler added a comment - 2021-11-22 14:30 > I don't know whether it's better to split (2) into a separate issue or handle within this one, but I hope that the final solution won't be downgraded to dealing with (1) only, like MDEV-14622 was previously downgraded to patching Spider, which of course didn't solve anything other than one of numerous side-effects of the root cause issue. elenst as there is already lots of information here about the Debian packaging problem, I suggest we handle 2 here and that's also why I have created MDEV-27107 for 1. > My suggested solution to (1) above allows to solve test issues by running > apt-get install mariadb-server && mysql_upgrade > this will guarantee that all tables are properly upgraded and there are no concurrently running mysql_upgrade's anymore. Buildbot scripts can be amended to do it. serg ok so that means to add mysql_upgrade in post-inst. What can we do with elenst above scenario? And do you have an idea how we can prevent access to the DB during post-inst mysql_upgrade run as I guess this is still required?

Sergei Golubchik added a comment - 2021-11-22 14:47 faust , that did not mean it at all. I said that if my suggestion from that link will be implemented then Buildbot can avoid mysql_upgrade concurrency problems by invoking mysql_upgrade explicitly. It does not say anything about post-inst.

Elena Stepanova added a comment - 2021-11-22 15:04 Invoking mysql_upgrade explicitly and hitting the deadlock in CIs is the least of our problems discussed here, because it's relatively easy to work around. It appears regularly in new or modified tests which are initially written optimistically, assuming it would work, but then we add all sorts of "wait till there is no mysql_upgrade process running [anymore] ", and that does the trick. Our big problem is a clash with other packages, which we can't do anything about other than stop testing the affected package altogether.

Otto Kekäläinen added a comment - 2021-11-23 05:43 Running mariadb-upgrade requires a mariadbd server that is online, so that it can accept connections and run the commands. Therefore the upgrade script is not part of the distro packaging, as there is no guarantee that an upgrade would actually end up with a new running server. On some systems the server might be intentionally shut down for the duration of the system update and in those cases it should not start at a random time. In some cases users might want the server to stay running while files update, and restart at a user defined time. Thus the mariadb-upgrade runs as part of the systemd script / SysV init, because only there we can be sure that the server is actually running. I have suggested a couple of times that the mariadbd binary should take care of its own upgrade. Relying on an external script and external stuff to trigger that script is error prone as we see here. I think it would be a better design if every time mariadbd starts it would automatically by itself do a sanity check and upgrade if it is needed. The upgrade actions of binary data / database has absolutely zero distro specific things and there is no need that distros / service scripts need to manage when and how the binary data format upgrade is triggered. Could mariadbd perhaps in a future release take care of its own upgrade so we would get rid of external mariadb-upgrade scripts?

Daniel Black added a comment - 2021-12-08 10:05 Looking at required and desired system server variables on what should happen on a temporary instances. May not be complete. Suggestions/edits welcome. innodb_buffer_pool_load_at_startup=0 - preserve unnecessary load innodb_buffer_pool_dump_at_shutdown=0 - our temporary usage isn't a standard workload innodb_force_recovery=0 - ensure the default is true innodb_buffer_pool_size - cap to a few G, especially if its just a mariadb-upgrade of system tables innodb page flushing options to more aggressively flush (mariadb-upgrade should be only altering a innodb table once) skip-networking - required skip-slave-start - (I'm assuming this stops the SQL thread) socket - non-standard required enforce_storage_engine=none skip-log-bin - disable binary logging of the mariabd-upgrade, replicas should instigate their own update wsrep_on = off flush-time = 30 (closing tables after seconds)

Michael Widenius added a comment - 2021-12-14 14:26 Otto, I don't think it is good to have the code in the server. This is perfectly doable in a system or user level script. Why do you think my proposal is not doable? (In other words, have a separate section in the systemd scripts that starts and stops the server specially for mysql_upgrade?) That is the easiest and safest thing to do). Wonder how PostgreSQL does it? (As PostgreSQL very often changes disk format and one has to dump and restore the database on major version upgrades)

Michael Widenius added a comment - 2021-12-14 15:31 This is my suggestion what Faustin (who is working on this) could start with. Let's first work on getting mysql_upgrade to support all options needed for this task: Add option '--check-if-upgrade-is-needed' to maria_upgrade the returns 0 or 1 to the operating system (trough exit call) if a full maria_upgrade should be run. Now we only check if the last run was done with the same MariaDB version as last time. Change the check so that we check if the major version has changed (10.1.xxx -> 10.2.xxx). Change the error message from: default one: "This installation of MariaDB is already upgraded to 10.7.2-MariaDB, use --force if you still need to run mysql_upgrade" to "This installation of MariaDB is already upgraded to 10.7, use --force if you still need to run mysql_upgrade" When coding this, take into account that there is a small change that we in the future may need to run mysql_upgrade for a minor version. We should have a function that check the version against a minor version (by default current_major_version.99, like 10.6.99) and if the internal version is smaller, then we still need to run upgrade. mysql_upgrade creates a 'mysql_upgrade_info' file in the data directory. When mysql_upgrade is started, take an advisory lock on the file (create it if it does not exists). This ensures we cannot have two mysql_upgrade run at the same time. Please create a separate Jira entry for the above and link it to this one. Future work: Find out why why running two mysql_upgrade in parllel causes lockups. If we can identify this and fix it, things will be much better as there is a chance that mysql_upgrade will now lock also for other cases than just another mysql_upgrade running. To make maria_upgrade faster, we should consider adding multiple thread support to mysql_check_table and run multiple check table & alter table in parallel. This should speed up mysql_upgrade for major version upgrades notable (in proportional to the number of CPU's in the system).

Daniel Black added a comment - 2021-12-15 21:02 monty can you expand on "If anyone else connects to mysqld/mariadbd during maria_upgrade, bad things can happen as the tables are not up to date for the system. The possible problems are hangs or wrong data from selects or errors from table updates (as the indexes may not be up to date)." I'm assuming the first sentence is referring to the system tables component of mysql_upgrade. The second sentence is related to the collation (and other?) user table updates. Why are collation updates needed? Where are they implemented? Is "the indexes may not up to date" referring to joining on a char column against another table? What is the consequence to the user if these user tables updates aren't performed? I'm mainly looking at these questions from a documentation point of view as to what sort of error/warning/notice to give to the user if for reasons the upgrade does only system tables updates. Other related subtasks related to minimizing unneeded mysql_upgrades: should mysql_install_db create a mysql_upgrade_info file for its current version? should mariabackup save/restore this mysql_upgrade_info? With a split of mysql_upgrade for system and user tables should these independent aspects be recorded separately rather than one encompassing mysql_upgrade_info file?

Sergei Golubchik added a comment - 2021-12-16 17:14 danblack , if we find a bug in a collation (say, "b" < "a") and we decide to fix it, then after the upgrade indexes that use this collation will be, basically, corrupted. In the index "b" will be stored before "a", while in the server "b" will be larger than "a" (after the bug fix). So all these tables will have to be repaired. Because of that, changing a collation is very expensive and we very rarely do it. May be never in MariaDB yet. But technically it can happen.

Marko Mäkelä added a comment - 2021-12-21 10:55 For what it is worth, I believe that after an incompatible change to a collation, it may become impossible to dump all records from an InnoDB table whose PRIMARY KEY would depend on the collation. This is because retrieving records from InnoDB involves storing and restoring a persistent cursor position, and that in turn will depend on key comparisons and the assumption that the index is sorted by the key.

Michael Widenius added a comment - 2022-01-20 15:16 Added --force --force option Pushed to 10.2 tree

Daniel Black added a comment - 2022-01-20 20:58 - edited monty , a second undocumented --force option is user unfriendly as it ties its functionality to the first --force. Perhaps a --skip-create-upgrade-info as a more descriptive name? Did you want to make mysql_install_db create the upgrade_info file or leave that for a separate ticket? Few review items on https://github.com/MariaDB/server/commit/0713614c3416eaa7a9faeac4707bbc810756b496 in client/mysql_upgrade.c left there from last time. Only two typos + why the use of STDCALL in calc_server_version.

Michael Widenius added a comment - 2022-01-24 17:14 - edited We have several other causes where you specify an option several times and it does more, like --version, --big --big (in mtr) etc. As this is something that one should never ever do in normal circumstances (because it leads to other (documented) troubles), adding a new flag is overkill. --force --force is also good from the point that it is backward compatible with older versions, so one can use this without having to check if the mysql_upgrade supports it or not. The option is documented in both mysql_upgrade and the KB, as it should be.

Daniel Black added a comment - 2022-01-25 03:12 ok.

Faustin Lammler added a comment - 2022-01-25 13:01 - edited > 2) Debian upgrade runs mysql_upgrade non-deterministically. It was already described in MDEV-14622 and it is still the case. Below is a proof-log of 10.5=>10.6 upgrade (official releases) I performed today on Debian 9. As you see after the package upgrade ends, sys schema wasn't created, general log is empty, which indicates mysql_upgrade wasn't even started. I have to run it manually to finalize the upgrade. When I did exactly the same on, say, Ubuntu 20.04, sys schema was there and the general log was full of mysql_upgrade stuff. I can't say whether the same happens every time on a certain debian/ubuntu flavor or it is sporadic, but it is definitely possible to end up with not upgraded schema. elenst did you noticed this behavior with other Debian/Ubuntu versions? I can confirm it on Debian 9 but I am not able to reproduce it with something else than Debian 9.

Elena Stepanova added a comment - 2022-01-25 14:58 Elena Stepanova did you noticed this behavior with other Debian/Ubuntu versions? I can't say one way or another. If I did, I wouldn't necessarily register it because I find it irrelevant. Unless somebody knows about a specific bug fixed in all other versions so that the fix has magically made this code behave deterministically, relying on better/more lucky timing of certain versions over others isn't wise.

Otto Kekäläinen added a comment - 2023-07-23 21:02 The argumentation in the comments of this Jira is basically pros and cons for implementing MDEV-30499 . After thinking about this long and hard, I still stand by the arguments I listed in https://jira.mariadb.org/browse/MDEV-27068?focusedCommentId=206749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-206749 and think it would be best if mariadbd server itself took care of the upgrade without relying on external scripts or users to run mariadb-upgrade. Such a dependency is simply too brittle.