Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.10, 10.11, 11.0, 11.1, 11.2
-
None
Description
I didn't try 10.6 or earlier servers, possibly they have the same problem. At least the same applies both to 6.x and 23.x Columnstore. It is also possible that it can happen with some other plugin/engine instead of the Columnstore, but so far I didn't encounter it. RocksDB in the scenario is replaceable with any engine/plugin with strict dependency on the server version.
Let's have Columnstore and RocksDB installed.
$ sudo rpm -qa | grep MariaDB
|
MariaDB-common-10.10.5-1.el8.x86_64
|
MariaDB-server-10.10.5-1.el8.x86_64
|
MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64
|
MariaDB-shared-10.10.5-1.el8.x86_64
|
MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64
|
MariaDB-client-10.10.5-1.el8.x86_64
|
For a more obvious effect, let's have a rocksdb-specific option in the config:
$ sudo cat /etc/my.cnf.d/rocksdb.cnf
|
[mariadb]
|
plugin-load-add=ha_rocksdb.so
|
rocksdb_deadlock_detect=1
|
Now upgrade:
$ sudo dnf -y upgrade MariaDB-server MariaDB-rocksdb-engine MariaDB-columnstore-engine
|
Running transaction
|
Preparing : 1/1
|
Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64 1/1
|
Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64 1/6
|
Upgrading : MariaDB-server-10.10.6-1.el8.x86_64 1/6
|
Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64 1/6
|
Running scriptlet: MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64 2/6
|
|
Mariab Columnstore uninstall completed
|
|
|
Upgrading : MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64 2/6
|
Running scriptlet: MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64 2/6
|
checking for engine columnstore...
|
columnstore doesn't exist
|
There was an error installing MariaDB ColumnStore engine plugin. Continue to install the engine though. Please resolve the issues and run necessary scripts manually.
|
Populating the engine initial system catalog.
|
|
|
Running scriptlet: MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64 3/6
|
Upgrading : MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64 3/6
|
Running scriptlet: MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64 3/6
|
Running scriptlet: MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64 4/6
|
Cleanup : MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64 4/6
|
Running scriptlet: MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64 4/6
|
Running scriptlet: MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64 5/6
|
Cleanup : MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64 5/6
|
Running scriptlet: MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64 5/6
|
Running scriptlet: MariaDB-server-10.10.5-1.el8.x86_64 6/6
|
Cleanup : MariaDB-server-10.10.5-1.el8.x86_64 6/6
|
Running scriptlet: MariaDB-server-10.10.5-1.el8.x86_64 6/6
|
Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64 6/6
|
failed
|
|
|
Running scriptlet: MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64 6/6
|
Running scriptlet: MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64 6/6
|
Running scriptlet: MariaDB-server-10.10.5-1.el8.x86_64 6/6
|
Verifying : MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64 1/6
|
Verifying : MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64 2/6
|
Verifying : MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64 3/6
|
Verifying : MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64 4/6
|
Verifying : MariaDB-server-10.10.6-1.el8.x86_64 5/6
|
Verifying : MariaDB-server-10.10.5-1.el8.x86_64 6/6
|
|
|
Upgraded:
|
MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64 MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64 MariaDB-server-10.10.6-1.el8.x86_64
|
|
|
Complete!
|
So, first the server upgrades from 10.10.5 to 10.10.6.
Then Columnstore starts upgrading, and in the process it wants to restart the server. But RocksDB is still 10.10.5, it doesn't load with the server 10.10.6, and since there is a rocksdb option, the server doesn't start at all.
Sep 15 18:02:39 rocky84-amd64 systemd[1]: Starting MariaDB 10.10.6 database server...
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] Starting MariaDB 10.10.6-MariaDB source revision cce155cc901a573d68ccda856bbf53e90edef15d as process 191772
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] mariadbd: Can't open shared library 'ha_rocksdb.so' (errno: 8, API version for STORAGE ENGINE plugin ROCKSDB
|
not supported by this version of the server)
|
...
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] /usr/sbin/mariadbd: unknown variable 'rocksdb_deadlock_detect=1'
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] Aborting
|
Columnstore somehow survives it, the process continues, and all three packages get upgraded at the end. However, the server remains down, even though it should have been started after upgrade (because it was running before the upgrade):
[buildbot@rocky84-amd64 ~]$ sudo systemctl status mariadb
|
● mariadb.service - MariaDB 10.10.6 database server
|
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
|
Drop-In: /etc/systemd/system/mariadb.service.d
|
└─migrated-from-my.cnf-settings.conf
|
Active: failed (Result: exit-code) since Fri 2023-09-15 18:02:39 UTC; 17s ago
|
Docs: man:mariadbd(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Main PID: 191772 (code=exited, status=7)
|
Status: "MariaDB server is down"
|
|
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: log sequence number 72336; transaction id 59
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] Plugin 'FEEDBACK' is disabled.
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] /usr/sbin/mariadbd: unknown variable 'rocksdb_deadlock_detect=1'
|
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] Aborting
|
Sep 15 18:02:39 rocky84-amd64 systemd[1]: mariadb.service: Main process exited, code=exited, status=7/NOTRUNNING
|
Sep 15 18:02:39 rocky84-amd64 systemd[1]: mariadb.service: Failed with result 'exit-code'.
|
Sep 15 18:02:39 rocky84-amd64 systemd[1]: Failed to start MariaDB 10.10.6 database server.
|
It can now be started manually, but any upgrade process checking for the running server will fail, and in general, a bunch of errors like this doesn't make a good impression.