[MDEV-32182] Columnstore restarts the server in the middle of upgrade, causes failures Created: 2023-09-15  Updated: 2023-11-28

Status: Open
Project: MariaDB Server
Component/s: Packaging, Storage Engine - ColumnStore
Affects Version/s: 10.10, 10.11, 11.0, 11.1, 11.2
Fix Version/s: 10.11, 11.0, 11.1, 11.2

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Unresolved Votes: 0
Labels: None


 Description   

I didn't try 10.6 or earlier servers, possibly they have the same problem. At least the same applies both to 6.x and 23.x Columnstore. It is also possible that it can happen with some other plugin/engine instead of the Columnstore, but so far I didn't encounter it. RocksDB in the scenario is replaceable with any engine/plugin with strict dependency on the server version.

Let's have Columnstore and RocksDB installed.

$ sudo rpm -qa | grep MariaDB
 
MariaDB-common-10.10.5-1.el8.x86_64
 
MariaDB-server-10.10.5-1.el8.x86_64
 
MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64
 
MariaDB-shared-10.10.5-1.el8.x86_64
 
MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64
 
MariaDB-client-10.10.5-1.el8.x86_64

For a more obvious effect, let's have a rocksdb-specific option in the config:

$ sudo cat /etc/my.cnf.d/rocksdb.cnf 
[mariadb]
 
plugin-load-add=ha_rocksdb.so
 
rocksdb_deadlock_detect=1

Now upgrade:

$ sudo dnf -y upgrade MariaDB-server MariaDB-rocksdb-engine MariaDB-columnstore-engine


Running transaction
 
  Preparing        :                                                                                                                                                                  1/1 
  Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64                                                                                                                              1/1 
  Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64                                                                                                                              1/6 
  Upgrading        : MariaDB-server-10.10.6-1.el8.x86_64                                                                                                                              1/6 
  Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64                                                                                                                              1/6 
  Running scriptlet: MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64                                                                                                            2/6 
 
Mariab Columnstore uninstall completed
 
 
 
  Upgrading        : MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64                                                                                                            2/6 
  Running scriptlet: MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64                                                                                                            2/6 
checking for engine columnstore...
 
columnstore doesn't exist
 
There was an error installing MariaDB ColumnStore engine plugin. Continue to install the engine though. Please resolve the issues and run necessary scripts manually.
 
Populating the engine initial system catalog.
 
 
 
  Running scriptlet: MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64                                                                                                                      3/6 
  Upgrading        : MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64                                                                                                                      3/6 
  Running scriptlet: MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64                                                                                                                      3/6 
  Running scriptlet: MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64                                                                                                                      4/6 
  Cleanup          : MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64                                                                                                                      4/6 
  Running scriptlet: MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64                                                                                                                      4/6 
  Running scriptlet: MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64                                                                                                            5/6 
  Cleanup          : MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64                                                                                                            5/6 
  Running scriptlet: MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64                                                                                                            5/6 
  Running scriptlet: MariaDB-server-10.10.5-1.el8.x86_64                                                                                                                              6/6 
  Cleanup          : MariaDB-server-10.10.5-1.el8.x86_64                                                                                                                              6/6 
  Running scriptlet: MariaDB-server-10.10.5-1.el8.x86_64                                                                                                                              6/6 
  Running scriptlet: MariaDB-server-10.10.6-1.el8.x86_64                                                                                                                              6/6 
failed
 
 
 
  Running scriptlet: MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64                                                                                                            6/6 
  Running scriptlet: MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64                                                                                                                      6/6 
  Running scriptlet: MariaDB-server-10.10.5-1.el8.x86_64                                                                                                                              6/6 
  Verifying        : MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64                                                                                                            1/6 
  Verifying        : MariaDB-columnstore-engine-10.10.5_6.4.7-1.el8.x86_64                                                                                                            2/6 
  Verifying        : MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64                                                                                                                      3/6 
  Verifying        : MariaDB-rocksdb-engine-10.10.5-1.el8.x86_64                                                                                                                      4/6 
  Verifying        : MariaDB-server-10.10.6-1.el8.x86_64                                                                                                                              5/6 
  Verifying        : MariaDB-server-10.10.5-1.el8.x86_64                                                                                                                              6/6 
 
 
Upgraded:
 
  MariaDB-columnstore-engine-10.10.6_6.4.7-1.el8.x86_64                  MariaDB-rocksdb-engine-10.10.6-1.el8.x86_64                  MariaDB-server-10.10.6-1.el8.x86_64            
 
 
Complete!

So, first the server upgrades from 10.10.5 to 10.10.6.
Then Columnstore starts upgrading, and in the process it wants to restart the server. But RocksDB is still 10.10.5, it doesn't load with the server 10.10.6, and since there is a rocksdb option, the server doesn't start at all. 

Sep 15 18:02:39 rocky84-amd64 systemd[1]: Starting MariaDB 10.10.6 database server...
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] Starting MariaDB 10.10.6-MariaDB source revision cce155cc901a573d68ccda856bbf53e90edef15d as process 191772
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] mariadbd: Can't open shared library 'ha_rocksdb.so' (errno: 8, API version for STORAGE ENGINE plugin ROCKSDB
 
 not supported by this version of the server)
 
...
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] /usr/sbin/mariadbd: unknown variable 'rocksdb_deadlock_detect=1'
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] Aborting

Columnstore somehow survives it, the process continues, and all three packages get upgraded at the end. However, the server remains down, even though it should have been started after upgrade (because it was running before the upgrade):

[buildbot@rocky84-amd64 ~]$ sudo systemctl status mariadb
 
● mariadb.service - MariaDB 10.10.6 database server
 
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
 
  Drop-In: /etc/systemd/system/mariadb.service.d
 
           └─migrated-from-my.cnf-settings.conf
 
   Active: failed (Result: exit-code) since Fri 2023-09-15 18:02:39 UTC; 17s ago
 
     Docs: man:mariadbd(8)
 
           https://mariadb.com/kb/en/library/systemd/
 
 Main PID: 191772 (code=exited, status=7)
 
   Status: "MariaDB server is down"
 
 
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: log sequence number 72336; transaction id 59
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [Note] Plugin 'FEEDBACK' is disabled.
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] /usr/sbin/mariadbd: unknown variable 'rocksdb_deadlock_detect=1'
 
Sep 15 18:02:39 rocky84-amd64 mariadbd[191772]: 2023-09-15 18:02:39 0 [ERROR] Aborting
 
Sep 15 18:02:39 rocky84-amd64 systemd[1]: mariadb.service: Main process exited, code=exited, status=7/NOTRUNNING
 
Sep 15 18:02:39 rocky84-amd64 systemd[1]: mariadb.service: Failed with result 'exit-code'.
 
Sep 15 18:02:39 rocky84-amd64 systemd[1]: Failed to start MariaDB 10.10.6 database server.

It can now be started manually, but any upgrade process checking for the running server will fail, and in general, a bunch of errors like this doesn't make a good impression.
Generated at Thu Feb 08 10:29:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.