[MCOL-4992] cmapi packages with single node installation causes DBRM readonly error Created: 2022-02-09  Updated: 2022-10-26  Resolved: 2022-08-17

Status: Closed
Project: MariaDB ColumnStore
Component/s: cmapi
Affects Version/s: 6.2.2, 6.3.1, cmapi-1.5, cmapi-1.6
Fix Version/s: cmapi-22.08.1

Type: Bug Priority: Major
Reporter: Daniel Lee (Inactive) Assignee: Alan Mologorsky
Resolution: Won't Fix Votes: 0
Labels: None

Sprint: 2021-17

 Description   

Build tested: MCS 6.2.2, 6.3.1
cmapi: 1.5, 1.6
OS: I have been testing with Ubuntu 18.04.

For single node installation, cmapi package is installed, the node would end up in a DBRM readonly state. Starting the cmapi server or not does not make a difference.

The simplest way to reproduce this is to just install all the deb packages, then run DDL statements. DBRM readonly error will occur.

Although cmapi is not required for single-node installation, there is a high chance that user's would just install all packages download, causing this issue.

I did not see the same issue with multiple-node installation, which requires cmapi.

So far, I did the following columnstore and cmapi combinations:

6.2.2-1 1.4 single-node passed
6.2.2-1 1.4 multi-node passed

6.2.2-1 1.5 singl-enode failed
6.2.2-1 1.5 multi-node passed

6.3.1-1 1.5 single-node failed
6.3.1-1 1.5 multi-node passed

6.3.1-1 1.6 single-node failed
6.3.1-1 1.6 multi-node passed

The issue was not identified previously because for single-node installation, cmapi was not installed since it was not required.

 Comments   
Comment by Daniel Lee (Inactive) [ 2022-02-12 ]

So far I tested only on Centos 8 and Ubuntu18. It worked on Centos 8 but failed on Ubuntu18.

This issue occurs as long as I installed the CMAPI package file. I did not add a node since it is a single cluster and I did not run any other CMAPI commands. I did not even need to start the CMAPI server service.

[ubuntu18:root~]# curl https://127.0.0.1:8640/cmapi/0.4.0/node/new_primary --header 'Content-Type:application/json' --header 'x-api-key:somekey123' -k | jq
 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 
                                 Dload  Upload   Total   Spent    Left  Speed
 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to 127.0.0.1 port 8640: Connection refused

If I do start the CMAPI server service, then

[ubuntu18:root~]# curl https://127.0.0.1:8640/cmapi/0.4.0/node/new_primary --header 'Content-Type:application/json' --header 'x-api-key:somekey123' -k | jq
 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 
                                 Dload  Upload   Total   Spent    Left  Speed
 
100    20  100    20    0     0    526      0 --:--:-- --:--:-- --:--:--   526
 
{
 
  "is_primary": true
 
}

The main point here is that this is a single node installation. CMAPI was not used and no cluster was configured. Simply installing the CMAPI would cause this DBRM read-only error.

Comment by Daniel Lee (Inactive) [ 2022-02-18 ]

More details on the issue:

When installing CMAPI, CMAPI will set the mariadb-columnstore service to masked, which would prevent it from starting. This becomes an issue for single-node installation, which uses systemctl to 'start' and 'stop' the mariadb-columnstore service. For multiple installation, ColumnStore uses CMAPI instead of systemctl to 'start' and 'shutdown' cluster. Therefore, multi-node cluster is not affected.

The following work around can be used to avoid this issue

1. Don't install CMAPI.
2. If already installed, set mariadb-columnstore sevice to unmasked, then start it using systemctl

[ubuntu18:root~]# systemctl start mariadb-columnstore
 
Failed to start mariadb-columnstore.service: Unit mariadb-columnstore.service is masked.
 
[ubuntu18:root~]# systemctl unmask mariadb-columnstore
 
Removed /etc/systemd/system/mariadb-columnstore.service.
 
[ubuntu18:root~]# systemctl start mariadb-columnstore
 
[ubuntu18:root~]# systemctl status mariadb-columnstore
 
● mariadb-columnstore.service - mariadb-columnstore
 
   Loaded: loaded (/lib/systemd/system/mariadb-columnstore.service; enabled; vendor preset: enabled)
 
   Active: active (exited) since Fri 2022-02-18 16:58:24 UTC; 13min ago
 
 Main PID: 5451 (code=exited, status=0/SUCCESS)
 
    Tasks: 0 (limit: 2336)
 
   CGroup: /mariadb-columnstore.service
 
 
 
Feb 18 16:58:23 vagrant systemd[1]: Starting mariadb-columnstore...
 
Feb 18 16:58:24 vagrant su[5632]: Successful su for mysql by root
 
Feb 18 16:58:24 vagrant su[5632]: + ??? root:mysql
 
Feb 18 16:58:24 vagrant su[5632]: pam_unix(su:session): session opened for user mysql by (uid=0)
 
Feb 18 16:58:24 vagrant su[5632]: pam_unix(su:session): session closed for user mysql
 
Feb 18 16:58:24 vagrant systemd[1]: Started mariadb-columnstore.

Comment by Alan Mologorsky [ 2022-06-14 ]

Looks like this is corner case when someone's trying to install packages in a blind manner: `sudo apt -y install ./*.deb`

So as ben.thompson suggested setting up mariadb-columnstore as a dependency for cmapi will solve the issue.

drrtuy What do you think? Looks like this could be a one liner improvement, but do we need that?

Generated at Thu Feb 08 02:54:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.