1.4.2 multi-node with gluster failing to start via postConfigure - gluster assign/unassign errors

Setting up a 2um/3pm gluster system on Centos 8 and it failed to start via postConfigure.

Setup 5 EC2 instances with 1.4.2 installed and post-install

Installed gluster on the PMS following this centos 8 procedure

https://computingforgeeks.com/setup-glusterfs-storage-with-heketi-on-centos-server/

sudo gluster --version
glusterfs 6.7

sudo mcsadmin getso

getsoftwareinfo Thu Feb 27 17:44:29 2020

SoftwareVersion = 1.4.2
SoftwareRelease = 1

kicked of postConfigure on pm1

sudo postConfigure

This is the MariaDB ColumnStore System Configuration and Installation tool.
It will Configure the MariaDB ColumnStore System and will perform a Package
Installation of all of the Servers within the System that is being configured.

IMPORTANT: This tool requires to run on the Performance Module #1

Prompting instructions:

Press 'enter' to accept a value in (), if available or
Enter one of the options within [], if available, or
Enter a new value

===== Setup System Server Type Configuration =====

There are 2 options when configuring the System Server Type: single and multi

'single' - Single-Server install is used when there will only be 1 server configured
on the system. It can also be used for production systems, if the plan is
to stay single-server.

'multi' - Multi-Server install is used when you want to configure multiple servers now or
in the future. With Multi-Server install, you can still configure just 1 server
now and add on addition servers/modules in the future.

Select the type of System Server install [1=single, 2=multi] (2) >

===== Setup System Module Type Configuration =====

There are 2 options when configuring the System Module Type: separate and combined

'separate' - User and Performance functionality on separate servers.

'combined' - User and Performance functionality on the same server

Select the type of System Module Install [1=separate, 2=combined] (1) >

Seperate Server Installation will be performed.

NOTE: Local Query Feature allows the ability to query data from a single Performance
Module. Check MariaDB ColumnStore Admin Guide for additional information.

Enable Local Query feature? [y,n] >

NOTE: The MariaDB ColumnStore Schema Sync feature will replicate all of the
schemas and InnoDB tables across the User Module nodes. This feature can be enabled
or disabled, for example, if you wish to configure your own replication post installation.

MariaDB ColumnStore Schema Sync feature is Enabled, do you want to leave enabled? [y,n] >

NOTE: MariaDB ColumnStore Replication Feature is enabled

Enter System Name (columnstore-1) >

===== Setup Storage Configuration =====

----- Setup Performance Module DBRoot Data Storage Mount Configuration -----

Columnstore supports the following storage options...
1 - internal. This uses the linux VFS to access files and does
not manage the filesystem.
2 - external *. If you have other mountable filesystems you would
like ColumnStore to use & manage, select this option.
3 - GlusterFS * Note: glusterd service must be running and enabled on
all PMs.
4 - S3-compatible cloud storage *. Note: that should be configured
before running postConfigure (see storagemanager.cnf)

• - This option enables data replication and server failover in a
  multi-node configuration.

These options are available on this system: [1, 2, 3, 4]
Select the type of data storage (1) > 3

===== Setup Memory Configuration =====

NOTE: Setting 'NumBlocksPct' to 70%
Setting 'TotalUmMemory' to 50%

===== Setup the Module Configuration =====

----- User Module Configuration -----

Enter number of User Modules [1,1024] (1) > 2

• • • User Module #1 Configuration ***

Enter Nic Interface #1 Host Name (unassigned) > um1
Enter Nic Interface #1 IP Address or hostname of um1 (0.0.0.0) > 172.30.0.128
Enter Nic Interface #2 Host Name (unassigned) >

• • • User Module #2 Configuration ***

Enter Nic Interface #1 Host Name (unassigned) > um2
Enter Nic Interface #1 IP Address or hostname of um2 (0.0.0.0) > 172.30.0.102
Enter Nic Interface #2 Host Name (unassigned) >

----- Performance Module Configuration -----

Enter number of Performance Modules [1,1024] (1) > 3

• • • Parent OAM Module Performance Module #1 Configuration ***

Enter Nic Interface #1 Host Name (ip-172-30-0-250.ec2.internal) > pm1
Enter Nic Interface #1 IP Address or hostname of pm1 (0.0.0.0) > 172.30.0.250
Enter Nic Interface #2 Host Name (unassigned) >
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm1' (1) >

• • • Performance Module #2 Configuration ***

Enter Nic Interface #1 Host Name (unassigned) > pm2
Enter Nic Interface #1 IP Address or hostname of pm2 (0.0.0.0) > 172.30.0.204
Enter Nic Interface #2 Host Name (unassigned) >
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm2' () > 2

• • • Performance Module #3 Configuration ***

Enter Nic Interface #1 Host Name (unassigned) > pm3
Enter Nic Interface #1 IP Address or hostname of pm3 (0.0.0.0) > 172.30.0.229
Enter Nic Interface #2 Host Name (unassigned) >
Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm3' () > 3

Next step is to enter the password to access the other Servers.
This is either user password or you can default to using a ssh key
If using a user password, the password needs to be the same on all Servers.

Enter password, hit 'enter' to default to using a ssh key, or 'exit' >

----- Performing Install on 'um1 / um1' -----

Install log file is located here: /tmp/columnstore_tmp_files/um1_binary_install.log

----- Performing Install on 'um2 / um2' -----

Install log file is located here: /tmp/columnstore_tmp_files/um2_binary_install.log

----- Performing Install on 'pm2 / pm2' -----

Install log file is located here: /tmp/columnstore_tmp_files/pm2_binary_install.log

----- Performing Install on 'pm3 / pm3' -----

Install log file is located here: /tmp/columnstore_tmp_files/pm3_binary_install.log

===== Configuring MariaDB ColumnStore Data Redundancy Functionality =====

----- Setup Data Redundancy Copy Count Configuration -----

Setup the Number of Copies: This is the total number of copies of the data
in the system. At least 2, but not more than the number of PMs(3), are required.

Enter Number of Copies [2-3] (2) > 3

----- Setup Data Redundancy Network Configuration -----

'existing' - This is specified when using previously configured network devices. (NIC Interface #1)
No additional network configuration is required with this option.

'dedicated' - This is specified when it is desired for Data Redundancy traffic to use
a separate network than one previously configured for ColumnStore.
You will be prompted to provide Hostname and IP information for each PM.

Select the data redundancy network [1=existing, 2=dedicated] (1) >

----- Performing Data Redundancy Configuration -----

gluster peer probe 172.30.0.250
gluster peer probe 172.30.0.204
gluster peer probe 172.30.0.229
Gluster create and start volume dbroot1...DONE
Gluster create and start volume dbroot2...DONE
Gluster create and start volume dbroot3...DONE

----- Data Redundancy Configuration Complete -----

===== Checking MariaDB ColumnStore System Logging Functionality =====

The MariaDB ColumnStore system logging is setup and working on local server

MariaDB ColumnStore System Configuration and Installation is Completed

===== MariaDB ColumnStore System Startup =====

System Configuration is complete.
Performing System Installation.

----- Starting MariaDB ColumnStore on local server -----

MariaDB ColumnStore successfully started

MariaDB ColumnStore Database Platform Starting, please wait .............

TIMEOUT: ProcMon not responding to getSystemStatus FAILED

IMPORTANT: There was a system startup failed, once issue has been resolved, rerun postConfigure

MariaDB ColumnStore System failed to start, check log files in /var/log/mariadb/columnstore
Enter the following command to define MariaDB ColumnStore Alias Commands

. /etc/profile.d/columnstoreAlias.sh

Enter 'mariadb' to access the MariaDB ColumnStore SQL console
Enter 'mcsadmin' to access the MariaDB ColumnStore Admin console

NOTE: The MariaDB ColumnStore Alias Commands are in /etc/profile.d/columnstoreAlias

[centos@ip-172-30-0-250 ~]$ sudo ma
sudo: ma: command not found
[centos@ip-172-30-0-250 ~]$ sudo mcsadmin

MariaDB ColumnStore Admin Console
enter 'help' for list of commands
enter 'exit' to exit the MariaDB ColumnStore Command Console
use up/down arrows to recall commands

Active Alarm Counts: Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0

Critical Active Alarms:

mcsadmin> getsystemi
getsysteminfo Thu Feb 27 17:12:41 2020

System columnstore-1

System and Module statuses

Component Status Last Status Change
------------ -------------------------- ------------------------
System FAILED Thu Feb 27 17:11:08 2020

Module um1 MAN_INIT Thu Feb 27 17:10:41 2020
Module um2 MAN_INIT Thu Feb 27 17:10:46 2020
Module pm1 MAN_INIT Thu Feb 27 17:10:36 2020
Module pm2 UP Thu Feb 27 17:09:36 2020
Module pm3 UP Thu Feb 27 17:09:36 2020

Active Parent OAM Performance Module is 'pm1'
Primary Front-End MariaDB ColumnStore Module is 'um1'
MariaDB ColumnStore Replication Feature is enabled

MariaDB ColumnStore Process statuses

Process Module Status Last Status Change Process ID
------------------ ------ --------------- ------------------------ ----------
ProcessMonitor um1 ACTIVE Thu Feb 27 17:10:17 2020 2789
ServerMonitor um1 ACTIVE Thu Feb 27 17:10:45 2020 3127
DBRMWorkerNode um1 INITIAL
ExeMgr um1 INITIAL
DDLProc um1 INITIAL
DMLProc um1 INITIAL
mysqld um1 ACTIVE Thu Feb 27 17:10:43 2020 3085

ProcessMonitor um2 ACTIVE Thu Feb 27 17:10:18 2020 2780
ServerMonitor um2 ACTIVE Thu Feb 27 17:10:49 2020 3136
DBRMWorkerNode um2 INITIAL
ExeMgr um2 INITIAL
DDLProc um2 INITIAL
DMLProc um2 INITIAL
mysqld um2 ACTIVE Thu Feb 27 17:10:47 2020 3094

ProcessMonitor pm1 ACTIVE Thu Feb 27 17:09:34 2020 3230
ProcessManager pm1 ACTIVE Thu Feb 27 17:10:56 2020 5542
DBRMControllerNode pm1 ACTIVE Thu Feb 27 17:10:50 2020 5478
ServerMonitor pm1 ACTIVE Thu Feb 27 17:10:51 2020 5517
DBRMWorkerNode pm1 ACTIVE Thu Feb 27 17:10:52 2020 5554
PrimProc pm1 ACTIVE Thu Feb 27 17:10:56 2020 5698
WriteEngineServer pm1 ACTIVE Thu Feb 27 17:10:57 2020 5752

ProcessMonitor pm2 ACTIVE Thu Feb 27 17:10:28 2020 2735
ProcessManager pm2 HOT_STANDBY Thu Feb 27 17:10:36 2020 2923
DBRMControllerNode pm2 INITIAL
ServerMonitor pm2 INITIAL
DBRMWorkerNode pm2 INITIAL
PrimProc pm2 INITIAL
WriteEngineServer pm2 INITIAL

ProcessMonitor pm3 ACTIVE Thu Feb 27 17:10:30 2020 2736
ProcessManager pm3 COLD_STANDBY Thu Feb 27 17:10:29 2020
DBRMControllerNode pm3 INITIAL
ServerMonitor pm3 INITIAL
DBRMWorkerNode pm3 INITIAL
PrimProc pm3 INITIAL
WriteEngineServer pm3 INITIAL

Active Alarm Counts: Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0
mcsadmin> exit

sudo cat crit.log
Feb 27 17:10:49 ip-172-30-0-250 ProcessMonitor[3230]: 49.797238 |0|0|0| C 18 CAL0000: *****MariaDB ColumnStore Process Restarting: ProcessManager, old PID = 3421
Feb 27 17:11:08 ip-172-30-0-250 ProcessManager[5542]: 08.048648 |0|0|0| C 17 CAL0000: startMgrProcessThread Exit with a failure, error returned from startSystemThread
[centos@ip-172-30-0-250 columnstore]$ sudo cat err.log
Feb 27 17:09:17 ip-172-30-0-250 ProcessMonitor[3230]: 17.603804 |0|0|0| E 18 CAL0000: glusterAssign mount failure: dbroot: 1 error: 1
Feb 27 17:09:17 ip-172-30-0-250 ProcessMonitor[3230]: 17.603903 |0|0|0| E 18 CAL0000: glusterAssign failed.
Feb 27 17:09:17 ip-172-30-0-250 ProcessMonitor[3230]: 17.609714 |0|0|0| E 18 CAL0000: Error assigning gluster dbroot# 1
Feb 27 17:09:17 ip-172-30-0-250 ProcessMonitor[3230]: 17.798631 |0|0|0| E 18 CAL0000: glusterUnassign mount failure: dbroot: 2 error: 32
Feb 27 17:09:17 ip-172-30-0-250 ProcessMonitor[3230]: 17.802900 |0|0|0| E 18 CAL0000: glusterUnassign mount failure: dbroot: 3 error: 32
Feb 27 17:10:47 ip-172-30-0-250 ProcessMonitor[3230]: 47.604827 |0|0|0| E 18 CAL0000: glusterAssign mount failure: dbroot: 1 error: 1
Feb 27 17:10:47 ip-172-30-0-250 ProcessMonitor[3230]: 47.604918 |0|0|0| E 18 CAL0000: glusterAssign failed.
Feb 27 17:10:47 ip-172-30-0-250 ProcessMonitor[3230]: 47.607492 |0|0|0| E 18 CAL0000: Error assigning gluster dbroot# 1
Feb 27 17:10:47 ip-172-30-0-250 ProcessMonitor[3230]: 47.756645 |0|0|0| E 18 CAL0000: glusterUnassign mount failure: dbroot: 2 error: 32
Feb 27 17:10:47 ip-172-30-0-250 ProcessMonitor[3230]: 47.761881 |0|0|0| E 18 CAL0000: glusterUnassign mount failure: dbroot: 3 error: 32
Feb 27 17:10:49 ip-172-30-0-250 ProcessMonitor[3230]: 49.797238 |0|0|0| C 18 CAL0000: *****MariaDB ColumnStore Process Restarting: ProcessManager, old PID = 3421
Feb 27 17:11:08 ip-172-30-0-250 ProcessManager[5542]: 08.048648 |0|0|0| C 17 CAL0000: startMgrProcessThread Exit with a failure, error returned from startSystemThread

ll /var/lib/columnstore/
data/ data1/ data2/ data3/ gluster/ local/
[root@ip-172-30-0-250 ~]# ll /var/lib/columnstore/
total 0
drwxr-xr-x. 3 root root 18 Feb 4 17:12 data
drwxr-xr-x. 3 root root 47 Feb 27 17:09 data1
drwxr-xr-x 2 root root 6 Feb 27 17:00 data2
drwxr-xr-x 2 root root 6 Feb 27 17:00 data3
drwxr-xr-x 5 root root 48 Feb 27 17:08 gluster
drwxr-xr-x. 3 root root 51 Feb 27 17:10 local
[root@ip-172-30-0-250 ~]# cd /var/lib/columnstore/
[root@ip-172-30-0-250 columnstore]# cd gluster/
[root@ip-172-30-0-250 gluster]# ll
total 0
drwxr-xr-x 3 root root 24 Feb 27 17:08 brick1
drwxr-xr-x 3 root root 24 Feb 27 17:08 brick2
drwxr-xr-x 3 root root 24 Feb 27 17:09 brick3
[root@ip-172-30-0-250 gluster]# cat /etc/fstab
LABEL=root / xfs defaults 0 0