in testing failover based on a user scenario, ran into a situation where the system become unusable.
First got the the problem in MCOL-1139 where had a defective PM1 Nd it was removed from the system. This left the storage in a bad state.
Then from pm3, did a shutdown and planned to rerun postConfigure to fix. But then I got the below error message..
so either need to document this issue that a pm1 server needs to be added back into the system or we change the logic where we require postConfigure from running on pm1.
getst
getstorageconfig Fri Jan 5 16:40:54 2018
System Storage Configuration
Performance Module (DBRoot) Storage Type = external
User Module Storage Type = internal
System Assigned DBRoot Count = 3
DBRoot IDs assigned to 'pm2' = 2
DBRoot IDs unassigned = 1, 3
Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot2: vol-0f2fa799bd8450525, /dev/sdh, /dev/xvdh
Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot1: vol-072d7d6b55ee398e9, /dev/sdg, /dev/xvdg
Amazon EC2 Volume Name/Device Name/Amazon Device Name for DBRoot3: vol-0a1492a16c1d4404e, /dev/sdi, /dev/xvdi
mcsadmin> shutd y
shutdownsystem Fri Jan 5 16:42:36 2018
This command stops the processing of applications on all Modules within the MariaDB ColumnStore System
Checking for active transactions
Stopping System...
Successful stop of System
Shutting Down System...
Successful shutdown of System
mcsadmin> exit
exit Fri Jan 5 16:43:23 2018
Exiting the MariaDB ColumnStore Admin Console
[mariadb-user@ip-172-30-0-129 ~]$ module
pm3
[mariadb-user@ip-172-30-0-129 ~]$
[mariadb-user@ip-172-30-0-129 ~]$
[mariadb-user@ip-172-30-0-129 ~]$ /home/mariadb-user/mariadb/columnstore/bin/postConfigure -i /home/mariadb-user/mariadb/columnstore
This is the MariaDB ColumnStore System Configuration and Installation tool.
It will Configure the MariaDB ColumnStore System and will perform a Package
Installation of all of the Servers within the System that is being configured.
IMPORTANT: This tool should only be run on the Parent OAM Module
which is a Performance Module, preferred Module #1
Prompting instructions:
Press 'enter' to accept a value in (), if available or
Enter one of the options within [], if available, or
Enter a new value
A copy of the MariaDB ColumnStore Configuration file has been saved during Package install.
It's Configured for a Multi-Server Install.
You have an option of utilizing the configuration data from that file or starting
with the MariaDB ColumnStore Configuration File that comes with the MariaDB ColumnStore Package.
You will only want to utilize the old configuration data when performing the same
type of install, i.e. Single or Multi-Server
Do you want to utilize the configuration data from the saved copy? [y,n] > n
===== Setup System Server Type Configuration =====
There are 2 options when configuring the System Server Type: single and multi
'single' - Single-Server install is used when there will only be 1 server configured
on the system. It can also be used for production systems, if the plan is
to stay single-server.
'multi' - Multi-Server install is used when you want to configure multiple servers now or
in the future. With Multi-Server install, you can still configure just 1 server
now and add on addition servers/modules in the future.
Select the type of System Server install [1=single, 2=multi] (2) >
===== Setup System Module Type Configuration =====
There are 2 options when configuring the System Module Type: separate and combined
'separate' - User and Performance functionality on separate servers.
'combined' - User and Performance functionality on the same server
Select the type of System Module Install [1=separate, 2=combined] (2) >
Combined Server Installation will be performed.
The Server will be configured as a Performance Module.
All MariaDB ColumnStore Processes will run on the Performance Modules.
NOTE: The MariaDB ColumnStore Schema Sync feature will replicate all of the
schemas and InnoDB tables across the User Module nodes. This feature can be enabled
or disabled, for example, if you wish to configure your own replication post installation.
MariaDB ColumnStore Schema Sync feature is Enabled, do you want to leave enabled? [y,n] 
>
NOTE: Configured to have ColumnStore use the Amazon AWS CLI Tools
NOTE: MariaDB ColumnStore Replication Feature is enabled
Enter System Name (1.1.2) >
ERROR: exiting, postConfigure can only run executed on pm1, current module is pm3
[mariadb-user@ip-172-30-0-129 ~]$