[MCOL-2173] altersystem-enablemodule gets Columnstore stack into a bad state Created: 2019-02-13  Updated: 2023-10-26  Resolved: 2020-04-15

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.2.3
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Daniel Lee (Inactive) Assignee: Ben Thompson (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None


 Description   

Build tested: 1.2.3-1

server commit:
61f32f2
engine commit:
46cc344

Started with a 1um2pm stack, I added a 3rd PM and it was successful. I added a dbroot and assigned it to pm3. PM3 was still in MAN_OFFLINE state. Enabling or disabling it still failed. Shutdownsystem and startsystem also did not help.

mcsadmin> altersystem-enablemodule pm3
altersystem-enablemodule Wed Feb 13 19:14:22 2019

This command starts the processing of applications on a Module within the MariaDB ColumnStore System
Do you want to proceed: (y or n) [n]: y

Enabling Modules
Successful enable of Modules

Performance Module(s) Enabled, run movePmDbrootConfig or assignDbrootPmConfig to assign dbroots, if needed

mcsadmin> getstorage
getstorageconfig Wed Feb 13 19:14:47 2019

System Storage Configuration

Performance Module (DBRoot) Storage Type = internal
System Assigned DBRoot Count = 2
DBRoot IDs assigned to 'pm1' = 1
DBRoot IDs assigned to 'pm2' = 2
DBRoot IDs assigned to 'pm3' =

mcsadmin> adddbroot 1
adddbroot Wed Feb 13 19:15:34 2019

New DBRoot IDs added = 3

mcsadmin>
mcsadmin> help assignDbrootPmConfig
help Wed Feb 13 19:16:00 2019

Command: assignDbrootPmConfig

Description: Assign unassigned DBroots to Performance Module
This command could be used after a Performance Module and/or DBRoot is added

Arguments: Required: List of Dbroot IDs to be assigned
Required: Performance Module Name

mcsadmin> assignDbrootPmConfig 3 pm3
assigndbrootpmconfig Wed Feb 13 19:16:13 2019

DBRoot IDs assigned to 'pm3' =

Changes being applied

DBRoot IDs assigned to 'pm3' = 3

Successfully Assigned DBRoots

mcsadmin> getprocessstatus
getprocessstatus Wed Feb 13 19:16:38 2019

MariaDB ColumnStore Process statuses

Process Module Status Last Status Change Process ID
------------------ ------ --------------- ------------------------ ----------
ProcessMonitor um1 ACTIVE Wed Feb 13 17:54:39 2019 18308
ServerMonitor um1 ACTIVE Wed Feb 13 17:54:51 2019 18686
DBRMWorkerNode um1 ACTIVE Wed Feb 13 17:54:51 2019 18734
ExeMgr um1 ACTIVE Wed Feb 13 17:55:01 2019 19788
DDLProc um1 ACTIVE Wed Feb 13 17:55:05 2019 19807
DMLProc um1 ACTIVE Wed Feb 13 17:55:09 2019 19853
mysqld um1 ACTIVE Wed Feb 13 17:54:59 2019 18642

ProcessMonitor pm1 ACTIVE Wed Feb 13 17:54:00 2019 17901
ProcessManager pm1 ACTIVE Wed Feb 13 17:54:06 2019 17967
DBRMControllerNode pm1 ACTIVE Wed Feb 13 17:54:42 2019 18789
ServerMonitor pm1 ACTIVE Wed Feb 13 17:54:44 2019 18811
DBRMWorkerNode pm1 ACTIVE Wed Feb 13 17:54:44 2019 18835
PrimProc pm1 ACTIVE Wed Feb 13 17:54:48 2019 18977
WriteEngineServer pm1 ACTIVE Wed Feb 13 17:54:49 2019 19022

ProcessMonitor pm2 ACTIVE Wed Feb 13 17:54:35 2019 18679
ProcessManager pm2 HOT_STANDBY Wed Feb 13 17:54:36 2019 18723
DBRMControllerNode pm2 COLD_STANDBY Wed Feb 13 17:54:50 2019
ServerMonitor pm2 ACTIVE Wed Feb 13 17:54:53 2019 18757
DBRMWorkerNode pm2 ACTIVE Wed Feb 13 17:54:54 2019 18773
PrimProc pm2 ACTIVE Wed Feb 13 17:54:58 2019 18790
WriteEngineServer pm2 ACTIVE Wed Feb 13 17:54:59 2019 18800

ProcessMonitor pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
ProcessManager pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
DBRMControllerNode pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
ServerMonitor pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
DBRMWorkerNode pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
PrimProc pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
WriteEngineServer pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019

mcsadmin> altersystem-enablemodule pm3
altersystem-enablemodule Wed Feb 13 19:16:58 2019

This command starts the processing of applications on a Module within the MariaDB ColumnStore System
Do you want to proceed: (y or n) [n]: y

Enabling Modules

        • enableModule Failed : Target in an invalid state
          mcsadmin> altersystem-disablemodule pm3
          altersystem-disablemodule Wed Feb 13 19:17:29 2019

This command stops the processing of applications on a Module within the MariaDB ColumnStore System
Do you want to proceed: (y or n) [n]: y

        • alterSystem-disableModule Failed : pm3 has dbroots still assigned and will not be disabled. Please run movePmDbrootConfig or unassignDbrootPmConfig.

mcsadmin> shutdownsystem
shutdownsystem Wed Feb 13 19:17:46 2019

This command stops the processing of applications on all Modules within the MariaDB ColumnStore System

Checking for active transactions
Do you want to proceed: (y or n) [n]: y

Stopping System...

        • stopSystem Failed : check log files

Shutting Down System...
Successful shutdown of System

mcsadmin> startsystem vagrant
startsystem Wed Feb 13 19:19:10 2019

startSystem command, 'columnstore' service is down, sending command to
start the 'columnstore' service on all modules

System being started, please wait........................

TIMEOUT: ProcMon not responding to getSystemStatus

        • startSystem Failed : check log files

TIMEOUT: ProcMon not responding to getSystemStatus

        • startSystem Failed : check log files
          mcsadmin> getprocessstatus
          getprocessstatus Wed Feb 13 19:25:03 2019

MariaDB ColumnStore Process statuses

Process Module Status Last Status Change Process ID
------------------ ------ --------------- ------------------------ ----------
ProcessMonitor um1 ACTIVE Wed Feb 13 19:20:07 2019 18268
ServerMonitor um1 INITIAL
DBRMWorkerNode um1 INITIAL
ExeMgr um1 INITIAL
DDLProc um1 INITIAL
DMLProc um1 INITIAL
mysqld um1 INITIAL

ProcessMonitor pm1 ACTIVE Wed Feb 13 19:19:53 2019 6217
ProcessManager pm1 ACTIVE Wed Feb 13 19:19:59 2019 6381
DBRMControllerNode pm1 INITIAL
ServerMonitor pm1 INITIAL
DBRMWorkerNode pm1 INITIAL
PrimProc pm1 INITIAL
WriteEngineServer pm1 INITIAL

ProcessMonitor pm2 ACTIVE Wed Feb 13 19:20:14 2019 19554
ProcessManager pm2 HOT_STANDBY Wed Feb 13 19:20:15 2019 19603
DBRMControllerNode pm2 INITIAL
ServerMonitor pm2 INITIAL
DBRMWorkerNode pm2 INITIAL
PrimProc pm2 INITIAL
WriteEngineServer pm2 INITIAL

ProcessMonitor pm3 INITIAL
ProcessManager pm3 INITIAL
DBRMControllerNode pm3 INITIAL
ServerMonitor pm3 INITIAL
DBRMWorkerNode pm3 INITIAL
PrimProc pm3 INITIAL
WriteEngineServer pm3 INITIAL

[root@localhost columnstore]# cat crit.log
Feb 13 19:23:25 localhost ProcessManager[6381]: 25.515784 |0|0|0| C 17 CAL0000: startMgrProcessThread Exit with a failure, not all ProcMons ACTIVE



 Comments   
Comment by Todd Stoffel (Inactive) [ 2020-04-15 ]

OAM is being deprecated and replaced by an enhanced API and the MaxScale orchestration project.

Generated at Thu Feb 08 02:34:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.