Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-2173

altersystem-enablemodule gets Columnstore stack into a bad state

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.2.3
    • Fix Version/s: N/A
    • Component/s: oam
    • Labels:
      None

      Description

      Build tested: 1.2.3-1

      server commit:
      61f32f2
      engine commit:
      46cc344

      Started with a 1um2pm stack, I added a 3rd PM and it was successful. I added a dbroot and assigned it to pm3. PM3 was still in MAN_OFFLINE state. Enabling or disabling it still failed. Shutdownsystem and startsystem also did not help.

      mcsadmin> altersystem-enablemodule pm3
      altersystem-enablemodule Wed Feb 13 19:14:22 2019

      This command starts the processing of applications on a Module within the MariaDB ColumnStore System
      Do you want to proceed: (y or n) [n]: y

      Enabling Modules
      Successful enable of Modules

      Performance Module(s) Enabled, run movePmDbrootConfig or assignDbrootPmConfig to assign dbroots, if needed

      mcsadmin> getstorage
      getstorageconfig Wed Feb 13 19:14:47 2019

      System Storage Configuration

      Performance Module (DBRoot) Storage Type = internal
      System Assigned DBRoot Count = 2
      DBRoot IDs assigned to 'pm1' = 1
      DBRoot IDs assigned to 'pm2' = 2
      DBRoot IDs assigned to 'pm3' =

      mcsadmin> adddbroot 1
      adddbroot Wed Feb 13 19:15:34 2019

      New DBRoot IDs added = 3

      mcsadmin>
      mcsadmin> help assignDbrootPmConfig
      help Wed Feb 13 19:16:00 2019

      Command: assignDbrootPmConfig

      Description: Assign unassigned DBroots to Performance Module
      This command could be used after a Performance Module and/or DBRoot is added

      Arguments: Required: List of Dbroot IDs to be assigned
      Required: Performance Module Name

      mcsadmin> assignDbrootPmConfig 3 pm3
      assigndbrootpmconfig Wed Feb 13 19:16:13 2019

      DBRoot IDs assigned to 'pm3' =

      Changes being applied

      DBRoot IDs assigned to 'pm3' = 3

      Successfully Assigned DBRoots

      mcsadmin> getprocessstatus
      getprocessstatus Wed Feb 13 19:16:38 2019

      MariaDB ColumnStore Process statuses

      Process Module Status Last Status Change Process ID
      ------------------ ------ --------------- ------------------------ ----------
      ProcessMonitor um1 ACTIVE Wed Feb 13 17:54:39 2019 18308
      ServerMonitor um1 ACTIVE Wed Feb 13 17:54:51 2019 18686
      DBRMWorkerNode um1 ACTIVE Wed Feb 13 17:54:51 2019 18734
      ExeMgr um1 ACTIVE Wed Feb 13 17:55:01 2019 19788
      DDLProc um1 ACTIVE Wed Feb 13 17:55:05 2019 19807
      DMLProc um1 ACTIVE Wed Feb 13 17:55:09 2019 19853
      mysqld um1 ACTIVE Wed Feb 13 17:54:59 2019 18642

      ProcessMonitor pm1 ACTIVE Wed Feb 13 17:54:00 2019 17901
      ProcessManager pm1 ACTIVE Wed Feb 13 17:54:06 2019 17967
      DBRMControllerNode pm1 ACTIVE Wed Feb 13 17:54:42 2019 18789
      ServerMonitor pm1 ACTIVE Wed Feb 13 17:54:44 2019 18811
      DBRMWorkerNode pm1 ACTIVE Wed Feb 13 17:54:44 2019 18835
      PrimProc pm1 ACTIVE Wed Feb 13 17:54:48 2019 18977
      WriteEngineServer pm1 ACTIVE Wed Feb 13 17:54:49 2019 19022

      ProcessMonitor pm2 ACTIVE Wed Feb 13 17:54:35 2019 18679
      ProcessManager pm2 HOT_STANDBY Wed Feb 13 17:54:36 2019 18723
      DBRMControllerNode pm2 COLD_STANDBY Wed Feb 13 17:54:50 2019
      ServerMonitor pm2 ACTIVE Wed Feb 13 17:54:53 2019 18757
      DBRMWorkerNode pm2 ACTIVE Wed Feb 13 17:54:54 2019 18773
      PrimProc pm2 ACTIVE Wed Feb 13 17:54:58 2019 18790
      WriteEngineServer pm2 ACTIVE Wed Feb 13 17:54:59 2019 18800

      ProcessMonitor pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
      ProcessManager pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
      DBRMControllerNode pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
      ServerMonitor pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
      DBRMWorkerNode pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
      PrimProc pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019
      WriteEngineServer pm3 MAN_OFFLINE Wed Feb 13 19:11:18 2019

      mcsadmin> altersystem-enablemodule pm3
      altersystem-enablemodule Wed Feb 13 19:16:58 2019

      This command starts the processing of applications on a Module within the MariaDB ColumnStore System
      Do you want to proceed: (y or n) [n]: y

      Enabling Modules

            • enableModule Failed : Target in an invalid state
              mcsadmin> altersystem-disablemodule pm3
              altersystem-disablemodule Wed Feb 13 19:17:29 2019

      This command stops the processing of applications on a Module within the MariaDB ColumnStore System
      Do you want to proceed: (y or n) [n]: y

            • alterSystem-disableModule Failed : pm3 has dbroots still assigned and will not be disabled. Please run movePmDbrootConfig or unassignDbrootPmConfig.

      mcsadmin> shutdownsystem
      shutdownsystem Wed Feb 13 19:17:46 2019

      This command stops the processing of applications on all Modules within the MariaDB ColumnStore System

      Checking for active transactions
      Do you want to proceed: (y or n) [n]: y

      Stopping System...

            • stopSystem Failed : check log files

      Shutting Down System...
      Successful shutdown of System

      mcsadmin> startsystem vagrant
      startsystem Wed Feb 13 19:19:10 2019

      startSystem command, 'columnstore' service is down, sending command to
      start the 'columnstore' service on all modules

      System being started, please wait........................

      TIMEOUT: ProcMon not responding to getSystemStatus

            • startSystem Failed : check log files

      TIMEOUT: ProcMon not responding to getSystemStatus

            • startSystem Failed : check log files
              mcsadmin> getprocessstatus
              getprocessstatus Wed Feb 13 19:25:03 2019

      MariaDB ColumnStore Process statuses

      Process Module Status Last Status Change Process ID
      ------------------ ------ --------------- ------------------------ ----------
      ProcessMonitor um1 ACTIVE Wed Feb 13 19:20:07 2019 18268
      ServerMonitor um1 INITIAL
      DBRMWorkerNode um1 INITIAL
      ExeMgr um1 INITIAL
      DDLProc um1 INITIAL
      DMLProc um1 INITIAL
      mysqld um1 INITIAL

      ProcessMonitor pm1 ACTIVE Wed Feb 13 19:19:53 2019 6217
      ProcessManager pm1 ACTIVE Wed Feb 13 19:19:59 2019 6381
      DBRMControllerNode pm1 INITIAL
      ServerMonitor pm1 INITIAL
      DBRMWorkerNode pm1 INITIAL
      PrimProc pm1 INITIAL
      WriteEngineServer pm1 INITIAL

      ProcessMonitor pm2 ACTIVE Wed Feb 13 19:20:14 2019 19554
      ProcessManager pm2 HOT_STANDBY Wed Feb 13 19:20:15 2019 19603
      DBRMControllerNode pm2 INITIAL
      ServerMonitor pm2 INITIAL
      DBRMWorkerNode pm2 INITIAL
      PrimProc pm2 INITIAL
      WriteEngineServer pm2 INITIAL

      ProcessMonitor pm3 INITIAL
      ProcessManager pm3 INITIAL
      DBRMControllerNode pm3 INITIAL
      ServerMonitor pm3 INITIAL
      DBRMWorkerNode pm3 INITIAL
      PrimProc pm3 INITIAL
      WriteEngineServer pm3 INITIAL

      [root@localhost columnstore]# cat crit.log
      Feb 13 19:23:25 localhost ProcessManager[6381]: 25.515784 |0|0|0| C 17 CAL0000: startMgrProcessThread Exit with a failure, not all ProcMons ACTIVE

        Attachments

          Activity

            People

            Assignee:
            ben.thompson Ben Thompson
            Reporter:
            dleeyh Daniel Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.