Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-1699

amazon EC2 install add module/dbroot failed to restart up

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.6
    • Fix Version/s: 1.1.6
    • Component/s: oam
    • Labels:
      None
    • Environment:
      amazon ec2 AMI 1um/2pm starting setup

      Description

      adding a um and successful did s startsystem at the end.
      added a pm and dbroot and startsystem failed.

      mcsadmin> alterSystem-enableModule pm3
      altersystem-enablemodule Thu Sep 6 16:17:22 2018

      This command starts the processing of applications on a Module within the MariaDB ColumnStore System
      Do you want to proceed: (y or n) [n]: y

      Enabling Modules
      Successful enable of Modules

      Performance Module(s) Enabled, run movePmDbrootConfig or assignDbrootPmConfig to assign dbroots, if needed

      mcsadmin> adddbroot 1
      adddbroot Thu Sep 6 16:18:15 2018

      Enter EBS storage size in GB, current setting is 50 :
      Please enter: 50
      Create AWS Volume for DBRoot #3
      Formatting DBRoot #3, please wait...

      New DBRoot IDs added = 3

      mcsadmin> assignDbrootPmConfig 3 pm3
      assigndbrootpmconfig Thu Sep 6 16:19:27 2018

      DBRoot IDs assigned to 'pm3' =

      Changes being applied

      DBRoot IDs assigned to 'pm3' = 3

      Successfully Assigned DBRoots

      mcsadmin> starts
      startsystem Thu Sep 6 16:19:32 2018

      System being started, please wait...

      TIMEOUT: ProcMon not responding to getSystemStatus

            • startSystem Failed : check log files

      mcsadmin> getsystemi
      getsysteminfo Thu Sep 6 16:24:26 2018

      System 1.1.6

      System and Module statuses

      Component Status Last Status Change
      ------------ -------------------------- ------------------------
      System FAILED Thu Sep 6 16:19:57 2018

      Module um1 ACTIVE Thu Sep 6 16:18:03 2018
      Module um2 ACTIVE Thu Sep 6 16:12:31 2018
      Module pm1 ACTIVE Thu Sep 6 15:13:10 2018
      Module pm2 ACTIVE Thu Sep 6 15:13:19 2018
      Module pm3 FAILED Thu Sep 6 16:19:53 2018

      Active Parent OAM Performance Module is 'pm1'
      Primary Front-End MariaDB ColumnStore Module is 'um1'
      MariaDB ColumnStore Replication Feature is enabled

      MariaDB ColumnStore Process statuses

      Process Module Status Last Status Change Process ID
      ------------------ ------ --------------- ------------------------ ----------
      ProcessMonitor um1 ACTIVE Thu Sep 6 15:12:56 2018 2562
      ServerMonitor um1 ACTIVE Thu Sep 6 15:13:09 2018 2939
      DBRMWorkerNode um1 ACTIVE Thu Sep 6 16:17:37 2018 22228
      ExeMgr um1 MAN_OFFLINE Thu Sep 6 16:17:53 2018
      DDLProc um1 ACTIVE Thu Sep 6 15:13:26 2018 4534
      DMLProc um1 ACTIVE Thu Sep 6 16:12:15 2018 20580
      mysqld um1 ACTIVE Thu Sep 6 16:18:03 2018 22593

      ProcessMonitor um2 ACTIVE Thu Sep 6 16:10:37 2018 2347
      ServerMonitor um2 ACTIVE Thu Sep 6 16:12:24 2018 3627
      DBRMWorkerNode um2 ACTIVE Thu Sep 6 16:17:42 2018 5124
      ExeMgr um2 MAN_OFFLINE Thu Sep 6 16:17:53 2018
      DDLProc um2 COLD_STANDBY Thu Sep 6 16:18:06 2018
      DMLProc um2 COLD_STANDBY Thu Sep 6 16:18:07 2018
      mysqld um2 ACTIVE Thu Sep 6 16:18:05 2018 5468

      ProcessMonitor pm1 ACTIVE Thu Sep 6 15:12:12 2018 4101
      ProcessManager pm1 ACTIVE Thu Sep 6 15:12:19 2018 4238
      DBRMControllerNode pm1 ACTIVE Thu Sep 6 16:17:34 2018 18315
      ServerMonitor pm1 ACTIVE Thu Sep 6 15:13:04 2018 5200
      DBRMWorkerNode pm1 ACTIVE Thu Sep 6 16:17:46 2018 18639
      DecomSvr pm1 ACTIVE Thu Sep 6 15:13:08 2018 5379
      PrimProc pm1 ACTIVE Thu Sep 6 16:17:55 2018 18988
      WriteEngineServer pm1 ACTIVE Thu Sep 6 16:18:04 2018 19370

      ProcessMonitor pm2 ACTIVE Thu Sep 6 15:12:56 2018 2027
      ProcessManager pm2 HOT_STANDBY Thu Sep 6 15:12:57 2018 2152
      DBRMControllerNode pm2 COLD_STANDBY Thu Sep 6 15:13:09 2018
      ServerMonitor pm2 ACTIVE Thu Sep 6 15:13:13 2018 2252
      DBRMWorkerNode pm2 ACTIVE Thu Sep 6 16:17:51 2018 6970
      DecomSvr pm2 ACTIVE Thu Sep 6 15:13:17 2018 2283
      PrimProc pm2 ACTIVE Thu Sep 6 16:17:56 2018 6992
      WriteEngineServer pm2 ACTIVE Thu Sep 6 16:18:05 2018 7028

      ProcessMonitor pm3 ACTIVE Thu Sep 6 16:16:29 2018 1933
      ProcessManager pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018
      DBRMControllerNode pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018
      ServerMonitor pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018
      DBRMWorkerNode pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018
      DecomSvr pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018
      PrimProc pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018
      WriteEngineServer pm3 MAN_OFFLINE Thu Sep 6 16:17:23 2018

      Sep 6 16:16:10 ip-172-31-32-225 ProcessMonitor[1933]: 10.529831 |0|0|0| D 18 CAL0000: checkDataMount called
      Sep 6 16:16:10 ip-172-31-32-225 ProcessMonitor[1933]: 10.535693 |0|0|0| W 18 CAL0000: No dbroots are configured in Columnstore.xml file
      Sep 6 16:16:10 ip-172-31-32-225 ProcessMonitor[1933]: 10.535720 |0|0|0| W 18 CAL0000: ERROR: checkDataMount to failed, module is disabled, continuing

      [root@ip-172-31-32-225 columnstore]# cat err.log
      Sep 6 16:19:53 ip-172-31-32-225 ProcessMonitor[1933]: 53.580757 |0|0|0| E 18 CAL0000: checkDataMount: mount failed, DBRoot: /home/mariadb-user/mariadb/columnstore/data3
      Sep 6 16:19:53 ip-172-31-32-225 ProcessMonitor[1933]: 53.582201 |0|0|0| C 18 CAL0000: checkDataMount error, startmodule failed
      [root@ip-172-31-32-225 columnstore]#

      pm1 logs for the add dbroot

      Sep 6 16:18:07 ip-172-31-46-167 ProcessManager[4238]: 07.966607 |0|0|0| I 17 CAL0000: ENABLEMODULE: ACK back to sender
      Sep 6 16:18:20 ip-172-31-46-167 oamcpp[20155]: 20.859796 |0|0|0| W 08 CAL0006: MCSVolume: attachvolume failed: available
      Sep 6 16:18:26 ip-172-31-46-167 oamcpp[30141]: 26.146397 |0|0|0| D 08 CAL0000: addDbroot format cmd: sudo mkfs.ext2 -F /dev/xvdi > /tmp/format.log 2>&1

        Attachments

          Activity

            People

            Assignee:
            dleeyh Daniel Lee
            Reporter:
            hill David Hill (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.