Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-916

Gluster failover: Stack did not recover completely after PM1 reboot

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 1.1.0
    • 1.1.1
    • ?
    • None
    • 2017-18, 2017-19, 2017-20, 2017-21

    Description

      Build tested: 1.1.0-1 beta

      [root@localhost columnstore]# cat crit.log
      Aug 29 14:38:48 localhost ProcessMonitor[6881]: 48.096282 |0|0|0| C 18 CAL0000: *****Calpont Process Restarting: ProcessManager, old PID = 6981
      Aug 29 15:33:54 localhost ProcessMonitor[6881]: 54.703490 |0|0|0| C 18 CAL0000: *****Calpont Process Restarting: ProcessManager, old PID = 7332
      Aug 29 15:35:23 localhost ProcessManager[11150]: 23.027245 |0|0|0| C 17 CAL0000: startMgrProcessThread Exit with a failure, not all ProcMons ACTIVE

      But I checked all 8 nodes and found that procmons are all running.

      maybe a one point procmon was not running

      Tried shutdownsystem from PM2 (active PM after failover). Command failed:

      Aug 29 17:00:50 localhost controllernode[11150]: 50.867853 |0|0|0| E 29 CAL0000: DBRM: error: SessionManager::clearSystemState() failed (network)
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.459731 |0|0|0| E 18 CAL0000: glusterUnassign failed.
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.496036 |0|0|0| E 18 CAL0000: Error unassigning gluster dbroot# 1
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.523484 |0|0|0| E 18 CAL0000: glusterUnassign failed.
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.531013 |0|0|0| E 18 CAL0000: Error unassigning gluster dbroot# 3
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.571047 |0|0|0| E 18 CAL0000: glusterUnassign failed.
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.597465 |0|0|0| E 18 CAL0000: Error unassigning gluster dbroot# 4
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.622959 |0|0|0| E 18 CAL0000: glusterUnassign failed.
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.640528 |0|0|0| E 18 CAL0000: Error unassigning gluster dbroot# 5
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.661192 |0|0|0| E 18 CAL0000: glusterUnassign failed.
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.670325 |0|0|0| E 18 CAL0000: Error unassigning gluster dbroot# 6
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.698744 |0|0|0| E 18 CAL0000: glusterUnassign failed.
      Aug 29 17:00:55 localhost ProcessMonitor[6881]: 55.711682 |0|0|0| E 18 CAL0000: Error unassigning gluster dbroot# 7

      Attachments

        Activity

          People

            dleeyh Daniel Lee (Inactive)
            dleeyh Daniel Lee (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.