centos 7 nightly regression test run

Trying:
Add code to reload the xml and retry when this occurs. It seems to help. We'll see if this issue disappears after this.

The issue has been seen since this modification.

It seems that the restart.sh script used in the regression tests (101 and 299) calls mysql-columnstore stop after it calls mcsadmin restart. This will cause mysqld to go into MAN_OFFLINE state temporarily. Such an action causes the PM to go into a DEGRADED state. Current code thinks that DEGRADED is invalid. DEGRADED is also used when one nic of a multi-nic PM is down. We want to keep such a PM in the list.

Sometimes, our scripts are so fast that some SQL leaks in before mysqld is rebooted, but doesn't get to that point in the processing until after the PM is DEGRADED. This causes ExeMgr to not include that PM in its cached list of active PM's. That SQL is then rejected because of the issue, and all subsequent is also rejected because the PM state is cached.

By code inspection, this problem can occur PM is in some start other than ACTIVE and some thread in ExeMgr calls one of the OAM_CACHE functions, which might trigger an oam cache reload. If a PM is in any but ACTIVE state, the oam cache reload logic ignores that PM, which is not what we want. I believe my retry logic fails because, during these fast moving tests, the retry is too soon and the PM is still in another state.

I added code so that if a PM is in BUSY_INIT (not probable), MAN_INIT (also not probable) or PID_UPDATE (possible, this state was more recently added to the code), the cache reload will retry a few times over 5 seconds to see if that PM goes ACTIVE. Also added that DEGRADED is a valid state, rather than rejecting it.

For review

Assigned it to Mr. Hill for regression test.

test299 passed regression testing.. issue not encountered.