[MCOL-4260] cmapi_server doesn't work with FQDN in k8s Created: 2020-08-20  Updated: 2020-11-12  Resolved: 2020-08-20

Status: Closed
Project: MariaDB ColumnStore
Component/s: cmapi
Affects Version/s: 1.5.3
Fix Version/s: N/A

Type: Bug Priority: Blocker
Reporter: Jens Röwekamp (Inactive) Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: SkySQLMVP
Environment:

Kubernetes

image: mariadb/skysql-enterprise-columnstore-handoff:cluster-dev
digest: 02b7298cadbd822c88de9db04f34c37f33561cd8ef0c6800acd97fc6bb9d10bb



 Description   

Adding the first node to a ColumnStore cluster through the cmapi server in k8s using its fully qualified domain name fails.

The same steps however succeed if localhost is used instead of the FQDN.

localhost works

[root@cs-test-mdb-cs-node-0 /]# ping localhost -c2
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.017 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.029 ms
 
--- localhost ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 54ms
rtt min/avg/max/mdev = 0.017/0.023/0.029/0.006 ms

[root@cs-test-mdb-cs-node-0 /]# curl -s https://localhost:8640/cmapi/0.4.0/cluster/status --header 'Content-Type:application/json' --header "x-api-key:somekey123" -k | jq .
{
  "timestamp": "2020-08-20 18:02:44.320358",
  "num_nodes": 0
}

[root@cs-test-mdb-cs-node-0 /]# curl -s -X PUT "https://localhost:8640/cmapi/0.4.0/cluster/add-node" --header 'Content-Type:application/json' --header "x-api-key:somekey123" --data '{"timeout":60, "node": "localhost"}' -k | jq .
{
  "timestamp": "2020-08-20 18:02:49.877532",
  "node_id": "localhost"
}

[root@cs-test-mdb-cs-node-0 /]# curl -s https://localhost:8640/cmapi/0.4.0/cluster/status --header 'Content-Type:application/json' --header "x-api-key:somekey123" -k | jq .
{
  "timestamp": "2020-08-20 18:03:57.401819",
  "localhost": {
    "timestamp": "2020-08-20 18:03:57.408225",
    "uptime": 1379,
    "dbrm_mode": "master",
    "cluster_mode": "readwrite",
    "dbroots": [
      "1"
    ],
    "module_id": 1,
    "services": [
      {
        "name": "workernode",
        "pid": 1215
      },
      {
        "name": "controllernode",
        "pid": 1272
      },
      {
        "name": "PrimProc",
        "pid": 1275
      },
      {
        "name": "ExeMgr",
        "pid": 1300
      },
      {
        "name": "WriteEngine",
        "pid": 1303
      },
      {
        "name": "DMLProc",
        "pid": 1319
      },
      {
        "name": "DDLProc",
        "pid": 1322
      }
    ]
  },
  "num_nodes": 1
}
[root@cs-test-mdb-cs-node-0 /]# exit

FQDN cs-test-mdb-cs-node-0.cs-test-mdb-clust doesn't work

dev-jump:~ $ kubectl logs cs-test-mdb-cs-node-0
Starting...
CMAPI PID = 848
Starting MariaDB.200820 18:06:24 mysqld_safe Logging to '/var/log/mariadb/columnstore/mariadb-error.log'.
200820 18:06:24 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
 SUCCESS!
 New Monit id: 6290541950c79652f3904ecad7f4f33c
 Stored in '/root/.monit.id'
Starting Monit 5.26.0 daemon with http interface at [localhost]:2812
Monit will delay for 60s on first start after reboot ...
'cs-test-mdb-cs-node-0' Monit 5.26.0 started

dev-jump:~ $ kubectl exec -it cs-test-mdb-cs-node-0 -- bash
[root@cs-test-mdb-cs-node-0 /]# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   4340   796 ?        Ss   18:05   0:00 /usr/bin/tini -- docker-entrypoint.sh /bin/sh -c cmapi-start && monit -I
root           7  0.0  0.0  14816  3300 ?        S    18:05   0:00 /bin/bash /usr/bin/docker-entrypoint.sh /bin/sh -c cmapi-start && monit -I
root          10  0.4  0.0 197144  2372 ?        Ssl  18:05   0:00 rsyslogd
root          12  0.0  0.0 109444  5280 ?        Sl   18:05   0:00 monit -I
root         848  2.6  0.6 1211824 51116 ?       Sl   18:06   0:02 /opt/cmapi/python/bin/python3 -m cmapi_server
root         905  0.0  0.0  15080  3444 ?        S    18:06   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/cs-test-mdb-cs-node-0.pid
mysql       1035  0.1  1.3 1700728 99968 ?       Sl   18:06   0:00 /usr/sbin/mariadbd --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mariadb/columnstore/mariadb-error.log --
root        1092  1.0  0.0  15052  3728 pts/0    Ss   18:07   0:00 bash
root        1107  0.0  0.0  50436  3884 pts/0    R+   18:07   0:00 ps -aux

[root@cs-test-mdb-cs-node-0 /]# ping cs-test-mdb-cs-node-0.cs-test-mdb-clust -c2
PING cs-test-mdb-cs-node-0.cs-test-mdb-clust.db00005735.svc.cluster.local (10.0.0.166) 56(84) bytes of data.
64 bytes from cs-test-mdb-cs-node-0.cs-test-mdb-clust.db00005735.svc.cluster.local (10.0.0.166): icmp_seq=1 ttl=64 time=0.014 ms
64 bytes from cs-test-mdb-cs-node-0.cs-test-mdb-clust.db00005735.svc.cluster.local (10.0.0.166): icmp_seq=2 ttl=64 time=0.038 ms
 
--- cs-test-mdb-cs-node-0.cs-test-mdb-clust.db00005735.svc.cluster.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 12ms
rtt min/avg/max/mdev = 0.014/0.026/0.038/0.012 ms

[root@cs-test-mdb-cs-node-0 /]# curl -s https://cs-test-mdb-cs-node-0.cs-test-mdb-clust:8640/cmapi/0.4.0/cluster/status --header 'Content-Type:application/json' --header "x-api-key:somekey123" -k | jq .
{
  "timestamp": "2020-08-20 18:08:06.374310",
  "num_nodes": 0
}

[root@cs-test-mdb-cs-node-0 /]# curl -s -X PUT "https://cs-test-mdb-cs-node-0.cs-test-mdb-clust:8640/cmapi/0.4.0/cluster/add-node" --header 'Content-Type:application/json' --header "x-api-key:somekey123" --data '{"timeout":60, "node": "cs-test-mdb-cs-node-0.cs-test-mdb-clust"}' -k | jq .
{
  "timestamp": "2020-08-20 18:08:12.795427",
  "node_id": "cs-test-mdb-cs-node-0.cs-test-mdb-clust"
}

[root@cs-test-mdb-cs-node-0 /]# sleep 120

[root@cs-test-mdb-cs-node-0 /]# curl -s https://cs-test-mdb-cs-node-0.cs-test-mdb-clust:8640/cmapi/0.4.0/cluster/status --header 'Content-Type:application/json' --header "x-api-key:somekey123" -k | jq .
{
  "timestamp": "2020-08-20 18:10:41.507126",
  "cs-test-mdb-cs-node-0.cs-test-mdb-clust": {
    "timestamp": "2020-08-20 18:10:41.515981",
    "uptime": 1783,
    "dbrm_mode": "offline",
    "cluster_mode": "readonly",
    "dbroots": [],
    "module_id": 1,
    "services": []
  },
  "num_nodes": 1
}

[root@cs-test-mdb-cs-node-0 /]# for i in $(ls /var/log/mariadb/columnstore/); do
>     echo "------"
> echo $i
> cat /var/log/mariadb/columnstore/$i
> echo
> done
------
archive
cat: /var/log/mariadb/columnstore/archive: Is a directory
 
------
cmapi_server.log
127.0.0.1 - - [20/Aug/2020 18:06:25] cmapi_server DEBUG    Using container dispatcher module
127.0.0.1 - - [20/Aug/2020 18:06:25] cmapi_server INFO     CMAPI Version: 1.5.3.1
[20/Aug/2020 18:06:25] root  get_module_net_address Module 1 network address 127.0.0.1
[20/Aug/2020 18:06:25] root  Loaded the config file, my name is None
[20/Aug/2020:18:06:25] ENGINE Bus STARTING
[20/Aug/2020 18:06:25] root  Starting the heartbeat listener
[20/Aug/2020 18:06:25] root  starting the monitor logic
[20/Aug/2020:18:06:25] ENGINE Serving on https://0.0.0.0:8640
[20/Aug/2020:18:06:25] ENGINE Bus STARTED
[20/Aug/2020 18:06:26] root  Failover monitoring is inactive; requires at least 3 nodes and a shared storage system
10.0.0.166 - - [20/Aug/2020 18:08:06] cmapi_server DEBUG    get_status starts
10.0.0.166 - - [20/Aug/2020 18:08:06] cmapi_server DEBUG    get_status returns {'timestamp': '2020-08-20 18:08:06.374310', 'num_nodes': 0}
10.0.0.166 - - [20/Aug/2020:18:08:06] "GET /cmapi/0.4.0/cluster/status HTTP/1.1" 200 59 "" "curl/7.61.1"
10.0.0.166 - - [20/Aug/2020 18:08:12] cmapi_server DEBUG    put_add_node starts
10.0.0.166 - - [20/Aug/2020 18:08:12] cmapi_server DEBUG    put_begin starts
10.0.0.166 - - [20/Aug/2020 18:08:12] cmapi_server DEBUG    put_begin JSON body {'id': 26562, 'timeout': 299}
10.0.0.166 - - [20/Aug/2020 18:08:12] cmapi_server DEBUG    put_begin returns {'timestamp': '2020-08-20 18:08:12.806544'}
10.0.0.166 - - [20/Aug/2020:18:08:12] "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43 "" "python-requests/2.23.0"
[20/Aug/2020 18:08:12] root  get_module_net_address Module 1 network address 127.0.0.1
[20/Aug/2020 18:08:12] root  add_node(): replacing 127.0.0.1/localhost with 10.0.0.166/cs-test-mdb-cs-node-0.cs-test-mdb-clust as this node's name. Be sure cs-test-mdb-cs-node-0.cs-test-mdb-clust resolves to 10.0.0.166 on all other nodes in the cluster.
10.0.0.166 - - [20/Aug/2020 18:08:12] cmapi_server DEBUG    put_config starts
10.0.0.166 - - [20/Aug/2020 18:08:12] cmapi_server DEBUG    put_config JSON body {'manager': 'cs-test-mdb-cs-node-0.cs-test-mdb-clust', 'revision': '1', 'timeout': 300, 'config': '<?xml version="1.0" ?>\n<Columnstore Version="V1.0.0">\n    <ExeMgr1>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8601</Port>\n        <Module>unassigned</Module>\n    </ExeMgr1>\n    <JobProc>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8602</Port>\n    </JobProc>\n    <ProcMgr>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8603</Port>\n    </ProcMgr>\n    <ProcMgr_Alarm>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8606</Port>\n    </ProcMgr_Alarm>\n    <ProcStatusControl>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8604</Port>\n    </ProcStatusControl>\n    <ProcStatusControlStandby>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8605</Port>\n    </ProcStatusControlStandby>\n    <localhost_ProcessMonitor>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8800</Port>\n    </localhost_ProcessMonitor>\n    <dm1_ProcessMonitor>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8800</Port>\n    </dm1_ProcessMonitor>\n    <um1_ProcessMonitor>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8800</Port>\n    </um1_ProcessMonitor>\n    <pm1_ProcessMonitor>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8800</Port>\n    </pm1_ProcessMonitor>\n    <dm1_ServerMonitor>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8622</Port>\n    </dm1_ServerMonitor>\n    <um1_ServerMonitor>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8622</Port>\n    </um1_ServerMonitor>\n    <pm1_ServerMonitor>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8622</Port>\n    </pm1_ServerMonitor>\n    <pm1_WriteEngineServer>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8630</Port>\n    </pm1_WriteEngineServer>\n    <DDLProc>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8612</Port>\n    </DDLProc>\n    <DMLProc>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8614</Port>\n    </DMLProc>\n    <BatchInsert>\n        <RowsPerBatch>10000</RowsPerBatch>\n    </BatchInsert>\n    <PrimitiveServers>\n        <Count>1</Count>\n        <ConnectionsPerPrimProc>2</ConnectionsPerPrimProc>\n        <ProcessorThreshold>128</ProcessorThreshold>\n        <ProcessorQueueSize>10K</ProcessorQueueSize>\n        <DebugLevel>0</DebugLevel>\n        <LBID_Shift>13</LBID_Shift>\n        <ColScanBufferSizeBlocks>512</ColScanBufferSizeBlocks>\n        <ColScanReadAheadBlocks>512</ColScanReadAheadBlocks>\n        <PrefetchThreshold>1</PrefetchThreshold>\n        <PTTrace>0</PTTrace>\n        <RotatingDestination>n</RotatingDestination>\n        <DirectIO>y</DirectIO>\n        <HighPriorityPercentage/>\n        <MediumPriorityPercentage/>\n        <LowPriorityPercentage/>\n    </PrimitiveServers>\n    <PMS1>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS1>\n    <PMS2>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS2>\n    <PMS3>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS3>\n    <PMS4>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS4>\n    <PMS5>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS5>\n    <PMS6>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS6>\n    <PMS7>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS7>\n    <PMS8>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS8>\n    <PMS9>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS9>\n    <PMS10>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS10>\n    <PMS11>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS11>\n    <PMS12>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS12>\n    <PMS13>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS13>\n    <PMS14>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS14>\n    <PMS15>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS15>\n    <PMS16>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS16>\n    <PMS17>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS17>\n    <PMS18>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS18>\n    <PMS19>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS19>\n    <PMS20>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS20>\n    <PMS21>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS21>\n    <PMS22>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS22>\n    <PMS23>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS23>\n    <PMS24>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS24>\n    <PMS25>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS25>\n    <PMS26>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS26>\n    <PMS27>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS27>\n    <PMS28>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS28>\n    <PMS29>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS29>\n    <PMS30>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS30>\n    <PMS31>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS31>\n    <PMS32>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8620</Port>\n    </PMS32>\n    <SystemConfig>\n        <SystemName>columnstore-1</SystemName>\n        <ParentOAMModuleName>pm1</ParentOAMModuleName>\n        <StandbyOAMModuleName>unassigned</StandbyOAMModuleName>\n        <PrimaryUMModuleName>pm1</PrimaryUMModuleName>\n        <ModuleHeartbeatPeriod>1</ModuleHeartbeatPeriod>\n        <ModuleHeartbeatCount>3</ModuleHeartbeatCount>\n        <ModuleProcMonWaitCount>12</ModuleProcMonWaitCount>\n        \t// 2.5 minutes\n        <DBRootCount>1</DBRootCount>\n        <DBRoot1>/var/lib/columnstore/data1</DBRoot1>\n        <DBRMRoot>/var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves</DBRMRoot>\n        <TableLockSaveFile>/var/lib/columnstore/data1/systemFiles/dbrm/tablelocks</TableLockSaveFile>\n        <DBRMTimeOut>15</DBRMTimeOut>\n        <DBRMSnapshotInterval>100000</DBRMSnapshotInterval>\n        <ExternalCriticalThreshold>90</ExternalCriticalThreshold>\n        <ExternalMajorThreshold>80</ExternalMajorThreshold>\n        <ExternalMinorThreshold>70</ExternalMinorThreshold>\n        <TransactionArchivePeriod>10</TransactionArchivePeriod>\n        <NMSIPAddress>0.0.0.0</NMSIPAddress>\n        <TempSaveSize>128M</TempSaveSize>\n        <WaitPeriod>10</WaitPeriod>\n        <ProcessRestartCount>10</ProcessRestartCount>\n        <ProcessRestartPeriod>120</ProcessRestartPeriod>\n        <SwapAction>restartSystem</SwapAction>\n        <ActivePmFailoverDisabled>n</ActivePmFailoverDisabled>\n        <MemoryCheckPercent>95</MemoryCheckPercent>\n        <DataFileLog>OFF</DataFileLog>\n        <hdfsRdwrScratch>/rdwrscratch</hdfsRdwrScratch>\n        <TempFileDir>/columnstore_tmp_files</TempFileDir>\n        <SystemTempFileDir>/tmp/columnstore_tmp_files</SystemTempFileDir>\n        <DataFilePlugin>libcloudio.so</DataFilePlugin>\n    </SystemConfig>\n    <SystemModuleConfig>\n        <ModuleType1>dm</ModuleType1>\n        <ModuleDesc1>Director Module</ModuleDesc1>\n        <RunType1>SIMPLEX</RunType1>\n        <ModuleCount1>0</ModuleCount1>\n        <ModuleIPAddr1-1-1>0.0.0.0</ModuleIPAddr1-1-1>\n        <ModuleHostName1-1-1>unassigned</ModuleHostName1-1-1>\n        <ModuleDisableState1-1>ENABLED</ModuleDisableState1-1>\n        <ModuleCPUCriticalThreshold1>0</ModuleCPUCriticalThreshold1>\n        <ModuleCPUMajorThreshold1>0</ModuleCPUMajorThreshold1>\n        <ModuleCPUMinorThreshold1>0</ModuleCPUMinorThreshold1>\n        <ModuleCPUMinorClearThreshold1>0</ModuleCPUMinorClearThreshold1>\n        <ModuleDiskCriticalThreshold1>90</ModuleDiskCriticalThreshold1>\n        <ModuleDiskMajorThreshold1>80</ModuleDiskMajorThreshold1>\n        <ModuleDiskMinorThreshold1>70</ModuleDiskMinorThreshold1>\n        <ModuleMemCriticalThreshold1>90</ModuleMemCriticalThreshold1>\n        <ModuleMemMajorThreshold1>0</ModuleMemMajorThreshold1>\n        <ModuleMemMinorThreshold1>0</ModuleMemMinorThreshold1>\n        <ModuleSwapCriticalThreshold1>90</ModuleSwapCriticalThreshold1>\n        <ModuleSwapMajorThreshold1>80</ModuleSwapMajorThreshold1>\n        <ModuleSwapMinorThreshold1>70</ModuleSwapMinorThreshold1>\n        <ModuleDiskMonitorFileSystem1-1>/</ModuleDiskMonitorFileSystem1-1>\n        <ModuleDBRootCount1-1>unassigned</ModuleDBRootCount1-1>\n        <ModuleDBRootID1-1-1>unassigned</ModuleDBRootID1-1-1>\n        <ModuleType2>um</ModuleType2>\n        <ModuleDesc2>User Module</ModuleDesc2>\n        <RunType2>SIMPLEX</RunType2>\n        <ModuleCount2>0</ModuleCount2>\n        <ModuleIPAddr1-1-2>0.0.0.0</ModuleIPAddr1-1-2>\n        <ModuleHostName1-1-2>unassigned</ModuleHostName1-1-2>\n        <ModuleDisableState1-2>ENABLED</ModuleDisableState1-2>\n        <ModuleCPUCriticalThreshold2>0</ModuleCPUCriticalThreshold2>\n        <ModuleCPUMajorThreshold2>0</ModuleCPUMajorThreshold2>\n        <ModuleCPUMinorThreshold2>0</ModuleCPUMinorThreshold2>\n        <ModuleCPUMinorClearThreshold2>0</ModuleCPUMinorClearThreshold2>\n        <ModuleDiskCriticalThreshold2>90</ModuleDiskCriticalThreshold2>\n        <ModuleDiskMajorThreshold2>80</ModuleDiskMajorThreshold2>\n        <ModuleDiskMinorThreshold2>70</ModuleDiskMinorThreshold2>\n        <ModuleMemCriticalThreshold2>90</ModuleMemCriticalThreshold2>\n        <ModuleMemMajorThreshold2>0</ModuleMemMajorThreshold2>\n        <ModuleMemMinorThreshold2>0</ModuleMemMinorThreshold2>\n        <ModuleSwapCriticalThreshold2>90</ModuleSwapCriticalThreshold2>\n        <ModuleSwapMajorThreshold2>80</ModuleSwapMajorThreshold2>\n        <ModuleSwapMinorThreshold2>70</ModuleSwapMinorThreshold2>\n        <ModuleDiskMonitorFileSystem1-2>/</ModuleDiskMonitorFileSystem1-2>\n        <ModuleDBRootCount1-2>unassigned</ModuleDBRootCount1-2>\n        <ModuleDBRootID1-1-2>unassigned</ModuleDBRootID1-1-2>\n        <ModuleType3>pm</ModuleType3>\n        <ModuleDesc3>Performance Module</ModuleDesc3>\n        <RunType3>SIMPLEX</RunType3>\n        <ModuleCount3>1</ModuleCount3>\n        <ModuleIPAddr1-1-3>10.0.0.166</ModuleIPAddr1-1-3>\n        <ModuleHostName1-1-3>cs-test-mdb-cs-node-0.cs-test-mdb-clust</ModuleHostName1-1-3>\n        <ModuleDisableState1-3>ENABLED</ModuleDisableState1-3>\n        <ModuleCPUCriticalThreshold3>0</ModuleCPUCriticalThreshold3>\n        <ModuleCPUMajorThreshold3>0</ModuleCPUMajorThreshold3>\n        <ModuleCPUMinorThreshold3>0</ModuleCPUMinorThreshold3>\n        <ModuleCPUMinorClearThreshold3>0</ModuleCPUMinorClearThreshold3>\n        <ModuleDiskCriticalThreshold3>90</ModuleDiskCriticalThreshold3>\n        <ModuleDiskMajorThreshold3>80</ModuleDiskMajorThreshold3>\n        <ModuleDiskMinorThreshold3>70</ModuleDiskMinorThreshold3>\n        <ModuleMemCriticalThreshold3>90</ModuleMemCriticalThreshold3>\n        <ModuleMemMajorThreshold3>0</ModuleMemMajorThreshold3>\n        <ModuleMemMinorThreshold3>0</ModuleMemMinorThreshold3>\n        <ModuleSwapCriticalThreshold3>90</ModuleSwapCriticalThreshold3>\n        <ModuleSwapMajorThreshold3>80</ModuleSwapMajorThreshold3>\n        <ModuleSwapMinorThreshold3>70</ModuleSwapMinorThreshold3>\n        <ModuleDiskMonitorFileSystem1-3>/</ModuleDiskMonitorFileSystem1-3>\n        <ModuleDBRootCount1-3>1</ModuleDBRootCount1-3>\n        <ModuleDBRootID1-1-3>1</ModuleDBRootID1-1-3>\n    </SystemModuleConfig>\n    <SystemExtDeviceConfig>\n        <Count>0</Count>\n        <Name1>unassigned</Name1>\n        <IPAddr1>0.0.0.0</IPAddr1>\n        <DisableState1>ENABLED</DisableState1>\n    </SystemExtDeviceConfig>\n    <SessionManager>\n        <MaxConcurrentTransactions>1000</MaxConcurrentTransactions>\n        <TxnIDFile>/var/lib/columnstore/data1/systemFiles/dbrm/SMTxnID</TxnIDFile>\n    </SessionManager>\n    <VersionBuffer>\n        <VersionBufferFileSize>1GB</VersionBufferFileSize>\n    </VersionBuffer>\n    <OIDManager>\n        <OIDBitmapFile>/var/lib/columnstore/data1/systemFiles/dbrm/oidbitmap</OIDBitmapFile>\n        <FirstOID>3000</FirstOID>\n    </OIDManager>\n    <WriteEngine>\n        <BulkRoot>/var/lib/columnstore/data/bulk</BulkRoot>\n        <BulkRollbackDir>/var/lib/columnstore/data1/systemFiles/bulkRollback</BulkRollbackDir>\n        <MaxFileSystemDiskUsagePct>98</MaxFileSystemDiskUsagePct>\n        <CompressedPaddingBlocks>1</CompressedPaddingBlocks>\n    </WriteEngine>\n    <DBRM_Controller>\n        <NumWorkers>1</NumWorkers>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8616</Port>\n    </DBRM_Controller>\n    <DBRM_Worker1>\n        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>\n        <Port>8700</Port>\n        <Module>pm1</Module>\n    </DBRM_Worker1>\n    <DBRM_Worker2>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker2>\n    <DBRM_Worker3>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker3>\n    <DBRM_Worker4>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker4>\n    <DBRM_Worker5>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker5>\n    <DBRM_Worker6>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker6>\n    <DBRM_Worker7>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker7>\n    <DBRM_Worker8>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker8>\n    <DBRM_Worker9>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker9>\n    <DBRM_Worker10>\n        <IPAddr>0.0.0.0</IPAddr>\n        <Port>8700</Port>\n        <Module>unassigned</Module>\n    </DBRM_Worker10>\n    <DBBC>\n        <NumCaches>1</NumCaches>\n        <IOMTracing>0</IOMTracing>\n        <BRPTracing>0</BRPTracing>\n        <ReportFrequency>65536</ReportFrequency>\n        <MaxOpenFiles>2K</MaxOpenFiles>\n        <DecreaseOpenFilesCount>200</DecreaseOpenFilesCount>\n        <FDCacheTrace>0</FDCacheTrace>\n        <NumBlocksPct>50</NumBlocksPct>\n    </DBBC>\n    <Installation>\n        <SystemStartupOffline>n</SystemStartupOffline>\n        <InitialInstallFlag>y</InitialInstallFlag>\n        <SingleServerInstall>y</SingleServerInstall>\n        <ServerTypeInstall>2</ServerTypeInstall>\n        <PMwithUM>n</PMwithUM>\n        <MySQLRep>n</MySQLRep>\n        <DBRootStorageType>StorageManager</DBRootStorageType>\n        <UMStorageType>internal</UMStorageType>\n        <EEPackageType>binary</EEPackageType>\n        <SystemLogConfigFile>/etc/rsyslog.d/columnstore.conf</SystemLogConfigFile>\n        <Cloud>unassigned</Cloud>\n        <AmazonVPCNextPrivateIP>autoassign</AmazonVPCNextPrivateIP>\n        <UMInstanceType>unassigned</UMInstanceType>\n        <UMSecurityGroup>unassigned</UMSecurityGroup>\n        <UMVolumeSize>unassigned</UMVolumeSize>\n        <UMVolumeType>gp2</UMVolumeType>\n        <UMVolumeIOPS>unassigned</UMVolumeIOPS>\n        <PMInstanceType>unassigned</PMInstanceType>\n        <PMVolumeSize>unassigned</PMVolumeSize>\n        <PMVolumeType>gp2</PMVolumeType>\n        <PMVolumeIOPS>unassigned</PMVolumeIOPS>\n        <AmazonPMFailover>y</AmazonPMFailover>\n        <AmazonAutoTagging>y</AmazonAutoTagging>\n        <AmazonElasticIPCount>0</AmazonElasticIPCount>\n        <AmazonElasticIPAddr1>unassigned</AmazonElasticIPAddr1>\n        <AmazonElasticModule1>unassigned</AmazonElasticModule1>\n        <DataRedundancyConfig>n</DataRedundancyConfig>\n        <DataRedundancyCopies>0</DataRedundancyCopies>\n        <DataRedundancyStorageType>unassigned</DataRedundancyStorageType>\n        <CoreFileFlag>n</CoreFileFlag>\n        <MySQLPort>3306</MySQLPort>\n        <AmazonDeviceName>/dev/xvd</AmazonDeviceName>\n        <LockFileDirectory>/var/lock/subsys</LockFileDirectory>\n        <ProfileFile>/etc/profile.d/columnstoreAlias.sh</ProfileFile>\n        <DataRedundancyNetworkType/>\n    </Installation>\n    <ExtentMap>\n        <FilesPerColumnPartition>4</FilesPerColumnPartition>\n        <ExtentsPerSegmentFile>2</ExtentsPerSegmentFile>\n        <BRM_UID>0x0</BRM_UID>\n    </ExtentMap>\n    <HashJoin>\n        <MaxBuckets>128</MaxBuckets>\n        <MaxElems>128K</MaxElems>\n        <FifoSizeLargeSide>64</FifoSizeLargeSide>\n        <PmMaxMemorySmallSide>1G</PmMaxMemorySmallSide>\n        <TotalUmMemory>25%</TotalUmMemory>\n        <TotalPmUmMemory>100%</TotalPmUmMemory>\n        <CPUniqueLimit>100</CPUniqueLimit>\n        <AllowDiskBasedJoin>N</AllowDiskBasedJoin>\n        <TempFileCompression>Y</TempFileCompression>\n    </HashJoin>\n    <JobList>\n        <FlushInterval>16K</FlushInterval>\n        <FifoSize>16</FifoSize>\n        <RequestSize>1</RequestSize>\n        <ThreadPoolSize>100</ThreadPoolSize>\n    </JobList>\n    <TupleWSDL>\n        <MaxSize>1M</MaxSize>\n        <InitialCapacity>1M</InitialCapacity>\n        <MaxBuckets>512</MaxBuckets>\n    </TupleWSDL>\n    <RowAggregation>\n\t</RowAggregation>\n    <CrossEngineSupport>\n        <Host>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Host>\n        <Port>3306</Port>\n        <User>cej</User>\n        <Password>C0lumnStore!</Password>\n        <TLSCA/>\n        <TLSClientCert/>\n        <TLSClientKey/>\n    </CrossEngineSupport>\n    <QueryStats>\n        <Enabled>N</Enabled>\n    </QueryStats>\n    <UserPriority>\n        <Enabled>N</Enabled>\n    </UserPriority>\n    <NetworkCompression>\n        <Enabled>Y</Enabled>\n    </NetworkCompression>\n    <QueryTele>\n        <Host>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Host>\n        <Port>0</Port>\n    </QueryTele>\n    <StorageManager>\n        <MaxSockets>30</MaxSockets>\n        <Enabled>Y</Enabled>\n    </StorageManager>\n    <DataRedundancyConfig>\n        <DBRoot1PMs/>\n    </DataRedundancyConfig>\n    <ConfigRevision>1</ConfigRevision>\n    <ClusterManager>cs-test-mdb-cs-node-0.cs-test-mdb-clust</ClusterManager>\n    <ClusterName>MyCluster</ClusterName>\n    <NextNodeId>2</NextNodeId>\n    <NextDBRootId>2</NextDBRootId>\n    <DesiredNodes>\n        <Node>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Node>\n    </DesiredNodes>\n    <ActiveNodes>\n        <Node>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Node>\n    </ActiveNodes>\n    <InactiveNodes/>\n    <PrimaryNode>cs-test-mdb-cs-node-0.cs-test-mdb-clust</PrimaryNode>\n    <SharedStorage>yes</SharedStorage>\n</Columnstore>', 'cs_config_filename': '/etc/columnstore/Columnstore.xml'}
[20/Aug/2020 18:08:12] root  This node is not in the current ActiveNodes section.  Not starting Columnstore.
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-dmlproc. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-dmlproc 0
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-ddlproc. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-ddlproc 0
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-primproc. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-primproc 0
[20/Aug/2020 18:08:12] root  This node has no entry in the list of desired nodes
[20/Aug/2020 18:08:12] root  Loaded the config file, my name is None
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-writeengineserver. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-writeengineserver 0
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-exemgr. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-exemgr 0
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-controllernode. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-controllernode 0
[20/Aug/2020 18:08:12] root  apply Running stop on mcs-workernode. With sudo False and is_primary False
[20/Aug/2020 18:08:12] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-workernode 0
[20/Aug/2020 18:08:14] root  apply Running stop on mcs-storagemanager. With sudo False and is_primary False
[20/Aug/2020 18:08:14] root  stop running /opt/cmapi/mcs_node_control/custom_dispatchers/container.sh stop mcs-storagemanager 0
10.0.0.166 - - [20/Aug/2020:18:08:14] "PUT /cmapi/0.4.0/node/config HTTP/1.1" 200 43 "" "python-requests/2.23.0"
[20/Aug/2020 18:08:14] root  broadcast_new_config(): successfully pushed the new config file to ['cs-test-mdb-cs-node-0.cs-test-mdb-clust']
10.0.0.166 - - [20/Aug/2020 18:08:14] cmapi_server DEBUG    put_commit starts
10.0.0.166 - - [20/Aug/2020 18:08:14] cmapi_server DEBUG    put_commit returns {'timestamp': '2020-08-20 18:08:14.211940'}
10.0.0.166 - - [20/Aug/2020:18:08:14] "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43 "" "python-requests/2.23.0"
10.0.0.166 - - [20/Aug/2020 18:08:14] cmapi_server DEBUG    put_add_node returns {'timestamp': '2020-08-20 18:08:12.795427', 'node_id': 'cs-test-mdb-cs-node-0.cs-test-mdb-clust'}
10.0.0.166 - - [20/Aug/2020:18:08:14] "PUT /cmapi/0.4.0/cluster/add-node HTTP/1.1" 200 97 "" "curl/7.61.1"
10.0.0.166 - - [20/Aug/2020 18:10:41] cmapi_server DEBUG    get_status starts
10.0.0.166 - - [20/Aug/2020 18:10:41] cmapi_server DEBUG    get_status starts
[20/Aug/2020 18:10:41] root  Cannot establish or use DBRM connection. [Errno 111] Connection refused
10.0.0.166 - - [20/Aug/2020 18:10:41] cmapi_server DEBUG    get_status returns {'timestamp': '2020-08-20 18:10:41.515981', 'uptime': 1783, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': [], 'module_id': 1, 'services': []}
10.0.0.166 - - [20/Aug/2020:18:10:41] "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 156 "" "curl/7.61.1"
10.0.0.166 - - [20/Aug/2020 18:10:41] cmapi_server DEBUG    get_status returns {'timestamp': '2020-08-20 18:10:41.507126', 'cs-test-mdb-cs-node-0.cs-test-mdb-clust': {'timestamp': '2020-08-20 18:10:41.515981', 'uptime': 1783, 'dbrm_mode': 'offline', 'cluster_mode': 'readonly', 'dbroots': [], 'module_id': 1, 'services': []}, 'num_nodes': 1}
10.0.0.166 - - [20/Aug/2020:18:10:41] "GET /cmapi/0.4.0/cluster/status HTTP/1.1" 200 262 "" "curl/7.61.1"
 
------
columnstore-init.log
Removing gs://mdb0002756-1595842761951-columnstore-db00005735-79658635/087b1561-a13e-4d19-ae01-70d375532e36_0_2097152_data1~000.dir~000.dir~008.dir~016.dir~000.dir~FILE000.cdf...
Removing gs://mdb0002756-1595842761951-columnstore-db00005735-79658635/00a05b2e-e403-4dbe-b9e1-50066a396af8_0_2097152_data1~000.dir~000.dir~008.dir~025.dir~000.dir~FILE000.cdf...
Removing gs://mdb0002756-1595842761951-columnstore-db00005735-79658635/007d515e-8e44-449d-98b2-c18dea916a9b_0_8_data1~systemFiles~dbrm~BRM_savesA_vss...
Removing gs://mdb0002756-1595842761951-columnstore-db00005735-79658635/00ceca06-c6ea-4377-af91-e69668a1e47e_0_3436_data1~systemFiles~dbrm~BRM_savesA_em...
Removing gs://mdb0002756-1595842761951-columnstore-db00005735-79658635/0252c561-d480-4b53-ac20-e22c20f0d781_0_1048576_data1~000.dir~000.dir~004.dir~007.dir~000.dir~FILE000.cdf...
CommandException: 5 files/objects could not be removed.
Starting Primary Node...
StorageManager PID = 318
loading BRM
Error reading data1/systemFiles/dbrm/BRM_saves_current: No such file or directory
workernode PID = 333
controllernode PID = 336
PrimProc PID = 339
ExeMgr PID = 371
WriteEngineServer PID = 374
DMLProc PID = 389
DDLProc PID = 392
Starting MariaDB.200820 18:06:17 mysqld_safe Logging to '/var/log/mariadb/columnstore/mariadb-error.log'.
200820 18:06:17 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
 SUCCESS!
Cross engine join user created
Replication user created
MaxScale user created
Stopping Primary Node...
Shutting down MariaDB. SUCCESS!
Saved to /var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves
Multi-node with local-storage detected.
Sending SIGTERM to Columnstore processes
/usr/bin/columnstore-init: line 262:   339 Terminated              $MCS_INSTALL_BIN/PrimProc &> $LOG_PREFIX/primproc.log
/usr/bin/columnstore-init: line 262:   371 Terminated              $MCS_INSTALL_BIN/ExeMgr &> $LOG_PREFIX/exemgr.log
/usr/bin/columnstore-init: line 262:   374 Terminated              $MCS_INSTALL_BIN/WriteEngineServer &> $LOG_PREFIX/writeengineserver.log
/usr/bin/columnstore-init: line 262:   389 Terminated              $MCS_INSTALL_BIN/DMLProc &> $LOG_PREFIX/dmlproc.log
/usr/bin/columnstore-init: line 262:   392 Terminated              $MCS_INSTALL_BIN/DDLProc &> $LOG_PREFIX/ddlproc.log
Sending SIGKILL to ColumnStore processes
Clearing shared memory...
 
------
controllernode.log
DBRM Controller: Waiting for threads to finish...
Exiting...
 
------
corefiles
cat: /var/log/mariadb/columnstore/corefiles: Is a directory
 
------
dbbuilder.log
Creating System Catalog...
 
Creating SYSTABLE
---------------------------------------
  Creating TableName column OID: 1001
  Creating TableName column dictionary
  Creating Schema column OID: 1002
  Creating Schema column dictionary
  Creating ObjectId column OID: 1003
  Creating CreateDate column OID: 1004
  Creating LastUpdate column OID: 1005
  Creating INIT column OID: 1006
  Creating NEXT column OID: 1007
  Creating NUMOFROWS column OID: 1008
  Creating AVGROWLEN column OID: 1009
  Creating NUMOFBLOCKS column OID: 1010
  Creating AUTOINCREMENT column OID: 1011
 
Creating SYSCOLUMN
---------------------------------------
  Creating Schema column OID: 1021
  Creating Schema column dictionary...
  Creating TableName column OID: 1022
  Creating TableName column dictionary...
  Creating ColumnName column OID: 1023
  Creating ColumnName column dictionary...
  Creating ObjectID column OID: 1024
  Creating DictOID column OID: 1025
  Creating ListOID column OID: 1026
  Creating TreeOID column OID: 1027
  Creating DataType column OID: 1028
  Creating ColumnLength column OID: 1029
  Creating ColumnPos column OID: 1030
  Creating LastUpdate column OID: 1031
  Creating DefaultValue column OID: 1032
  Creating DefaultValue column dictionary...
  Creating Nullable column OID: 1033
  Creating Scale column OID: 1034
  Creating Precision column OID: 1035
  Creating AutoInc column OID: 1036
  Creating DISTCOUNT column OID: 1037
  Creating NULLCOUNT column OID: 1038
  Creating MINVALUE column OID: 1039
  Creating MINVALUE column dictionary...
  Creating MAXVALUE column OID: 1040
  Creating MAXVALUE column dictionary...
  Creating CompressionType column OID: 1041
  Creating NEXTVALUE column OID: 1042
System Catalog creation took: 0.452925 seconds to complete.
 
System Catalog created
 
Saved to /var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves
 
------
ddlproc.log
DDLProc is ready...
 
------
debug.log
Aug 20 18:05:57 cs-test-mdb-cs-node-0 StorageManager[318]: Using the config file found at /etc/columnstore/storagemanager.cnf
Aug 20 18:06:12 cs-test-mdb-cs-node-0 StorageManager[327]: Using the config file found at /etc/columnstore/storagemanager.cnf
Aug 20 18:06:12 cs-test-mdb-cs-node-0 IDBFile[333]: 12.941968 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:12 cs-test-mdb-cs-node-0 IDBFile[348]: 12.974113 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:12 cs-test-mdb-cs-node-0 StorageManager[318]: Ownership: taking ownership of data1
Aug 20 18:06:13 cs-test-mdb-cs-node-0 controllernode[348]: 13.034426 |0|0|0| D 29 CAL0000: TableLockServer::load(): could not open the save file/var/lib/columnstore/data1/systemFiles/dbrm/tablelocks
Aug 20 18:06:13 cs-test-mdb-cs-node-0 IDBFile[339]: 13.036026 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:14 cs-test-mdb-cs-node-0 IDBFile[374]: 14.004685 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:16 cs-test-mdb-cs-node-0 IDBFile[389]: 16.047648 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:16 cs-test-mdb-cs-node-0 IDBFile[392]: 16.057264 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:16 cs-test-mdb-cs-node-0 IDBFile[393]: 16.067991 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:16 cs-test-mdb-cs-node-0 IDBFile[427]: 16.758494 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:19 cs-test-mdb-cs-node-0 IDBFile[831]: 19.771999 |0|0|0| D 35 CAL0002: IDBFactory::installPlugin: installed filesystem plugin libcloudio.so
Aug 20 18:06:19 cs-test-mdb-cs-node-0 StorageManager[318]: SessionManager Caught Signal 15
Aug 20 18:06:19 cs-test-mdb-cs-node-0 StorageManager[318]: Shutdown StorageManager...
Aug 20 18:06:20 cs-test-mdb-cs-node-0 StorageManager[318]: Ownership: releasing ownership of data1
 
------
dmlproc.log
DMLProc is ready...
 
------
exemgr.log
Starting ExeMgr: st = 50, qs = 20, mx = 95, cf = /etc/columnstore/Columnstore.xml
 
------
info.log
Aug 20 18:05:57 cs-test-mdb-cs-node-0 StorageManager[318]: Cache/cache_size = 8589934592
Aug 20 18:05:58 cs-test-mdb-cs-node-0 StorageManager[318]: S3Storage: S3 connectivity & permissions are OK
Aug 20 18:05:58 cs-test-mdb-cs-node-0 StorageManager[318]: max_concurrent_downloads = 21
Aug 20 18:05:58 cs-test-mdb-cs-node-0 StorageManager[318]: max_concurrent_uploads = 21
Aug 20 18:05:58 cs-test-mdb-cs-node-0 StorageManager[318]: SessionManager waiting for sockets.
Aug 20 18:06:16 cs-test-mdb-cs-node-0 DMLProc[389]: 16.065888 |0|0|0| I 20 CAL0002: DMLProc starts rollbackAll.
Aug 20 18:06:16 cs-test-mdb-cs-node-0 DMLProc[389]: 16.154070 |0|0|0| I 20 CAL0002: DMLProc will rollback 0 tables.
Aug 20 18:06:16 cs-test-mdb-cs-node-0 DMLProc[389]: 16.166109 |0|0|0| I 20 CAL0002: DMLProc finished rollbackAll.
Aug 20 18:06:20 cs-test-mdb-cs-node-0 StorageManager[318]: StorageManager Shutdown Complete.
 
------
mariadb-error.log
2020-08-20 18:06:17 0 [Warning] Plugin 'Columnstore' is of maturity level gamma while the server is stable
2020-08-20 18:06:17 0 [Warning] Plugin 'COLUMNSTORE_COLUMNS' is of maturity level gamma while the server is stable
2020-08-20 18:06:17 0 [Warning] Plugin 'COLUMNSTORE_TABLES' is of maturity level gamma while the server is stable
2020-08-20 18:06:17 0 [Warning] Plugin 'COLUMNSTORE_FILES' is of maturity level gamma while the server is stable
2020-08-20 18:06:17 0 [Warning] Plugin 'COLUMNSTORE_EXTENTS' is of maturity level gamma while the server is stable
200820 18:06:17 Columnstore: Started; Version: 1.5.4-1
2020-08-20 18:06:17 0 [Note] mariadbd: Aria engine: starting recovery
recovered pages: 0% 10% 20% 30% 43% 53% 88% 99% 100% (0.0 seconds); tables to flush: 2 1 0
 (0.0 seconds);
2020-08-20 18:06:17 0 [Note] mariadbd: Aria engine: recovery done
2020-08-20 18:06:17 0 [Note] InnoDB: Using Linux native AIO
2020-08-20 18:06:17 0 [Note] InnoDB: Uses event mutexes
2020-08-20 18:06:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-20 18:06:17 0 [Note] InnoDB: Number of pools: 1
2020-08-20 18:06:17 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2020-08-20 18:06:17 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-08-20 18:06:17 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2020-08-20 18:06:17 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-20 18:06:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-20 18:06:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=56438
2020-08-20 18:06:17 0 [Note] InnoDB: 128 rollback segments are active.
2020-08-20 18:06:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-08-20 18:06:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-20 18:06:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-20 18:06:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-20 18:06:17 0 [Note] InnoDB: 10.5.6 started; log sequence number 56450; transaction id 49
2020-08-20 18:06:17 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-20 18:06:17 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-08-20 18:06:17 0 [Note] InnoDB: Buffer pool(s) load completed at 200820 18:06:17
2020-08-20 18:06:17 0 [Note] Server socket created on IP: '::'.
2020-08-20 18:06:17 0 [Note] Reading of all Master_info entries succeeded
2020-08-20 18:06:17 0 [Note] Added new Master_info '' to hash table
2020-08-20 18:06:17 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.5.6-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
2020-08-20 18:06:18 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2020-08-20 18:06:18 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-08-20 18:06:18 0 [Note] InnoDB: FTS optimize thread exiting.
2020-08-20 18:06:18 0 [Note] InnoDB: Starting shutdown...
2020-08-20 18:06:18 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-08-20 18:06:18 0 [Note] InnoDB: Buffer pool(s) dump completed at 200820 18:06:18
2020-08-20 18:06:19 0 [Note] InnoDB: Shutdown completed; log sequence number 56474; transaction id 52
2020-08-20 18:06:19 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-08-20 18:06:19 0 [Note] /usr/sbin/mariadbd: Shutdown complete
 
2020-08-20 18:06:24 0 [Warning] Plugin 'Columnstore' is of maturity level gamma while the server is stable
2020-08-20 18:06:24 0 [Warning] Plugin 'COLUMNSTORE_COLUMNS' is of maturity level gamma while the server is stable
2020-08-20 18:06:24 0 [Warning] Plugin 'COLUMNSTORE_TABLES' is of maturity level gamma while the server is stable
2020-08-20 18:06:24 0 [Warning] Plugin 'COLUMNSTORE_FILES' is of maturity level gamma while the server is stable
2020-08-20 18:06:24 0 [Warning] Plugin 'COLUMNSTORE_EXTENTS' is of maturity level gamma while the server is stable
200820 18:06:24 Columnstore: Started; Version: 1.5.4-1
2020-08-20 18:06:24 0 [Note] InnoDB: Using Linux native AIO
2020-08-20 18:06:24 0 [Note] InnoDB: Uses event mutexes
2020-08-20 18:06:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-20 18:06:24 0 [Note] InnoDB: Number of pools: 1
2020-08-20 18:06:24 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2020-08-20 18:06:24 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-08-20 18:06:24 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2020-08-20 18:06:24 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-20 18:06:24 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-20 18:06:24 0 [Note] InnoDB: 128 rollback segments are active.
2020-08-20 18:06:24 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-20 18:06:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-20 18:06:24 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-20 18:06:24 0 [Note] InnoDB: 10.5.6 started; log sequence number 56474; transaction id 49
2020-08-20 18:06:24 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-20 18:06:24 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-08-20 18:06:24 0 [Note] InnoDB: Buffer pool(s) load completed at 200820 18:06:24
2020-08-20 18:06:24 0 [Note] Server socket created on IP: '::'.
2020-08-20 18:06:24 0 [Note] Reading of all Master_info entries succeeded
2020-08-20 18:06:24 0 [Note] Added new Master_info '' to hash table
2020-08-20 18:06:24 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.5.6-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
2020-08-20 18:06:25 4 [Note] Start binlog_dump to slave_server(126), pos(, 4), using_gtid(1), gtid('')
2020-08-20 18:06:28 6 [Note] Start binlog_dump to slave_server(126), pos(mariadb-bin.000002, 344), using_gtid(1), gtid('0-125-11')
 
------
primproc.log
Starting PrimitiveServer: st = 1, sq = 10, pw = 128, pq = 10240, nb = 478531, nt = 4, nc = 1, ra = 512, db = 128, mb = 512, rd = 0, tr = 0, ss = 67108864, bp = 4
 
------
storagemanager.log
 
------
trace
cat: /var/log/mariadb/columnstore/trace: Is a directory
 
------
workernode.log
"/tmp/columnstore_tmp_files/rdwrscratch"
 
------
writeengineserver.log
WriteEngineServer is ready

[root@cs-test-mdb-cs-node-0 /]# cat /etc/columnstore/Columnstore.xml
<?xml version="1.0" ?>
<Columnstore Version="V1.0.0">
    <ExeMgr1>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8601</Port>
        <Module>unassigned</Module>
    </ExeMgr1>
    <JobProc>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8602</Port>
    </JobProc>
    <ProcMgr>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8603</Port>
    </ProcMgr>
    <ProcMgr_Alarm>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8606</Port>
    </ProcMgr_Alarm>
    <ProcStatusControl>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8604</Port>
    </ProcStatusControl>
    <ProcStatusControlStandby>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8605</Port>
    </ProcStatusControlStandby>
    <localhost_ProcessMonitor>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8800</Port>
    </localhost_ProcessMonitor>
    <dm1_ProcessMonitor>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8800</Port>
    </dm1_ProcessMonitor>
    <um1_ProcessMonitor>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8800</Port>
    </um1_ProcessMonitor>
    <pm1_ProcessMonitor>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8800</Port>
    </pm1_ProcessMonitor>
    <dm1_ServerMonitor>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8622</Port>
    </dm1_ServerMonitor>
    <um1_ServerMonitor>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8622</Port>
    </um1_ServerMonitor>
    <pm1_ServerMonitor>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8622</Port>
    </pm1_ServerMonitor>
    <pm1_WriteEngineServer>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8630</Port>
    </pm1_WriteEngineServer>
    <DDLProc>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8612</Port>
    </DDLProc>
    <DMLProc>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8614</Port>
    </DMLProc>
    <BatchInsert>
        <RowsPerBatch>10000</RowsPerBatch>
    </BatchInsert>
    <PrimitiveServers>
        <Count>1</Count>
        <ConnectionsPerPrimProc>2</ConnectionsPerPrimProc>
        <ProcessorThreshold>128</ProcessorThreshold>
        <ProcessorQueueSize>10K</ProcessorQueueSize>
        <DebugLevel>0</DebugLevel>
        <LBID_Shift>13</LBID_Shift>
        <ColScanBufferSizeBlocks>512</ColScanBufferSizeBlocks>
        <ColScanReadAheadBlocks>512</ColScanReadAheadBlocks>
        <PrefetchThreshold>1</PrefetchThreshold>
        <PTTrace>0</PTTrace>
        <RotatingDestination>n</RotatingDestination>
        <DirectIO>y</DirectIO>
        <HighPriorityPercentage/>
        <MediumPriorityPercentage/>
        <LowPriorityPercentage/>
    </PrimitiveServers>
    <PMS1>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS1>
    <PMS2>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS2>
    <PMS3>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS3>
    <PMS4>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS4>
    <PMS5>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS5>
    <PMS6>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS6>
    <PMS7>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS7>
    <PMS8>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS8>
    <PMS9>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS9>
    <PMS10>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS10>
    <PMS11>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS11>
    <PMS12>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS12>
    <PMS13>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS13>
    <PMS14>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS14>
    <PMS15>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS15>
    <PMS16>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS16>
    <PMS17>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS17>
    <PMS18>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS18>
    <PMS19>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS19>
    <PMS20>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS20>
    <PMS21>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS21>
    <PMS22>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS22>
    <PMS23>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS23>
    <PMS24>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS24>
    <PMS25>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS25>
    <PMS26>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS26>
    <PMS27>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS27>
    <PMS28>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS28>
    <PMS29>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS29>
    <PMS30>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS30>
    <PMS31>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS31>
    <PMS32>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8620</Port>
    </PMS32>
    <SystemConfig>
        <SystemName>columnstore-1</SystemName>
        <ParentOAMModuleName>pm1</ParentOAMModuleName>
        <StandbyOAMModuleName>unassigned</StandbyOAMModuleName>
        <PrimaryUMModuleName>pm1</PrimaryUMModuleName>
        <ModuleHeartbeatPeriod>1</ModuleHeartbeatPeriod>
        <ModuleHeartbeatCount>3</ModuleHeartbeatCount>
        <ModuleProcMonWaitCount>12</ModuleProcMonWaitCount>
                // 2.5 minutes
        <DBRootCount>1</DBRootCount>
        <DBRoot1>/var/lib/columnstore/data1</DBRoot1>
        <DBRMRoot>/var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves</DBRMRoot>
        <TableLockSaveFile>/var/lib/columnstore/data1/systemFiles/dbrm/tablelocks</TableLockSaveFile>
        <DBRMTimeOut>15</DBRMTimeOut>
        <DBRMSnapshotInterval>100000</DBRMSnapshotInterval>
        <ExternalCriticalThreshold>90</ExternalCriticalThreshold>
        <ExternalMajorThreshold>80</ExternalMajorThreshold>
        <ExternalMinorThreshold>70</ExternalMinorThreshold>
        <TransactionArchivePeriod>10</TransactionArchivePeriod>
        <NMSIPAddress>0.0.0.0</NMSIPAddress>
        <TempSaveSize>128M</TempSaveSize>
        <WaitPeriod>10</WaitPeriod>
        <ProcessRestartCount>10</ProcessRestartCount>
        <ProcessRestartPeriod>120</ProcessRestartPeriod>
        <SwapAction>restartSystem</SwapAction>
        <ActivePmFailoverDisabled>n</ActivePmFailoverDisabled>
        <MemoryCheckPercent>95</MemoryCheckPercent>
        <DataFileLog>OFF</DataFileLog>
        <hdfsRdwrScratch>/rdwrscratch</hdfsRdwrScratch>
        <TempFileDir>/columnstore_tmp_files</TempFileDir>
        <SystemTempFileDir>/tmp/columnstore_tmp_files</SystemTempFileDir>
        <DataFilePlugin>libcloudio.so</DataFilePlugin>
    </SystemConfig>
    <SystemModuleConfig>
        <ModuleType1>dm</ModuleType1>
        <ModuleDesc1>Director Module</ModuleDesc1>
        <RunType1>SIMPLEX</RunType1>
        <ModuleCount1>0</ModuleCount1>
        <ModuleIPAddr1-1-1>0.0.0.0</ModuleIPAddr1-1-1>
        <ModuleHostName1-1-1>unassigned</ModuleHostName1-1-1>
        <ModuleDisableState1-1>ENABLED</ModuleDisableState1-1>
        <ModuleCPUCriticalThreshold1>0</ModuleCPUCriticalThreshold1>
        <ModuleCPUMajorThreshold1>0</ModuleCPUMajorThreshold1>
        <ModuleCPUMinorThreshold1>0</ModuleCPUMinorThreshold1>
        <ModuleCPUMinorClearThreshold1>0</ModuleCPUMinorClearThreshold1>
        <ModuleDiskCriticalThreshold1>90</ModuleDiskCriticalThreshold1>
        <ModuleDiskMajorThreshold1>80</ModuleDiskMajorThreshold1>
        <ModuleDiskMinorThreshold1>70</ModuleDiskMinorThreshold1>
        <ModuleMemCriticalThreshold1>90</ModuleMemCriticalThreshold1>
        <ModuleMemMajorThreshold1>0</ModuleMemMajorThreshold1>
        <ModuleMemMinorThreshold1>0</ModuleMemMinorThreshold1>
        <ModuleSwapCriticalThreshold1>90</ModuleSwapCriticalThreshold1>
        <ModuleSwapMajorThreshold1>80</ModuleSwapMajorThreshold1>
        <ModuleSwapMinorThreshold1>70</ModuleSwapMinorThreshold1>
        <ModuleDiskMonitorFileSystem1-1>/</ModuleDiskMonitorFileSystem1-1>
        <ModuleDBRootCount1-1>unassigned</ModuleDBRootCount1-1>
        <ModuleDBRootID1-1-1>unassigned</ModuleDBRootID1-1-1>
        <ModuleType2>um</ModuleType2>
        <ModuleDesc2>User Module</ModuleDesc2>
        <RunType2>SIMPLEX</RunType2>
        <ModuleCount2>0</ModuleCount2>
        <ModuleIPAddr1-1-2>0.0.0.0</ModuleIPAddr1-1-2>
        <ModuleHostName1-1-2>unassigned</ModuleHostName1-1-2>
        <ModuleDisableState1-2>ENABLED</ModuleDisableState1-2>
        <ModuleCPUCriticalThreshold2>0</ModuleCPUCriticalThreshold2>
        <ModuleCPUMajorThreshold2>0</ModuleCPUMajorThreshold2>
        <ModuleCPUMinorThreshold2>0</ModuleCPUMinorThreshold2>
        <ModuleCPUMinorClearThreshold2>0</ModuleCPUMinorClearThreshold2>
        <ModuleDiskCriticalThreshold2>90</ModuleDiskCriticalThreshold2>
        <ModuleDiskMajorThreshold2>80</ModuleDiskMajorThreshold2>
        <ModuleDiskMinorThreshold2>70</ModuleDiskMinorThreshold2>
        <ModuleMemCriticalThreshold2>90</ModuleMemCriticalThreshold2>
        <ModuleMemMajorThreshold2>0</ModuleMemMajorThreshold2>
        <ModuleMemMinorThreshold2>0</ModuleMemMinorThreshold2>
        <ModuleSwapCriticalThreshold2>90</ModuleSwapCriticalThreshold2>
        <ModuleSwapMajorThreshold2>80</ModuleSwapMajorThreshold2>
        <ModuleSwapMinorThreshold2>70</ModuleSwapMinorThreshold2>
        <ModuleDiskMonitorFileSystem1-2>/</ModuleDiskMonitorFileSystem1-2>
        <ModuleDBRootCount1-2>unassigned</ModuleDBRootCount1-2>
        <ModuleDBRootID1-1-2>unassigned</ModuleDBRootID1-1-2>
        <ModuleType3>pm</ModuleType3>
        <ModuleDesc3>Performance Module</ModuleDesc3>
        <RunType3>SIMPLEX</RunType3>
        <ModuleCount3>1</ModuleCount3>
        <ModuleIPAddr1-1-3>10.0.0.166</ModuleIPAddr1-1-3>
        <ModuleHostName1-1-3>cs-test-mdb-cs-node-0.cs-test-mdb-clust</ModuleHostName1-1-3>
        <ModuleDisableState1-3>ENABLED</ModuleDisableState1-3>
        <ModuleCPUCriticalThreshold3>0</ModuleCPUCriticalThreshold3>
        <ModuleCPUMajorThreshold3>0</ModuleCPUMajorThreshold3>
        <ModuleCPUMinorThreshold3>0</ModuleCPUMinorThreshold3>
        <ModuleCPUMinorClearThreshold3>0</ModuleCPUMinorClearThreshold3>
        <ModuleDiskCriticalThreshold3>90</ModuleDiskCriticalThreshold3>
        <ModuleDiskMajorThreshold3>80</ModuleDiskMajorThreshold3>
        <ModuleDiskMinorThreshold3>70</ModuleDiskMinorThreshold3>
        <ModuleMemCriticalThreshold3>90</ModuleMemCriticalThreshold3>
        <ModuleMemMajorThreshold3>0</ModuleMemMajorThreshold3>
        <ModuleMemMinorThreshold3>0</ModuleMemMinorThreshold3>
        <ModuleSwapCriticalThreshold3>90</ModuleSwapCriticalThreshold3>
        <ModuleSwapMajorThreshold3>80</ModuleSwapMajorThreshold3>
        <ModuleSwapMinorThreshold3>70</ModuleSwapMinorThreshold3>
        <ModuleDiskMonitorFileSystem1-3>/</ModuleDiskMonitorFileSystem1-3>
        <ModuleDBRootCount1-3>1</ModuleDBRootCount1-3>
        <ModuleDBRootID1-1-3>1</ModuleDBRootID1-1-3>
    </SystemModuleConfig>
    <SystemExtDeviceConfig>
        <Count>0</Count>
        <Name1>unassigned</Name1>
        <IPAddr1>0.0.0.0</IPAddr1>
        <DisableState1>ENABLED</DisableState1>
    </SystemExtDeviceConfig>
    <SessionManager>
        <MaxConcurrentTransactions>1000</MaxConcurrentTransactions>
        <TxnIDFile>/var/lib/columnstore/data1/systemFiles/dbrm/SMTxnID</TxnIDFile>
    </SessionManager>
    <VersionBuffer>
        <VersionBufferFileSize>1GB</VersionBufferFileSize>
    </VersionBuffer>
    <OIDManager>
        <OIDBitmapFile>/var/lib/columnstore/data1/systemFiles/dbrm/oidbitmap</OIDBitmapFile>
        <FirstOID>3000</FirstOID>
    </OIDManager>
    <WriteEngine>
        <BulkRoot>/var/lib/columnstore/data/bulk</BulkRoot>
        <BulkRollbackDir>/var/lib/columnstore/data1/systemFiles/bulkRollback</BulkRollbackDir>
        <MaxFileSystemDiskUsagePct>98</MaxFileSystemDiskUsagePct>
        <CompressedPaddingBlocks>1</CompressedPaddingBlocks>
    </WriteEngine>
    <DBRM_Controller>
        <NumWorkers>1</NumWorkers>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8616</Port>
    </DBRM_Controller>
    <DBRM_Worker1>
        <IPAddr>cs-test-mdb-cs-node-0.cs-test-mdb-clust</IPAddr>
        <Port>8700</Port>
        <Module>pm1</Module>
    </DBRM_Worker1>
    <DBRM_Worker2>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker2>
    <DBRM_Worker3>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker3>
    <DBRM_Worker4>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker4>
    <DBRM_Worker5>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker5>
    <DBRM_Worker6>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker6>
    <DBRM_Worker7>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker7>
    <DBRM_Worker8>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker8>
    <DBRM_Worker9>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker9>
    <DBRM_Worker10>
        <IPAddr>0.0.0.0</IPAddr>
        <Port>8700</Port>
        <Module>unassigned</Module>
    </DBRM_Worker10>
    <DBBC>
        <NumCaches>1</NumCaches>
        <IOMTracing>0</IOMTracing>
        <BRPTracing>0</BRPTracing>
        <ReportFrequency>65536</ReportFrequency>
        <MaxOpenFiles>2K</MaxOpenFiles>
        <DecreaseOpenFilesCount>200</DecreaseOpenFilesCount>
        <FDCacheTrace>0</FDCacheTrace>
        <NumBlocksPct>50</NumBlocksPct>
    </DBBC>
    <Installation>
        <SystemStartupOffline>n</SystemStartupOffline>
        <InitialInstallFlag>y</InitialInstallFlag>
        <SingleServerInstall>y</SingleServerInstall>
        <ServerTypeInstall>2</ServerTypeInstall>
        <PMwithUM>n</PMwithUM>
        <MySQLRep>n</MySQLRep>
        <DBRootStorageType>StorageManager</DBRootStorageType>
        <UMStorageType>internal</UMStorageType>
        <EEPackageType>binary</EEPackageType>
        <SystemLogConfigFile>/etc/rsyslog.d/columnstore.conf</SystemLogConfigFile>
        <Cloud>unassigned</Cloud>
        <AmazonVPCNextPrivateIP>autoassign</AmazonVPCNextPrivateIP>
        <UMInstanceType>unassigned</UMInstanceType>
        <UMSecurityGroup>unassigned</UMSecurityGroup>
        <UMVolumeSize>unassigned</UMVolumeSize>
        <UMVolumeType>gp2</UMVolumeType>
        <UMVolumeIOPS>unassigned</UMVolumeIOPS>
        <PMInstanceType>unassigned</PMInstanceType>
        <PMVolumeSize>unassigned</PMVolumeSize>
        <PMVolumeType>gp2</PMVolumeType>
        <PMVolumeIOPS>unassigned</PMVolumeIOPS>
        <AmazonPMFailover>y</AmazonPMFailover>
        <AmazonAutoTagging>y</AmazonAutoTagging>
        <AmazonElasticIPCount>0</AmazonElasticIPCount>
        <AmazonElasticIPAddr1>unassigned</AmazonElasticIPAddr1>
        <AmazonElasticModule1>unassigned</AmazonElasticModule1>
        <DataRedundancyConfig>n</DataRedundancyConfig>
        <DataRedundancyCopies>0</DataRedundancyCopies>
        <DataRedundancyStorageType>unassigned</DataRedundancyStorageType>
        <CoreFileFlag>n</CoreFileFlag>
        <MySQLPort>3306</MySQLPort>
        <AmazonDeviceName>/dev/xvd</AmazonDeviceName>
        <LockFileDirectory>/var/lock/subsys</LockFileDirectory>
        <ProfileFile>/etc/profile.d/columnstoreAlias.sh</ProfileFile>
        <DataRedundancyNetworkType/>
    </Installation>
    <ExtentMap>
        <FilesPerColumnPartition>4</FilesPerColumnPartition>
        <ExtentsPerSegmentFile>2</ExtentsPerSegmentFile>
        <BRM_UID>0x0</BRM_UID>
    </ExtentMap>
    <HashJoin>
        <MaxBuckets>128</MaxBuckets>
        <MaxElems>128K</MaxElems>
        <FifoSizeLargeSide>64</FifoSizeLargeSide>
        <PmMaxMemorySmallSide>1G</PmMaxMemorySmallSide>
        <TotalUmMemory>25%</TotalUmMemory>
        <TotalPmUmMemory>100%</TotalPmUmMemory>
        <CPUniqueLimit>100</CPUniqueLimit>
        <AllowDiskBasedJoin>N</AllowDiskBasedJoin>
        <TempFileCompression>Y</TempFileCompression>
    </HashJoin>
    <JobList>
        <FlushInterval>16K</FlushInterval>
        <FifoSize>16</FifoSize>
        <RequestSize>1</RequestSize>
        <ThreadPoolSize>100</ThreadPoolSize>
    </JobList>
    <TupleWSDL>
        <MaxSize>1M</MaxSize>
        <InitialCapacity>1M</InitialCapacity>
        <MaxBuckets>512</MaxBuckets>
    </TupleWSDL>
    <RowAggregation>
        </RowAggregation>
    <CrossEngineSupport>
        <Host>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Host>
        <Port>3306</Port>
        <User>cej</User>
        <Password>C0lumnStore!</Password>
        <TLSCA/>
        <TLSClientCert/>
        <TLSClientKey/>
    </CrossEngineSupport>
    <QueryStats>
        <Enabled>N</Enabled>
    </QueryStats>
    <UserPriority>
        <Enabled>N</Enabled>
    </UserPriority>
    <NetworkCompression>
        <Enabled>Y</Enabled>
    </NetworkCompression>
    <QueryTele>
        <Host>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Host>
        <Port>0</Port>
    </QueryTele>
    <StorageManager>
        <MaxSockets>30</MaxSockets>
        <Enabled>Y</Enabled>
    </StorageManager>
    <DataRedundancyConfig>
        <DBRoot1PMs/>
    </DataRedundancyConfig>
    <ConfigRevision>1</ConfigRevision>
    <ClusterManager>cs-test-mdb-cs-node-0.cs-test-mdb-clust</ClusterManager>
    <ClusterName>MyCluster</ClusterName>
    <NextNodeId>2</NextNodeId>
    <NextDBRootId>2</NextDBRootId>
    <DesiredNodes>
        <Node>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Node>
    </DesiredNodes>
    <ActiveNodes>
        <Node>cs-test-mdb-cs-node-0.cs-test-mdb-clust</Node>
    </ActiveNodes>
    <InactiveNodes/>
    <PrimaryNode>cs-test-mdb-cs-node-0.cs-test-mdb-clust</PrimaryNode>
    <SharedStorage>yes</SharedStorage>



 Comments   
Comment by Jens Röwekamp (Inactive) [ 2020-08-20 ]

Something is wrong with the shorter DNS record in k8s.
If the complete long DNS record "cs-test-mdb-cs-node-0.cs-test-mdb-clust.db00005735.svc.cluster.local" is used instead everything works okay.

Generated at Thu Feb 08 02:49:00 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.