20/Aug/2025 17:43:26 [INFO] (root) {MainThread} CMAPI Version: 23.10.6.1
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} Created self signed sertificate for CMAPI API access.
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} Detected 7 MCS services.MCS version is 22.08.x and higher
20/Aug/2025 17:43:26 [DEBUG] (root) {MainThread} Waiting for command to finish.
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} Detected 255 SYSTEMD version.
20/Aug/2025 17:43:26 [DEBUG] (root) {MainThread} get_module_net_address Module 1 network address 127.0.0.1
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} Loaded the config file, my name is None
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} Failover is turned ON by default or in CMAPI config file.
20/Aug/2025 17:43:26 [INFO] (cherrypy.error) {MainThread} ENGINE Bus STARTING
20/Aug/2025 17:43:26 [INFO] (cherrypy.error) {MainThread} ENGINE Serving on https://0.0.0.0:8640
20/Aug/2025 17:43:26 [INFO] (cherrypy.error) {MainThread} ENGINE Bus STARTED
20/Aug/2025 17:43:26 [INFO] (cherrypy.error) {HTTPServer Thread-3} ENGINE Client ('127.0.0.1', 45854) lost — peer dropped the TLS connection suddenly, during handshake: (8, 'EOF occurred in violation of protocol (_ssl.c:1129)')
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} get_current_config_file(): seeking the current config file
20/Aug/2025 17:43:26 [DEBUG] (root) {MainThread} get_module_net_address Module 1 network address 127.0.0.1
20/Aug/2025 17:43:26 [INFO] (root) {MainThread} wait_for_deactivation_or_put_config: Was removed from the cluster, safe to continue.
20/Aug/2025 17:43:26 [WARNING] (root) {MainThread} Something went wrong while trying to detect dbrm protocol.
Seems "controllernode" process isn't started.
This is just a notification, not a problem.
Next detection will started at first node\cluster status check.
This can cause extra 5 seconds delay while
first attempt to get status.
Traceback (most recent call last):
  File "/usr/share/columnstore/cmapi/cmapi_server/__main__.py", line 215, in <module>
    dbrm_socket.connect()
  File "/usr/share/columnstore/cmapi/mcs_node_control/models/dbrm_socket.py", line 208, in connect
    self._socket.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused
20/Aug/2025 17:43:26 [INFO] (cherrypy.error) {MainThread} ENGINE Starting Failover monitor thread.
20/Aug/2025 17:43:26 [INFO] (heartbeater) {HeartBeater} Starting the heartbeat listener.
20/Aug/2025 17:43:26 [INFO] (node_monitor) {NodeMonitor} Starting the monitor logic
20/Aug/2025 17:43:27 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 127.0.0.1 ACCESS "GET /cmapi/ready HTTP/1.1" code 200, bytes 17, user-agent "curl/8.5.0"
20/Aug/2025 17:43:27 [INFO] (node_monitor) {NodeMonitor} Failover support is inactive; requires at least 3 nodes and a shared storage system
20/Aug/2025 17:43:31 [DEBUG] (root) {NodeMonitor} get_module_net_address Module 1 network address 127.0.0.1
20/Aug/2025 17:43:31 [INFO] (root) {NodeMonitor} Loaded the config file, my name is None
20/Aug/2025 17:43:36 [INFO] (access_logger) {CP Server Thread-6} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/cluster/status". uid: 99331e6b-2565-4981-9a7b-4c721eb9adf2
20/Aug/2025 17:43:36 [WARNING] (cmapi_server) {CP Server Thread-6} No API key in the configuration. Adding it into the config.
20/Aug/2025 17:43:36 [DEBUG] (cmapi_server) {CP Server Thread-6} get_status starts
20/Aug/2025 17:43:36 [DEBUG] (cmapi_server) {CP Server Thread-6} Cluster status command called. Getting status.
20/Aug/2025 17:43:36 [DEBUG] (cmapi_server) {CP Server Thread-6} Successfully finished getting cluster status.
20/Aug/2025 17:43:36 [DEBUG] (cmapi_server) {CP Server Thread-6} get_status returns {'timestamp': '2025-08-20 17:43:36.701653', 'num_nodes': 0}
20/Aug/2025 17:43:36 [INFO] (access_logger) {CP Server Thread-6} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/cluster/status" in 0.0015 seconds. uid: 99331e6b-2565-4981-9a7b-4c721eb9adf2
20/Aug/2025 17:43:36 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/cluster/status HTTP/1.1" code 200, bytes 59, user-agent "ansible-httpget"
20/Aug/2025 17:43:37 [INFO] (access_logger) {CP Server Thread-7} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/cluster/node". uid: e9e94cf3-8c1a-462c-9933-4c554086dc76
20/Aug/2025 17:43:37 [DEBUG] (cmapi_server) {CP Server Thread-7} add_node starts
20/Aug/2025 17:43:37 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 17:43:37 [INFO] (access_logger) {CP Server Thread-8} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin". uid: 80ab6e9e-c8b7-4f6c-9f87-9ab182a57082
20/Aug/2025 17:43:37 [DEBUG] (cmapi_server) {CP Server Thread-8} put_begin starts
20/Aug/2025 17:43:37 [DEBUG] (cmapi_server) {CP Server Thread-8} put_begin JSON body {'id': 753345, 'timeout': 299}
20/Aug/2025 17:43:37 [DEBUG] (cmapi_server) {CP Server Thread-8} put_begin returns {'timestamp': '2025-08-20 17:43:37.153828'}
20/Aug/2025 17:43:37 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin" in 0.0005 seconds. uid: 80ab6e9e-c8b7-4f6c-9f87-9ab182a57082
20/Aug/2025 17:43:37 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/begin HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 17:43:37 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43
20/Aug/2025 17:43:37 [DEBUG] (cmapi_server) {CP Server Thread-7} Cluster add node command called. Adding node ip-172-31-47-126.us-west-2.compute.internal.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-7} get_module_net_address Module 1 network address 127.0.0.1
20/Aug/2025 17:43:37 [INFO] (root) {CP Server Thread-7} add_node(): replacing 127.0.0.1/localhost with 172.31.47.126/ip-172-31-47-126.us-west-2.compute.internal as this node's name. Be sure ip-172-31-47-126.us-west-2.compute.internal resolves to 172.31.47.126 on all other nodes in the cluster.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-7} get_module_net_address Module 1 network address 172.31.47.126
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-7} No .secrets file found so not distrinuting it.
20/Aug/2025 17:43:37 [DEBUG] (asyncio) {CP Server Thread-7} Using selector: EpollSelector
20/Aug/2025 17:43:37 [INFO] (access_logger) {CP Server Thread-10} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config". uid: a72962e5-b142-45e9-a169-5294c6081562
20/Aug/2025 17:43:37 [DEBUG] (cmapi_server) {CP Server Thread-10} put_config starts
20/Aug/2025 17:43:37 [INFO] (root) {CP Server Thread-10}  -- Matching against ModuleIPAddr1-1-3, which says 172.31.47.126
20/Aug/2025 17:43:37 [INFO] (root) {CP Server Thread-10} Wrote "pm1" to /var/lib/columnstore/local/module
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-ddlproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-ddlproc" with "systemctl stop mcs-ddlproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 17:43:37 [INFO] (root) {CP Server Thread-10} Trying to gracefully stop DMLProc using DBRM commands.
20/Aug/2025 17:43:37 [ERROR] (root) {CP Server Thread-10} Cannot set SS_ROLLBACK and SS_SHUTDOWN_PENDING via DBRM, graceful auto stop of DMLProc failed. Try a regular stop method.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-dmlproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-dmlproc" with "systemctl stop mcs-dmlproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "inactive" state
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-primproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-primproc" with "systemctl stop mcs-primproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-writeengineserver".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-writeengineserver" with "systemctl stop mcs-writeengineserver".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-controllernode".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-controllernode" with "systemctl stop mcs-controllernode".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "disable" on service "mcs-workernode@1.service" with "systemctl disable mcs-workernode@1.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-workernode@1.service mcs-workernode@2.service" with "systemctl stop mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode" with "systemctl show -p ActiveState --value mcs-workernode".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode" is in "inactive" state
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-storagemanager".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-storagemanager" with "systemctl stop mcs-storagemanager".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-storagemanager" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-storagemanager" with "systemctl show -p ActiveState --value mcs-storagemanager".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-storagemanager" is in "inactive" state
20/Aug/2025 17:43:37 [INFO] (root) {CP Server Thread-10} Skip starting StorageManager with systemd dispatcher.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "enable" on service "mcs-workernode@1.service" with "systemctl enable mcs-workernode@1.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode@1.service" is in "inactive" state
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-workernode@1.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-workernode@1.service" with "systemctl start mcs-workernode@1.service".
20/Aug/2025 17:43:37 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:37 [INFO] (root) {NodeMonitor} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-workernode@1.service.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode@1.service" is in "active" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for all workernodes to come up before starting controllernode on a primary.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for "['DBRM_Worker1']"....30
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-controllernode".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-controllernode" with "systemctl start mcs-controllernode".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-controllernode.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "active" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-primproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-primproc" with "systemctl start mcs-primproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-primproc.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "active" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-writeengineserver".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-writeengineserver" with "systemctl start mcs-writeengineserver".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-writeengineserver.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "active" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "inactive" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-dmlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-dmlproc" with "systemctl start mcs-dmlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-dmlproc.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "active" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-ddlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-ddlproc" with "systemctl start mcs-ddlproc".
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:39 [INFO] (root) {CP Server Thread-10} Attempt to run dbbuilder on primary node
20/Aug/2025 17:43:39 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:40 [INFO] (root) {NodeMonitor} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-ddlproc.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "active" state
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-10} Node is ready to accept queries.
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config" in 3.1057 seconds. uid: a72962e5-b142-45e9-a169-5294c6081562
20/Aug/2025 17:43:40 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/config HTTP/1.1" code 200, bytes 43, user-agent "Python/3.9 aiohttp/3.11.16"
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-7} Node ip-172-31-47-126.us-west-2.compute.internal config put successfull.
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-7} Successfully finished adding node ip-172-31-47-126.us-west-2.compute.internal.
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-12} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit". uid: 027c8a4a-8bbb-4b17-9c93-47975cbbfeed
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-12} put_commit starts
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-12} put_commit returns {'timestamp': '2025-08-20 17:43:40.289896'}
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-12} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit" in 0.0006 seconds. uid: 027c8a4a-8bbb-4b17-9c93-47975cbbfeed
20/Aug/2025 17:43:40 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/commit HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-7} add_node returns {'timestamp': '2025-08-20 17:43:37.155345', 'node_id': 'ip-172-31-47-126.us-west-2.compute.internal'}
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/cluster/node" in 3.1451 seconds. uid: e9e94cf3-8c1a-462c-9933-4c554086dc76
20/Aug/2025 17:43:40 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/cluster/node HTTP/1.1" code 200, bytes 101, user-agent "ansible-httpget"
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-4} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/cluster/node". uid: 5f8723ad-83b1-4b40-947c-6c7f3ebe0019
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-4} add_node starts
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-5} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin". uid: fd70caf2-6f22-4dd3-a1c2-a0b4fca0bb4c
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-5} put_begin starts
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-5} put_begin JSON body {'id': 159885, 'timeout': 299}
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-5} put_begin returns {'timestamp': '2025-08-20 17:43:40.623854'}
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-5} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin" in 0.0006 seconds. uid: fd70caf2-6f22-4dd3-a1c2-a0b4fca0bb4c
20/Aug/2025 17:43:40 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/begin HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-4} Cluster add node command called. Adding node ip-172-31-43-121.us-west-2.compute.internal.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-4} Nothing to replace, DBRM_Controller/IPAddr isn't localhost.
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-4} Using ip address 172.31.43.121 and hostname ip-172-31-43-121.us-west-2.compute.internal
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-4} Waiting for Maxscale to choose the new repl master...
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} Starting new HTTPS connection (1): 172.31.47.126:8640
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/new_primary". uid: e159b3f0-2701-405c-99fe-b9f1d06b00de
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-8} get_new_primary starts
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-8} get_new_primary returns {'is_primary': True}
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/new_primary" in 0.0116 seconds. uid: e159b3f0-2701-405c-99fe-b9f1d06b00de
20/Aug/2025 17:43:40 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" code 200, bytes 20, user-agent "python-requests/2.32.3"
20/Aug/2025 17:43:40 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} https://172.31.47.126:8640 "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" 200 20
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-4} The new replication master is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-4} _move_primary_node(): dbroot 1 is assigned to ('ip-172-31-47-126.us-west-2.compute.internal', '172.31.47.126')
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-4} get_module_net_address Module 1 network address 172.31.47.126
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-4} No .secrets file found so not distrinuting it.
20/Aug/2025 17:43:40 [DEBUG] (asyncio) {CP Server Thread-4} Using selector: EpollSelector
20/Aug/2025 17:43:40 [INFO] (access_logger) {CP Server Thread-10} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config". uid: d92f10ef-87a8-4773-ba2c-81c8c1adeacc
20/Aug/2025 17:43:40 [DEBUG] (cmapi_server) {CP Server Thread-10} put_config starts
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-10}  -- Matching against ModuleIPAddr1-1-3, which says 172.31.47.126
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-10} Wrote "pm1" to /var/lib/columnstore/local/module
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-ddlproc".
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-ddlproc" with "systemctl stop mcs-ddlproc".
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:40 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-10} Trying to gracefully stop DMLProc using DBRM commands.
20/Aug/2025 17:43:40 [INFO] (root) {CP Server Thread-10} DBRM._send_command: Command set_system_state was requested with value 24
20/Aug/2025 17:43:41 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:43:42 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 172.31.43.121 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-12} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em". uid: fa93bb0a-cd69-4a18-abc7-719d603df05f
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes starts
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns em from S3.
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns.
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-12} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em" in 0.1099 seconds. uid: fa93bb0a-cd69-4a18-abc7-719d603df05f
20/Aug/2025 17:43:42 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/em HTTP/1.1" code 200, bytes 4940, user-agent "Python-urllib/3.12"
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-13} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal". uid: 44883954-7dd1-48df-b656-8bb4603809ec
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes starts
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns journal from S3.
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns.
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal" in 0.0874 seconds. uid: 44883954-7dd1-48df-b656-8bb4603809ec
20/Aug/2025 17:43:42 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/journal HTTP/1.1" code 200, bytes -, user-agent "Python-urllib/3.12"
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-7} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm". uid: 0f394f95-7252-4438-bc31-7c2f45695a3a
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes starts
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes returns vbbm from S3.
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes returns.
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm" in 0.0933 seconds. uid: 0f394f95-7252-4438-bc31-7c2f45695a3a
20/Aug/2025 17:43:42 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vbbm HTTP/1.1" code 200, bytes 12, user-agent "Python-urllib/3.12"
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-5} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss". uid: 95166169-5003-4555-a032-34cf6bcfce6f
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-5} get_brm_bytes starts
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-5} get_brm_bytes returns vss from S3.
20/Aug/2025 17:43:42 [DEBUG] (cmapi_server) {CP Server Thread-5} get_brm_bytes returns.
20/Aug/2025 17:43:42 [INFO] (access_logger) {CP Server Thread-5} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss" in 0.1019 seconds. uid: 95166169-5003-4555-a032-34cf6bcfce6f
20/Aug/2025 17:43:42 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vss HTTP/1.1" code 200, bytes 8, user-agent "Python-urllib/3.12"
20/Aug/2025 17:43:43 [INFO] (root) {CP Server Thread-4} Node ip-172-31-43-121.us-west-2.compute.internal config put successfull.
20/Aug/2025 17:43:45 [INFO] (root) {CP Server Thread-10} Detected "new" DBRM bytestream protocol
20/Aug/2025 17:43:45 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop in 10 seconds
20/Aug/2025 17:43:45 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop. Seconds left 10.
20/Aug/2025 17:43:46 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop. Seconds left 9.
20/Aug/2025 17:43:47 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop. Seconds left 8.
20/Aug/2025 17:43:47 [INFO] (root) {CP Server Thread-10} DMLProc gracefully stopped by DBRM command.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-primproc".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-primproc" with "systemctl stop mcs-primproc".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-writeengineserver".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-writeengineserver" with "systemctl stop mcs-writeengineserver".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-controllernode".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-controllernode" with "systemctl stop mcs-controllernode".
20/Aug/2025 17:43:47 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Call "disable" on service "mcs-workernode@1.service" with "systemctl disable mcs-workernode@1.service".
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-workernode@1.service mcs-workernode@2.service" with "systemctl stop mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 17:43:59 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:43:59 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:43:59 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 96370d57-eebb-4317-a29a-d7bd6a4862f7
20/Aug/2025 17:43:59 [DEBUG] (cmapi_server) {CP Server Thread-8} get_primary starts
20/Aug/2025 17:43:59 [DEBUG] (cmapi_server) {CP Server Thread-8} get_primary returns {'is_primary': 'True'}
20/Aug/2025 17:43:59 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0071 seconds. uid: 96370d57-eebb-4317-a29a-d7bd6a4862f7
20/Aug/2025 17:43:59 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode" is running.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode" with "systemctl show -p ActiveState --value mcs-workernode".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode" is in "inactive" state
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-storagemanager".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-storagemanager" with "systemctl stop mcs-storagemanager".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-storagemanager" is running.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-storagemanager" with "systemctl show -p ActiveState --value mcs-storagemanager".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Service "mcs-storagemanager" is in "inactive" state
20/Aug/2025 17:44:00 [INFO] (root) {CP Server Thread-10} Skip starting StorageManager with systemd dispatcher.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Call "enable" on service "mcs-workernode@1.service" with "systemctl enable mcs-workernode@1.service".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode@1.service" is in "inactive" state
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-workernode@1.service".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-workernode@1.service" with "systemctl start mcs-workernode@1.service".
20/Aug/2025 17:44:00 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:01 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-9} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:01 [INFO] (access_logger) {CP Server Thread-11} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 9130f8c8-ca8b-48ff-b94b-36adbb5d0d85
20/Aug/2025 17:44:01 [DEBUG] (cmapi_server) {CP Server Thread-11} get_primary starts
20/Aug/2025 17:44:01 [DEBUG] (cmapi_server) {CP Server Thread-11} get_primary returns {'is_primary': 'True'}
20/Aug/2025 17:44:01 [INFO] (access_logger) {CP Server Thread-11} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0081 seconds. uid: 9130f8c8-ca8b-48ff-b94b-36adbb5d0d85
20/Aug/2025 17:44:01 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:02 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-workernode@1.service.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode@1.service" is in "active" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for all workernodes to come up before starting controllernode on a primary.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for "['DBRM_Worker1', 'DBRM_Worker2']"....30
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-controllernode".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-controllernode" with "systemctl start mcs-controllernode".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-controllernode.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "active" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-primproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-primproc" with "systemctl start mcs-primproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-primproc.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "active" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-writeengineserver".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-writeengineserver" with "systemctl start mcs-writeengineserver".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-writeengineserver.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "active" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "inactive" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-dmlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-dmlproc" with "systemctl start mcs-dmlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-dmlproc.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "active" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-ddlproc" with "systemctl start mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-10} Attempt to run dbbuilder on primary node
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [ERROR] (root) {CP Server Thread-10} Calling "su -s /bin/sh -c "/usr/bin/dbbuilder 7" mysql" finished with return code: "1" and stderr+stdout "".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-ddlproc.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "active" state
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-10} Node is ready to accept queries.
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config" in 21.9073 seconds. uid: d92f10ef-87a8-4773-ba2c-81c8c1adeacc
20/Aug/2025 17:44:02 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/config HTTP/1.1" code 200, bytes 43, user-agent "Python/3.9 aiohttp/3.11.16"
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-4} Node ip-172-31-47-126.us-west-2.compute.internal config put successfull.
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-4} Successfully finished adding node ip-172-31-43-121.us-west-2.compute.internal.
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-13} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit". uid: 920a9712-403f-4f80-a824-e6ef128d4fd3
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-13} put_commit starts
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-13} put_commit returns {'timestamp': '2025-08-20 17:44:02.582772'}
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit" in 0.0006 seconds. uid: 920a9712-403f-4f80-a824-e6ef128d4fd3
20/Aug/2025 17:44:02 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/commit HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-4} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-4} add_node returns {'timestamp': '2025-08-20 17:43:40.631837', 'node_id': 'ip-172-31-43-121.us-west-2.compute.internal'}
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/cluster/node" in 21.9740 seconds. uid: 5f8723ad-83b1-4b40-947c-6c7f3ebe0019
20/Aug/2025 17:44:02 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/cluster/node HTTP/1.1" code 200, bytes 101, user-agent "ansible-httpget"
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-5} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/cluster/node". uid: 5e000f5b-c650-4588-b4e2-7b160bccc775
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-5} add_node starts
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): ip-172-31-45-153.us-west-2.compute.internal:8640
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://ip-172-31-45-153.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-6} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin". uid: 3f730b3a-f930-4120-9b89-3fa7bea88a69
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-6} put_begin starts
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-6} put_begin JSON body {'id': 826051, 'timeout': 299}
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-6} put_begin returns {'timestamp': '2025-08-20 17:44:02.921234'}
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-6} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin" in 0.0006 seconds. uid: 3f730b3a-f930-4120-9b89-3fa7bea88a69
20/Aug/2025 17:44:02 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/begin HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/begin HTTP/1.1" 200 43
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-5} Cluster add node command called. Adding node ip-172-31-45-153.us-west-2.compute.internal.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-5} Nothing to replace, DBRM_Controller/IPAddr isn't localhost.
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-5} Using ip address 172.31.45.153 and hostname ip-172-31-45-153.us-west-2.compute.internal
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-5} Waiting for Maxscale to choose the new repl master...
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): 172.31.47.126:8640
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-9} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/new_primary". uid: c490ac86-622e-4236-9f45-44a616162afa
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-9} get_new_primary starts
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-9} get_new_primary returns {'is_primary': True}
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-9} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/new_primary" in 0.0114 seconds. uid: c490ac86-622e-4236-9f45-44a616162afa
20/Aug/2025 17:44:02 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-9} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" code 200, bytes 20, user-agent "python-requests/2.32.3"
20/Aug/2025 17:44:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://172.31.47.126:8640 "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" 200 20
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-5} The new replication master is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-5} _move_primary_node(): dbroot 1 is assigned to ('ip-172-31-47-126.us-west-2.compute.internal', '172.31.47.126')
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-5} get_module_net_address Module 1 network address 172.31.47.126
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-5} No .secrets file found so not distrinuting it.
20/Aug/2025 17:44:02 [DEBUG] (asyncio) {CP Server Thread-5} Using selector: EpollSelector
20/Aug/2025 17:44:02 [INFO] (access_logger) {CP Server Thread-10} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config". uid: 7b311b92-81f3-4c9c-9591-9d98786cade0
20/Aug/2025 17:44:02 [DEBUG] (cmapi_server) {CP Server Thread-10} put_config starts
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-10}  -- Matching against ModuleIPAddr1-1-3, which says 172.31.47.126
20/Aug/2025 17:44:02 [INFO] (root) {CP Server Thread-10} Wrote "pm1" to /var/lib/columnstore/local/module
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-ddlproc" with "systemctl stop mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:44:02 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:03 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 17:44:03 [INFO] (root) {CP Server Thread-10} Trying to gracefully stop DMLProc using DBRM commands.
20/Aug/2025 17:44:03 [INFO] (root) {CP Server Thread-10} DBRM._send_command: Command set_system_state was requested with value 24
20/Aug/2025 17:44:03 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop in 10 seconds
20/Aug/2025 17:44:03 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop. Seconds left 10.
20/Aug/2025 17:44:03 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:44:03 [INFO] (node_monitor) {NodeMonitor} Failover support is active, monitoring nodes ['ip-172-31-43-121.us-west-2.compute.internal', 'ip-172-31-45-153.us-west-2.compute.internal', 'ip-172-31-47-126.us-west-2.compute.internal']
20/Aug/2025 17:44:04 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop. Seconds left 9.
20/Aug/2025 17:44:04 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.45.153 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:04 [INFO] (access_logger) {CP Server Thread-13} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/em". uid: 19bf03c8-52a7-4762-8627-ea720c895c78
20/Aug/2025 17:44:04 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes starts
20/Aug/2025 17:44:04 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns em from S3.
20/Aug/2025 17:44:04 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns.
20/Aug/2025 17:44:04 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/em" in 0.0943 seconds. uid: 19bf03c8-52a7-4762-8627-ea720c895c78
20/Aug/2025 17:44:04 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/em HTTP/1.1" code 200, bytes 4940, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:04 [INFO] (access_logger) {CP Server Thread-7} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/journal". uid: 9dc8a687-3aea-46c7-adfd-ff6a800d5d23
20/Aug/2025 17:44:04 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes starts
20/Aug/2025 17:44:04 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes returns journal from S3.
20/Aug/2025 17:44:04 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes returns.
20/Aug/2025 17:44:04 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/journal" in 0.1103 seconds. uid: 9dc8a687-3aea-46c7-adfd-ff6a800d5d23
20/Aug/2025 17:44:04 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/journal HTTP/1.1" code 200, bytes 5, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:05 [INFO] (access_logger) {CP Server Thread-4} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vbbm". uid: cc059397-71b7-44d3-86d1-7528106ebaaa
20/Aug/2025 17:44:05 [DEBUG] (cmapi_server) {CP Server Thread-4} get_brm_bytes starts
20/Aug/2025 17:44:05 [DEBUG] (cmapi_server) {CP Server Thread-4} get_brm_bytes returns vbbm from S3.
20/Aug/2025 17:44:05 [INFO] (root) {CP Server Thread-10} Waiting for DMLProc to stop. Seconds left 8.
20/Aug/2025 17:44:05 [INFO] (root) {CP Server Thread-10} DMLProc gracefully stopped by DBRM command.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-primproc".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-primproc" with "systemctl stop mcs-primproc".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-writeengineserver".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-writeengineserver" with "systemctl stop mcs-writeengineserver".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-controllernode".
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-controllernode" with "systemctl stop mcs-controllernode".
20/Aug/2025 17:44:05 [DEBUG] (cmapi_server) {CP Server Thread-4} get_brm_bytes returns.
20/Aug/2025 17:44:05 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vbbm" in 0.1114 seconds. uid: cc059397-71b7-44d3-86d1-7528106ebaaa
20/Aug/2025 17:44:05 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/vbbm HTTP/1.1" code 200, bytes 12, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:05 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:05 [INFO] (access_logger) {CP Server Thread-6} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vss". uid: f4837533-cc46-4d50-a47a-4811c45da271
20/Aug/2025 17:44:05 [DEBUG] (cmapi_server) {CP Server Thread-6} get_brm_bytes starts
20/Aug/2025 17:44:05 [DEBUG] (cmapi_server) {CP Server Thread-6} get_brm_bytes returns vss from S3.
20/Aug/2025 17:44:05 [DEBUG] (cmapi_server) {CP Server Thread-6} get_brm_bytes returns.
20/Aug/2025 17:44:05 [INFO] (access_logger) {CP Server Thread-6} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vss" in 0.0783 seconds. uid: f4837533-cc46-4d50-a47a-4811c45da271
20/Aug/2025 17:44:05 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/vss HTTP/1.1" code 200, bytes 8, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:06 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.43.121 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-9} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em". uid: a4d78eca-2dac-4fac-9481-43b663ae0774
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes starts
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes returns em from S3.
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes returns.
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-9} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em" in 0.0939 seconds. uid: a4d78eca-2dac-4fac-9481-43b663ae0774
20/Aug/2025 17:44:06 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-9} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/em HTTP/1.1" code 200, bytes 4940, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-11} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal". uid: e57974e5-ddf1-4f69-921e-ed6aaff4bdc7
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes starts
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes returns journal from S3.
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes returns.
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-11} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal" in 0.0918 seconds. uid: e57974e5-ddf1-4f69-921e-ed6aaff4bdc7
20/Aug/2025 17:44:06 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/journal HTTP/1.1" code 200, bytes 5, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-12} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm". uid: c4a1bf7a-3a88-4b9d-83f4-8b0d533b797b
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes starts
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns vbbm from S3.
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns.
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-12} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm" in 0.1099 seconds. uid: c4a1bf7a-3a88-4b9d-83f4-8b0d533b797b
20/Aug/2025 17:44:06 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vbbm HTTP/1.1" code 200, bytes 12, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-13} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss". uid: 918921a9-0b93-42c3-bf8c-26de47ceeb1e
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes starts
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns vss from S3.
20/Aug/2025 17:44:06 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns.
20/Aug/2025 17:44:06 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss" in 0.1017 seconds. uid: 918921a9-0b93-42c3-bf8c-26de47ceeb1e
20/Aug/2025 17:44:06 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vss HTTP/1.1" code 200, bytes 8, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:14 [INFO] (root) {CP Server Thread-5} Node ip-172-31-45-153.us-west-2.compute.internal config put successfull.
20/Aug/2025 17:44:16 [INFO] (root) {CP Server Thread-5} Node ip-172-31-43-121.us-west-2.compute.internal config put successfull.
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Call "disable" on service "mcs-workernode@1.service" with "systemctl disable mcs-workernode@1.service".
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-workernode@1.service mcs-workernode@2.service" with "systemctl stop mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 17:44:18 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:18 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:18 [INFO] (access_logger) {CP Server Thread-4} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: d657ae23-8a94-4deb-8829-40c84e1a7691
20/Aug/2025 17:44:18 [DEBUG] (cmapi_server) {CP Server Thread-4} get_primary starts
20/Aug/2025 17:44:18 [DEBUG] (cmapi_server) {CP Server Thread-4} get_primary returns {'is_primary': 'True'}
20/Aug/2025 17:44:18 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0088 seconds. uid: d657ae23-8a94-4deb-8829-40c84e1a7691
20/Aug/2025 17:44:18 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode" is running.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode" with "systemctl show -p ActiveState --value mcs-workernode".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode" is in "inactive" state
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Stopping "mcs-storagemanager".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Call "stop" on service "mcs-storagemanager" with "systemctl stop mcs-storagemanager".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-storagemanager" is running.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-storagemanager" with "systemctl show -p ActiveState --value mcs-storagemanager".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Service "mcs-storagemanager" is in "inactive" state
20/Aug/2025 17:44:19 [INFO] (root) {CP Server Thread-10} Skip starting StorageManager with systemd dispatcher.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Call "enable" on service "mcs-workernode@1.service" with "systemctl enable mcs-workernode@1.service".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode@1.service" is in "inactive" state
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-workernode@1.service".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-workernode@1.service" with "systemctl start mcs-workernode@1.service".
20/Aug/2025 17:44:19 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:21 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 1711685c-eb36-4687-a369-44c4390c5a95
20/Aug/2025 17:44:21 [DEBUG] (cmapi_server) {CP Server Thread-8} get_primary starts
20/Aug/2025 17:44:21 [DEBUG] (cmapi_server) {CP Server Thread-8} get_primary returns {'is_primary': 'True'}
20/Aug/2025 17:44:21 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0080 seconds. uid: 1711685c-eb36-4687-a369-44c4390c5a95
20/Aug/2025 17:44:21 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-workernode@1.service.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-workernode@1.service" is in "active" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for all workernodes to come up before starting controllernode on a primary.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for "['DBRM_Worker1', 'DBRM_Worker2', 'DBRM_Worker3']"....30
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-controllernode".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-controllernode" with "systemctl start mcs-controllernode".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-controllernode.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-controllernode" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-controllernode" is in "active" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-primproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-primproc" with "systemctl start mcs-primproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-primproc.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-primproc" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-primproc" is in "active" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-writeengineserver".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-writeengineserver" with "systemctl start mcs-writeengineserver".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-writeengineserver.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-writeengineserver" is in "active" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "inactive" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-dmlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-dmlproc" with "systemctl start mcs-dmlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-dmlproc.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-dmlproc" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-dmlproc" is in "active" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Starting "mcs-ddlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Call "start" on service "mcs-ddlproc" with "systemctl start mcs-ddlproc".
20/Aug/2025 17:44:21 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:21 [INFO] (root) {CP Server Thread-10} Attempt to run dbbuilder on primary node
20/Aug/2025 17:44:22 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:22 [ERROR] (root) {CP Server Thread-10} Calling "su -s /bin/sh -c "/usr/bin/dbbuilder 7" mysql" finished with return code: "1" and stderr+stdout "".
20/Aug/2025 17:44:22 [DEBUG] (root) {CP Server Thread-10} Successfully started mcs-ddlproc.
20/Aug/2025 17:44:22 [DEBUG] (root) {CP Server Thread-10} Checking if "mcs-ddlproc" is running.
20/Aug/2025 17:44:22 [DEBUG] (root) {CP Server Thread-10} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 17:44:22 [DEBUG] (root) {CP Server Thread-10} Waiting for command to finish.
20/Aug/2025 17:44:22 [DEBUG] (root) {CP Server Thread-10} Service "mcs-ddlproc" is in "active" state
20/Aug/2025 17:44:22 [DEBUG] (cmapi_server) {CP Server Thread-10} Node is ready to accept queries.
20/Aug/2025 17:44:22 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config" in 19.1082 seconds. uid: 7b311b92-81f3-4c9c-9591-9d98786cade0
20/Aug/2025 17:44:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/config HTTP/1.1" code 200, bytes 43, user-agent "Python/3.9 aiohttp/3.11.16"
20/Aug/2025 17:44:22 [INFO] (root) {CP Server Thread-5} Node ip-172-31-47-126.us-west-2.compute.internal config put successfull.
20/Aug/2025 17:44:22 [DEBUG] (cmapi_server) {CP Server Thread-5} Successfully finished adding node ip-172-31-45-153.us-west-2.compute.internal.
20/Aug/2025 17:44:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): ip-172-31-45-153.us-west-2.compute.internal:8640
20/Aug/2025 17:44:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://ip-172-31-45-153.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43
20/Aug/2025 17:44:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 17:44:22 [INFO] (access_logger) {CP Server Thread-11} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit". uid: 450dc815-3272-4eed-a863-9dd6ebc33cd7
20/Aug/2025 17:44:22 [DEBUG] (cmapi_server) {CP Server Thread-11} put_commit starts
20/Aug/2025 17:44:22 [DEBUG] (cmapi_server) {CP Server Thread-11} put_commit returns {'timestamp': '2025-08-20 17:44:22.091838'}
20/Aug/2025 17:44:22 [INFO] (access_logger) {CP Server Thread-11} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit" in 0.0006 seconds. uid: 450dc815-3272-4eed-a863-9dd6ebc33cd7
20/Aug/2025 17:44:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/commit HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 17:44:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43
20/Aug/2025 17:44:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 17:44:22 [INFO] (root) {NodeMonitor} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 17:44:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-5} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "PUT /cmapi/0.4.0/node/commit HTTP/1.1" 200 43
20/Aug/2025 17:44:22 [DEBUG] (cmapi_server) {CP Server Thread-5} add_node returns {'timestamp': '2025-08-20 17:44:02.929366', 'node_id': 'ip-172-31-45-153.us-west-2.compute.internal'}
20/Aug/2025 17:44:22 [INFO] (access_logger) {CP Server Thread-5} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/cluster/node" in 19.1926 seconds. uid: 5e000f5b-c650-4588-b4e2-7b160bccc775
20/Aug/2025 17:44:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/cluster/node HTTP/1.1" code 200, bytes 101, user-agent "ansible-httpget"
20/Aug/2025 21:32:11 [INFO] (access_logger) {CP Server Thread-13} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/cluster/status". uid: 1ad8931c-f7f0-47af-ba8a-778c74a8d88b
20/Aug/2025 21:32:11 [DEBUG] (cmapi_server) {CP Server Thread-13} get_status starts
20/Aug/2025 21:32:11 [DEBUG] (cmapi_server) {CP Server Thread-13} Cluster status command called. Getting status.
20/Aug/2025 21:32:11 [DEBUG] (urllib3.connectionpool) {CP Server Thread-13} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 21:32:11 [INFO] (access_logger) {CP Server Thread-7} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/status". uid: c87bf297-8f5a-4fe5-9e75-6295f5505974
20/Aug/2025 21:32:11 [DEBUG] (cmapi_server) {CP Server Thread-7} get_status starts
20/Aug/2025 21:32:11 [INFO] (root) {CP Server Thread-7} DBRM._send_command: Command readonly was requested with value None
20/Aug/2025 21:32:11 [DEBUG] (cmapi_server) {CP Server Thread-7} get_status returns {'timestamp': '2025-08-20 21:32:11.828663', 'uptime': 13872, 'dbrm_mode': 'master', 'cluster_mode': 'readwrite', 'dbroots': ['1'], 'module_id': 1, 'services': [{'name': 'StorageManager', 'pid': 8841}, {'name': 'workernode', 'pid': 8879}, {'name': 'controllernode', 'pid': 8890}, {'name': 'PrimProc', 'pid': 8907}, {'name': 'WriteEngineServer', 'pid': 9034}, {'name': 'DMLProc', 'pid': 9043}, {'name': 'DDLProc', 'pid': 9074}]}
20/Aug/2025 21:32:11 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/status" in 0.0731 seconds. uid: c87bf297-8f5a-4fe5-9e75-6295f5505974
20/Aug/2025 21:32:11 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/node/status HTTP/1.1" code 200, bytes 426, user-agent "python-requests/2.32.3"
20/Aug/2025 21:32:11 [DEBUG] (urllib3.connectionpool) {CP Server Thread-13} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 426
20/Aug/2025 21:32:11 [DEBUG] (urllib3.connectionpool) {CP Server Thread-13} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 21:32:16 [DEBUG] (urllib3.connectionpool) {CP Server Thread-13} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 315
20/Aug/2025 21:32:16 [DEBUG] (urllib3.connectionpool) {CP Server Thread-13} Starting new HTTPS connection (1): ip-172-31-45-153.us-west-2.compute.internal:8640
20/Aug/2025 21:32:22 [DEBUG] (urllib3.connectionpool) {CP Server Thread-13} https://ip-172-31-45-153.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 315
20/Aug/2025 21:32:22 [DEBUG] (cmapi_server) {CP Server Thread-13} Successfully finished getting cluster status.
20/Aug/2025 21:32:22 [DEBUG] (cmapi_server) {CP Server Thread-13} get_status returns {'timestamp': '2025-08-20 21:32:11.817506', 'ip-172-31-47-126.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:32:11.828663', 'uptime': 13872, 'dbrm_mode': 'master', 'cluster_mode': 'readwrite', 'dbroots': ['1'], 'module_id': 1, 'services': [{'name': 'StorageManager', 'pid': 8841}, {'name': 'workernode', 'pid': 8879}, {'name': 'controllernode', 'pid': 8890}, {'name': 'PrimProc', 'pid': 8907}, {'name': 'WriteEngineServer', 'pid': 9034}, {'name': 'DMLProc', 'pid': 9043}, {'name': 'DDLProc', 'pid': 9074}]}, 'ip-172-31-43-121.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:32:11.911411', 'uptime': 13871, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': ['2'], 'module_id': 2, 'services': [{'name': 'StorageManager', 'pid': 7657}, {'name': 'workernode', 'pid': 7694}, {'name': 'PrimProc', 'pid': 7720}, {'name': 'WriteEngineServer', 'pid': 7838}]}, 'ip-172-31-45-153.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:32:17.008101', 'uptime': 13878, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': ['3'], 'module_id': 3, 'services': [{'name': 'StorageManager', 'pid': 7324}, {'name': 'workernode', 'pid': 7365}, {'name': 'PrimProc', 'pid': 7387}, {'name': 'WriteEngineServer', 'pid': 7505}]}, 'num_nodes': 3}
20/Aug/2025 21:32:22 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/cluster/status" in 10.2778 seconds. uid: 1ad8931c-f7f0-47af-ba8a-778c74a8d88b
20/Aug/2025 21:32:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/cluster/status HTTP/1.1" code 200, bytes 1262, user-agent "python-requests/2.32.3"
20/Aug/2025 21:32:38 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/cluster/status". uid: 7273668c-634a-4c99-ba28-12ca62422b66
20/Aug/2025 21:32:38 [DEBUG] (cmapi_server) {CP Server Thread-8} get_status starts
20/Aug/2025 21:32:38 [DEBUG] (cmapi_server) {CP Server Thread-8} Cluster status command called. Getting status.
20/Aug/2025 21:32:38 [DEBUG] (urllib3.connectionpool) {CP Server Thread-8} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 21:32:38 [INFO] (access_logger) {CP Server Thread-10} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/status". uid: f850b66d-0c9c-4ba2-a221-0331bde0c9b0
20/Aug/2025 21:32:38 [DEBUG] (cmapi_server) {CP Server Thread-10} get_status starts
20/Aug/2025 21:32:38 [INFO] (root) {CP Server Thread-10} DBRM._send_command: Command readonly was requested with value None
20/Aug/2025 21:32:38 [DEBUG] (cmapi_server) {CP Server Thread-10} get_status returns {'timestamp': '2025-08-20 21:32:38.661751', 'uptime': 13899, 'dbrm_mode': 'master', 'cluster_mode': 'readwrite', 'dbroots': ['1'], 'module_id': 1, 'services': [{'name': 'StorageManager', 'pid': 8841}, {'name': 'workernode', 'pid': 8879}, {'name': 'controllernode', 'pid': 8890}, {'name': 'PrimProc', 'pid': 8907}, {'name': 'WriteEngineServer', 'pid': 9034}, {'name': 'DMLProc', 'pid': 9043}, {'name': 'DDLProc', 'pid': 9074}]}
20/Aug/2025 21:32:38 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/status" in 0.0605 seconds. uid: f850b66d-0c9c-4ba2-a221-0331bde0c9b0
20/Aug/2025 21:32:38 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/node/status HTTP/1.1" code 200, bytes 426, user-agent "python-requests/2.32.3"
20/Aug/2025 21:32:38 [DEBUG] (urllib3.connectionpool) {CP Server Thread-8} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 426
20/Aug/2025 21:32:38 [DEBUG] (urllib3.connectionpool) {CP Server Thread-8} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 21:32:38 [DEBUG] (urllib3.connectionpool) {CP Server Thread-8} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 315
20/Aug/2025 21:32:38 [DEBUG] (urllib3.connectionpool) {CP Server Thread-8} Starting new HTTPS connection (1): ip-172-31-45-153.us-west-2.compute.internal:8640
20/Aug/2025 21:32:38 [DEBUG] (urllib3.connectionpool) {CP Server Thread-8} https://ip-172-31-45-153.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 315
20/Aug/2025 21:32:38 [DEBUG] (cmapi_server) {CP Server Thread-8} Successfully finished getting cluster status.
20/Aug/2025 21:32:38 [DEBUG] (cmapi_server) {CP Server Thread-8} get_status returns {'timestamp': '2025-08-20 21:32:38.655744', 'ip-172-31-47-126.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:32:38.661751', 'uptime': 13899, 'dbrm_mode': 'master', 'cluster_mode': 'readwrite', 'dbroots': ['1'], 'module_id': 1, 'services': [{'name': 'StorageManager', 'pid': 8841}, {'name': 'workernode', 'pid': 8879}, {'name': 'controllernode', 'pid': 8890}, {'name': 'PrimProc', 'pid': 8907}, {'name': 'WriteEngineServer', 'pid': 9034}, {'name': 'DMLProc', 'pid': 9043}, {'name': 'DDLProc', 'pid': 9074}]}, 'ip-172-31-43-121.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:32:38.728343', 'uptime': 13898, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': ['2'], 'module_id': 2, 'services': [{'name': 'StorageManager', 'pid': 7657}, {'name': 'workernode', 'pid': 7694}, {'name': 'PrimProc', 'pid': 7720}, {'name': 'WriteEngineServer', 'pid': 7838}]}, 'ip-172-31-45-153.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:32:38.794163', 'uptime': 13899, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': ['3'], 'module_id': 3, 'services': [{'name': 'StorageManager', 'pid': 7324}, {'name': 'workernode', 'pid': 7365}, {'name': 'PrimProc', 'pid': 7387}, {'name': 'WriteEngineServer', 'pid': 7505}]}, 'num_nodes': 3}
20/Aug/2025 21:32:38 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/cluster/status" in 0.2008 seconds. uid: 7273668c-634a-4c99-ba28-12ca62422b66
20/Aug/2025 21:32:38 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/cluster/status HTTP/1.1" code 200, bytes 1262, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:01 [INFO] (access_logger) {CP Server Thread-12} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin". uid: 5b057110-6480-4365-8182-355328d0aef7
20/Aug/2025 21:33:01 [DEBUG] (cmapi_server) {CP Server Thread-12} put_begin starts
20/Aug/2025 21:33:01 [DEBUG] (cmapi_server) {CP Server Thread-12} put_begin JSON body {'id': 822464, 'timeout': 86399}
20/Aug/2025 21:33:01 [DEBUG] (cmapi_server) {CP Server Thread-12} put_begin returns {'timestamp': '2025-08-20 21:33:01.992211'}
20/Aug/2025 21:33:01 [INFO] (access_logger) {CP Server Thread-12} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/begin" in 0.0007 seconds. uid: 5b057110-6480-4365-8182-355328d0aef7
20/Aug/2025 21:33:01 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/begin HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:02 [INFO] (access_logger) {CP Server Thread-7} Got incoming DELETE request from "127.0.0.1" to "/cmapi/0.4.0/cluster/node". uid: fc807045-1e68-4be9-affc-346a8258fb85
20/Aug/2025 21:33:02 [DEBUG] (cmapi_server) {CP Server Thread-7} remove_node starts
20/Aug/2025 21:33:02 [DEBUG] (cmapi_server) {CP Server Thread-7} Cluster remove node command called. Removing node ip-172-31-45-153.us-west-2.compute.internal.
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-7} Waiting for Maxscale to choose the new repl master...
20/Aug/2025 21:33:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} Starting new HTTPS connection (1): 172.31.47.126:8640
20/Aug/2025 21:33:02 [INFO] (access_logger) {CP Server Thread-4} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/new_primary". uid: bd41725a-24a5-4fc2-a493-b51c6543a55d
20/Aug/2025 21:33:02 [DEBUG] (cmapi_server) {CP Server Thread-4} get_new_primary starts
20/Aug/2025 21:33:02 [DEBUG] (cmapi_server) {CP Server Thread-4} get_new_primary returns {'is_primary': True}
20/Aug/2025 21:33:02 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/new_primary" in 0.0121 seconds. uid: bd41725a-24a5-4fc2-a493-b51c6543a55d
20/Aug/2025 21:33:02 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" code 200, bytes 20, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:02 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} https://172.31.47.126:8640 "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" 200 20
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-7} The new replication master is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-7} _move_primary_node(): dbroot 1 is assigned to ('ip-172-31-47-126.us-west-2.compute.internal', '172.31.47.126')
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-7} get_module_net_address Module 1 network address 172.31.47.126
20/Aug/2025 21:33:02 [DEBUG] (asyncio) {CP Server Thread-7} Using selector: EpollSelector
20/Aug/2025 21:33:02 [INFO] (access_logger) {CP Server Thread-6} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config". uid: 461e28f6-4904-4f5c-904d-410566abf126
20/Aug/2025 21:33:02 [DEBUG] (cmapi_server) {CP Server Thread-6} put_config starts
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-6}  -- Matching against ModuleIPAddr1-1-3, which says 172.31.47.126
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-6} Wrote "pm1" to /var/lib/columnstore/local/module
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-ddlproc".
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-ddlproc" with "systemctl stop mcs-ddlproc".
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-ddlproc" is running.
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:02 [DEBUG] (root) {CP Server Thread-6} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-6} Trying to gracefully stop DMLProc using DBRM commands.
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-6} DBRM._send_command: Command set_system_state was requested with value 24
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop in 10 seconds
20/Aug/2025 21:33:02 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop. Seconds left 10.
20/Aug/2025 21:33:02 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 21:33:03 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop. Seconds left 9.
20/Aug/2025 21:33:03 [INFO] (root) {CP Server Thread-6} DMLProc gracefully stopped by DBRM command.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-primproc".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-primproc" with "systemctl stop mcs-primproc".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-primproc" is running.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-writeengineserver".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-writeengineserver" with "systemctl stop mcs-writeengineserver".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-controllernode".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-controllernode" with "systemctl stop mcs-controllernode".
20/Aug/2025 21:33:03 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:05 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 172.31.43.121 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-9} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em". uid: 82149b19-a4bb-4610-8f49-6ca72477fa48
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes starts
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes returns em from S3.
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes returns.
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-9} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em" in 0.2177 seconds. uid: 82149b19-a4bb-4610-8f49-6ca72477fa48
20/Aug/2025 21:33:05 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-9} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/em HTTP/1.1" code 200, bytes 4940, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal". uid: bb397ecf-dfd9-4058-9cd5-2549e5257589
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-8} get_brm_bytes starts
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-8} get_brm_bytes returns journal from S3.
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-8} get_brm_bytes returns.
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal" in 0.1067 seconds. uid: bb397ecf-dfd9-4058-9cd5-2549e5257589
20/Aug/2025 21:33:05 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/journal HTTP/1.1" code 200, bytes 12781, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-11} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm". uid: cef70d98-82d0-4ca6-b5b4-18e2e5f22c88
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes starts
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes returns vbbm from S3.
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes returns.
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-11} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm" in 0.0821 seconds. uid: cef70d98-82d0-4ca6-b5b4-18e2e5f22c88
20/Aug/2025 21:33:05 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vbbm HTTP/1.1" code 200, bytes 12, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:05 [INFO] (access_logger) {CP Server Thread-12} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss". uid: 176be563-fead-41dd-8633-c736dee13c07
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes starts
20/Aug/2025 21:33:05 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns vss from S3.
20/Aug/2025 21:33:06 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns.
20/Aug/2025 21:33:06 [INFO] (access_logger) {CP Server Thread-12} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss" in 0.0923 seconds. uid: 176be563-fead-41dd-8633-c736dee13c07
20/Aug/2025 21:33:06 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vss HTTP/1.1" code 200, bytes 8, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-controllernode" is running.
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Call "disable" on service "mcs-workernode@1.service" with "systemctl disable mcs-workernode@1.service".
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-workernode@1.service mcs-workernode@2.service" with "systemctl stop mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 21:33:11 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:11 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:11 [INFO] (access_logger) {CP Server Thread-4} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 015ae5d8-be04-4060-8c61-f570e74ca895
20/Aug/2025 21:33:11 [DEBUG] (cmapi_server) {CP Server Thread-4} get_primary starts
20/Aug/2025 21:33:11 [DEBUG] (cmapi_server) {CP Server Thread-4} get_primary returns {'is_primary': 'True'}
20/Aug/2025 21:33:11 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0080 seconds. uid: 015ae5d8-be04-4060-8c61-f570e74ca895
20/Aug/2025 21:33:11 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-workernode" is running.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-workernode" with "systemctl show -p ActiveState --value mcs-workernode".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Service "mcs-workernode" is in "inactive" state
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-storagemanager".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-storagemanager" with "systemctl stop mcs-storagemanager".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-storagemanager" is running.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-storagemanager" with "systemctl show -p ActiveState --value mcs-storagemanager".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Service "mcs-storagemanager" is in "inactive" state
20/Aug/2025 21:33:12 [INFO] (root) {CP Server Thread-6} Skip starting StorageManager with systemd dispatcher.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Call "enable" on service "mcs-workernode@1.service" with "systemctl enable mcs-workernode@1.service".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Service "mcs-workernode@1.service" is in "inactive" state
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-workernode@1.service".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-workernode@1.service" with "systemctl start mcs-workernode@1.service".
20/Aug/2025 21:33:12 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:14 [INFO] (access_logger) {CP Server Thread-10} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 55eb36a0-a35a-4686-8d63-6fc12b802493
20/Aug/2025 21:33:14 [DEBUG] (cmapi_server) {CP Server Thread-10} get_primary starts
20/Aug/2025 21:33:14 [DEBUG] (cmapi_server) {CP Server Thread-10} get_primary returns {'is_primary': 'True'}
20/Aug/2025 21:33:14 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0085 seconds. uid: 55eb36a0-a35a-4686-8d63-6fc12b802493
20/Aug/2025 21:33:14 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-workernode@1.service.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-workernode@1.service" is in "active" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for all workernodes to come up before starting controllernode on a primary.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for "['DBRM_Worker1', 'DBRM_Worker2']"....30
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-controllernode" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-controllernode".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-controllernode" with "systemctl start mcs-controllernode".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-controllernode.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-controllernode" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-controllernode" is in "active" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-primproc" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-primproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-primproc" with "systemctl start mcs-primproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-primproc.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-primproc" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-primproc" is in "active" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-writeengineserver".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-writeengineserver" with "systemctl start mcs-writeengineserver".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-writeengineserver.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-writeengineserver" is in "active" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-dmlproc" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-dmlproc" is in "inactive" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-dmlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-dmlproc" with "systemctl start mcs-dmlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-dmlproc.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-dmlproc" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-dmlproc" is in "active" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-ddlproc" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-ddlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-ddlproc" with "systemctl start mcs-ddlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [INFO] (root) {CP Server Thread-6} Attempt to run dbbuilder on primary node
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [ERROR] (root) {CP Server Thread-6} Calling "su -s /bin/sh -c "/usr/bin/dbbuilder 7" mysql" finished with return code: "1" and stderr+stdout "".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-ddlproc.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-ddlproc" is running.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:14 [DEBUG] (root) {CP Server Thread-6} Service "mcs-ddlproc" is in "active" state
20/Aug/2025 21:33:14 [DEBUG] (cmapi_server) {CP Server Thread-6} Node is ready to accept queries.
20/Aug/2025 21:33:14 [INFO] (access_logger) {CP Server Thread-6} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/config" in 12.7946 seconds. uid: 461e28f6-4904-4f5c-904d-410566abf126
20/Aug/2025 21:33:14 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/config HTTP/1.1" code 200, bytes 43, user-agent "Python/3.9 aiohttp/3.11.16"
20/Aug/2025 21:33:14 [INFO] (root) {CP Server Thread-7} Node ip-172-31-47-126.us-west-2.compute.internal config put successfull.
20/Aug/2025 21:33:15 [INFO] (root) {CP Server Thread-7} Node ip-172-31-43-121.us-west-2.compute.internal config put successfull.
20/Aug/2025 21:33:15 [DEBUG] (cmapi_server) {CP Server Thread-7} Successfully finished removing node ip-172-31-45-153.us-west-2.compute.internal.
20/Aug/2025 21:33:15 [DEBUG] (cmapi_server) {CP Server Thread-7} remove_node returns {'timestamp': '2025-08-20 21:33:02.005740', 'node_id': 'ip-172-31-45-153.us-west-2.compute.internal'}
20/Aug/2025 21:33:15 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming DELETE request from "127.0.0.1" to "/cmapi/0.4.0/cluster/node" in 13.6098 seconds. uid: fc807045-1e68-4be9-affc-346a8258fb85
20/Aug/2025 21:33:15 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 127.0.0.1 ACCESS "DELETE /cmapi/0.4.0/cluster/node HTTP/1.1" code 200, bytes 101, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:15 [INFO] (access_logger) {CP Server Thread-11} Got incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit". uid: 02ce7e84-98ba-4238-9af2-aefac31c4323
20/Aug/2025 21:33:15 [DEBUG] (cmapi_server) {CP Server Thread-11} put_commit starts
20/Aug/2025 21:33:15 [DEBUG] (cmapi_server) {CP Server Thread-11} put_commit returns {'timestamp': '2025-08-20 21:33:15.621915'}
20/Aug/2025 21:33:15 [INFO] (access_logger) {CP Server Thread-11} Finished processing incoming PUT request from "172.31.47.126" to "/cmapi/0.4.0/node/commit" in 0.0007 seconds. uid: 02ce7e84-98ba-4238-9af2-aefac31c4323
20/Aug/2025 21:33:15 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 172.31.47.126 ACCESS "PUT /cmapi/0.4.0/node/commit HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:19 [INFO] (access_logger) {CP Server Thread-5} Got incoming PUT request from "172.31.43.121" to "/cmapi/0.4.0/node/begin". uid: 62c6c98d-0add-4459-a048-baab2df68d6c
20/Aug/2025 21:33:19 [DEBUG] (cmapi_server) {CP Server Thread-5} put_begin starts
20/Aug/2025 21:33:19 [DEBUG] (cmapi_server) {CP Server Thread-5} put_begin JSON body {'id': 645537, 'timeout': 286}
20/Aug/2025 21:33:19 [DEBUG] (cmapi_server) {CP Server Thread-5} put_begin returns {'timestamp': '2025-08-20 21:33:19.094309'}
20/Aug/2025 21:33:19 [INFO] (access_logger) {CP Server Thread-5} Finished processing incoming PUT request from "172.31.43.121" to "/cmapi/0.4.0/node/begin" in 0.0006 seconds. uid: 62c6c98d-0add-4459-a048-baab2df68d6c
20/Aug/2025 21:33:19 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 172.31.43.121 ACCESS "PUT /cmapi/0.4.0/node/begin HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:19 [INFO] (access_logger) {CP Server Thread-13} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/new_primary". uid: 5a4eae0c-4df2-4300-87ba-8a02e21a9a18
20/Aug/2025 21:33:19 [DEBUG] (cmapi_server) {CP Server Thread-13} get_new_primary starts
20/Aug/2025 21:33:19 [DEBUG] (cmapi_server) {CP Server Thread-13} get_new_primary returns {'is_primary': True}
20/Aug/2025 21:33:19 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/new_primary" in 0.0123 seconds. uid: 5a4eae0c-4df2-4300-87ba-8a02e21a9a18
20/Aug/2025 21:33:19 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/new_primary HTTP/1.1" code 200, bytes 20, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:19 [INFO] (access_logger) {CP Server Thread-6} Got incoming PUT request from "172.31.43.121" to "/cmapi/0.4.0/node/config". uid: 2c487b9f-a5b2-4eb7-adae-991054a0084a
20/Aug/2025 21:33:19 [DEBUG] (cmapi_server) {CP Server Thread-6} put_config starts
20/Aug/2025 21:33:19 [INFO] (root) {CP Server Thread-6}  -- Matching against ModuleIPAddr1-1-3, which says 172.31.47.126
20/Aug/2025 21:33:19 [INFO] (root) {CP Server Thread-6} Wrote "pm1" to /var/lib/columnstore/local/module
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-ddlproc".
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-ddlproc" with "systemctl stop mcs-ddlproc".
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-ddlproc" is running.
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:19 [DEBUG] (root) {CP Server Thread-6} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 21:33:19 [INFO] (root) {CP Server Thread-6} Trying to gracefully stop DMLProc using DBRM commands.
20/Aug/2025 21:33:19 [INFO] (root) {CP Server Thread-6} DBRM._send_command: Command set_system_state was requested with value 24
20/Aug/2025 21:33:19 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop in 10 seconds
20/Aug/2025 21:33:19 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop. Seconds left 10.
20/Aug/2025 21:33:19 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 21:33:20 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop. Seconds left 9.
20/Aug/2025 21:33:21 [INFO] (root) {CP Server Thread-6} Waiting for DMLProc to stop. Seconds left 8.
20/Aug/2025 21:33:21 [INFO] (root) {CP Server Thread-6} DMLProc gracefully stopped by DBRM command.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-primproc".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-primproc" with "systemctl stop mcs-primproc".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-primproc" is running.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-writeengineserver".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-writeengineserver" with "systemctl stop mcs-writeengineserver".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-controllernode".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-controllernode" with "systemctl stop mcs-controllernode".
20/Aug/2025 21:33:21 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:21 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-9} 172.31.45.153 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-7} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/em". uid: b8fd25ba-a47b-4e31-9731-01ef529ac1c0
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes starts
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes returns em from S3.
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-7} get_brm_bytes returns.
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/em" in 0.1054 seconds. uid: b8fd25ba-a47b-4e31-9731-01ef529ac1c0
20/Aug/2025 21:33:21 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/em HTTP/1.1" code 200, bytes 4940, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/journal". uid: 7150c4af-73f2-40ff-92bc-256a4d5f2ca3
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-8} get_brm_bytes starts
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-8} get_brm_bytes returns journal from S3.
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-8} get_brm_bytes returns.
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/journal" in 0.1026 seconds. uid: 7150c4af-73f2-40ff-92bc-256a4d5f2ca3
20/Aug/2025 21:33:21 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/journal HTTP/1.1" code 200, bytes 5, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-11} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vbbm". uid: d54ca447-29ce-44b0-89cd-5ee0b7b0239c
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes starts
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes returns vbbm from S3.
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-11} get_brm_bytes returns.
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-11} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vbbm" in 0.1040 seconds. uid: d54ca447-29ce-44b0-89cd-5ee0b7b0239c
20/Aug/2025 21:33:21 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-11} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/vbbm HTTP/1.1" code 200, bytes 1860, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:21 [INFO] (access_logger) {CP Server Thread-12} Got incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vss". uid: 949fe7e5-a872-4046-b36d-b7d5f377d9e6
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes starts
20/Aug/2025 21:33:21 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns vss from S3.
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-12} get_brm_bytes returns.
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-12} Finished processing incoming GET request from "172.31.45.153" to "/cmapi/0.4.0/node/meta/vss" in 0.1061 seconds. uid: 949fe7e5-a872-4046-b36d-b7d5f377d9e6
20/Aug/2025 21:33:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-12} 172.31.45.153 ACCESS "GET /cmapi/0.4.0/node/meta/vss HTTP/1.1" code 200, bytes 2768, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 172.31.43.121 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-4} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em". uid: f066a0ab-24e4-4184-9a24-a6845c149583
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-4} get_brm_bytes starts
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-4} get_brm_bytes returns em from S3.
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-4} get_brm_bytes returns.
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/em" in 0.0942 seconds. uid: f066a0ab-24e4-4184-9a24-a6845c149583
20/Aug/2025 21:33:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/em HTTP/1.1" code 200, bytes 4940, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-13} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal". uid: 185eae5f-4c29-415b-9e8e-320183d19e2f
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes starts
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns journal from S3.
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-13} get_brm_bytes returns.
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-13} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/journal" in 0.1110 seconds. uid: 185eae5f-4c29-415b-9e8e-320183d19e2f
20/Aug/2025 21:33:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/journal HTTP/1.1" code 200, bytes 5, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-10} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm". uid: 3e3b34e3-63f0-4bdf-927d-59fd3aad5269
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-10} get_brm_bytes starts
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-10} get_brm_bytes returns vbbm from S3.
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-10} get_brm_bytes returns.
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vbbm" in 0.1070 seconds. uid: 3e3b34e3-63f0-4bdf-927d-59fd3aad5269
20/Aug/2025 21:33:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vbbm HTTP/1.1" code 200, bytes 1860, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-9} Got incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss". uid: 2519f826-69c2-40d5-8285-dcbdd2a6e73d
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes starts
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes returns vss from S3.
20/Aug/2025 21:33:22 [DEBUG] (cmapi_server) {CP Server Thread-9} get_brm_bytes returns.
20/Aug/2025 21:33:22 [INFO] (access_logger) {CP Server Thread-9} Finished processing incoming GET request from "172.31.43.121" to "/cmapi/0.4.0/node/meta/vss" in 0.1116 seconds. uid: 2519f826-69c2-40d5-8285-dcbdd2a6e73d
20/Aug/2025 21:33:22 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-9} 172.31.43.121 ACCESS "GET /cmapi/0.4.0/node/meta/vss HTTP/1.1" code 200, bytes 2768, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:29 [INFO] (access_logger) {CP Server Thread-7} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/cluster/status". uid: af4852af-2d56-472c-92eb-2b81598d17d6
20/Aug/2025 21:33:29 [DEBUG] (cmapi_server) {CP Server Thread-7} get_status starts
20/Aug/2025 21:33:29 [DEBUG] (cmapi_server) {CP Server Thread-7} Cluster status command called. Getting status.
20/Aug/2025 21:33:29 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} Starting new HTTPS connection (1): ip-172-31-47-126.us-west-2.compute.internal:8640
20/Aug/2025 21:33:29 [INFO] (access_logger) {CP Server Thread-8} Got incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/status". uid: cac6954c-416d-483b-8fb6-243f71b24ea8
20/Aug/2025 21:33:29 [DEBUG] (cmapi_server) {CP Server Thread-8} get_status starts
20/Aug/2025 21:33:29 [ERROR] (root) {CP Server Thread-8} Cannot establish or use DBRM connection.
Traceback (most recent call last):
  File "/usr/share/columnstore/cmapi/mcs_node_control/models/node_status.py", line 38, in get_cluster_mode
    with DBRM() as dbrm:
  File "/usr/share/columnstore/cmapi/mcs_node_control/models/dbrm.py", line 54, in __enter__
    self.connect()
  File "/usr/share/columnstore/cmapi/mcs_node_control/models/dbrm.py", line 48, in connect
    self.dbrm_socket.connect(dbrm_host, dbrm_port)
  File "/usr/share/columnstore/cmapi/mcs_node_control/models/dbrm_socket.py", line 208, in connect
    self._socket.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused
20/Aug/2025 21:33:29 [DEBUG] (cmapi_server) {CP Server Thread-8} get_status returns {'timestamp': '2025-08-20 21:33:29.403882', 'uptime': 13950, 'dbrm_mode': 'master', 'cluster_mode': 'readonly', 'dbroots': ['1'], 'module_id': 1, 'services': [{'name': 'StorageManager', 'pid': 14381}, {'name': 'workernode', 'pid': 14480}, {'name': 'controllernode', 'pid': 14491}]}
20/Aug/2025 21:33:29 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming GET request from "172.31.47.126" to "/cmapi/0.4.0/node/status" in 0.0540 seconds. uid: cac6954c-416d-483b-8fb6-243f71b24ea8
20/Aug/2025 21:33:29 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.47.126 ACCESS "GET /cmapi/0.4.0/node/status HTTP/1.1" code 200, bytes 281, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:29 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} https://ip-172-31-47-126.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 281
20/Aug/2025 21:33:29 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} Starting new HTTPS connection (1): ip-172-31-43-121.us-west-2.compute.internal:8640
20/Aug/2025 21:33:29 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} https://ip-172-31-43-121.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 319
20/Aug/2025 21:33:29 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} Starting new HTTPS connection (1): ip-172-31-45-153.us-west-2.compute.internal:8640
20/Aug/2025 21:33:29 [DEBUG] (urllib3.connectionpool) {CP Server Thread-7} https://ip-172-31-45-153.us-west-2.compute.internal:8640 "GET /cmapi/0.4.0/node/status HTTP/1.1" 200 319
20/Aug/2025 21:33:29 [DEBUG] (cmapi_server) {CP Server Thread-7} Successfully finished getting cluster status.
20/Aug/2025 21:33:29 [DEBUG] (cmapi_server) {CP Server Thread-7} get_status returns {'timestamp': '2025-08-20 21:33:29.397097', 'ip-172-31-47-126.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:33:29.403882', 'uptime': 13950, 'dbrm_mode': 'master', 'cluster_mode': 'readonly', 'dbroots': ['1'], 'module_id': 1, 'services': [{'name': 'StorageManager', 'pid': 14381}, {'name': 'workernode', 'pid': 14480}, {'name': 'controllernode', 'pid': 14491}]}, 'ip-172-31-43-121.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:33:29.467990', 'uptime': 13949, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': ['2'], 'module_id': 2, 'services': [{'name': 'StorageManager', 'pid': 12650}, {'name': 'workernode', 'pid': 12690}, {'name': 'PrimProc', 'pid': 12712}, {'name': 'WriteEngineServer', 'pid': 12830}]}, 'ip-172-31-45-153.us-west-2.compute.internal': {'timestamp': '2025-08-20 21:33:29.527868', 'uptime': 13950, 'dbrm_mode': 'slave', 'cluster_mode': 'readonly', 'dbroots': ['3'], 'module_id': 3, 'services': [{'name': 'StorageManager', 'pid': 12037}, {'name': 'workernode', 'pid': 12228}, {'name': 'PrimProc', 'pid': 12250}, {'name': 'WriteEngineServer', 'pid': 12368}]}, 'num_nodes': 3}
20/Aug/2025 21:33:29 [INFO] (access_logger) {CP Server Thread-7} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/cluster/status" in 0.1851 seconds. uid: af4852af-2d56-472c-92eb-2b81598d17d6
20/Aug/2025 21:33:29 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-7} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/cluster/status HTTP/1.1" code 200, bytes 1125, user-agent "python-requests/2.32.3"
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-controllernode" is running.
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Call "disable" on service "mcs-workernode@1.service" with "systemctl disable mcs-workernode@1.service".
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-workernode@1.service mcs-workernode@2.service" with "systemctl stop mcs-workernode@1.service mcs-workernode@2.service".
20/Aug/2025 21:33:30 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:30 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-5} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:30 [INFO] (access_logger) {CP Server Thread-4} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 3987ffef-1bbd-41bf-b8dd-ca21fbc5cf03
20/Aug/2025 21:33:30 [DEBUG] (cmapi_server) {CP Server Thread-4} get_primary starts
20/Aug/2025 21:33:30 [DEBUG] (cmapi_server) {CP Server Thread-4} get_primary returns {'is_primary': 'True'}
20/Aug/2025 21:33:30 [INFO] (access_logger) {CP Server Thread-4} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0078 seconds. uid: 3987ffef-1bbd-41bf-b8dd-ca21fbc5cf03
20/Aug/2025 21:33:30 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-4} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-workernode" is running.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-workernode" with "systemctl show -p ActiveState --value mcs-workernode".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Service "mcs-workernode" is in "inactive" state
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Stopping "mcs-storagemanager".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Call "stop" on service "mcs-storagemanager" with "systemctl stop mcs-storagemanager".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-storagemanager" is running.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-storagemanager" with "systemctl show -p ActiveState --value mcs-storagemanager".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Service "mcs-storagemanager" is in "inactive" state
20/Aug/2025 21:33:31 [INFO] (root) {CP Server Thread-6} Skip starting StorageManager with systemd dispatcher.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Call "enable" on service "mcs-workernode@1.service" with "systemctl enable mcs-workernode@1.service".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Service "mcs-workernode@1.service" is in "inactive" state
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-workernode@1.service".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-workernode@1.service" with "systemctl start mcs-workernode@1.service".
20/Aug/2025 21:33:31 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-13} 127.0.0.1 ACCESS "POST /notfound HTTP/1.1" code 404, bytes 513, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:33 [INFO] (access_logger) {CP Server Thread-10} Got incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary". uid: 2640d931-1df3-4c52-8582-bf7784effca9
20/Aug/2025 21:33:33 [DEBUG] (cmapi_server) {CP Server Thread-10} get_primary starts
20/Aug/2025 21:33:33 [DEBUG] (cmapi_server) {CP Server Thread-10} get_primary returns {'is_primary': 'True'}
20/Aug/2025 21:33:33 [INFO] (access_logger) {CP Server Thread-10} Finished processing incoming GET request from "127.0.0.1" to "/cmapi/0.4.0/node/primary" in 0.0075 seconds. uid: 2640d931-1df3-4c52-8582-bf7784effca9
20/Aug/2025 21:33:33 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-10} 127.0.0.1 ACCESS "GET /cmapi/0.4.0/node/primary HTTP/1.1" code 200, bytes 22, user-agent "Python-urllib/3.12"
20/Aug/2025 21:33:33 [INFO] (root) {HeartBeater} Loaded the config file, my name is ip-172-31-47-126.us-west-2.compute.internal
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-workernode@1.service.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-workernode@1.service" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-workernode@1.service" with "systemctl show -p ActiveState --value mcs-workernode@1.service".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-workernode@1.service" is in "active" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for all workernodes to come up before starting controllernode on a primary.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for "['DBRM_Worker1', 'DBRM_Worker2', 'DBRM_Worker3']"....30
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-controllernode" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-controllernode" is in "inactive" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-controllernode".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-controllernode" with "systemctl start mcs-controllernode".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-controllernode.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-controllernode" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-controllernode" with "systemctl show -p ActiveState --value mcs-controllernode".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-controllernode" is in "active" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-primproc" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-primproc" is in "inactive" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-primproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-primproc" with "systemctl start mcs-primproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-primproc.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-primproc" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-primproc" with "systemctl show -p ActiveState --value mcs-primproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-primproc" is in "active" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-writeengineserver" is in "inactive" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-writeengineserver".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-writeengineserver" with "systemctl start mcs-writeengineserver".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-writeengineserver.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-writeengineserver" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-writeengineserver" with "systemctl show -p ActiveState --value mcs-writeengineserver".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-writeengineserver" is in "active" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-dmlproc" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-dmlproc" is in "inactive" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-dmlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-dmlproc" with "systemctl start mcs-dmlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-dmlproc.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-dmlproc" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-dmlproc" with "systemctl show -p ActiveState --value mcs-dmlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-dmlproc" is in "active" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for controllernode to come up before starting ddlproc/dmlproc on non-primary nodes.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-ddlproc" is running.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Service "mcs-ddlproc" is in "inactive" state
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Starting "mcs-ddlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Call "start" on service "mcs-ddlproc" with "systemctl start mcs-ddlproc".
20/Aug/2025 21:33:33 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:33 [INFO] (root) {CP Server Thread-6} Attempt to run dbbuilder on primary node
20/Aug/2025 21:33:34 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:34 [ERROR] (root) {CP Server Thread-6} Calling "su -s /bin/sh -c "/usr/bin/dbbuilder 7" mysql" finished with return code: "1" and stderr+stdout "".
20/Aug/2025 21:33:34 [DEBUG] (root) {CP Server Thread-6} Successfully started mcs-ddlproc.
20/Aug/2025 21:33:34 [DEBUG] (root) {CP Server Thread-6} Checking if "mcs-ddlproc" is running.
20/Aug/2025 21:33:34 [DEBUG] (root) {CP Server Thread-6} Call "show -p ActiveState --value" on service "mcs-ddlproc" with "systemctl show -p ActiveState --value mcs-ddlproc".
20/Aug/2025 21:33:34 [DEBUG] (root) {CP Server Thread-6} Waiting for command to finish.
20/Aug/2025 21:33:34 [DEBUG] (root) {CP Server Thread-6} Service "mcs-ddlproc" is in "active" state
20/Aug/2025 21:33:34 [DEBUG] (cmapi_server) {CP Server Thread-6} Node is ready to accept queries.
20/Aug/2025 21:33:34 [INFO] (access_logger) {CP Server Thread-6} Finished processing incoming PUT request from "172.31.43.121" to "/cmapi/0.4.0/node/config" in 14.9258 seconds. uid: 2c487b9f-a5b2-4eb7-adae-991054a0084a
20/Aug/2025 21:33:34 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-6} 172.31.43.121 ACCESS "PUT /cmapi/0.4.0/node/config HTTP/1.1" code 200, bytes 43, user-agent "Python/3.9 aiohttp/3.11.16"
20/Aug/2025 21:33:34 [INFO] (access_logger) {CP Server Thread-8} Got incoming PUT request from "172.31.43.121" to "/cmapi/0.4.0/node/commit". uid: eeddf675-5ce4-4e18-81ef-e645684336ca
20/Aug/2025 21:33:34 [DEBUG] (cmapi_server) {CP Server Thread-8} put_commit starts
20/Aug/2025 21:33:34 [DEBUG] (cmapi_server) {CP Server Thread-8} put_commit returns {'timestamp': '2025-08-20 21:33:34.079128'}
20/Aug/2025 21:33:34 [INFO] (access_logger) {CP Server Thread-8} Finished processing incoming PUT request from "172.31.43.121" to "/cmapi/0.4.0/node/commit" in 0.0005 seconds. uid: eeddf675-5ce4-4e18-81ef-e645684336ca
20/Aug/2025 21:33:34 [INFO] (cherrypy.access.131723105933584) {CP Server Thread-8} 172.31.43.121 ACCESS "PUT /cmapi/0.4.0/node/commit HTTP/1.1" code 200, bytes 43, user-agent "python-requests/2.32.3"