FAILOVER: DMLProc failed when hot-standby module is out of service

Build tested: 1.4.4-1 (Jenkins 20200508)

Stack: 3pm combo with glusterfs
OS: Centos 7.6

With a newly installed stack that past sanity test, I took PM2 out of service. After the failover process is completed, DMLProc is in FAILED state. Please see OAM output below.

Logs attached. No log files for PM2 since it was down

Steps:

1. Install a 3pm combo stack with glusterfs
2. take PM2 offline (vagrant halt -f pm2)

------
getprocessstatus Tue May 12 17:12:48 2020

MariaDB ColumnStore Process statuses

Process Module Status Last Status Change Process ID
------------------ ------ --------------- ------------------------ ----------
ProcessMonitor pm1 ACTIVE Tue May 12 16:51:12 2020 4025
ProcessManager pm1 ACTIVE Tue May 12 16:51:19 2020 4246
DBRMControllerNode pm1 ACTIVE Tue May 12 16:52:51 2020 6089
ServerMonitor pm1 ACTIVE Tue May 12 16:52:53 2020 6119
DBRMWorkerNode pm1 ACTIVE Tue May 12 16:52:54 2020 6156
PrimProc pm1 ACTIVE Tue May 12 16:53:00 2020 6254
ExeMgr pm1 ACTIVE Tue May 12 16:58:18 2020 17118
WriteEngineServer pm1 ACTIVE Tue May 12 16:53:22 2020 8240
DDLProc pm1 ACTIVE Tue May 12 16:58:28 2020 17289
DMLProc pm1 FAILED Tue May 12 16:58:54 2020 17379
mysqld pm1 ACTIVE Tue May 12 16:54:02 2020 11047

ProcessMonitor pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
ProcessManager pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
DBRMControllerNode pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
ServerMonitor pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
DBRMWorkerNode pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
PrimProc pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
ExeMgr pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
WriteEngineServer pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
DDLProc pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
DMLProc pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020
mysqld pm2 AUTO_OFFLINE Tue May 12 16:58:12 2020

ProcessMonitor pm3 ACTIVE Tue May 12 16:52:38 2020 3528
ProcessManager pm3 COLD_STANDBY Tue May 12 16:58:46 2020
DBRMControllerNode pm3 COLD_STANDBY Tue May 12 16:58:46 2020
ServerMonitor pm3 ACTIVE Tue May 12 16:53:03 2020 3905
DBRMWorkerNode pm3 ACTIVE Tue May 12 16:53:05 2020 3923
PrimProc pm3 ACTIVE Tue May 12 16:53:09 2020 3946
ExeMgr pm3 ACTIVE Tue May 12 16:58:22 2020 5615
WriteEngineServer pm3 ACTIVE Tue May 12 16:53:23 2020 4281
DDLProc pm3 COLD_STANDBY Tue May 12 16:58:46 2020
DMLProc pm3 COLD_STANDBY Tue May 12 16:58:46 2020
mysqld pm3 ACTIVE Tue May 12 16:58:45 2020 4681