mcsapi hangs trying to create a bulk insert

Reported by customer:

We are using the python pymcsapi to connect to our MariaDB ColumnStore instance and bulk insert data. There was an issue with our script and it failed to remove the lock on the tables. After running the `/usr/bin/cleartablelock` script to clear the table locks the pymcsapi can no longer create a bulk insert. It just hangs indefinitely.

They have attached the support log to the Case they opened. The 1 strange thing I saw in the log is that the WritEngineServer is using what I think is why to much memory. In 2 cases Write before a PrimProc crashed, it showed using 20% and after it showed 28%. This is not good if the default for the UM-Memory and PM-Cache to be configured to use 75% as default, its causing the memory usage to go into critical use, over 90%. Don't know fi that is related to the issue at hand.

And as as mentioned above, there was a PrimProc crash that occured after the first critcal memory usage was reported. Again, dont know if its related. There was no trace or corefile reported.

Here is some output from the logs

Aug 14 16:50:35 swdc-staging-ec2-db-clmstr ProcessMonitor[1263]: 35.870838 |0|0|0| C 18 CAL0000: *****MariaDB ColumnStore Process Restarting: PrimProc, old PID = 4443

Aug 14 16:50:37 swdc-staging-ec2-db-clmstr joblist[3288]: 37.109300 |0|0|0| C 05 CAL0000: /__w/1/s/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE/storage/columnstore/columnstore/dbcon/joblist/distributedenginecomm.cpp @ 432 DEC: lost connection to 127.0.0.1
Aug 14 16:51:06 swdc-staging-ec2-db-clmstr joblist[2054]: 06.369377 |2147483648|0|0| C 05 CAL0000: st: 0 TupleBPS::sendPrimitiveMessages() caught an exception: DistributedEngineComm::write: Broken Pipe error
Aug 14 16:51:06 swdc-staging-ec2-db-clmstr controllernode[2054]: 06.370850 |0|0|0| C 29 CAL0000: IDB-2044: An internal error occurred. Check the error log file & contact support.

Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.836493 |0|0|0| I 09 CAL0000: Local Memory above Critical Memory threshold with a percentage of 93 ; Swap 0
Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.838751 |0|0|0| I 09 CAL0000: Local-Memory usage at percentage of 93 , Alarm set: 7
Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.856567 |0|0|0| I 09 CAL0000: Memory Usage for Process: ExeMgr : Memory Used 23029 : %% Used 3
Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.856644 |0|0|0| I 09 CAL0000: Memory Usage for Process: mysqld : Memory Used 23767 : %% Used 3
Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.856689 |0|0|0| I 09 CAL0000: Memory Usage for Process: StorageManager : Memory Used 36839 : %% Used 5
Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.856756 |0|0|0| I 09 CAL0000: Memory Usage for Process: WriteEngineServ : Memory Used 155988 : %% Used 20
Aug 14 16:44:19 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 19.856811 |0|0|0| I 09 CAL0000: Memory Usage for Process: PrimProc : Memory Used 417313 : %% Used 53

Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.740873 |0|0|0| I 09 CAL0000: Local Memory above Critical Memory threshold with a percentage of 92 ; Swap 0
Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.743104 |0|0|0| I 09 CAL0000: Local-Memory usage at percentage of 92 , Alarm set: 7
Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.763986 |0|0|0| I 09 CAL0000: Memory Usage for Process: ExeMgr : Memory Used 14842 : %% Used 2
Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.764073 |0|0|0| I 09 CAL0000: Memory Usage for Process: mysqld : Memory Used 23677 : %% Used 3
Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.764133 |0|0|0| I 09 CAL0000: Memory Usage for Process: StorageManager : Memory Used 38967 : %% Used 5
Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.764171 |0|0|0| I 09 CAL0000: Memory Usage for Process: WriteEngineServ : Memory Used 223817 : %% Used 28
Aug 14 17:22:36 swdc-staging-ec2-db-clmstr ServerMonitor[2072]: 36.764210 |0|0|0| I 09 CAL0000: Memory Usage for Process: PrimProc : Memory Used 354907 : %% Used 45