The issue that Daniel reproduced can easily be reproduced on a single server 1.1 stack.
I have failed to reproduce it on a single server 1.0 stack.

This implies there's a good possibility the bug was introduced in 1.1. I will attempt to ascertain the exact patch that did it.

Meantime, perhaps the customer may want to try 1.0.12 and see if that helps.

A git bisect shows with some certainty the regression happened during the transition from MariaDB 10.1 -> 10.2 in ColumnStore 1.1. It is difficult to pinpoint exactly what triggered it as finding matching server and engine code revisions is very difficult (and not all of them compile on my test machine).

Recommended next step is to look at what the server is asking of the engine in both 1.0 and 1.1 for these queries to spot any differences.

In 02/17, we added threadpooling to ExeMgr to increase performance. The connections to the connector, DMLProc, or DDLProc are thread-pooled and limited to 50 threads. Any additional connections are delayed until a thread becomes available. By removing the 50 limit, and allowing the threadpool to grow to an unspecified number of threads, the issue clears up.

DMLProc still calls ExeMgr even though there's no explicit query – it still needs to do system catalog queries.

Why waiting for an ExeMgr thread causes DMLProc to wait forever is not yet clear. However, this is the first progress we've made on this issue so I thought a status update was needed. I should be able to clear this all up in one day more.

Since they are inserts every one of them will flush the system catalog cache (for extent metadata update) and since they are on the UM they will get the system catalog via ExeMgr rather than direct from PrimProc (there are two access methods in the code).

I believe ExeMgr waits forever because it is waiting on PrimProc (MCOL-1062). This can be observed using Poor Man's Profiler (https://poormansprofiler.org/) on PrimProc and ExeMgr and looking at what stage the threads are stuck. PrimProc is trying to get a threadpool thread in itself for BPP searches but can't because none are available. None can be freed up.

A work around is to shut the system down. Be sure it is completely down. Then add the following to the Columnstore.xml:

<ServerThreads>200</ServerThreads>
<ServerQueueSize>400</ServerQueueSize>
such that it looks like this:

<Columnstore Version="V1.0.0">
<ExeMgr1>
<ServerThreads>200</ServerThreads>
<ServerQueueSize>400</ServerQueueSize>
<IPAddr>127.0.0.1</IPAddr>
<Port>8601</Port>
<Module>pm1</Module>
</ExeMgr1>

Restart the system

The defaults here are 50/100. You could make them bigger than 200/400.

After some experiments, it becomes clear that the accept loop in ExeMgr must have unfettered access to new threads, so that is what I did.

Build verified: 1.1.3-1 Github source

/root/columnstore/mariadb-columnstore-server
commit e0ae0d2fecf9941887478d9aa669c8b2d1092090
Merge: 21ec50194e 2490ddf50e
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Fri Jan 19 12:39:05 2018 -0600

Merge pull request #84 from mariadb-corporation/MCOL-1159

MCOL-1159 Merge mariadb-10.2.12

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit c74d5de21d6571c0b0e9a12dacaf77856d332e63
Merge: 201813d6 63adbd0f
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Mon Jan 22 09:42:34 2018 -0600

Merge pull request #375 from mariadb-corporation/dev-1.1-build-fix

Fix missing compiler flag from 1.0 -> 1.1 merge

No longer reproducing the issue originally reported (see test case in comment #1)