Document information about thread pools

We should probably document this:

There appears to be some confusion on how to control the ThreadPools of ExeMgr and DMLProc. This has changed since 1.1.3-2

The settings
<ServerThreads>200</ServerThreads>
<ServerQueueSize>400</ServerQueueSize>
no longer exist in <ExeMgr>

Queue size can no longer be set for ExeMgr. It is hard-coded to 0 (unlimited). If someone is running an older version that uses these settings, you should advise that ServerQueueSize be set to 0.

The way the generic ThreadPool works is:
There are two settings. ThreadPoolSize is the maximum number of threads the pool will generate or maintain. Queuesize is the maximum number of items the threadpool will hold, including the running threads. Any items over the ThreadPoolSize are queued waiting for a thread to become available. If queue size is exceeded, the caller is blocked until a queue spot becomes available.

This all is different if queue size is set to 0 (which it is for ExeMgr). In this case, there is no queue, and there is no maximum on the number of threads that can be concurrently running in the pool. ThreadPoolSize, in this case, controls the number of threads to maintain in the pool if the number of running threads dips below ThreadPoolSize. The idle threads to be shut down are not closed immediately. There's a 10 minute grace period in case the work load spikes real soon, thus alleviating the overhead of destroying and immediately re-creating threads.

This is not the same behavior as the PriorityThreadPool as used in PrimProc.

So the only issue any more is the number of threads to have hanging around when there's little work to do. <ThreadPoolSize> controls this.

To set the ThreadPool size, use
<ExeMgr1>
<ThreadPoolSize>200</ThreadPoolSize> // default 50

There's a debug monitor for ThreadPool. For ExeMgr server ThreadPool, the setting
<ExeMgr1>
<ThreadPoolDebug>Y</ThreadPoolDebug>
controls this.

Use this setting with care, as it can become quite verbose and fill up disk drives over time.

The new logs will be found in
/var/log/mariadb/columnstore/trace/ThreadPool_ExeMgrServer

There are two threadpools of interest in ExeMgr. One is that discussed above – the ServerPool which handles connections from the connector and DDL/DMLProc – one per session or so.

The other is Joblist, which handles all the transient threads each query needs to process. Its name internally is jobstepThreadPool. It's controlled by settings in the <JobList> section of the XML:
<JobList>
<ThreadPoolSize>100</ThreadPoolSize> // default is 100

Queue size is 0 and not settable.

Debug for ExeMgr's jobstepThreadPool is turned on by
<JobList>
<ThreadPoolDebug>Y</ThreadPoolDebug>

The logs will be in:
/var/log/mariadb/columnstore/trace/ThreadPool_ExeMgrJobList

DMLProc also use a jobstepThreadPool
DMLProc's jobstepThreadPool's size is controlled by the same setting:
<JobList>
<ThreadPoolSize>100</ThreadPoolSize>

Debug for DMLProc is controlled by
<JobList>
<DMLThreadPoolDebug>Y</DMLThreadPoolDebug>

The logs will be in:
/var/log/mariadb/columnstore/trace/ThreadPool_DMLProcJobList

These logs can get large and are not auto-archived or deleted. Once the problem occurs and is logged, this logging should be turned off to save disk. Old logs should be manually deleted.

The logs may be cryptic and will take some understanding of the workings of class ThreadPool.