[MCOL-513] analyze and implement thread pools and memory buffers for performance optimization Created: 2017-01-18  Updated: 2017-05-22  Resolved: 2017-05-22

Status: Closed
Project: MariaDB ColumnStore
Component/s: ExeMgr
Affects Version/s: None
Fix Version/s: 1.1.0

Type: Task Priority: Major
Reporter: David Thompson (Inactive) Assignee: David Hill (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Sprint: 2017-2, 2017-3, 2017-4, 2017-5, 2017-6, 2017-7, 2017-8, 2017-9, 2017-10

 Description   

Analyze code and implement thread pools and memory buffers where this will benefit query or write performance.



 Comments   
Comment by David Hall (Inactive) [ 2017-02-17 ]

class ThreadPool has been modified:
add a handle to be returned by Invoke.
new methods Join(handle) and Join(vector of handles)
A monitor class was added for debugging.
The BeginThread() method has been streamlined.
new feature: if queuesize is 0, then maxthreads is a suggestion, not a hard rule. If more threads are needed, they are created. No one is rejected. After 10 minutes of idle, threads will end until we have idled down to maxthreads.

JobList destructor spun up threads to end jobstep threads. This seemed excessive, especially since most of the time, there was little or nothing to do. I experimented with ending things in a loop instead and discovered it to be generally faster.

All jobsteps are now run in jobstepThreadPool as well as all the threads the jobsteps create.

A separate ThreadPool is used to handle mysqld connections and yet another to handle FEMsgHandler.

No ThreadPool is created for DistributedEngineComm. It appears these are long lived threads and a pool would just muddy things up.

Comment by Andrew Hutchings (Inactive) [ 2017-02-20 ]

Reviewed the code and it looks good after a minor change made. Moving to test.

Running normal regressions tests should be adequate for this. Biggest things to look for are crashes and hangs.

Comment by Daniel Lee (Inactive) [ 2017-05-05 ]

Assigned it to Mr. Hill for regression tests.

Comment by David Thompson (Inactive) [ 2017-05-22 ]

This has been tested.

Generated at Thu Feb 08 02:21:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.