[MCOL-824] Constant TIME_WAIT Connections Created: 2017-07-21  Updated: 2017-11-27  Resolved: 2017-11-27

Status: Closed
Project: MariaDB ColumnStore
Component/s: DDLProc, DMLProc, ExeMgr
Affects Version/s: 1.0.9
Fix Version/s: Icebox

Type: Bug Priority: Critical
Reporter: Yago Gonzalez Rozas Assignee: David Thompson (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

VMWare - CentOS 7

Attachments: File columnstoreSupportReport.traqus_db1.tar.gz    

 Description   

We have been using MariaDB ColumnStore consistently since version 1.0.6. After we installed the VMWare tools to check the usage of our machine, we have seen random shutdowns of the database. The issue seems to come from a lot of TCP connections that there three components create. Currently, if we shutdown the database, the TCP connections come down to 5, and as soon as we start it, they go to nearly 600, mostly on TIME_WAIT, eating all TCP connections

 Comments   
Comment by Daniel Lee (Inactive) [ 2017-07-21 ]

Aloha, Yago;

We are looking into the issues. Please help us by providing the following information.

1) Is this a single-server or multi-server installation?
2) By "random shutdowns of the database", did you mean ColumnStore in the VM, or the VM itself was shutdown?
3) Do you know what initiated the shutdown? ColumnStore does not do self-shutdown.
4) Do you have other processes that interacts/controls the operation of ColumnStore?
5) Please execute the columnstoreSupport tool on UM1 (multiple-server) or PM1(single-server) and provide us the report
/usr/local/mariadb/columnstore/columnstoreSupport -a

Mahalo

Daniel

Comment by Yago Gonzalez Rozas [ 2017-08-01 ]

Daniel,

Sorry for the delay.

a) Multi with only one node
b) Columnstore, and we have started seeing it on a second machine with multi threading.
c) It just blocks the database, it is not complete shutdown.
d) No, only query
e) columnstoreSupportReport.traqus_db1.tar.gz

Comment by David Thompson (Inactive) [ 2017-08-09 ]

ygonroz In looking at the 2nd traqus_db1 report i see a local memory alert. It looks like you have a java process consuming a decent percentage of the memory on the box. This is fine but you'll need to override the default memory settings of ColumnStore to limit it to allow whatever overhead you need for other processes: https://mariadb.com/kb/en/mariadb/mariadb-columnstore-performance-related-configuration-settings/#memory-management-numblockspct-and-totalummemory

Aug 1 10:53:37 773-traqus ServerMonitor[24414]: 37.505265 |0|0|0| I 09 CAL0000: Local Memory above Critical Memory threshold with a percentage of 100 ; Swap 0
Aug 1 10:53:37 773-traqus ServerMonitor[24414]: 37.569381 |0|0|0| I 09 CAL0000: Memory Usage for Process: python : Memory Used 2154 : % Used 1
Aug 1 10:53:37 773-traqus ServerMonitor[24414]: 37.569431 |0|0|0| I 09 CAL0000: Memory Usage for Process: rsyslogd : Memory Used 2393 : % Used 1
Aug 1 10:53:37 773-traqus ServerMonitor[24414]: 37.569455 |0|0|0| I 09 CAL0000: Memory Usage for Process: systemd-journald : Memory Used 3366 : % Used 1
Aug 1 10:53:37 773-traqus ServerMonitor[24414]: 37.569477 |0|0|0| I 09 CAL0000: Memory Usage for Process: mysqld : Memory Used 17944 : % Used 1
Aug 1 10:53:37 773-traqus ServerMonitor[24414]: 37.569498 |0|0|0| I 09 CAL0000: Memory Usage for Process: java : Memory Used 658145 : % Used 27

Comment by David Thompson (Inactive) [ 2017-08-09 ]

Also columnstore utilizes multiple function based processes so it's normal to see a decent number of connections but 600 seems high. Do you have a list of the ports for review? Which 3 processes were the problem?

Comment by David Thompson (Inactive) [ 2017-09-13 ]

ygonroz did you have a chance to review my comments above?

Generated at Thu Feb 08 02:24:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.