[MCOL-247] mysqld hung and stopsystem stop it Created: 2016-07-01  Updated: 2016-10-20  Resolved: 2016-10-20

Status: Closed
Project: MariaDB ColumnStore
Component/s: MariaDB Server
Affects Version/s: 1.0.1
Fix Version/s: Icebox

Type: Bug Priority: Critical
Reporter: Daniel Lee (Inactive) Assignee: David Thompson (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None

Sprint: 2016-19, 2016-20

 Description   

mscadmin> getsoft
getsoftwareinfo Fri Jul 1 20:21:02 2016

Name : mariadb-columnstore-platform
Version : 1.0
Release : 1
Architecture: x86_64
Install Date: Fri 24 Jun 2016 07:28:00 PM UTC

I was executing an LDI for a 10g lineitem InnoDB table. It ran over night and did not finished. I used the stopsystem to stop the system. ColumnStore admin console showed everything and up and normal, but when I tried to start the MySQL client, I got:

[root@ip-172-30-0-214 dbt3]# mcsmysql testdb
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock' (2 "No such file or directory")

The /usr/local/mariadb/columnstore/mysql/lib/mysql directory was empty and the mysql.sock file does not exist.
root 3067 1 0 04:06 ? 00:00:00 /bin/sh /usr/local/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/usr/local/mariadb/columnstore/mysql/db --pid-file=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-214.pid --ledir=/usr/local/mariadb/columnstore/mysql//sbin
mysql 3247 3067 18 04:06 ? 02:55:18 /usr/local/mariadb/columnstore/mysql//sbin/mysqld --basedir=/usr/local/mariadb/columnstore/mysql/ --datadir=/usr/local/mariadb/columnstore/mysql/db --plugin-dir=/usr/local/mariadb/columnstore/mysql/lib64/mysql/plugin --user=mysql --log-error=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-214.err --pid-file=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-214.pid --socket=/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306

I used kill -9 to kill the two mysqld processed and restarted ColumnStore again. ColumnStore is now operational.

 Comments   
Comment by David Hall (Inactive) [ 2016-08-18 ]

This may be related MCOL-66 and may be fixed

Comment by Dipti Joshi (Inactive) [ 2016-08-30 ]

dleeyh Please test this against 1.0.2 and validate if the issues has been resolved.

Comment by Daniel Lee (Inactive) [ 2016-09-15 ]

Build tested:

mscadmin> getsoft
getsoftwareinfo Thu Sep 15 16:37:03 2016

Name : mariadb-columnstore-platform
Version : 1.0.3
Release : 1
Architecture: x86_64
Install Date: Thu 15 Sep 2016 03:50:22 PM CDT
Group : Applications
Size : 25431329
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.
Signature : (none)
Source RPM : mariadb-columnstore-1.0.3-1.src.rpm
Build Date : Thu 15 Sep 2016 09:56:11 AM CDT

The same issue still exists in this 1.0.3 build

Comment by Daniel Lee (Inactive) [ 2016-10-17 ]

Primary key and indexes for the lineitem table.

ALTER TABLE lineitem ADD PRIMARY KEY (l_orderkey, l_linenumber);CREATE INDEX i_l_shipdate ON lineitem (l_shipdate);
CREATE INDEX i_l_suppkey_partkey ON lineitem (l_partkey, l_suppkey);
CREATE INDEX i_l_partkey ON lineitem (l_partkey);
CREATE INDEX i_l_suppkey ON lineitem (l_suppkey);
CREATE INDEX i_l_receiptdate ON lineitem (l_receiptdate);
CREATE INDEX i_l_orderkey ON lineitem (l_orderkey);
CREATE INDEX i_l_orderkey_quantity ON lineitem (l_orderkey, l_quantity);
CREATE INDEX i_l_commitdate ON lineitem (l_commitdate);

Comment by David Hill (Inactive) [ 2016-10-18 ]

there is logic in the mysql-Columnstore script that should do a force shutdown when it doesn't shutdown cleanly..
I guess that logic needs to be checked and see why it's not working... it does a kill-9 in there.

Comment by David Hall (Inactive) [ 2016-10-18 ]

I ran this test today and I also got the mysqld won't shut down issue. However, after kill -9, it wouldn't come up clean. It tried forever (it appears) to rollback the transaction and never (1 hour) became active. I had to blow away and reinstall.

Comment by David Hall (Inactive) [ 2016-10-19 ]

I ran this test against a mariadb only reference machine. The same thing happens. This should be turned into an MDEV.

Comment by David Hall (Inactive) [ 2016-10-20 ]

I left it sit, and after about 5 hours, mysqld finally stopped normally. I guess it takes that long to roll it back.

Comment by David Thompson (Inactive) [ 2016-10-20 ]

If you google load data infile very large file, lots of articles on why this is a bad idea. This is old and for mysql but is pretty good:
http://stackoverflow.com/questions/2463602/mysql-load-data-infile-acceleration

Generally break the file into smaller chunks.

Comment by David Thompson (Inactive) [ 2016-10-20 ]

Not a bug in column store and basically a limitation of innodb.

Generated at Thu Feb 08 02:19:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.