[MCOL-1197] CPIMPORT fails and Abandoned when data row is too BIG to Handle Created: 2018-02-05  Updated: 2018-05-22  Resolved: 2018-05-22

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: 1.1.2
Fix Version/s: 1.1.5

Type: Bug Priority: Critical
Reporter: Aziz Vahora Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: centos6

Attachments: File MCOL-1197.sql.gz    
Sprint: 2018-08, 2018-09, 2018-10, 2018-11

 Description   

terminate called after throwing an instance of 'std::runtime_error'
what(): Data Row too BIG to handle!!

mcsadmin restartSystem
restartsystem Mon Feb 5 17:07:18 2018
The following tables are locked:
LockID Name Process PID Session CreationTime State DBRoots
4 rtt.rtt_webservicelog cpimport 25972 BulkLoad 2018-02-05 05:00:02 PM Abandoned1
Your options are:
Cancel – Cancel the restart request
Wait – Wait for write operations to end and then restart
Force – Force a restart
What would you like to do: [Cancel]: Force

when cpimport fails, if does not properly rollback the transaction leaving the table locked and no way to kill the PID except to restart the system.
pid is not found on the system.
show open tables shows the lock but PID is 0.

 Comments   
Comment by Andrew Hutchings (Inactive) [ 2018-02-05 ]

Hi,

Do you have a test case for this? Usually you would hit a buffer error with a rollback before hitting this.

Comment by Aziz Vahora [ 2018-02-07 ]

Hey Andrew,
Test case is as following:

create table test (id int,listofitems text) engine=innodb;
create table test_cs (id int,listofitems text) engine=columnstore;

insert a row into test with very very large text, usually more than 1 million characters.

select from test table | cpimport -m1 -e 10 rtt test -s '\t' -f STDIN

This is where cpimport fails and process never releases lock on the test_cs table.

Comment by Andrew Hutchings (Inactive) [ 2018-02-07 ]

Confirmed. Test case:

1. gunzip the file MCOL-1197.sql.gz
2. in MariaDB Server:

set global max_allowed_packet=8388608

3. /usr/local/mariadb/columnstore/mysql/bin/mysql -uroot --host=127.0.0.1 test < ~/tmp/MCOL-1197.sql
4. in MariaDB Server:

CREATE TABLE t2 (a int, b longtext) engine=columnstore;

5. Execute the following:

/usr/local/mariadb/columnstore/mysql/bin/mysql -uroot --host=127.0.0.1 --skip-column-names -Ns test --execute "select * from t1" | sudo /usr/local/mariadb/columnstore/bin/cpimport -m1 -e 10 test t2 -s '\t' -f STDIN

It will fail as described. Increasing cpimport's buffers does not seem to help.

Comment by Andrew Hutchings (Inactive) [ 2018-05-09 ]

Made cpimport's '-c' flag work for this. If < 1MB it sets it to 1MB, otherwise it will use the selected size.

For QA. See the third comment. To make it work after the patch add '-c 8388608' to cpimport.

Comment by Daniel Lee (Inactive) [ 2018-05-22 ]

Build verified: 1.1.5-1 source

/root/columnstore/mariadb-columnstore-server
commit 0c983bff02172849a174dde46b62d76aa66485f8
Merge: 6b8a674 d5e6d89
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Thu Apr 26 16:16:51 2018 -0500

Merge pull request #112 from mariadb-corporation/davidhilldallas-patch-3

update to 1.1.5

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 1ea5198e0e9ecc2a8d13e6b44bf6c632f8561199
Merge: 4533116 59858aa
Author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Date: Fri May 18 12:37:47 2018 +0100

Merge pull request #475 from drrtuy/MCOL-1415

MCOL-1415

Reproduced the issue in 1.1.4-1 and verified the fix using the -c option.

Generated at Thu Feb 08 02:26:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.