Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
1.1.4
-
None
-
None
-
CS 1.1.2 on CentOS 7
-
2018-07, 2018-08
Description
1. Start bulk load of a large CSV file with pentaho bulk load adapter. File contains 121 million rows (9 GB in size).
2. Pentaho CSV input module writes data on chunks and passes them to the Bulk Loader. This process is visible from the following log part:
2018/03/23 14:51:12 - CSV file input.0 - Line number : 99400000
|
2018/03/23 14:51:16 - MariaDB ColumnStore Bulk Loader.0 - Linenr 99400000
|
2018/03/23 14:51:18 - CSV file input.0 - Line number : 99450000
|
2018/03/23 14:51:18 - MariaDB ColumnStore Bulk Loader.0 - Linenr 99450000
|
2018/03/23 14:51:20 - CSV file input.0 - Line number : 99500000
|
2018/03/23 14:51:24 - MariaDB ColumnStore Bulk Loader.0 - Linenr 99500000
|
2018/03/23 14:51:26 - CSV file input.0 - Line number : 99550000
|
3. After around 2 hours work import process freezes on client side - server where pentaho job runs.
4. This is the time when on PM1 the following message is recorded:
Mar 23 12:53:07 mariadb-59f24c1f-215-1 writeengine[3596]: 07.588935 |0|147|0| I 19 CAL0080: Compression Handling: Compressed data does not fit, caused a chunk shifting @line:928 filename:/usr/lo
|
cal/mariadb/columnstore/data1/000.dir/000.dir/016.dir/135.dir/001.dir/FILE000.cdf, chunkId:0 data size:2030689/available:1859584 -- shifting SUCCESS
|
Mar 23 12:53:07 mariadb-59f24c1f-215-1 kernel: WriteEngineServ[14997]: segfault at 7f4cb80000c8 ip 00007f4cb80000c8 sp 00007f4cc0d22c78 error 15
|
Mar 23 12:53:08 mariadb-59f24c1f-215-1 ProcessMonitor[984]: 08.300933 |0|0|0| C 18 CAL0000: *****Calpont Process Restarting: WriteEngineServer, old PID = 3596
|
Mar 23 12:53:09 mariadb-59f24c1f-215-1 ProcessMonitor[984]: 09.557196 |0|0|0| I 18 CAL0000: Calpont Process WriteEngineServer restarted successfully!!
|
Mar 23 12:53:09 mariadb-59f24c1f-215-1 ProcessManager[1089]: 09.565387 |0|0|0| I 17 CAL0000: MSG RECEIVED: Process Restarted on pm1/WriteEngineServer
|
Mar 23 12:53:09 mariadb-59f24c1f-215-1 ProcessMonitor[984]: 09.644656 |0|0|0| I 18 CAL0000: MSG RECEIVED: Re-Init process request on: cpimport
|
Mar 23 12:53:09 mariadb-59f24c1f-215-1 ProcessMonitor[984]: 09.673504 |0|0|0| I 18 CAL0000: PROCREINITPROCESS: completed, no ack to ProcMgr
|
Result:
1. Pentaho job never finishes.
2. The table in CS where data must be loaded is empty and also it is locked by mcsapi process. Lock on this table is not released after termination of the petaho process. It looks like the lock is never released. Restart of the CS is needed. Or cleartablelock also can be used.
Expected:
1. It is expected that error on PM is propagated to all involved parties (in this case mcsapi and adapter). Final behavior must be - Pentaho job finished with error message.
2. Table lock must be released when import operation was not completed by some reason.
Note: The CSV file has been succesfully imported into CS using
LOAD DATA INFILE |
and data in it are homogeneous.
Attachments
Issue Links
- relates to
-
-
- Closed
-