[MCOL-1305] Bulk import of large CSV file failed because of SIGSEGV on Pentaho server. Created: 2018-03-26  Updated: 2023-10-26  Resolved: 2018-04-12

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 1.1.4
Fix Version/s: 1.1.4

Type: Bug Priority: Major
Reporter: Elena Kotsinova (Inactive) Assignee: Elena Kotsinova (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

CS 1.1.3 on CentOS 7

Attachments: Text File hs_err_pid32515.log    
Sprint: 2018-07, 2018-08

 Description   

1. Start bulk load of a large CSV file with pentaho bulk load adapter. File contains 121 million rows (9 GB in size).
2. Pentaho CSV input module writes data on chunks and passes them to the Bulk Loader.
3. After last chunk has been read the error is displayed on the server where pentaho job was run:

2018/03/26 15:06:05 - MariaDB ColumnStore Bulk Loader.0 - Linenr 121250000
 
2018/03/26 15:06:06 - CSV file input.0 - Finished processing (I=121270191, O=0, R=0, W=121270191, U=0, E=0)
 
#
 
# A fatal error has been detected by the Java Runtime Environment:
 
#
 
#  SIGSEGV (0xb) at pc=0x00007fe93dee54dc, pid=32515, tid=0x00007fe9255ea700
 
#
 
# JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build 1.8.0_161-b14)
 
# Java VM: OpenJDK 64-Bit Server VM (25.161-b14 mixed mode linux-amd64 compressed oops)
 
# Problematic frame:
 
# C  [libc.so.6+0x804dc]  cfree+0x1c
 
#
 
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
 
#
 
# An error report file with more information is saved as:
 
# /home/elena.kotsinova/downloads/data-integration/hs_err_pid32515.log
 
#
 
# If you would like to submit a bug report, please visit:
 
#   http://bugreport.java.com/bugreport/crash.jsp
 
# The crash happened outside the Java Virtual Machine in native code.
 
# See problematic frame for where to report the bug.

See attached hs_err_pid32515.log

Result:
The data are not loaded into database.

 Comments   
Comment by Andrew Hutchings (Inactive) [ 2018-03-26 ]

I think this is a duplicate of MCOL-1245. In which case it is already fixed in 1.1.4. Is it possible to re-test with a build from the latest develop-1.1?

Comment by Elena Kotsinova (Inactive) [ 2018-03-26 ]

Not sure that both are related.
Issue is connected with data volume. I tested with half of the volume and the load passed successfully.

Comment by Andrew Hutchings (Inactive) [ 2018-03-26 ]

data volume can trigger it too. The easy trigger is basically going over a certain amount of extents and having at least 1 PM remote from where the API is executed. This will cause the HWM packet at commit time to be long enough to trigger compression on it. Which in-turn will cause the crash or other weirdness.

Comment by David Thompson (Inactive) [ 2018-04-02 ]

Can you retest with 1.1.4 mcsapi?

Comment by Elena Kotsinova (Inactive) [ 2018-04-12 ]

load of 9GB flat file - 121070191 records, 10 columns finished with no errors.
Verified for PDI adapter v.1.1.4 Revision: 493914b and CS v.1.1.4 build from 2 April 2018

Generated at Thu Feb 08 02:27:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.