[MCOL-3791] cpimport crashes intermittantly Created: 2020-02-13  Updated: 2020-03-13  Resolved: 2020-02-19

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: 1.4.2
Fix Version/s: 1.4.3

Type: Bug Priority: Critical
Reporter: David Hall (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Sprint: 2020-2, 2020-3

 Description   

running the regression test001, queries/working_tpch1_compareLogOnly/misc/bug4947.sql always fails with two cpimport crashes. For some reason, it won't fail if run independently.

I've narrowed the cause down to the change from MCOL-498 which allowed for partial filling of files, rather thanfilling with NULLs to the full file size. cpimport, at least for uncompressed tables, uses filesize to determine how much space is left.



 Comments   
Comment by David Hall (Inactive) [ 2020-02-13 ]

I tried adding
<DBRoot1>
<PreallocSpace>OFF</PreallocSpace>
</DBRoot1>

But it still breaks.

Here's some debug logs
2020-02-13 17:42:59 (7800) INFO2 : Initializing import: Table-tpch1.orders2...
2020-02-13 17:42:59 (7800) INFO2 : Initializing import: Table-tpch1.orders2; Col-o_orderkey; OID-45757; hwm-740; file-/var/lib/columnstore/data1/000.dir/000.dir/178.dir/189.dir/000.dir/FILE000.cdf
2020-02-13 17:42:59 (7800) INFO2 : Init raw data offsets in column file OID-45757; DBRoot-1; part-0; seg-0; begByte-6062080; endByte-6037504; availFileSize--24576
2020-02-13 17:42:59 (7800) INFO2 : Initializing import: Table-tpch1.orders2; Col-o_orderstatus; OID-45758; hwm-185; file-/var/lib/columnstore/data1/000.dir/000.dir/178.dir/190.dir/000.dir/FILE000.cdf
2020-02-13 17:42:59 (7800) INFO2 : Initializing import: Table-tpch1.orders2; Col-o_totalprice; OID-45759; hwm-1480; file-/var/lib/columnstore/data1/000.dir/000.dir/178.dir/191.dir/000.dir/FILE000.cdf
2020-02-13 17:42:59 (7800) INFO2 : Init raw data offsets in column file OID-45759; DBRoot-1; part-0; seg-0; begByte-12124160; endByte-12066816; availFileSize--57344
2020-02-13 17:42:59 (7800) INFO2 : Initializing import: Table-tpch1.orders2; Col-o_orderpriority; OID-45760; hwm-1480; file-/var/lib/columnstore/data1/000.dir/000.dir/178.dir/192.dir/000.dir/FILE000.cdf
2020-02-13 17:42:59 (7800) INFO2 : Opening existing store file for o_orderpriority; OID-45761; DBRoot-1; part-0; seg-0; hwm-0; file-/var/lib/columnstore/data1/000.dir/000.dir/178.dir/193.dir/000.dir/FILE000.cdf
2020-02-13 17:42:59 (7800) INFO2 : Init raw data offsets in column file OID-45760; DBRoot-1; part-0; seg-0; begByte-12124160; endByte-12066816; availFileSize--57344

2020-02-13 17:42:59 (7800) INFO : PreProcessing check completed
2020-02-13 17:42:59 (7800) INFO : preProcess completed, run time for this step : 0.0147669 seconds
2020-02-13 17:42:59 (7800) INFO : No of Read Threads Spawned = 1
2020-02-13 17:42:59 (7800) INFO : No of Parse Threads Spawned = 3
2020-02-13 17:42:59 (7800) INFO : Reading input from STDIN to import into table tpch1.orders2...
2020-02-13 17:42:59 (7800) INFO2 : Start reading and loading table tpch1.orders2
2020-02-13 17:43:00 (7800) INFO2 : Dctnry rowsExtentCheck (filling extent): OID-45760; DBRoot-1; part-0; seg-0; spaceRequired-57344; dataInBuffer-0; availSpace-57344; fColWidth-8
2020-02-13 17:43:00 (7800) INFO2 : Dctnry rowsExtentCheck: OID-45760; DBRoot-1; part-0; seg-0; Changing nRows from 30804 to -7168

Notice that begByte (read HWM) is greater than endbyte (read filesize). This causes ColumnBufferManagerDctnry::rowsExtentCheck to calculate a negative number for rows that will fit (spaceRequired = -57344) and then everything goes downhill from there. I think for non-dictionary columns, this may not be a problem as ColumnBufferManager::rowsExtentCheck() is a no-op.

Comment by David Hall (Inactive) [ 2020-02-14 ]

So the questions are:

  1. How does hwm (times blocksize 8192) become > filesize? Seems to me that's a bit alarming?
  2. Why does dictionary columns use the difference here when non-dict columns don't care?
  3. How come <PreallocSpace>OFF</PreallocSpace> didn't stop the problem? Is the problem new or old?
  4. Does this affect compressed files?
  5. What to do?
Comment by Roman [ 2020-02-14 ]

1. Preallocation code adds extra block of EMPTY values in this particular case into non-compressed files.
2. Didn't quite understand the question.
3. It should be How come <PreallocSpace>ON</PreallocSpace> not OFF.
4. No, it doesn't.
5. It is a very good question. Nothing code-wise. We should document this behavior.

Comment by Daniel Lee (Inactive) [ 2020-02-19 ]

Build verified: 1.4.3-1 source

engine commit:
1a65d34

Executed regression test001 test. BUG4947 passed.

Generated at Thu Feb 08 02:45:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.