[MCOL-1160] Bulk write API doesn't start new block for dictionary Created: 2018-01-15  Updated: 2023-10-26  Resolved: 2018-01-31

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1.3

Type: Bug Priority: Major
Reporter: Andrew Hutchings (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MCOL-1154 mcsapi causes data corruption if a NU... Closed
Sprint: 2018-02, 2018-03

 Description   

The bulk write API continues the same block for dictionary entries. PrimProc uses a block level cache and this isn't flushed on a bulk write so new appended entries aren't seen by PrimProc showing CpNoTf until restart.

test.t1 is a table with an int and varchar(64).

Test case:

    std::string table("t1");
 
    std::string db("test");
 
    mcsapi::ColumnStoreDriver* driver;
 
    mcsapi::ColumnStoreBulkInsert* bulk;
 
    driver = new mcsapi::ColumnStoreDriver();
 
    bulk = driver->createBulkInsert(db, table, 0, 0);
 
    bulk->setColumn(0, 1);
 
    bulk->setColumn(1, "preload");
 
    bulk->writeRow();
 
    bulk->commit();
 
    mysql_query(my_con, "SELECT * FROM mcol1154");
 
    MYSQL_RES* result = mysql_store_result(my_con);
 
    mysql_free_result(result);
 
    delete bulk;
 
    bulk = driver->createBulkInsert(db, table, 0, 0);
 
    bulk->setColumn(0, 1);
 
    bulk->setColumn(1, "hello world1");
 
    bulk->writeRow();
 
    bulk->setColumn(0, 2);
 
    bulk->setColumn(1, "hello world4");
 
    bulk->writeRow();
 
    bulk->setColumn(0, 3);
 
    bulk->setColumn(1, "hello world9");
 
    bulk->writeRow();
 
    bulk->setColumn(0, 4);
 
    bulk->setColumn(1, "hello world16");
 
    bulk->writeRow();
 
    bulk->commit();
 
    delete bulk;
 
    delete driver;



 Comments   
Comment by Andrew Hutchings (Inactive) [ 2018-01-15 ]

workaround:

select calflushcache();

This will flush the PrimProc cache and the values will show correctly.

Comment by Andrew Hutchings (Inactive) [ 2018-01-15 ]

cpimport solves this by sending a PrimProc flush for the dictionary blocks in TableInfo::setParseComplete() calling cacheutils::flushPrimProcallverBlocks()

Comment by Andrew Hutchings (Inactive) [ 2018-01-23 ]

Pull requests open for the API and engine. Both are needed for the test to pass (although mixed API/engine versions will still work).

For QA: You need the patch in both engine and API. There is a test for this in the API's built-in regression suite.

Comment by Daniel Lee (Inactive) [ 2018-01-31 ]

Build verified: Github source 1.1.3-1
[root@localhost ~]# cat mariadb-columnstore-1.1.3-1-centos7.x86_64.bin.tar.txt
/root/columnstore/mariadb-columnstore-server
commit 99cdb0a4b5626426402a2be2572844409e4db18d
Merge: f56e806 936a78c
Author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Date: Wed Jan 31 09:35:48 2018 +0200
Merge pull request #91 from mariadb-corporation/MCOL-964
MCOL-964 Use cache_table name only if from view
/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit dd79644ae7003754b014bfbee1f33763286f8ad2
Merge: 70993d4 2b944eb
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Jan 30 15:32:43 2018 -0600
Merge pull request #381 from mariadb-corporation/MCOL-1160
MCOL-1160 Track and flush dictionary blocks
[root@localhost mariadb-columnstore-api]# git show
commit f053df67f2efe6ea64b67be0839b31d3a57d1784
Merge: 8246470 4279cd6
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Jan 30 15:33:40 2018 -0600
Merge pull request #41 from mariadb-corporation/MCOL-1160
MCOL-1160 Fix dictionary flushing
All 24 API tests passed.

Test mcol1160 is test #16 in the test suite.

Generated at Thu Feb 08 02:26:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.