[MCOL-1176] 10Mio Row test fails, only 9988608 rows are written to ColumnStore Created: 2018-01-24 Updated: 2023-10-26 Resolved: 2018-01-31 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | None |
| Affects Version/s: | 1.1.2 |
| Fix Version/s: | 1.1.3 |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Jens Röwekamp (Inactive) | Assignee: | Daniel Lee (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Debian 9 no other dists tested |
||
| Sprint: | 2018-02, 2018-03 |
| Description |
|
When expanding the 1Mio row python test [1] to 10Mio rows, only 9988608 rows are written to ColumnStore. The first 11392 rows are somehow dropped. Same behaviour for the 10Mio row java test [2]. Here also only 9988608 rows are written. Therefore, probably a bug in the C++ implementation or Swig. [1] https://github.com/mariadb-corporation/mariadb-columnstore-api/blob/MCOL-1091/python/test/test_million_row.py |
| Comments |
| Comment by Andrew Hutchings (Inactive) [ 2018-01-24 ] |
|
Confirmed in the C++ API too. Assigned to me |
| Comment by Andrew Hutchings (Inactive) [ 2018-01-24 ] |
|
10,000,000 rows means there will be 2 extents, the first with 8388608. The writeRow() that triggers the extent rollover will have 11392 rows left to write. So these must be getting dropped when the new extent is being created. |
| Comment by Andrew Hutchings (Inactive) [ 2018-01-25 ] |
|
I think this affects DML INSERTS too but I haven't yet been able to build a test to prove it. I did try to get the API to insert just under 8388608 rows to try and push it over the edge with DML and it creates an extra unused extent which fires an error when read. |
| Comment by Andrew Hutchings (Inactive) [ 2018-01-25 ] |
|
DML Insert is not affected and the just under 8388608 row problem was due to cpimport and bulk write skipping the first block even if it is empty. It is part of the same problem. |
| Comment by Andrew Hutchings (Inactive) [ 2018-01-25 ] |
|
Made this a blocker as this is a data loss issue with no good workaround. |
| Comment by Andrew Hutchings (Inactive) [ 2018-01-30 ] |
|
Branches in API and engine to be merged. For QA: There is a test in the API's built-in regression suite. |
| Comment by Daniel Lee (Inactive) [ 2018-01-31 ] |
|
Build verified: Github source 1.1.3-1 [root@localhost ~]# cat mariadb-columnstore-1.1.3-1-centos7.x86_64.bin.tar.txt Merge pull request #91 from mariadb-corporation/ /root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine Merge pull request #381 from mariadb-corporation/ [root@localhost mariadb-columnstore-api]# git show Merge pull request #41 from mariadb-corporation/ All 24 API tests passed. |