[MCOL-1662] WriteEngine bulk methods do not version dictionaries correctly Created: 2018-08-20 Updated: 2023-10-26 Resolved: 2019-01-04 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | None |
| Affects Version/s: | 1.1.5, 1.2.1 |
| Fix Version/s: | 1.1.7, 1.2.3 |
| Type: | Bug | Priority: | Critical |
| Reporter: | markus makela | Assignee: | Patrick LeBlanc (Inactive) |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Environment: |
single node docker image |
||
| Issue Links: |
|
||||||||||||
| Sprint: | 2018-16, 2018-17, 2018-18, 2018-19, 2018-20, 2018-21 | ||||||||||||
| Description |
|
The following test case creates a table, inserts a row into it via mcsapi and updates it after the bulk load has committed.
Once the update is done, the data in the table is corrupted:
Manually updating it afterwards doesn't fix it, it still remains invalid:
Additional note from Andrew: This also happens when using INSERT...SELECT and LDI when cpimport mode is not used. |
| Comments |
| Comment by markus makela [ 2018-08-20 ] | |||||||||||||||||||||||||||
|
Deleting bulk and driver has no effect and neither does reconnecting before updating. | |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-08-28 ] | |||||||||||||||||||||||||||
|
workaround, execute this at the end:
| |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-08-28 ] | |||||||||||||||||||||||||||
|
My notes: | |||||||||||||||||||||||||||
| Comment by Dipti Joshi (Inactive) [ 2018-11-21 ] | |||||||||||||||||||||||||||
|
markus makela Have you tried the work around that Andrew proposed on August 28th | |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-11-26 ] | |||||||||||||||||||||||||||
|
This is a much wider problem than I first anticipated:
| |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-11-26 ] | |||||||||||||||||||||||||||
|
This is a regression from 1.0 -> 1.1 | |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-11-26 ] | |||||||||||||||||||||||||||
|
The regression in INSERT...SELECT is due to switching away from using versioning for bulk inserts in write engine. I highly suspect we need to add extra checkpointing information in the BRM to fix this properly. | |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-12-14 ] | |||||||||||||||||||||||||||
|
Looks like dictionaries don't get versioned correctly. writeVBEntry() isn't called on them so they get stuck on version 0. | |||||||||||||||||||||||||||
| Comment by Andrew Hutchings (Inactive) [ 2018-12-17 ] | |||||||||||||||||||||||||||
|
Three branches for this one:
UPDATE queries were not updating the version number on dictionary blocks which was causing PrimProc cache to think they hadn't been touched, hence the bad cache issue. UPDATE is the perfect candidate for this because it does a PrimProc read/cache before updating the entry. The fix makes sure that version buffer is turned on for update queries, therefore allowing the block version to be bumped on commit. For QA: there is a test in the comments which has been converted into a regression test. There is also a test in the API's test suite. |