[MCOL-1411] Document how to decide value for VersionBufferFileSize Created: 2018-05-14  Updated: 2023-07-02  Resolved: 2023-07-02

Status: Closed
Project: MariaDB ColumnStore
Component/s: N/A
Affects Version/s: 1.1.4
Fix Version/s: Icebox

Type: Task Priority: Minor
Reporter: Geoff Montee (Inactive) Assignee: Richard Stracke
Resolution: Won't Do Votes: 1
Labels: documentation

Issue Links:
Relates
relates to MCOL-2054 Version buffer overflow could get MCS... Closed

 Description   

The documentation has some details on VersionBufferFileSize:

Version buffer file management

If the following error is received, most likely with a transaction LOAD DATA INFILE or INSERT INTO SELECT then it is recommended to break up the load into multiple smaller chunks, increase the VersionBufferFileSize setting, or consider a non transactional LOAD DATA INFILE or to use cpimport.

ERROR 1815 (HY000) at line 1 in file: 'ldi.sql': Internal error: CAL0006: IDB-2008: The version buffer overflowed. Increase VersionBufferFileSize or limit the rows to be processed.

The VersionBufferFileSize setting is updated in the ColumnStore.xml typically located under /usr/local/mariadb/columnstore/etc. This dictates the size of the version buffer file on disk which provides DML transactional consistency. The default value is '1GB' which reserves up to a 1 Gigabyte file size. Modify this on the PM1 node and restart the system if you require a larger value.

https://mariadb.com/kb/en/library/columnstore-batch-insert-mode/#version-buffer-file-management

There are also some more details here:

https://mariadb.com/kb/en/library/columnstore-storage-architecture/#how-the-version-buffer-files-work

We should probably also add details on how to come up with an optimal value. It sounds like each block updated by a transaction requires 40 bytes in the version buffer, so it sounds like the optimal value might depend on:

  • The size of the file or result set being loaded.
  • The number of columns being loaded.
  • The length of the data in each column.

If the optimal value depends on many different factors, is there a good way to generalize a method to determine the optimal value, or is it something that should be set by trial and error?

Can the error message in the log give some clue on how to decide the optimal value?:

May 14 14:14:07 ip-10-59-10-180 node[18880]: 07.686906 |0|0|0| C 30 CAL0000: VBBM::getBlocks(): version buffer overflow. Increase VersionBufferFileSize. Overflow occured in aged blocks. Requested NumBlocks:VbOid:vbFBO:lastFBO = 1:0:14400:14499 lbid locked is 290157568



 Comments   
Comment by Andrew Hutchings (Inactive) [ 2018-05-14 ]

Should also be updated to say that an UPDATE or DELETE is way more likely to require a large version buffer. LDI and INSERT...SELECT only use version buffer if they are part of a transaction.

Comment by Todd Stoffel (Inactive) [ 2023-07-02 ]

The "create date" on this ticket is pre-convergence with MariaDB server. If the issue still exists in a modern version of the engine/plugin please submit a new ticket.

Generated at Thu Feb 08 02:28:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.