[MCOL-3901] DBRM_READ_ONLY got read_only mode after version buffer overflow error Created: 2020-03-24  Updated: 2023-10-25  Due: 2020-03-24  Resolved: 2023-10-25

Status: Closed
Project: MariaDB ColumnStore
Component/s: DMLProc
Affects Version/s: 1.2.5
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: Allen Lee (Inactive) Assignee: Unassigned
Resolution: Won't Fix Votes: 3
Labels: None
Environment:

Red Hat Enterprise Linux 7.5
Virtualized, On Premise

Attachments: File columnstoreSupportReport - 2.7z    

 Description   

Customer reported they started to hit version buffer overflow error after Mar 21 06:03:34.
Once they hit this error, they hit another error with "Create table failed due to WE: Error updating calpontsys.systable: a BRM Begin copy error."
After restarting MCS, cluster got READ_ONLY mode.

Mar 21 06:03:34 nvm002314 workernode[29567]: 34.563294 |0|0|0| C 30 CAL0000: VBBM::getBlocks(): version buffer overflow. Increase VersionBufferFileSize. Overflow occured in aged blocks. Requested NumBlocks:VbOid:vbFBO:lastFBO = 50:0:112800:112899 lbid locked is 1689015296
 
Mar 21 06:03:34 nvm002314 ddlpackageproc[2510]: 34.567453 |0|0|0| E 23 CAL0009: Error in dropping column from systables.
 
Mar 21 06:30:34 nvm002314 ddlpackageproc[2510]: 34.574847 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 06:43:03 nvm002314 ddlpackageproc[2510]: 03.714655 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 06:46:46 nvm002314 workernode[29567]: 46.933233 |0|0|0| C 30 CAL0000: VBBM::getBlocks(): version buffer overflow. Increase VersionBufferFileSize. Overflow occured in aged blocks. Requested NumBlocks:VbOid:vbFBO:lastFBO = 336:1:106600:106999 lbid locked is 1689039872
 
Mar 21 06:46:46 nvm002314 dmlpackageproc[2563]: 46.981965 |0|0|0| E 21 CAL0002: Delete Failed:  IDB-2008: The version buffer overflowed. Increase VersionBufferFileSize or limit the rows to be processed.
 
Mar 21 07:00:22 nvm002314 ddlpackageproc[2510]: 22.692432 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 12:20:57 nvm002314 ddlpackageproc[2510]: 57.912176 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 21:05:11 nvm002314 workernode[29567]: 11.495991 |0|0|0| C 30 CAL0000: VBBM::getBlocks(): version buffer overflow. Increase VersionBufferFileSize. Overflow occured in aged blocks. Requested NumBlocks:VbOid:vbFBO:lastFBO = 1:1:106600:106699 lbid locked is 1689039872
 
Mar 21 21:05:11 nvm002314 dmlpackageproc[2563]: 11.541648 |0|0|0| E 21 CAL0002: Update Failed:  IDB-2008: The version buffer overflowed. Increase VersionBufferFileSize or limit the rows to be processed.
 
Mar 21 21:16:48 nvm002314 ddlpackageproc[2510]: 48.252063 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 23:20:28 nvm002314 ddlpackageproc[2510]: 28.726261 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 23:25:32 nvm002314 DMLProc[14662]: 32.713136 |0|0|0| C 20 CAL0002:  problem with rollback transaction 1498443and DBRM is setting to readonly and table lock is not released: Error encountered while opening version buffer file oid:dbroot = 0:4 and error message:No such file or directory (WriteEngine returns error 1)
 
Mar 21 23:25:32 nvm002314 DMLProc[14662]: 32.720120 |0|0|0| C 20 CAL0002:  problem with rollback transaction 1500786and DBRM is setting to readonly and table lock is not released: Can't read DBRM isReadWrite [ DBRM is in READ-ONLY mode]
 
Mar 21 23:31:39 nvm002314 ddlpackageproc[14526]: 39.432797 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 23:37:34 nvm002314 ddlpackageproc[14526]: 34.238514 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 21 23:40:38 nvm002314 DMLProc[10355]: 38.175194 |0|0|0| C 20 CAL0002:  problem with rollback transaction 1498443and DBRM is setting to readonly and table lock is not released: Error encountered while opening version buffer file oid:dbroot = 0:3 and error message:No such file or directory (WriteEngine returns error 1)
 
Mar 21 23:40:38 nvm002314 DMLProc[10355]: 38.184515 |0|0|0| C 20 CAL0002:  problem with rollback transaction 1500786and DBRM is setting to readonly and table lock is not released: Can't read DBRM isReadWrite [ DBRM is in READ-ONLY mode]
 
Mar 21 23:53:29 nvm002314 ddlpackageproc[10200]: 29.427302 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 22 00:00:21 nvm002314 ddlpackageproc[10200]: 21.847798 |0|0|0| E 23 CAL0009: (2)Create table failed due to  WE: Error updating calpontsys.systable: a BRM Begin copy error.
 
Mar 22 00:00:28 nvm002314 workernode[4231]: 28.395074 |0|0|0| C 30 CAL0000: VBBM::getBlocks(): version buffer overflow. Increase VersionBufferFileSize. Overflow occured in aged blocks. Requested NumBlocks:VbOid:vbFBO:lastFBO = 64:1:106600:106699 lbid locked is 1689039872
 
Mar 22 00:00:28 nvm002314 dmlpackageproc[10355]: 28.464383 |0|0|0| E 21 CAL0002: Delete Failed:  IDB-2008: The version buffer overflowed. Increase VersionBufferFileSize or limit the rows to be processed.
 
Mar 22 01:34:18 nvm002314 DMLProc[6357]: 18.350789 |0|0|0| C 20 CAL0002:  problem with rollback transaction 1498443and DBRM is setting to readonly and table lock is not released: Error encountered while opening version buffer file oid:dbroot = 0:2 and error message:No such file or directory (WriteEngine returns error 1)
 
Mar 22 01:34:18 nvm002314 DMLProc[6357]: 18.359866 |0|0|0| C 20 CAL0002:  problem with rollback transaction 1500786and DBRM is setting to readonly and table lock is not released: Can't read DBRM isReadWrite [ DBRM is in READ-ONLY mode]

It does look like https://jira.mariadb.org/browse/MCOL-2054, but wanted to make sure it is the different version.
Asked customer to clean up files under /usr/local/mariadb/columnstore/data1/systemFiles/dbrm/DMLLog_, but customer said there were no files with name staring DMLLog_.

After they hit "The version buffer overflowed. Increase VersionBufferFileSize or limit the rows to be processed." error, I recommended to increase it it up to 2GB before it was 1GB. But, it didn't help.

 Comments   
Comment by Roman [ 2020-03-26 ]

The overflow of version buffer had been caused by the commits that didn't rollback.

Generated at Thu Feb 08 02:46:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.