Details
-
New Feature
-
Status: Closed (View Workflow)
-
Minor
-
Resolution: Fixed
-
6.2.1
-
None
-
None
-
2021-17
Description
System cat. files are not compressed for some reason, that means, those files do not have a special header with `lbid`, `colwidth` and so on. We restore extents for this files from binary blob (initial state) and HWM is not calculating for them because `ChunkManager` does not have an API to read not compressed files block by block.
For big data bases - system files will grow as well and HWM will be greater than initial state, this should be supported.
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Sprint | 2021-17 [ 614 ] | |
| Affects Version/s | 6.2.1 [ 26035 ] |
| Description |
System cat. files are not compressed for some reason, that means, those files do not have a special header with `lbid`, `colwidth` and so on. We restore extents for this files from binary blob (initial state) and HWM is not calculating for them.
For big data bases system files will grow as well and HWM will greater than initial state, this should be supported. |
System cat. files are not compressed for some reason, that means, those files do not have a special header with `lbid`, `colwidth` and so on. We restore extents for this files from binary blob (initial state) and HWM is not calculating for them because `ChunkManager` does not have an API to read not compressed files block by block.
For big data bases system files will grow as well and HWM will greater than initial state, this should be supported. |
| Description |
System cat. files are not compressed for some reason, that means, those files do not have a special header with `lbid`, `colwidth` and so on. We restore extents for this files from binary blob (initial state) and HWM is not calculating for them because `ChunkManager` does not have an API to read not compressed files block by block.
For big data bases system files will grow as well and HWM will greater than initial state, this should be supported. |
System cat. files are not compressed for some reason, that means, those files do not have a special header with `lbid`, `colwidth` and so on. We restore extents for this files from binary blob (initial state) and HWM is not calculating for them because `ChunkManager` does not have an API to read not compressed files block by block.
For big data bases - system files will grow as well and HWM will be greater than initial state, this should be supported. |
| Fix Version/s | 6.4.1 [ 26046 ] |
| Comment | [ A comment with security level 'Developers' was removed. ] |
| Assignee | Denis Khalikov [ JIRAUSER48434 ] | Roman [ drrtuy ] |
| Status | In Progress [ 3 ] | In Review [ 10002 ] |
| Assignee | Roman [ drrtuy ] | Leonid Fedorov [ JIRAUSER48443 ] |
| Assignee | Leonid Fedorov [ JIRAUSER48443 ] | Roman [ drrtuy ] |
| Status | In Review [ 10002 ] | In Testing [ 10301 ] |
| Assignee | Roman [ drrtuy ] | Daniel Lee [ dleeyh ] |
| Rank | Ranked lower |
| Assignee | Daniel Lee [ dleeyh ] | Denis Khalikov [ JIRAUSER48434 ] |
| Status | In Testing [ 10301 ] | Stalled [ 10000 ] |
Build tested (#4749)
Tested new build with 10gb, 20gb, and 50gb datasets on both 1PM and 3PM configurations.
Passed.
| Component/s | advanced tools [ 13515 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Stalled [ 10000 ] | Closed [ 6 ] |
| Rank | Ranked lower |
| Rank | Ranked lower |
| Rank | Ranked lower |
| Component/s | advanced tools [ 13515 ] |
Build tested (#4661)
Executed tests using Docker containers.
Test scenario
create 1gb dbt databaseload databaseselect count(*) from lineitemstop ColumnStorerm BRM_saves_emmcsRebuildEM -vstart ColumnStoreselect count(*) from lineitemcpimport 1gb lineitemselect count(*) from lineitemDataset size: 1 gb DBT3
1PM local storage - PASSED
1PM S3 storage - PASSED
3PM local storage - PASSED
3PM S3 storage - PASSED
Dataset size: 10 gb DBT3
1PM local storage - FAILED *
1PM S3 storage - FAILED *
3PM local storage - PASSED
3PM S3 storage - PASSED
I am not sure if the issue is in the actual data files, or in this tool.
...Setting a HWM for [OID: 3060, partition: 0, segment: 0, col width: 8, lbid:2061312, hwm: 0, isDict: 1]Extent is created, allocated size 8192 actual LBID 2069504For [OID: 3060, partition: 0, segment: 0, col width: 8, lbid:2069504, hwm: 124032, isDict: 1]Setting a HWM for [OID: 3060, partition: 0, segment: 0, col width: 8, lbid:2069504, hwm: 124032, isDict: 1]Cannot set local HWM: ExtentMap::setLocalHWM(): new HWM is past the end of the file for OID 3060; partition 0; segment 0; HWM 124032Completed.The two failed cases also failed in VMs, not just in Docker containers.