[MCOL-4451] Problems with INSERT and SELECT operations when the number of rows falls between 100 millions and 1 billion Created: 2020-12-15 Updated: 2021-04-19 Resolved: 2020-12-22 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | installation |
| Affects Version/s: | 5.4.3 |
| Fix Version/s: | 5.4.3 |
| Type: | Bug | Priority: | Major |
| Reporter: | David Hill (Inactive) | Assignee: | Unassigned |
| Resolution: | Not a Bug | Votes: | 0 |
| Labels: | None | ||
| Environment: |
1 maxscale |
||
| Description |
|
Customer reported: Description: During some tests, we found that working with an increasing batch of data (1K, 10K, 100K, 1M, 10M, 100M, 1B rows) the ColumnStore worked as expected till we launched a SQL statement that acts on more than 100M rows. In this case, the operation stopped with the following error: #1815 - Internal error: InetStreamSocket::readToMagic: Remote is closed The SQL statement and the corresponding log are in the attached file. Date/time: 2020-12-04 16:21:06 /usr/bin/ExeMgr(+0x24ad0)[0x55ab2a6d3ad0] |
| Comments |
| Comment by David Hill (Inactive) [ 2020-12-15 ] |
|
I did recommended using cpimport and I got this followup We know that UPDATE and DELETE are not optimized in ColumnStore, but it fails the same way even with a SELECT operation only, and this is not a matter of optimization but probably it is due to an anomaly that is triggered by the number of rows involved in the operation. Since we ordinarily deal with billions of rows, for us it is critical to solve this problem also in a simple SELECT statement. |
| Comment by David Hill (Inactive) [ 2020-12-22 ] |
|
So this one looks like its related to Memory setting on their 3 pm setup Right on the call of memory overusage, I looked at the config files and did see that the Columnstore memory settings are 2 high. These are the 2 settings for Columnstore related to memory. These 2 settings should be a max of 75% as shown in following Document. They are at 150%. So these 2 need to be tweaked down to equal 75% between the 2. <TotalUmMemory>100%</TotalUmMemory> This is the memory size for InnoDB data. So if you are utilizing InnoDB database, then the combination of the 2 above plus this should be over 85%. This will leave space to temporary storage that is used at times. innodb_buffer_pool_size 53687091200 Server memory size is 80gb I will close and can be reopened if something different comes up. |