[MCOL-1997] Performance degradation after running out of disk space Created: 2018-12-06 Updated: 2021-01-16 Resolved: 2021-01-16 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | N/A |
| Affects Version/s: | 1.1.6 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Critical |
| Reporter: | Jorick Schram | Assignee: | Unassigned |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | performance | ||
| Environment: |
AWS EC2 m5.xlarge, Amazon Linux 2 AMI. |
||
| Description |
|
Our instance ran out of disk space because of unneeded binlogs filling up. After purging those and disabling the binlog, the performance of columnstore reads remained sluggish at best for two large tables (having 27 and 66 million rows). Queries that used to take somewhere between 0.5 and 1.5 seconds easily took between 30 and 90 seconds resulting in lots of timeouts on our applications. Rebooting the server didn't help either. No issues were reported by `SHOW TABLE STATUS` nor `CHECK TABLE`. In the end we created fresh tables and copied data with `INSERT INTO fresh SELECT FROM original` and then a `RENAME original TO old, fresh TO original` and performance was back to normal (fast). Most probably, the original tables were damaged somehow when the server ran out of disk space, but we couldn't see this being reported other than having a performance degradation. |
| Comments |
| Comment by Roman [ 2019-03-15 ] |
|
Greetings, It is hard to say something without looking into the broken system. Do you have any logs by chance? |
| Comment by Jorick Schram [ 2019-03-15 ] |
|
Hi Roman, No, not after all these months we don't have these logs anymore. |