[MXS-3778] MaxScale crashes when using Cache filter Created: 2021-09-21 Updated: 2022-11-19 Resolved: 2022-11-19 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | cache |
| Affects Version/s: | 6.1.1 |
| Fix Version/s: | 6.2.0 |
| Type: | Bug | Priority: | Major |
| Reporter: | Philip Hoyer | Assignee: | Johan Wikman |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Debian 10, MaxScale 6.1.1, MariaDB 10.3.29 |
||
| Sprint: | MXS-SPRINT-142, MXS-SPRINT-143 |
| Description |
|
When using the Cache filter it seems that MaxScale crashes on certain events, especially when changes in the database are made (INSERT, UPDATE, DELETE). For example our web application responds with a 500 error, when adding a forum post. Might be related to
|
| Comments |
| Comment by markus makela [ 2021-09-21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
It's possible that this is indeed | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by markus makela [ 2021-09-23 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Philip Hoyer have you tested if this happens with 2.5.15? | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Philip Hoyer [ 2021-09-24 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes, it happened with 2.5.15 first, then we updated MaxScale to 6.1.1 and the result were the same. I have noticed in the query log, that the crash usually happens (but not always!) when a DELETE statement is execute, which should invalidate the cache. In the following example with version 2.5.15 the first DELETE statement for table usr_data_multi successfully invalidates the cache, but the second DELETE statement for table usr_pref fails:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2021-09-28 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Philip Hoyer 6.1.2 that contains a fix for | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Philip Hoyer [ 2021-09-29 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes, with 6.1.2 it is still crashing unfortunately. But I have noticed a new "debug assert" in the log:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2021-09-29 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks. The debug assert is caused by the 6.1.2 packages being by mistake debug and not release packages. That we are addressing at this very moment. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2021-10-25 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Philip Hoyer Could you paste your cache configuration? Remove any sensitive data. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Philip Hoyer [ 2021-10-25 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2021-10-25 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Philip Hoyer No need for the config, I have found the bug. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2021-10-26 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Philip Hoyer Thanks for the config, although it turned out that I found the bug before I actually noticed that you had pasted it. Apparently you have a statement, conceptually similar to
When the result is cached, a note is made that this result depends upon usr_data_multi and usr_pref. Internally, there is a mapping from each table (for which at least one result has been cached) to all results depending on that table. When the statement DELETE FROM usr_data_multi ... is made, all results that depend upon usr_data_multi are evicted from the cache. However, the bug was that the note made for usr_pref was not removed, which caused freed memory to be accessed (with a subsequent crash) when the statement DELETE FROM usr_pref ... was made. Anyway, the bug has now been fixed and will be in the next 6.1 release. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2021-10-26 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
If a statement, such as
that depends upon two tables, is cached, it will be invalidated if either of those tables is modified, e.g. DELETE FROM tbl2. However, there would then be a crash if there is a subsequent invalidating statement addressing the other table, e.g. DELETE FROM tbl1. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brock Wilson (Inactive) [ 2022-11-18 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This appears to still happen in MaxScale 22.08.02
MaxScale config
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brock Wilson (Inactive) [ 2022-11-18 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
customer is reporting issue in 22.08.2, RHEL 7.9 Backend is MySQL 5.7. not sure if that matters, but making a note of it. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by markus makela [ 2022-11-19 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bwilson please include the whole stacktrace. We can't know if it's the same issue unless we have the whole stacktrace or a way to reproduce it. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by markus makela [ 2022-11-19 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
bwilson please open a new bug report for this. If it's a regression, we'll close it as a duplicate but right now it seems to be a different bug. |