[MDEV-23343] aria_log_control Gets Corrupted Preventing MariaDB from Starting Created: 2020-07-30 Updated: 2021-06-19 |
|
| Status: | Open |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - Aria |
| Affects Version/s: | 10.4.13 |
| Fix Version/s: | 10.4 |
| Type: | Bug | Priority: | Major |
| Reporter: | Denis Melnikov | Assignee: | Michael Widenius |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Environment: |
CentOS Linux release 7.6.1810 (Core) |
||
| Description |
|
On start the server writes in its log:
And it hangs in this state indefinitely. A fun fact is that another server with the same data has been upgraded from 10.3.23 upto 10.4.13, too, it shares the same config, and it hangs as well. The config is as follows:
UPD: Workaround is to remove aria_log_control before starting. |
| Comments |
| Comment by Vincent Milum Jr [ 2020-07-31 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I'm experiencing this exact same issue, hanging at what appears to be the exact same spot (based on the mysql.log file), and also running MariaDB 10.4.13 on CentOS EL7. I SCPed the database and config files over to a FreeBSD 12.1 install, and it started right up. Some extra notes: My instance was an upgrade from 10.3.11. When first upgrading, the 10.4 instance worked just fine. After running the mysql_upgrade program and restarting the instance, THAT is when it started to hang. Prior to running mysql_upgrade on 10.4, it would continue to function as normal. Setting skip-grant-tables (a suggestion I found on a random forum) allows the instance to start up... but obviously, this is a highly broken config. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Denis Melnikov [ 2020-07-31 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
@darkain, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vincent Milum Jr [ 2020-07-31 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
That actually did the trick! Removing that file allowed MariaDB to start. And yes, the file seems to get corrupt every single time, too, and needs to be re-deleted. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vincent Milum Jr [ 2020-07-31 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
from https://mariadb.com/kb/en/aria-faq/ "Aria has a log control file (aria_log_control) and log files (aria_log.%). The log files can be automatically purged when not needed or purged on demand (after backup)." I guess I'll just update my startup scripts to purge these files before launching the MariaDB process as a temporary work-around for this bug. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vincent Milum Jr [ 2020-08-01 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
@Denis Were you running MariaDB inside of a container of some kind? Docker or LXC/LXD or something else? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Denis Melnikov [ 2020-08-01 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
No, just a VM in VMWare ESXi. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-08-05 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
So, you remove the aria log control file, start the server, don't do anything there, shut it down properly, it says in the error log that shutdown was complete, and next time you start it it hangs because the newly created log control file is corrupt? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Denis Melnikov [ 2020-08-05 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Correct. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-08-05 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Please paste or attach the error log covering all these steps – successful start (after removing the log), clean shutdown, and the next unsuccessful start. darkain, do you also have TokuDB for a default engine and libjemalloc.so.1 As you understand, there must be something more to it than just a clean server restart, something specific to your (both of you) environment, operation or data, otherwise we would have thousands of such reports. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Denis Melnikov [ 2020-08-05 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The egrul.err pasted below, log_warnings=9.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Vincent Milum Jr [ 2020-08-05 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
In my particular case, these are being launch as LXC containers. They don't use systemd, instead they use a basic init script. We don't use TokuDB, we're a 100% InnoDB house. jemalloc is also not used. Also of note, this first test instance that I'm having this issue on is a very small instance only used for personal development at $DayJob. We're talking less than 10MB of InnoDB data, and the only Aria data is the system tables. As mentioned above, when I SCP the data to another OS, it loads up perfectly. So the data doesn't appear to be corrupt at all. I've built ~10 different CentOS VMs from fresh ISOs to attempt to reproduce the issue on my personal lab machines, with no luck so far. I'm not sure what customizations were done on $DayJob OS installs that might effect this issue, as that pre-dates my time there. In my particular case, the containers each use a dedicated ext4 volume from a thick provisioned file on the host.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Denis Melnikov [ 2020-09-07 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I can upload binary DB files (some 50 GB) if you are interested. |