[MDEV-23629] Random MariaDB Buffer Overflows Created: 2020-08-30  Updated: 2020-08-31

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.1.45
Fix Version/s: 10.1

Type: Bug Priority: Major
Reporter: Lewis Cawte Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Ubuntu Server 16.04.7

Attachments: HTML File crash-mysql-30Aug     HTML File crash-mysql-31Aug     Text File mysql.log    

 Description   

We've had a couple of these in the past, can't seem to find any pattern or cause to them so I'm now seeking help. Attached is the syslog entries (cut after MariaDB goes into recovery) from today's crash - hoping someone can help dial in what the issue is. It's a reasonable sized server with about 10k databases totalling about 450GB in files, so the restarts after these take a few hours.

 Comments   
Comment by Elena Stepanova [ 2020-08-31 ]

We have a few of similar issues - MDEV-23127, MDEV-23126, MDEV-23256.
Do you do FLUSH QUERY CACHE and/or change query cache type at runtime? If you do, as a workaround – try not to.
If you never do any of these and are getting the crashes anyway, maybe it could be an acceptable workaround to disable the query cache at all.

On a separate but possibly (although unlikely) related note – what is it exactly that takes several hours upon restart? Could you please provide the full error log of the startup processes – from the moment the server begins startup and up to when it is "ready to accept connections"?

Comment by Lewis Cawte [ 2020-08-31 ]

FLUSH QUERY CACHE isn't run manually, and I'm fairly sure its not run by any code (although, at the time of writing, the grep has been running through our entire codebase to double check and has yet to finish)

Aug 30 16:18:42 bellow mysqld[2446]: 2020-08-30 16:18:42 139958808316160 [Note] InnoDB: Processed 911306 .ibd/.isl files
 
Aug 30 16:18:50 bellow mysqld[2446]: 2020-08-30 16:18:50 139958808316160 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
 
Aug 30 16:18:50 bellow mysqld[2446]: 2020-08-30 16:18:50 139958808316160 [Note] InnoDB: Read redo log up to LSN=2167961924096
 
Aug 30 16:18:51 bellow mysqld[2446]: 2020-08-30 16:18:51 139958808316160 [Note] InnoDB: Starting final batch to recover 1505 pages from redo log

It gets a bit messy after that. I'll pull the rest of the logs for you later, I just need to extract them from the mess that is syslog.

Comment by Lewis Cawte [ 2020-08-31 ]

crash-mysql-30Aug crash-mysql-31Aug
We had another crash today, today's log is a bit cleaner.

Comment by Elena Stepanova [ 2020-08-31 ]

This one looks different, more InnoDB-ish. And the 2-hour crash recovery also happens in InnoDB.

Generated at Thu Feb 08 09:23:50 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.