[MDEV-19450] buffer overflow detected and data corruption Created: 2019-05-13  Updated: 2023-05-12  Resolved: 2023-05-12

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.11, 10.3.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Sathish Kumar S Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Ubuntu 16.04.5 LTS


Attachments: Text File mysql_buffer_overflow_stack_trace.txt     File prod.cnf    

 Description   

We are getting Assertion failure every time memory usage of MySQL spikes up to overall system's memory.

Every time Assertion failure reports different file in code location "/home/buildbot/buildbot/build/mariadb-10.3.11/storage", but every time we see same **buffer overflow detected** statement in mysqld log.

We only see "buffer overflow detected" like its a pattern. I've attached stack trace as well my.cnf file.

We are having below server with hardware configuration,
Ubuntu 16.04.5 LTS
Memory: 72G
Cores: 36
Our Scenario: writing a 50k as a batch insert, we insert about more than 100G of data.

      • We have a few suspects, but still, we are unable to get convinced what was the issue.

1. We are writing about 100 inserts with "on duplicate key update" - is that possible to get all indexes back into memory as pages(old/young pages) and it's not clearing
2. I was trying to insert same 50k inserts for about 2m records, I see, there was memory spike about to 2%(based on data size), every time I run it, again and again, it gets added up.

Important: when we see this issue, table data also getting corrupted. that's the huge worries on our side.



 Comments   
Comment by Sathish Kumar S [ 2019-05-13 ]

Please share feedback as well as any suggestions to explore as part of trial and error to debug more.

Comment by Marko Mäkelä [ 2023-04-14 ]

This could be two bugs in one. The root cause of the crash is some corruption in the adaptive hash index:

2019-05-06 07:35:08 0x7f171e37a700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.11/storage/innobase/btr/btr0sea.cc line 1415
InnoDB: Failing assertion: index->id == btr_page_get_index_id(page)

There have been many fixes to that subsystem since the MariaDB 10.3.11 and 10.3.12 releases. Is this reproducible with a newer version? If yes, can you produce fully resolved stack traces of all threads corresponding to the crash? You will likely have to enable core dumps and produce the stack traces from the core dump.

That the built-in stack trace reporter crashes is a secondary issue.

Generated at Thu Feb 08 08:51:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.