[MDEV-25316] Memory corruption assertion failure for INSERT ... SELECT involving two Spider tables Created: 2021-04-01  Updated: 2021-06-03  Resolved: 2021-05-27

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.4.17
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Valerii Kravchuk Assignee: Valerii Kravchuk
Resolution: Cannot Reproduce Votes: 0
Labels: need_feedback
Environment:

Docker image from https://hub.docker.com/_/mariadb



 Description   

Statement like this:

INSERT INTO spider_table (<columns list> )
  SELECT <constant>, Id, UNIX_TIMESTAMP(), <columns list>
  FROM other_spider_table WHERE ...;

trying to insert some 20+ rows ends up with the malloc-related assertion failure:

...
mariadb-archive-0 mysqld corrupted size vs. prev_size
mariadb-archive-0 mysqld 210226 14:25:16 [ERROR] mysqld got signal 6 ;
mariadb-archive-0 mysqld This could be because you hit a bug. It is also possible that this binary
mariadb-archive-0 mysqld or one of the libraries it was linked against is corrupt, improperly built,
mariadb-archive-0 mysqld or misconfigured. This error can also be caused by malfunctioning hardware.
mariadb-archive-0 mysqld
mariadb-archive-0 mysqld To report this bug, see https://mariadb.com/kb/en/reporting-bugs
mariadb-archive-0 mysqld
mariadb-archive-0 mysqld We will try our best to scrape up some info that will hopefully help
mariadb-archive-0 mysqld diagnose the problem, but since we have already crashed,
mariadb-archive-0 mysqld something is definitely wrong and this may fail.
mariadb-archive-0 mysqld
mariadb-archive-0 mysqld Server version: 10.4.17-MariaDB-1:10.4.17+maria~focal-log
mariadb-archive-0 mysqld key_buffer_size=134217728
mariadb-archive-0 mysqld read_buffer_size=2097152
mariadb-archive-0 mysqld max_used_connections=24
mariadb-archive-0 mysqld max_threads=3002
mariadb-archive-0 mysqld thread_count=51
mariadb-archive-0 mysqld It is possible that mysqld could use up to
mariadb-archive-0 mysqld key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18649564 K  bytes of memory
mariadb-archive-0 mysqld Hope that's ok; if not, decrease some variables in the equation.
mariadb-archive-0 mysqld
mariadb-archive-0 mysqld Thread pointer: 0x7fc310000c48
mariadb-archive-0 mysqld Attempting backtrace. You can use the following information to find out
mariadb-archive-0 mysqld where mysqld died. If you see no messages after this, something went
mariadb-archive-0 mysqld terribly wrong...
mariadb-archive-0 mysqld stack_bottom = 0x7fc43429add8 thread_stack 0x49000
mariadb-archive-0 mysqld 2021-02-26 14:25:36 0 [Note] InnoDB: Buffer pool(s) load completed at 210226 14:25:36
...

There is no assertion failure if we LIMIT number of rows to, say, 5, or if we use intermediate non-Spider table to store SELECT result.

I can not reproduce this locally outside of Docker image. Getting the core dump in a hope to see more details about the place in the code where assertion failure happens is problematic, as usual steps seem not to work for this Docker image (suggestions are appreciated).



 Comments   
Comment by Daniel Black [ 2021-04-12 ]

valerii I assume this was you where asking about in getting a core dump within a container. It it repeatable outside a container?

Do you know where 'corrupted size vs. prev_size' comes from? I can't find it in the server codebase.

Comment by Valerii Kravchuk [ 2021-04-12 ]

Customer uses that Docker image listed in the Environment, crash happened only there (sometimes, depending on numerous conditions), and I was not able to reproduce it with my tests locally.

The assertion failure is in glibc I assume, while freeing memory, like in this discussion:

https://stackoverflow.com/questions/49628615/understanding-corrupted-size-vs-prev-size-glibc-error

At least this is what Google searches gives me.

Comment by Daniel Black [ 2021-04-13 ]

note: the environment variable UBSAN_OPTIONS=print_stacktrace=1 to get strack traces directly from instrumentation.

Comment by Roel Van de Paar [ 2021-04-13 ]

danblack Ah yes, forgot to mention that. valerii I recommend setting all these:

export ASAN_OPTIONS=quarantine_size_mb=512:atexit=1:detect_invalid_pointer_pairs=3:dump_instruction_bytes=1:abort_on_error=1
export UBSAN_OPTIONS=print_stacktrace=1

Generated at Thu Feb 08 09:36:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.