[MDEV-24605] Assertion failure in file storage/innobase/log/log0log.cc line 631 (signal 6) Created: 2021-01-16  Updated: 2022-02-17

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.8
Fix Version/s: 10.5

Type: Bug Priority: Major
Reporter: Alexander Mikhailov Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 2
Labels: None
Environment:

FreeBSD 12.2-RELEASE-p2, mariadb105-server compiled from ports, options INNOBASE and GSSAPI_NONE only.
data and log directory on zfs
zfs list -r -o name,primarycache,recsize,compression zroot/db
NAME PRIMARYCACHE RECSIZE COMPRESS
zroot/db metadata 16K off
zroot/db/logs metadata 128K off



 Description   

2021-01-16 12:16:31 0xb3b3e0f00 InnoDB: Assertion failure in file storage/innobase/log/log0log.cc line 631
InnoDB: Failing assertion: lsn >= log_sys.get_flushed_lsn()
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
210116 12:16:31 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.5.8-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=53
max_threads=102
thread_count=55
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 355401 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xb431f78d8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdc509f38 thread_stack 0x49000
0x129f5ec <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc3911e <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd
0x801983b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xb49110fb0): RENAME TABLE `items_prices` TO `items_prices_old`, `items_prices_new` TO `items_prices`

Connection ID (thread ID): 1974000
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,i

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.



 Comments   
Comment by Alexander Mikhailov [ 2021-01-16 ]

I'm using a following procedure every 10-15 mins to recalculate prices based on suppliers data (changes every 5-10 mins)

  • delete new table if existst (DELETE TABLE new IF EXISTS)
  • create new table from old (CREATE TABLE new LIKE current)
  • calculate and inserting into new table
  • rename tables in one statement (RENAME TABLE current TO old, new TO current)

Sometimes mariadb GOT signal 6, started from 10.5.6 version

Comment by Alexander Mikhailov [ 2021-01-16 ]

mariadb startup options. As I can see in storage/innobase/log/log0log.cc around line 631 it could be connected to disabled double write (on zfs it only lowers performance)

--user=mysql \
--pid-file=/var/run/mysql/mysql.pid \
--datadir=/db \
--innodb-log-group-home-dir=/db/logs \
--skip-networking \
--socket=/var/run/mysql/mysql.sock \
--skip-name-resolve \
--max-connections=100 \
--max-user-connections=25 \
--tmpdir=/tmpfs \
--disable-log-bin \
--skip-innodb-doublewrite \
--innodb-flush-method=O_DSYNC \
--innodb-buffer-pool-size=12884901888 \
--innodb-log-file-size=3221225472 \
--innodb-io-capacity=10000 \
--innodb-io-capacity-max=20000 \
--innodb-flush-neighbors=0 \
--innodb-doublewrite=0 \
--innodb-flush-log-at-trx-commit=2 \
--innodb-sort-buffer-size=2097152 \
--metadata-locks-hash-instances=256 \
--join-buffer-size=67108864 \
--sort-buffer-size=2097152 \
--tmp-table-size=4194304 \
--max-heap-table-size=4194304 \
--table-definition-cache=2000 --table-open-cache=64000 \
--max-allowed-packet=134217728 \
--query-cache-type=0 --query-cache-size=0 \
--wait-timeout=120 --interactive-timeout=600 \
--transaction-isolation=READ-COMMITTED \
--sql-mode=NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,TIME_ROUND_FRACTIONAL \
--log-error=/var/log/mysql/error.log \
--collation-server=utf8_general_ci --character-set-server=utf8 \
--default-time-zone='+03:00'

Comment by Alexander Mikhailov [ 2021-07-04 ]

Still getting this issue on versions 10.5.7 - 10.5.11

Latest unaffected version is 10.5.6

Comment by Alexander Mikhailov [ 2021-09-22 ]

Still getting same error on 10.5.12 with MDEV-26537 patch

Comment by RG [ 2021-12-14 ]

Same error in mariadb-server-10.5 on Debian 11 (10.5.12-0+deb11u1)

2021-12-14 18:44:43 0x7f4389ffb700  InnoDB: Assertion failure in file ./storage/innobase/log/log0log.cc line 633
InnoDB: Failing assertion: lsn >= log_sys.get_flushed_lsn()
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
 
211214 18:44:43 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.12-MariaDB-0+deb11u1
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=658
max_threads=15936
thread_count=660
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35098964 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x30000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x560405da0a0e]
Printing to addr2line failed
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5604058aba45]
 
2021-12-14 18:45:08 27620710 [Note] InnoDB: Number of pools: 2
 
2021-12-14 18:48:52 0 [Warning] InnoDB: A long semaphore wait:
--Thread 139937284155136 has waited at btr0cur.cc line 6292 for 244.00 seconds the semaphore:
X-lock on RW-latch at 0x7f53f7969178 created in file buf0buf.cc line 1227
a writer (thread id 0) has reserved it in mode  SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time write locked in file buf0flu.cc line 837

Comment by RG [ 2021-12-16 ]

Same error in 10.5.13 from Mariadb repo

Comment by Alexander Mikhailov [ 2022-02-17 ]

It seems this bug could be the same as recently fixed MDEV-27754

Generated at Thu Feb 08 09:31:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.