[MDEV-21440] Aria log crash after shutdow+restart on ma_loghandler.c:7162: translog_read_record_header_from_buffer: Assertion `translog_is_LSN_chunk(page[page_offset])' failed Created: 2020-01-08  Updated: 2023-10-01

Status: Stalled
Project: MariaDB Server
Component/s: Storage Engine - Aria
Affects Version/s: 10.5
Fix Version/s: 10.5

Type: Bug Priority: Major
Reporter: Jan Lindström (Inactive) Assignee: Oleksandr Byelkin
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File crash.tgz     File rpl_type_inet6.tar.gz     File type_inet.rpl_type_inet6_mysqld_1_err     File type_inet.rpl_type_inet6_mysqld_2_err    

 Description   

I repeated this using 10.5 ES with /mtr --suite=galera,wsrep,sys_vars --force --parallel=auto --big-test --max-test-fail=0 but it does not repeat on every run. Furthermore, note that failing test case is not always the same.

galera.galera_restart_on_unknown_option 'innodb' w1 [ fail ]
 
        Test ended at 2020-01-08 12:46:26
 
 
 
CURRENT_TEST: galera.galera_restart_on_unknown_option
 
 
 
 
 
Server [mysqld.2 - pid: 1001, winpid: 1001, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
2020-01-08 12:45:51 0 [Note] /home/jan/mysql/10.5e-features/sql/mysqld (initiated by: root[root] @ localhost [127.0.0.1]): Normal shutdown
 
2020-01-08 12:45:51 0 [Note] WSREP: Shutdown replication
 
2020-01-08 12:45:51 0 [Note] WSREP: Server status change synced -> disconnecting
 
2020-01-08 12:45:51 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-01-08 12:45:51 0 [Note] WSREP: Closing send monitor...
 
2020-01-08 12:45:51 0 [Note] WSREP: Closed send monitor.
 
2020-01-08 12:45:51 0 [Note] WSREP: gcomm: terminating thread
 
2020-01-08 12:45:51 0 [Note] WSREP: gcomm: joining thread
 
2020-01-08 12:45:51 0 [Note] WSREP: gcomm: closing backend
 
2020-01-08 12:45:52 0 [Note] WSREP: view(view_id(NON_PRIM,b191c033,6) memb {
 
	eb58aeb4,0
 
} joined {
 
} left {
 
} partitioned {
 
	b191c033,0
 
})
 
2020-01-08 12:45:52 0 [Note] WSREP: PC protocol downgrade 1 -> 0
 
2020-01-08 12:45:52 0 [Note] WSREP: view((empty))
 
2020-01-08 12:45:52 0 [Note] WSREP: gcomm: closed
 
2020-01-08 12:45:52 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
2020-01-08 12:45:52 0 [Note] WSREP: Flow-control interval: [16, 16]
 
2020-01-08 12:45:52 0 [Note] WSREP: Received NON-PRIMARY.
 
2020-01-08 12:45:52 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 216)
 
2020-01-08 12:45:52 0 [Note] WSREP: New SELF-LEAVE.
 
2020-01-08 12:45:52 0 [Note] WSREP: Flow-control interval: [0, 0]
 
2020-01-08 12:45:52 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
 
2020-01-08 12:45:52 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
 
2020-01-08 12:45:52 15 [Note] WSREP: ####### processing CC -1, local, ordered
 
2020-01-08 12:45:52 0 [Note] WSREP: RECV thread exiting 0: Success
 
2020-01-08 12:45:52 15 [Note] WSREP: ####### My UUID: eb58aeb4-3203-11ea-ba66-eb3107bcd6e3
 
2020-01-08 12:45:52 15 [Note] WSREP: ####### ST not required
 
2020-01-08 12:45:52 0 [Note] WSREP: recv_thread() joined.
 
2020-01-08 12:45:52 0 [Note] WSREP: Closing replication queue.
 
2020-01-08 12:45:52 0 [Note] WSREP: Closing slave action queue.
 
2020-01-08 12:45:52 15 [Note] WSREP: ================================================
 
View:
 
  id: b1935aa1-3203-11ea-b8be-9bd032caaa2e:-1
 
  status: non-primary
 
  protocol_version: 4
 
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
  final: no
 
  own_index: 0
 
  members(1):
 
	0: eb58aeb4-3203-11ea-ba66-eb3107bcd6e3, jan-HP-ZBook-15u-G5
 
=================================================
 
2020-01-08 12:45:52 15 [Note] WSREP: Non-primary view
 
2020-01-08 12:45:52 15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-01-08 12:45:52 15 [Note] WSREP: ####### processing CC -1, local, ordered
 
2020-01-08 12:45:52 15 [Note] WSREP: ####### My UUID: eb58aeb4-3203-11ea-ba66-eb3107bcd6e3
 
2020-01-08 12:45:52 15 [Note] WSREP: ####### ST not required
 
2020-01-08 12:45:52 15 [Note] WSREP: ================================================
 
View:
 
  id: b1935aa1-3203-11ea-b8be-9bd032caaa2e:-1
 
  status: non-primary
 
  protocol_version: 4
 
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 
  final: yes
 
  own_index: -1
 
  members(0):
 
=================================================
 
2020-01-08 12:45:52 15 [Note] WSREP: Non-primary view
 
2020-01-08 12:45:52 15 [Note] WSREP: Server status change disconnecting -> disconnected
 
2020-01-08 12:45:52 15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-01-08 12:45:52 15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2020-01-08 12:45:52 15 [Note] WSREP: Applier thread exiting ret: 0 thd: 15
 
2020-01-08 12:45:52 15 [Warning] Aborted connection 15 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
 
2020-01-08 12:45:52 1 [Note] WSREP: rollbacker thread exiting 1
 
2020-01-08 12:45:52 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
 
2020-01-08 12:45:52 0 [Note] Event Scheduler: Purging the queue. 0 events
 
2020-01-08 12:45:52 0 [Note] WSREP: dtor state: CLOSED
 
2020-01-08 12:45:52 0 [Note] WSREP: MemPool(TrxHandleSlave): hit ratio: 0, misses: 73, in use: 73, in pool: 0
 
2020-01-08 12:45:52 0 [Note] WSREP: mon: entered 35 oooe fraction 0 oool fraction 0
 
2020-01-08 12:45:52 0 [Note] WSREP: mon: entered 35 oooe fraction 0.285714 oool fraction 0.0571429
 
2020-01-08 12:45:52 0 [Note] WSREP: mon: entered 40 oooe fraction 0 oool fraction 0.025
 
2020-01-08 12:45:52 0 [Note] WSREP: cert index usage at exit 38
 
2020-01-08 12:45:52 0 [Note] WSREP: cert trx map usage at exit 73
 
2020-01-08 12:45:52 0 [Note] WSREP: deps set usage at exit 0
 
2020-01-08 12:45:52 0 [Note] WSREP: avg deps dist 1.46575
 
2020-01-08 12:45:52 0 [Note] WSREP: avg cert interval 83.4658
 
2020-01-08 12:45:52 0 [Note] WSREP: cert index size 38
 
2020-01-08 12:45:52 0 [Note] WSREP: Service thread queue flushed.
 
2020-01-08 12:45:52 0 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
 
2020-01-08 12:45:52 0 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0.833333, misses: 1, in use: 0, in pool: 1
 
2020-01-08 12:45:52 0 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: -1)
 
2020-01-08 12:45:52 0 [Note] WSREP: Flushing memory map to disk...
 
2020-01-08 12:45:52 0 [Note] InnoDB: FTS optimize thread exiting.
 
2020-01-08 12:45:52 0 [Note] InnoDB: Starting shutdown...
 
2020-01-08 12:45:52 0 [Note] InnoDB: Dumping buffer pool(s) to /home/jan/mysql/10.5e-features/mysql-test/var/1/mysqld.2/data/ib_buffer_pool
 
2020-01-08 12:45:52 0 [Note] InnoDB: Instance 0, restricted to 123 pages due to innodb_buf_pool_dump_pct=25
 
2020-01-08 12:45:52 0 [Note] InnoDB: Buffer pool(s) dump completed at 200108 12:45:52
 
2020-01-08 12:45:53 0 [Note] InnoDB: Shutdown completed; log sequence number 3541071; transaction id 641
 
2020-01-08 12:45:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 
2020-01-08 12:45:53 0 [Note] Debug sync points hit:                   7936
 
2020-01-08 12:45:53 0 [Note] Debug sync points executed:              0
 
2020-01-08 12:45:53 0 [Note] Debug sync points max active per thread: 0
 
2020-01-08 12:45:53 0 [Note] /home/jan/mysql/10.5e-features/sql/mysqld: Shutdown complete
 
 
 
2020-01-08 12:46:25 0 [Warning] WSREP: option --wsrep-causal-reads is deprecated
 
2020-01-08 12:46:25 0 [Note] /home/jan/mysql/10.5e-features/sql/mysqld (mysqld 10.5.0-1-MariaDB-debug-log) starting as process 1002 ...
 
2020-01-08 12:46:25 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4190)
 
2020-01-08 12:46:25 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
 
2020-01-08 12:46:25 0 [Note] Plugin 'partition' is disabled.
 
mysqld: /home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c:7162: translog_read_record_header_from_buffer: Assertion `translog_is_LSN_chunk(page[page_offset])' failed.
 
200108 12:46:25 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.5.0-1-MariaDB-debug-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=153
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63649 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
/home/jan/mysql/10.5e-features/sql/mysqld(my_print_stacktrace+0x44)[0x5653bdfcb580]
 
/home/jan/mysql/10.5e-features/sql/mysqld(handle_fatal_signal+0x3b9)[0x5653bd6bdba3]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x15540)[0x7f362f6ec540]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f362f1cf3eb]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f362f1ae899]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x25769)[0x7f362f1ae769]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x37006)[0x7f362f1c0006]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0xe6723a)[0x5653bd96523a]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0xe67669)[0x5653bd965669]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0xe863c2)[0x5653bd9843c2]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0xe7f274)[0x5653bd97d274]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0xe7ee68)[0x5653bd97ce68]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0xe4b1bd)[0x5653bd9491bd]
 
/home/jan/mysql/10.5e-features/sql/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x100)[0x5653bd6c1539]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0x88f154)[0x5653bd38d154]
 
/home/jan/mysql/10.5e-features/sql/mysqld(_Z11plugin_initPiPPci+0xaea)[0x5653bd38df64]
 
/home/jan/mysql/10.5e-features/sql/mysqld(+0x72cff6)[0x5653bd22aff6]
 
/home/jan/mysql/10.5e-features/sql/mysqld(_Z11mysqld_mainiPPc+0x73c)[0x5653bd22c2b3]
 
/home/jan/mysql/10.5e-features/sql/mysqld(main+0x24)[0x5653bd22094d]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f362f1b01e3]
 
/home/jan/mysql/10.5e-features/sql/mysqld(_start+0x2e)[0x5653bd22086e]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /home/jan/mysql/10.5e-features/mysql-test/var/1/mysqld.2/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             127634               127634               processes 
Max open files            1024                 1024                 files     
Max locked memory         67108864             67108864             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       127634               127634               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/share/apport/apport %p %s %c %d %P
 
 
 
----------SERVER LOG END-------------
 
mysqltest failed but provided no output
 
The result from queries just before the failure was:
 
< snip >
 
connection node_2;
 
CREATE TABLE t1 (f1 INTEGER PRIMARY KEY, f2 CHAR(1));
 
INSERT INTO t1 VALUES (1, 'a'), (2, 'a'), (3, 'a');
 
SELECT * FROM t1;
 
f1	f2
 
1	a
 
2	a
 
3	a
 
Shutting down server ...
 
connection node_1;
 
UPDATE t1 SET f2 = 'b' WHERE f1 > 1;
 
UPDATE t1 SET f2 = 'c' WHERE f1 > 2;
 
SELECT * FROM t1;
 
f1	f2
 
1	a
 
2	b
 
3	c
 
connection node_2;
 
Starting server ...
 
Starting server ...
 
 
 
 
 
 
 
 - found 'core' (0/5)
 
 
 
Trying 'dbx' to get a backtrace
 
 
 
Trying 'gdb' to get a backtrace from coredump /home/jan/mysql/10.5e-features/mysql-test/var/1/log/galera.galera_restart_on_unknown_option-innodb/mysqld.2/data/core
 
Core generated by '/home/jan/mysql/10.5e-features/sql/mysqld'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
[New LWP 1002]
 
[New LWP 1003]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 
Core was generated by `/home/jan/mysql/10.5e-features/sql/mysqld --defaults-group-suffix=.2 --defaults'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
56	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
 
[Current thread is 1 (Thread 0x7f362f185b80 (LWP 1002))]
 
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x00005653bdfcb676 in my_write_core (sig=6) at /home/jan/mysql/10.5e-features/mysys/stacktrace.c:518
 
#2  0x00005653bd6bdf2d in handle_fatal_signal (sig=6) at /home/jan/mysql/10.5e-features/sql/signal_handler.cc:343
 
#3  <signal handler called>
 
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#5  0x00007f362f1ae899 in __GI_abort () at abort.c:79
 
#6  0x00007f362f1ae769 in __assert_fail_base (fmt=0x7f362f342fb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5653be2d44b8 "translog_is_LSN_chunk(page[page_offset])", file=0x5653be2cf718 "/home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c", line=7162, function=<optimized out>) at assert.c:92
 
#7  0x00007f362f1c0006 in __GI___assert_fail (assertion=0x5653be2d44b8 "translog_is_LSN_chunk(page[page_offset])", file=0x5653be2cf718 "/home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c", line=7162, function=0x5653be2d67e0 <__PRETTY_FUNCTION__.19805> "translog_read_record_header_from_buffer") at assert.c:101
 
#8  0x00005653bd96523a in translog_read_record_header_from_buffer (page=0x7ffd2e4466e0 "\005", page_offset=2747, buff=0x7ffd2e448bb0, scanner=0x0) at /home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c:7162
 
#9  0x00005653bd965669 in translog_read_record_header (lsn=4295011003, buff=0x7ffd2e448bb0) at /home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c:7229
 
#10 0x00005653bd9843c2 in parse_checkpoint_record (lsn=4295011003) at /home/jan/mysql/10.5e-features/storage/maria/ma_recovery.c:3256
 
#11 0x00005653bd97d274 in maria_apply_log (from_lsn=0, end_lsn=0, apply=MARIA_LOG_APPLY, trace_file=0x0, should_run_undo_phase=1 '\001', skip_DDLs_arg=1 '\001', take_checkpoints=1 '\001', warnings_count=0x7ffd2e4490b0) at /home/jan/mysql/10.5e-features/storage/maria/ma_recovery.c:339
 
#12 0x00005653bd97ce68 in maria_recovery_from_log () at /home/jan/mysql/10.5e-features/storage/maria/ma_recovery.c:239
 
#13 0x00005653bd9491bd in ha_maria_init (p=0x5653c01a0490) at /home/jan/mysql/10.5e-features/storage/maria/ha_maria.cc:3665
 
#14 0x00005653bd6c1539 in ha_initialize_handlerton (plugin=0x5653c0190150) at /home/jan/mysql/10.5e-features/sql/handler.cc:550
 
#15 0x00005653bd38d154 in plugin_initialize (tmp_root=0x7ffd2e449550, plugin=0x5653c0190150, argc=0x5653beb6eef8 <remaining_argc>, argv=0x5653c0047c40, options_only=false) at /home/jan/mysql/10.5e-features/sql/sql_plugin.cc:1452
 
#16 0x00005653bd38df64 in plugin_init (argc=0x5653beb6eef8 <remaining_argc>, argv=0x5653c0047c40, flags=0) at /home/jan/mysql/10.5e-features/sql/sql_plugin.cc:1734
 
#17 0x00005653bd22aff6 in init_server_components () at /home/jan/mysql/10.5e-features/sql/mysqld.cc:5040
 
#18 0x00005653bd22c2b3 in mysqld_main (argc=170, argv=0x5653c0047c40) at /home/jan/mysql/10.5e-features/sql/mysqld.cc:5583
 
#19 0x00005653bd22094d in main (argc=23, argv=0x7ffd2e449dc8) at /home/jan/mysql/10.5e-features/sql/main.cc:25
 
 
 
Thread 2 (Thread 0x7f362fae9700 (LWP 1003)):
 
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f362fae8e90, clockid=<optimized out>, expected=0, futex_word=0x5653bf41b5c8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
 
#1  __pthread_cond_wait_common (abstime=0x7f362fae8e90, clockid=<optimized out>, mutex=0x5653bf41b508 <LOCK_timer+40>, cond=0x5653bf41b5a0 <COND_timer>) at pthread_cond_wait.c:520
 
#2  __pthread_cond_timedwait (cond=0x5653bf41b5a0 <COND_timer>, mutex=0x5653bf41b508 <LOCK_timer+40>, abstime=0x7f362fae8e90) at pthread_cond_wait.c:656
 
#3  0x00005653bdfd1279 in safe_cond_timedwait (cond=0x5653bf41b5a0 <COND_timer>, mp=0x5653bf41b4e0 <LOCK_timer>, abstime=0x7f362fae8e90, file=0x5653be482540 "/home/jan/mysql/10.5e-features/include/mysql/psi/mysql_thread.h", line=1204) at /home/jan/mysql/10.5e-features/mysys/thr_mutex.c:546
 
#4  0x00005653bdfd22fd in inline_mysql_cond_timedwait (that=0x5653bf41b5a0 <COND_timer>, mutex=0x5653bf41b4e0 <LOCK_timer>, abstime=0x7f362fae8e90, src_file=0x5653be482580 "/home/jan/mysql/10.5e-features/mysys/thr_timer.c", src_line=321) at /home/jan/mysql/10.5e-features/include/mysql/psi/mysql_thread.h:1204
 
#5  0x00005653bdfd3000 in timer_handler (arg=0x0) at /home/jan/mysql/10.5e-features/mysys/thr_timer.c:321
 
#6  0x00005653bda70a26 in pfs_spawn_thread (arg=0x5653c018d820) at /home/jan/mysql/10.5e-features/storage/perfschema/pfs.cc:1862
 
#7  0x00007f362f6e0669 in start_thread (arg=<optimized out>) at pthread_create.c:479
 
#8  0x00007f362f2ab323 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
Thread 1 (Thread 0x7f362f185b80 (LWP 1002)):
 
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x00005653bdfcb676 in my_write_core (sig=6) at /home/jan/mysql/10.5e-features/mysys/stacktrace.c:518
 
#2  0x00005653bd6bdf2d in handle_fatal_signal (sig=6) at /home/jan/mysql/10.5e-features/sql/signal_handler.cc:343
 
#3  <signal handler called>
 
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#5  0x00007f362f1ae899 in __GI_abort () at abort.c:79
 
#6  0x00007f362f1ae769 in __assert_fail_base (fmt=0x7f362f342fb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5653be2d44b8 "translog_is_LSN_chunk(page[page_offset])", file=0x5653be2cf718 "/home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c", line=7162, function=<optimized out>) at assert.c:92
 
#7  0x00007f362f1c0006 in __GI___assert_fail (assertion=0x5653be2d44b8 "translog_is_LSN_chunk(page[page_offset])", file=0x5653be2cf718 "/home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c", line=7162, function=0x5653be2d67e0 <__PRETTY_FUNCTION__.19805> "translog_read_record_header_from_buffer") at assert.c:101
 
#8  0x00005653bd96523a in translog_read_record_header_from_buffer (page=0x7ffd2e4466e0 "\005", page_offset=2747, buff=0x7ffd2e448bb0, scanner=0x0) at /home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c:7162
 
#9  0x00005653bd965669 in translog_read_record_header (lsn=4295011003, buff=0x7ffd2e448bb0) at /home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c:7229
 
#10 0x00005653bd9843c2 in parse_checkpoint_record (lsn=4295011003) at /home/jan/mysql/10.5e-features/storage/maria/ma_recovery.c:3256
 
#11 0x00005653bd97d274 in maria_apply_log (from_lsn=0, end_lsn=0, apply=MARIA_LOG_APPLY, trace_file=0x0, should_run_undo_phase=1 '\001', skip_DDLs_arg=1 '\001', take_checkpoints=1 '\001', warnings_count=0x7ffd2e4490b0) at /home/jan/mysql/10.5e-features/storage/maria/ma_recovery.c:339
 
#12 0x00005653bd97ce68 in maria_recovery_from_log () at /home/jan/mysql/10.5e-features/storage/maria/ma_recovery.c:239
 
#13 0x00005653bd9491bd in ha_maria_init (p=0x5653c01a0490) at /home/jan/mysql/10.5e-features/storage/maria/ha_maria.cc:3665
 
#14 0x00005653bd6c1539 in ha_initialize_handlerton (plugin=0x5653c0190150) at /home/jan/mysql/10.5e-features/sql/handler.cc:550
 
#15 0x00005653bd38d154 in plugin_initialize (tmp_root=0x7ffd2e449550, plugin=0x5653c0190150, argc=0x5653beb6eef8 <remaining_argc>, argv=0x5653c0047c40, options_only=false) at /home/jan/mysql/10.5e-features/sql/sql_plugin.cc:1452
 
#16 0x00005653bd38df64 in plugin_init (argc=0x5653beb6eef8 <remaining_argc>, argv=0x5653c0047c40, flags=0) at /home/jan/mysql/10.5e-features/sql/sql_plugin.cc:1734
 
#17 0x00005653bd22aff6 in init_server_components () at /home/jan/mysql/10.5e-features/sql/mysqld.cc:5040
 
#18 0x00005653bd22c2b3 in mysqld_main (argc=170, argv=0x5653c0047c40) at /home/jan/mysql/10.5e-features/sql/mysqld.cc:5583
 
#19 0x00005653bd22094d in main (argc=23, argv=0x7ffd2e449dc8) at /home/jan/mysql/10.5e-features/sql/main.cc:25
 
 
 
Compressed file /home/jan/mysql/10.5e-features/mysql-test/var/1/log/galera.galera_restart_on_unknown_option-innodb/mysqld.2/data/core
 
 - saving '/home/jan/mysql/10.5e-features/mysql-test/var/1/log/galera.galera_restart_on_unknown_option-innodb/' to '/home/jan/mysql/10.5e-features/mysql-test/var/log/galera.galera_restart_on_unknown_option-innodb/'
 
***Warnings generated in error logs during shutdown after running tests: galera.galera_restart_on_unknown_option
 
 
 
mysqld: /home/jan/mysql/10.5e-features/storage/maria/ma_loghandler.c:7162: translog_read_record_header_from_buffer: Assertion `translog_is_LSN_chunk(page[page_offset])' failed.
 
Attempting backtrace. You can use the following information to find out



 Comments   
Comment by Oleksandr Byelkin [ 2020-01-09 ]

The log of the slave (mysqld.2) was not clearly written, probably server was killed before it flushed pages, aria_dump_log output:

  Chunk 1 43420:
 
  Filler till the page end
 
    WARNING: non filler character met before page end (page + 0x0abb: 0x1d) (stop interpretation)!!!

Comment by Alice Sherepa [ 2020-02-12 ]

10.5 1a6f708ec594ac0ae2dd3

#2  <signal handler called>
 
#3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#4  0x00007f357f2fe535 in __GI_abort () at abort.c:79
 
#5  0x00007f357f2fe40f in __assert_fail_base (fmt=0x7f357f460ee0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x565012212498 "translog_is_LSN_chunk(page[page_offset])", file=0x56501220d6f0 "/10.5/src/storage/maria/ma_loghandler.c", line=7162, function=<optimized out>) at assert.c:92
 
#6  0x00007f357f30c102 in __GI___assert_fail (assertion=0x565012212498 "translog_is_LSN_chunk(page[page_offset])", file=0x56501220d6f0 "/10.5/src/storage/maria/ma_loghandler.c", line=7162, function=0x5650122147c0 <__PRETTY_FUNCTION__.19931> "translog_read_record_header_from_buffer") at assert.c:101
 
#7  0x00005650117c3642 in translog_read_record_header_from_buffer (page=0x7f357d1f5080 "\002", page_offset=5320, buff=0x7ffdd2e220f0, scanner=0x0) at /10.5/src/storage/maria/ma_loghandler.c:7162
 
#8  0x00005650117c3a55 in translog_read_record_header (lsn=4294989000, buff=0x7ffdd2e220f0) at /10.5/src/storage/maria/ma_loghandler.c:7229
 
#9  0x00005650117e23c7 in parse_checkpoint_record (lsn=4294989000) at /10.5/src/storage/maria/ma_recovery.c:3256
 
#10 0x00005650117db372 in maria_apply_log (from_lsn=0, end_lsn=0, apply=MARIA_LOG_APPLY, trace_file=0x0, should_run_undo_phase=1 '\001', skip_DDLs_arg=1 '\001', take_checkpoints=1 '\001', warnings_count=0x7ffdd2e225f0) at /10.5/src/storage/maria/ma_recovery.c:339
 
#11 0x00005650117daf6a in maria_recovery_from_log () at /10.5/src/storage/maria/ma_recovery.c:239
 
#12 0x00005650117a7a87 in ha_maria_init (p=0x565013c1c7a0) at /10.5/src/storage/maria/ha_maria.cc:3680
 
#13 0x000056501151f85d in ha_initialize_handlerton (plugin=0x565013c0c860) at /10.5/src/sql/handler.cc:549
 
#14 0x00005650111f8907 in plugin_initialize (tmp_root=0x7ffdd2e22a90, plugin=0x565013c0c860, argc=0x565012b076d8 <remaining_argc>, argv=0x565013bcf538, options_only=false) at /10.5/src/sql/sql_plugin.cc:1450
 
#15 0x00005650111f96a5 in plugin_init (argc=0x565012b076d8 <remaining_argc>, argv=0x565013bcf538, flags=0) at /10.5/src/sql/sql_plugin.cc:1731
 
#16 0x000056501109d0a7 in init_server_components () at /10.5/src/sql/mysqld.cc:5016
 
#17 0x000056501109e2fa in mysqld_main (argc=16, argv=0x565013bcf538) at /10.5/src/sql/mysqld.cc:5544
 
#18 0x0000565011092bf5 in main (argc=3, argv=0x7ffdd2e232d8) at /10.5/src/sql/main.cc:25

Comment by Michael Widenius [ 2020-06-08 ]

I was examining the logs in crash.tgz and here is my comments:

The recovery logs in mysqld.1/data are correct

The recovery logs in mysqld.2/data are corrupted:

  • aria_read_log -d shows that the redo logs starts at LSN 1,0x2007 and ends at LSN 1,0xa999
    However "aria_read_log -h . --print-log-control-file" shows:
    Aria file version: 1
    Block size: 8192
    maria_uuid: 24ee3135-3202-11ea-ac6c-00e04ca2f8fe
    last_checkpoint_lsn: (1,0xaabb)
    last_log_number: 1
    trid: 618
    recovery_failuers: 0

The corruptions comes from that last_checkpoint_lsn is bigger then end LSN!
The size of the aria_log.000001 file is 49152 = 0xc000, but it's filled with 0xff starting from 0xaae8
The data at 0xaabb (checkpoint position) doesn't look like a full checkpoint. There is a lot of filler bytes (0xff) just before the checkpoint that shouldn't be there.

My guess is that Galera when synchronizing the slave files copied the aria log files life but without taking into account that the log files are written to at the same time and that the last block can be written many times (one can't copy the log by just a big copy and expect things to work).
The right way is to copy the aria_log_control file first and then the aria_log.00000* files. Doing it this way will ensure that recorded checkpoint exists in the log file and that Aria recovery has everything it needs.
To make it really safe aria check points should be disabled during the copy of the logs, but the above should be enough for most cases.

Comment by Oleksandr Byelkin [ 2020-06-09 ]

OK, I will take it

Comment by Jan Lindström (Inactive) [ 2020-06-09 ]

I find problematic the idea that some files need to be copied during SST on certain order. That is clear sign that something is wrong with checkpoint writing and/or recovery. How certain we can be that reordering the files will really fix this problem? Maybe, more detailed description how checkpoint is written and how copy ordering effects recovery is required here.

Comment by Angelique Sklavounos (Inactive) [ 2023-05-11 ]

This assertion was seen here:
https://buildbot.mariadb.org/#/builders/534/builds/6093/steps/7/logs/stdio

10.11 1916bf2a02c1a2

 
***Warnings generated in error logs during shutdown after running tests: type_inet.rpl_type_inet6
 
mariadbd: /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_loghandler.c:7161: translog_read_record_header_from_buffer: Assertion `translog_is_LSN_chunk(page[page_offset])' failed.

Error logs and var dir attached. rpl_type_inet6.tar.gz type_inet.rpl_type_inet6_mysqld_1_err type_inet.rpl_type_inet6_mysqld_2_err 

type_inet.rpl_type_inet6 'stmt'          w3 [ fail ]
 
        Test ended at 2023-05-11 07:40:19
 
CURRENT_TEST: type_inet.rpl_type_inet6
 
Failed to start mysqld.1
 
mysqltest failed but provided no output
 
 - found 'core' (0/1)
 
Core generated by '/home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mariadbd'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
[New LWP 342440]
 
[New LWP 342502]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 
Core was generated by `/home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mariadbd --defaults-group-s'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  0x00007f775337da7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
 
[Current thread is 1 (Thread 0x7f7752fd3f40 (LWP 342440))]
 
#0  0x00007f775337da7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
 
#1  0x000055cab5eecc3e in my_write_core (sig=6) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/stacktrace.c:424
 
#2  0x000055cab562a02d in handle_fatal_signal (sig=6) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/signal_handler.cc:360
 
#3  <signal handler called>
 
#4  0x00007f775337da7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
 
#5  0x00007f7753329476 in raise () from /lib/x86_64-linux-gnu/libc.so.6
 
#6  0x00007f775330f7f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6
 
#7  0x00007f775330f71b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
#8  0x00007f7753320e96 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
 
#9  0x000055cab586ee0b in translog_read_record_header_from_buffer (page=0x7fffe81dc9d0 "4", page_offset=1429, buff=0x7fffe81dee90, scanner=0x0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_loghandler.c:7161
 
#10 0x000055cab586f1a4 in translog_read_record_header (lsn=4295394709, buff=0x7fffe81dee90) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_loghandler.c:7228
 
#11 0x000055cab58897e7 in parse_checkpoint_record (lsn=4295394709) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_recovery.c:3295
 
#12 0x000055cab588254a in maria_apply_log (from_lsn=0, end_redo_lsn=0, end_undo_lsn=0, apply=MARIA_LOG_APPLY, trace_file=0x0, skip_DDLs_arg=1 '\001', take_checkpoints=1 '\001', warnings_count=0x7fffe81df360) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_recovery.c:342
 
#13 0x000055cab588218f in maria_recovery_from_log () at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_recovery.c:240
 
#14 0x000055cab5858671 in ha_maria_init (p=0x55cab86fec78) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ha_maria.cc:3913
 
#15 0x000055cab562d845 in ha_initialize_handlerton (plugin=0x55cab86ee5e0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/handler.cc:649
 
#16 0x000055cab5295d30 in plugin_initialize (tmp_root=0x7fffe81df780, plugin=0x55cab86ee5e0, argc=0x55cab6cf7a58 <remaining_argc>, argv=0x55cab8551870, options_only=false) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/sql_plugin.cc:1465
 
#17 0x000055cab5296b91 in plugin_init (argc=0x55cab6cf7a58 <remaining_argc>, argv=0x55cab8551870, flags=0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/sql_plugin.cc:1758
 
#18 0x000055cab50e3cfa in init_server_components () at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mysqld.cc:5208
 
#19 0x000055cab50e504c in mysqld_main (argc=141, argv=0x55cab8551870) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mysqld.cc:5826
 
#20 0x000055cab50d9b3d in main (argc=8, argv=0x7fffe81e1c18) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/main.cc:34
 
Thread 2 (Thread 0x7f774edb2640 (LWP 342502)):
 
#0  0x00007f7753378197 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#1  0x00007f775337af1b in pthread_cond_timedwait () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#2  0x000055cab5ef2157 in safe_cond_timedwait (cond=0x55cab75c6460 <COND_timer>, mp=0x55cab75c63a0 <LOCK_timer>, abstime=0x7f774edb1b70, file=0x55cab646d4b8 "/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/my_thr_init.c", line=609) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/thr_mutex.c:548
 
        error = 32631
 
        save_state = {global = {__data = {__lock = 1, __count = 0, __owner = 342502, __nusers = 1, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000\346\071\005\000\001\000\000\000\002", '\000' <repeats 22 times>, __align = 1}, mutex = {__data = {__lock = 1, __count = 0, __owner = 342502, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000\346\071\005\000\001\000\000\000\003", '\000' <repeats 22 times>, __align = 1}, file = 0x55cab646e308 "/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/thr_timer.c", name = 0x55cab646e34b "LOCK_timer", line = 302, count = 0, create_flags = 0, active_flags = 0, id = 10, thread = 140150400820800, locked_mutex = 0x7f77480022f8, used_mutex = 0x7f7748002380, prev = 0x0, next = 0x0}
 
#3  0x000055cab5eeb2e7 in psi_cond_timedwait (that=0x55cab75c6460 <COND_timer>, mutex=0x55cab75c63a0 <LOCK_timer>, abstime=0x7f774edb1b70, file=0x55cab646e308 "/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/thr_timer.c", line=321) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/my_thr_init.c:609
 
        state = {m_flags = 7, m_operation = (PSI_COND_TIMEDWAIT | unknown: 0x7f76), m_cond = 0x55cab8631cc0, m_mutex = 0x7f774f192cc0, m_thread = 0x7f77528f6d40, m_timer_start = 2351919348814600, m_timer = 0x55cab5961b4e <my_timer_cycles()>, m_wait = 0x7f77528f7670}
 
        locker = 0x7f774edb1aa0
 
        result = 32631
 
#4  0x000055cab5ef2e44 in inline_mysql_cond_timedwait (that=0x55cab75c6460 <COND_timer>, mutex=0x55cab75c63a0 <LOCK_timer>, abstime=0x7f774edb1b70, src_file=0x55cab646e308 "/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/thr_timer.c", src_line=321) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/include/mysql/psi/mysql_thread.h:1086
 
No locals.
 
#5  0x000055cab5ef396d in timer_handler (arg=0x0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/thr_timer.c:321
 
        error = 21962
 
        top_time = 0x55cab75c64e0 <max_timer_data>
 
        now = {tv_sec = 1683790818, tv_nsec = 724317000}
 
        abstime = {tv_sec = 2147483647, tv_nsec = 0}
 
#6  0x000055cab592d489 in pfs_spawn_thread (arg=0x55cab86d8898) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/perfschema/pfs.cc:2201
 
        typed_arg = 0x55cab86d8898
 
        user_arg = 0x0
 
        user_start_routine = 0x55cab5ef3818 <timer_handler>
 
        pfs = 0x7f77528f6d40
 
        klass = 0x55cab856dac0
 
#7  0x00007f775337bb43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#8  0x00007f775340cbb4 in clone () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
Thread 1 (Thread 0x7f7752fd3f40 (LWP 342440)):
 
#0  0x00007f775337da7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#1  0x000055cab5eecc3e in my_write_core (sig=6) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysys/stacktrace.c:424
 
No locals.
 
#2  0x000055cab562a02d in handle_fatal_signal (sig=6) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/signal_handler.cc:360
 
        curr_time = 1683790818
 
        tm = {tm_sec = 18, tm_min = 40, tm_hour = 7, tm_mday = 11, tm_mon = 4, tm_year = 123, tm_wday = 4, tm_yday = 130, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x7f77534c0b1b "UTC"}
 
        thd = 0x0
 
        print_invalid_query_pointer = false
 
#3  <signal handler called>
 
No symbol table info available.
 
#4  0x00007f775337da7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#5  0x00007f7753329476 in raise () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#6  0x00007f775330f7f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#7  0x00007f775330f71b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#8  0x00007f7753320e96 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
 
No symbol table info available.
 
#9  0x000055cab586ee0b in translog_read_record_header_from_buffer (page=0x7fffe81dc9d0 "4", page_offset=1429, buff=0x7fffe81dee90, scanner=0x0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_loghandler.c:7161
 
        res = 32631
 
        __PRETTY_FUNCTION__ = "translog_read_record_header_from_buffer"
 
#10 0x000055cab586f1a4 in translog_read_record_header (lsn=4295394709, buff=0x7fffe81dee90) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_loghandler.c:7228
 
        psize_buff = {dummy = 13835058055298940980, buffer = "4\000\000\001\000\000\000\300\366\004l,   \t column_name, ordinal_position, column_default, is_nullable,          data_type, character_maximum_length, character_octet_length,          numeric_precision, numeric_scale, character_"...}
 
        page = 0x7fffe81dc9d0 "4"
 
        res = 4294967295
 
        page_offset = 1429
 
        direct_link = 0x0
 
        addr = 4295393280
 
        data = {addr = 0x7fffe81dc9b0, was_recovered = 0 '\000'}
 
        __PRETTY_FUNCTION__ = "translog_read_record_header"
 
#11 0x000055cab58897e7 in parse_checkpoint_record (lsn=4295394709) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_recovery.c:3295
 
        i = 18446744073709551615
 
        nb_dirty_pages = 18446744073709551615
 
        rec = {lsn = 4295394709, groups = 0x7fffe81deed0, short_trid = 61808, record_length = 32767, header = "\265\254ASw\177\000\000\320\356\035\350\377\177\000\000\300\357\035\350\377\177\000\000p\361\035\350\377\177\000\000\224iRSw\177", '\000' <repeats 26 times>, "x\364\035\350\377\177", '\000' <repeats 18 times>, "\260\372\035\350\377\177", '\000' <repeats 42 times>, "P\361\035\350\377\177\000\000X\361\035\350\377\177\000\000`\361\035\350\377\177\000\000h\361\035\350\377\177\000\000\270\372\035\350\377\177\000\000\000\000\000\000\000\000\000\000\300\372\035\350\377\177\000\000\372<\016\265\312U\000\000"..., groups_no = 0, chunk0_pages = 127, type = 3894276816, chunk0_data_addr = 94329117190072, compressed_LSN_economy = 0, non_header_data_start_offset = 0, non_header_data_len = 0, chunk0_data_len = 0}
 
        start_address = -1
 
        len = -1
 
        nb_active_transactions = 4294967295
 
        nb_committed_transactions = 4294967295
 
        nb_tables = 4294967295
 
        ptr = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>
 
        minimum_rec_lsn_of_active_transactions = -1
 
        minimum_rec_lsn_of_dirty_pages = -1
 
        next_dirty_page_in_pool = 0xffffffffffffffff
 
        __PRETTY_FUNCTION__ = "parse_checkpoint_record"
 
#12 0x000055cab588254a in maria_apply_log (from_lsn=0, end_redo_lsn=0, end_undo_lsn=0, apply=MARIA_LOG_APPLY, trace_file=0x0, skip_DDLs_arg=1 '\001', take_checkpoints=1 '\001', warnings_count=0x7fffe81df360) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_recovery.c:342
 
        error = 0
 
        uncommitted_trans = 21962
 
        old_now = 0
 
        abort_message_printed = 0 '\000'
 
        __PRETTY_FUNCTION__ = "maria_apply_log"
 
#13 0x000055cab588218f in maria_recovery_from_log () at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_recovery.c:240
 
        res = 1
 
        trace_file = 0x0
 
        warnings_count = 0
 
        __PRETTY_FUNCTION__ = "maria_recovery_from_log"
 
#14 0x000055cab5858671 in ha_maria_init (p=0x55cab86fec78) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ha_maria.cc:3913
 
        res = 0
 
        tmp = 0
 
        log_dir = 0x55cab6cf19c0 <mysql_real_data_home> "/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/3/mysqld.1/data/"
 
#15 0x000055cab562d845 in ha_initialize_handlerton (plugin=0x55cab86ee5e0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/handler.cc:649
 
        hton = 0x55cab86fec78
 
        tmp = 21962
 
        fslot = 94329125459336
 
#16 0x000055cab5295d30 in plugin_initialize (tmp_root=0x7fffe81df780, plugin=0x55cab86ee5e0, argc=0x55cab6cf7a58 <remaining_argc>, argv=0x55cab8551870, options_only=false) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/sql_plugin.cc:1465
 
        ret = 1
 
        __PRETTY_FUNCTION__ = "int plugin_initialize(MEM_ROOT*, st_plugin_int*, int*, char**, bool)"
 
        state = 4
 
#17 0x000055cab5296b91 in plugin_init (argc=0x55cab6cf7a58 <remaining_argc>, argv=0x55cab8551870, flags=0) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/sql_plugin.cc:1758
 
        plugin_table_engine = true
 
        opts_only = false
 
        idx = 4
 
        hash = 0x55cab6cfb880 <plugin_hash+128>
 
        i = 4
 
        builtins = 0x55cab6ae0ce0 <mysql_optional_plugins+64>
 
        plugin = 0x55cab6ae3728 <builtin_maria_partition_plugin+104>
 
        tmp = {name = {str = 0x55cab628c66e "partition", length = 9}, plugin = 0x55cab6ae36c0 <builtin_maria_partition_plugin>, plugin_dl = 0x0, ptr_backup = 0x0, nbackups = 0, state = 4, ref_count = 0, locks_total = 0, data = 0x0, mem_root = {free = 0x0, used = 0x0, pre_alloc = 0x0, min_malloc = 0, block_size = 0, block_num = 0, first_block_usage = 0, flags = 0, error_handler = 0x0, psi_key = 0}, system_vars = 0x0, load_option = PLUGIN_ON}
 
        plugin_ptr = 0x55cab86ee5e0
 
        reap = 0x7fffe81df498
 
        tmp_root = {free = 0x55cab86f9f48, used = 0x55cab86fb6f8, pre_alloc = 0x55cab86f0418, min_malloc = 32, block_size = 4088, block_num = 8, first_block_usage = 0, flags = 0, error_handler = 0x0, psi_key = 0}
 
        reaped_mandatory_plugin = false
 
        mandatory = false
 
        opt_plugin_load_list_iter = {<base_ilist_iterator> = {list = 0x55cab6cfb660 <opt_plugin_load_list>, el = 0x55cab6cfb660 <opt_plugin_load_list>, current = 0x55cab6cfb668 <opt_plugin_load_list+8>}, <No data fields>}
 
        plugin_table_engine_name_buf = "Aria\000\177\000\000\255\254:\265\312U\000\000@\305d\270\312U\000\000\240\370\035\350\377\177\000\000\220\370\035\350\377\377\377\377x\244:\265\312U\000\000\340K\320\266\000U\000\000\000\224x\\~\221\005\222\260"
 
        plugin_table_engine_name = {str = 0x7fffe81df850 "Aria", length = 4}
 
        MyISAM = {str = 0x55cab603dc8b "MyISAM", length = 6}
 
        __PRETTY_FUNCTION__ = "int plugin_init(int*, char**, int)"
 
#18 0x000055cab50e3cfa in init_server_components () at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mysqld.cc:5208
 
        __PRETTY_FUNCTION__ = "int init_server_components()"
 
#19 0x000055cab50e504c in mysqld_main (argc=141, argv=0x55cab8551870) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mysqld.cc:5826
 
        please_close_stdin = true
 
        ho_error = 0
 
        new_thread_stack_size = 299008
 
        user = 0x8000000000000006 <error: Cannot access memory at address 0x8000000000000006>
 
#20 0x000055cab50d9b3d in main (argc=8, argv=0x7fffe81e1c18) at /home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/main.cc:34
 
No locals.
 
Compressed file /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/3/log/type_inet.rpl_type_inet6-stmt/mysqld.1/data/core
 
 - saving '/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/3/log/type_inet.rpl_type_inet6-stmt/' to '/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/log/type_inet.rpl_type_inet6-stmt/'

From master error log:

mariadbd: /home/buildbot/amd64-ubuntu-2204-debug-ps/build/storage/maria/ma_loghandler.c:7161: translog_read_record_header_from_buffer: Assertion `translog_is_LSN_chunk(page[page_offset])' failed.
 
230511  7:40:18 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.11.4-MariaDB-debug-log source revision: 1916bf2a02c1a2e4de8526fcbd0c2637c3921f33
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=153
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63922 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
mysys/stacktrace.c:215(my_print_stacktrace)[0x55cab5eecb45]
 
sql/signal_handler.cc:241(handle_fatal_signal)[0x55cab5629ca9]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f7753329520]
 
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f775337da7c]
 
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f7753329476]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f775330f7f3]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7f775330f71b]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7f7753320e96]
 
maria/ma_loghandler.c:7162(translog_read_record_header_from_buffer)[0x55cab586ee0b]
 
maria/ma_loghandler.c:7228(translog_read_record_header)[0x55cab586f1a4]
 
maria/ma_recovery.c:3295(parse_checkpoint_record)[0x55cab58897e7]
 
maria/ma_recovery.c:342(maria_apply_log)[0x55cab588254a]
 
maria/ma_recovery.c:240(maria_recovery_from_log)[0x55cab588218f]
 
maria/ha_maria.cc:3912(ha_maria_init(void*))[0x55cab5858671]
 
sql/handler.cc:649(ha_initialize_handlerton(st_plugin_int*))[0x55cab562d845]
 
sql/sql_plugin.cc:1465(plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool))[0x55cab5295d30]
 
sql/sql_plugin.cc:1758(plugin_init(int*, char**, int))[0x55cab5296b91]
 
sql/mysqld.cc:5208(init_server_components())[0x55cab50e3cfa]
 
sql/mysqld.cc:5826(mysqld_main(int, char**))[0x55cab50e504c]
 
sql/main.cc:34(main)[0x55cab50d9b3d]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f7753310d90]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f7753310e40]
 
/home/buildbot/amd64-ubuntu-2204-debug-ps/build/sql/mariadbd(_start+0x25)[0x55cab50d9a55]
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...

Generated at Thu Feb 08 09:07:09 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.