[MDEV-18549] InnoDB: Failing assertion: opt_no_lock during mariabackup --backup ... Created: 2019-02-12  Updated: 2023-05-16  Resolved: 2019-02-14

Status: Closed
Project: MariaDB Server
Component/s: Backup, mariabackup
Affects Version/s: 10.4.3
Fix Version/s: 10.4.3

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Matthias Leich
Resolution: Fixed Votes: 0
Labels: None
Environment:

Linux Ubuntu 17.10 but most probably unimportant


Attachments: File 000000.log     File MDEV-18549.yy     File RQG_MDEV-18549.sh    
Issue Links:
Relates
relates to MDEV-23655 mariabackup crash 10.4.13 Closed

 Description   

Problem found during RQG testing of mariabackup.
bb-10.4-monty, origin/bb-10.4-monty ea07b5227feeccba1429b687e731033e1a6d560b 2019-02-05T20:08:11+02:00
One connection runs "chaotic" DDL.
Another connection runs mariabackup.
...
Executing backup: bld_debug/extra/mariabackup/mariabackup --host=127.0.0.1 --user=root --password=''  --innodb-use-native-aio=0  --port=19720 --backup --datadir=/dev/shm/vardir/1549967567/22/1/data --target-dir=/dev/shm/vardir/1549967567/22/1_clone/data
2019-02-12 11:33:27 Connecting to MySQL server host: 127.0.0.1, user: root, password: set, port: 19720, socket: not set
[00] 2019-02-12 11:33:27 Using server version 10.4.3-MariaDB-debug-log
...
2019-02-12 11:33:31 0x7feefeb29740  InnoDB: Assertion failure in file extra/mariabackup/xtrabackup.cc line 647
InnoDB: Failing assertion: opt_no_lock
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
190212 11:33:31 [ERROR] mysqld got signal 6 ;
...
stack_bottom = 0x0 thread_stack 0x49000
bld_debug/extra/mariabackup/mariabackup(my_print_stacktrace+0x40)[0x55beb64884f9]
bld_debug/extra/mariabackup/mariabackup(handle_fatal_signal+0x3e7)[0x55beb5d58dce]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13150)[0x7feefe726150]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7feefcfaa0bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16d)[0x7feefcfabf5d]
bld_debug/extra/mariabackup/mariabackup(+0x10308b9)[0x55beb61f98b9]
bld_debug/extra/mariabackup/mariabackup(+0x6581a7)[0x55beb58211a7]
bld_debug/extra/mariabackup/mariabackup(+0xec09e6)[0x55beb60899e6]
bld_debug/extra/mariabackup/mariabackup(+0xec2ef4)[0x55beb608bef4]
bld_debug/extra/mariabackup/mariabackup(+0xec6bce)[0x55beb608fbce]
bld_debug/extra/mariabackup/mariabackup(+0xec7635)[0x55beb6090635]
bld_debug/extra/mariabackup/mariabackup(+0x65b9fb)[0x55beb58249fb]
bld_debug/extra/mariabackup/mariabackup(+0x65bcb1)[0x55beb5824cb1]
ut/ut0mem.cc:42(ut_strlcpy(char*, char const*, unsigned long))[0x55beb582771e]
mariabackup/xtrabackup.cc:649(backup_file_op_fail(unsigned long, unsigned char const*, unsigned char const*, unsigned long, unsigned char const*, unsigned long))[0x55beb5828612]
log/log0recv.cc:493(fil_name_parse(unsigned char*, unsigned char const*, unsigned long, unsigned long, mlog_id_t, bool))[0x55beb582f7ae]
log/log0recv.cc:1187(recv_parse_or_apply_log_rec_body(mlog_id_t, unsigned char*, unsigned char*, unsigned long, unsigned long, bool, buf_block_t*, mtr_t*))[0x55beb582effe]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7feefcf941c1]
bld_debug/extra/mariabackup/mariabackup(_start+0x2a)[0x55beb581eaea]
 
RQG YY grammar:
query:
    ALTER TABLE t2 ADD PRIMARY KEY IF NOT EXISTS ( col2 , col_text(9) ) |
    ALTER TABLE t2 ADD PRIMARY KEY IF NOT EXISTS ( col_text(9) , col2 ) |
    ALTER TABLE t2 DROP PRIMARY KEY |
    ALTER TABLE t2 DROP PRIMARY KEY |
    ALTER TABLE t2 DROP PRIMARY KEY |
    ALTER TABLE t2 DROP PRIMARY KEY ;
 
query_init:
    CREATE TABLE IF NOT EXISTS t2 ( col1 INT, col2 INT, col_int INTEGER , col_string VARCHAR(19), col_varchar VARCHAR(500), col_text TEXT ) ENGINE = InnoDB ROW_FORMAT = Compressed ; thread_connect ;
 
thread_connect:
    SET AUTOCOMMIT = 0; SET SESSION lock_wait_timeout = 2 ; SET SESSION innodb_lock_wait_timeout = 1 ;



 Comments   
Comment by Vladislav Vaintroub [ 2019-02-13 ]

mleich, could you check if you get the same with 10.3 ?

Comment by Matthias Leich [ 2019-02-13 ]

No replay of the problem on
10.3 7293ce0ee81f05b1ec3ac9ddcc88bfbee4030e55 2019-02-04T17:52:39+03:00
10.3 4e7ee166a9c76eb3546356aabfd2dbc759671cd0 2019-02-11T14:42:48+02:00
or (30 brute force replay attempts per tree is limited) hitting it is extreme rare.

Replay of the problem on
10.4 eb1d7aeeea2978f5be10cf95e6f638f184cfe4ad 2019-02-08T07:58:29+02:00
But it seems to be less fast than on bb-10.4-monty.

Comment by Matthias Leich [ 2019-02-14 ]

How to replay the problem?
1. Please use a Linux.
2. Clone my experimental RQG 
     git clone https://github.com/mleich1/rqg --branch experimental RQG_mleich1
     cd RQG_mleich1
3. Place  MDEV-18549.yy and RQG_MDEV-19549.sh in that directory.
4. Please adjust RQG_MDEV-19549.sh to your environment.
5. ./RQG_MDEV-19549.sh <path to your MariaDB binaries>
                                                     Example from my box: /home/mleich/Server/10.4/bld_debug
6. grep 'InnoDB: Failing assertion: opt_no_lock' storage/mariabackup/rqg.log
    will show if you have hit the problem reported here.
    You might hit some other problem or maybe no failure at all.
    I need usually a runtime of ~ 3.5 minutes.

Generated at Thu Feb 08 08:44:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.