[MDEV-20737] Aria recovery with encryption fails on redo phase: Got error 192 when executing record redo_insert_row_blobs Created: 2019-10-03  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Encryption, Storage Engine - Aria
Affects Version/s: 10.1, 10.2, 10.3, 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Michael Widenius
Resolution: Unresolved Votes: 0
Labels: None


 Description   

10.4 9b5cdeeb

 
2019-10-03 18:00:29 0 [Note] /data/bld/10.4/bin/mysqld (mysqld 10.4.9-MariaDB-debug-log) starting as process 4641 ...
 
2019-10-03 18:00:29 0 [Note] mysqld: Aria engine: starting recovery
 
recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80%
 
2019-10-03 18:00:39 0 [ERROR] mysqld: failed to decrypt './test/t5'  rc: -1  dstlen: 0  size: 8172
 
 
 
Got error 192 when executing record redo_insert_row_blobs
 
2019-10-03 18:00:39 0 [ERROR] mysqld: Aria engine: Redo phase failed
 
tables to flush: 9 8 7 6 5 4 3 2 1 0
 
2019-10-03 18:00:39 0 [ERROR] mysqld: Aria engine: closing of tables failed
 
2019-10-03 18:00:39 0 [ERROR] mysqld: Aria recovery failed. Please run aria_chk -r on all Aria tables and delete all aria_log.######## files
 
2019-10-03 18:00:39 0 [ERROR] Plugin 'Aria' init function returned error.
 
2019-10-03 18:00:39 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.

To reproduce (Use RQG repo and branch as described, other versions may behave differently):

git clone https://github.com/MariaDB/randgen --branch mdev20737 rqg-mdev20737
 
cd rqg-mdev20737
 
perl ./runall-new.pl --basedir=<your basedir> --threads=8 --duration=250 --engine=Aria --seed=1 --scenario=CrashUpgrade --grammar=mdev20737.yy --skip-gendata --gendata-advanced --mysqld=--aria-encrypt-tables=1 --mysqld=--file-key-management --mysqld=--file-key-management-filename=`pwd`/data/file_key_management_keys.txt --mysqld=--plugin-load-add=file_key_management --vardir=/dev/shm/vardir_mdev20737

Remember to set the basedir in the command line above.

Sometimes it can fail instead with

Got error 192 when executing record redo_index_new_page

or

Got error 192 when executing record redo_new_row_tail

or

Got error 126 when executing undo undo_key_insert

Just re-run it if it happens.

If it doesn't fail as needed in 1-2 attempts, try pointing vardir at disk rather than shm. It fails for me almost every time either way, so I can't say how more probable it is on disk, but it seems that it can be.

Reproducible on 10.1-10.5.
Couldn't reproduce without encryption.

 Comments   
Comment by Elena Stepanova [ 2020-01-10 ]

Here is an easier way to reproduce the same or similar problem.

  • Start the fresh clean server with 

    --aria-encrypt-tables=1  --file-key-management --file-key-management-filename=`pwd`/mysql-test/std_data/keys.txt --plugin-load-add=file_key_management

  • Download and execute lsn.sql script: 

    bin/mysql < lsn.sql

    (but with user, password and protocol suitable for your server)

  • Kill the server with SIGKILL
  • Try to start the server again with the same options.

The error produced upon startup is very similar to the one in the description. The difference is that on 10.3+ there is an extra line about wrong LSN:

10.4 98316320

 
2020-01-10 17:53:55 0 [Note] mysqld: Aria engine: starting recovery
 
recovered pages: 0%
 
Table ./test/t5.MAD has wrong LSN: (564759,0x1aa9f625) on page: 1377
 
2020-01-10 17:53:55 0 [ERROR] mysqld: failed to decrypt './test/t5'  rc: -1  dstlen: 0  size: 8172
 
 
 
Got error 192 when executing record redo_insert_row_blobs
 
2020-01-10 17:53:55 0 [ERROR] mysqld: Aria engine: Redo phase failed
 
tables to flush: 5 4 3 2 1 0
 
2020-01-10 17:53:55 0 [ERROR] mysqld: Aria recovery failed. Please run aria_chk -r on all Aria tables and delete all aria_log.######## files
 
2020-01-10 17:53:55 0 [ERROR] Plugin 'Aria' init function returned error.
 
2020-01-10 17:53:55 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.

10.2 also fails, but without the "has wrong LSN" line.

Generated at Thu Feb 08 09:01:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.