Elena Stepanova added a comment - 2021-12-21 21:44 mleich , But you really don't load the provider plugin on the upgraded server, do you? I don't see it anywhere in the test options. In this case, there is no need of rr or a stress test to repeat this. Create an LZO-compressed table on 10.5, shut down, start 10.7 without lzo provider, observe the error.

Sergei Golubchik added a comment - 2021-12-21 21:56 And how is it a bug then?

Matthias Leich added a comment - 2021-12-22 15:51 - edited Assigning --mysqld=--plugin-load-add=provider_lzo.so --mysqld=--plugin-load-add=provider_bzip2.so --mysqld=--plugin-load-add=provider_lzma --mysqld=--plugin-load-add=provider_snappy --mysqld=--plugin-load-add=provider_lz4 to both server starts (RQG just does that) has the following effect 1. The 10.5 server comes up even though having error messages like [ERROR] mysqld: Can't open shared library '/data/Server_bin/10.5D_asan/lib/plugin/provider_lzo.so' (errno: 2, cannot open shared object file: No such file or directory) within the error log. The message by itself is correct because lib/plugin/provider_lzo.so does really not exist. 2. The effects described on top "MariaDB tried to use the LZO compression, but its provider plugin ..." disappears.

Marko Mäkelä added a comment - 2022-04-25 05:23 MDEV-27887 is another attempt at explaining how this is a bug. Basically, as noted in MDEV-15912 and MDEV-23755 , an upgrade after a normal shutdown is expected to work. Earlier, there had been instructions to execute a slow shutdown ( innodb_fast_shutdown=0 ) to ensure that the history of all transactions will be purged. As noted in MDEV-27887 , it is possible that some history of old transactions remains to be processed after the upgrade. If the compression libraries are unavailable, this would lead to an obscure error message not on the first SQL-layer access to affected tables, but possibly before any table has been accessed from SQL.

Marko Mäkelä added a comment - 2022-04-25 05:52 Here is a test case change that repeats the problem: ./mtr innodb.innodb-page_compression_snappy 10.7 9e314fcf6e666ce6fdbdd8ca1ae23d6c4b389b21 At line 105: query 'SET GLOBAL innodb_fast_shutdown=0' failed: <Unknown> (2006): Server has gone away These changes are also speeding up the test by working around MDEV-24813 (no locks will be acquired on InnoDB temporary tables) and by actually triggering a log checkpoint via page flushing. diff --git a/mysql-test/suite/innodb/include/innodb-page-compression.inc b/mysql-test/suite/innodb/include/innodb-page-compression.inc index b16edcf2a28..df93fe4cfd0 100644 --- a/mysql-test/suite/innodb/include/innodb-page-compression.inc +++ b/mysql-test/suite/innodb/include/innodb-page-compression.inc @@ -4,7 +4,7 @@ # This may be triggered on a slow system or one that lacks native AIO. call mtr.add_suppression("InnoDB: Trying to delete tablespace.*pending operations"); --enable_query_log -create table innodb_normal (c1 int not null auto_increment primary key, b char(200)) engine=innodb; +create temporary table innodb_normal (c1 int not null auto_increment primary key, b char(200)) engine=innodb; create table innodb_page_compressed1 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=1; create table innodb_page_compressed2 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=2; create table innodb_page_compressed3 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=3; @@ -15,6 +15,10 @@ create table innodb_page_compressed7 (c1 int not null auto_increment primary key create table innodb_page_compressed8 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=8; create table innodb_page_compressed9 (c1 int not null auto_increment primary key, b char(200)) engine=innodb page_compressed=1 page_compression_level=9; +connect (prevent_purge,localhost,root); +START TRANSACTION WITH CONSISTENT SNAPSHOT; +connection default; + --disable_query_log begin; let $i = 2000; @@ -40,20 +44,15 @@ insert into innodb_page_compressed9 select * from innodb_normal; commit; --enable_query_log -select count(*) from innodb_page_compressed1; -select count(*) from innodb_page_compressed3; -select count(*) from innodb_page_compressed4; -select count(*) from innodb_page_compressed5; -select count(*) from innodb_page_compressed6; -select count(*) from innodb_page_compressed6; -select count(*) from innodb_page_compressed7; -select count(*) from innodb_page_compressed8; -select count(*) from innodb_page_compressed9; - # # Wait until pages are really compressed # -let $wait_condition= select variable_value > 0 from information_schema.global_status where variable_name = 'INNODB_NUM_PAGES_PAGE_COMPRESSED'; +SET GLOBAL innodb_max_dirty_pages_pct=0.0; + +let $wait_condition = +SELECT variable_value = 0 +FROM information_schema.global_status +WHERE variable_name = 'INNODB_BUFFER_POOL_PAGES_DIRTY'; --source include/wait_condition.inc --let $MYSQLD_DATADIR=`select @@datadir` @@ -62,12 +61,7 @@ let $wait_condition= select variable_value > 0 from information_schema.global_st --source include/shutdown_mysqld.inc ---let t1_IBD = $MYSQLD_DATADIR/test/innodb_normal.ibd ---let SEARCH_RANGE = 10000000 --let SEARCH_PATTERN=AaAaAaAa ---echo # innodb_normal expected FOUND --- let SEARCH_FILE=$t1_IBD --- source include/search_pattern_in_file.inc --let t1_IBD = $MYSQLD_DATADIR/test/innodb_page_compressed1.ibd --echo # innodb_page_compressed1 page compressed expected NOT FOUND -- let SEARCH_FILE=$t1_IBD @@ -105,7 +99,11 @@ let $wait_condition= select variable_value > 0 from information_schema.global_st -- let SEARCH_FILE=$t1_IBD -- source include/search_pattern_in_file.inc +let $restart_parameters = --disable-provider-snappy -- source include/start_mysqld.inc +disconnect prevent_purge; +SET GLOBAL innodb_fast_shutdown=0; +-- source include/restart_mysqld.inc select count(*) from innodb_page_compressed1; select count(*) from innodb_page_compressed3; @@ -120,7 +118,6 @@ select count(*) from innodb_page_compressed9; let $wait_condition= select variable_value > 0 from information_schema.global_status where variable_name = 'INNODB_NUM_PAGES_PAGE_DECOMPRESSED'; --source include/wait_condition.inc -drop table innodb_normal; drop table innodb_page_compressed1; drop table innodb_page_compressed2; drop table innodb_page_compressed3; If you remove the $restart_parameters , the test should not crash, but the SELECT statements after the restart should report errors. Side note: The size of this test could be reduced further, by writing just one record to each table, now that the test should no longer depend on a small buffer pool size. I will do the test case cleanup separately.

Marko Mäkelä added a comment - 2022-04-25 06:19 Sorry, I made a mistake in my test: the relevant snippet should have been let $restart_parameters = --disable-provider-snappy; -- source include/start_mysqld.inc disconnect prevent_purge; SET GLOBAL innodb_fast_shutdown=0; -- source include/restart_mysqld.inc A semicolon was missing semicolon at the end of the let line, and the SQL statement was being submitted while no server was running. In a corrected form, this test does show a form of corruption even though it does not lead to a server crash: let $restart_parameters = --disable-provider-snappy; -- source include/start_mysqld.inc disconnect prevent_purge; SET GLOBAL innodb_fast_shutdown=0; let $restart_parameters =; -- source include/restart_mysqld.inc   select count (*) from innodb_page_compressed1; During the slow shutdown, the compressed tables are inaccessible, and a number of errors like the following will be emitted by purge: 10.7 9e314fcf6e666ce6fdbdd8ca1ae23d6c4b389b21 2022-04-25 9:04:09 0 [ERROR] InnoDB: Failed to read page 2 from file './test/innodb_page_compressed9.ibd': Table is compressed or encrypted but uncompress or decrypt failed. 2022-04-25 9:04:09 0 [ERROR] InnoDB: Failed to read page 3 from file './test/innodb_page_compressed9.ibd': Table is compressed or encrypted but uncompress or decrypt failed. 2022-04-25 9:04:09 0 [ERROR] InnoDB: Failed to read page 69 from file './test/innodb_page_compressed9.ibd': Table is compressed or encrypted but uncompress or decrypt failed. 2022-04-25 9:04:09 0 [ERROR] InnoDB: Failed to read page 122 from file './test/innodb_page_compressed9.ibd': Table is compressed or encrypted but uncompress or decrypt failed. After the restart, the first leaf page that will be accessed by the SELECT (for me: page number 4) shows that the history had not been reset during the slow shutdown: 00000060: 0200 1b69 6e66 696d 756d 0003 000b 0000 ...infimum...... 00000070: 7375 7072 656d 756d 0000 0010 00df 8000 supremum........ 00000080: 0001 0000 0000 0036 9400 0001 4701 1041 .......6....G..A 00000090: 6141 6141 6141 6141 6141 6141 6141 6141 aAaAaAaAaAaAaAaA The next-record pointer at 0x61 points to 0x63+0x1b = 0x7e. At that offset we can find (c1,DB_TRX_ID,DB_ROLL_PTR,b)=(1,0x36,0x94…,'AaAa…'). If the history had been purged as expected, the DB_TRX_ID and DB_ROLL_PTR would be 0 and 2⁵⁵. I think that a proper fix of this bug would be to make InnoDB refuse to start up if any of the undo log records traversed during trx_lists_init_at_db_start() point to compressed tables whose algorithm has not been loaded. Such a fix would only be possible if the .frm file stores the required compression algorithm. InnoDB itself only stores it in individual data pages, not in its data dictionary.

Marko Mäkelä added a comment - 2022-04-25 08:00 I introduced a smaller, non-restarting test innodb.innodb_page_compressed that will replace the file that the above patch is for.