[MDEV-29694] Remove the InnoDB change buffer - Jira

Details

Type: Task
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Fixed
Fix Version/s: 11.0.1
Component/s: Storage Engine - InnoDB
Labels:

Description

~~MDEV-27734~~ disabled the problematic InnoDB change buffer by default, and ~~MDEV-27735~~ deprecated the parameter in MariaDB Server 10.9.

The next logical step is to remove the parameter altogether, along with any code that would create the change buffer or add records to it. An attempt to downgrade to an earlier version will be caught and prevented by ~~MDEV-30106~~.

In order to be able to simplify the buffer pool interfaces, upgrading will be split in two parts: First, check if an upgrade is needed. If it is, we will apply all redo log, possibly upgrade the redo log to the current format, and then attempt to upgrade the change buffer:

Apply any buffered changes and clear the corresponding change buffer bitmap bits.
Create an empty the change buffer root page.
Free any pages that were allocated to the change buffer.
Reset the change buffer root page, to mark the upgrade as completed and to prevent a downgrade to an earlier version.

If an error occurs during before the final upgrade step, it should be possible to downgrade to MariaDB Server 10.8 or a later version (using the currently latest redo log format that was changed in ~~MDEV-14425~~).

As noted in ~~MDEV-11634~~, the change buffer may improve performance in some cases, but it is also missing a lot of potential. The main motivation for removing the change buffer in its current form is that difficult-to-reproduce corruption bugs keep popping up, such as ~~MDEV-24449~~, ~~MDEV-26917~~, ~~MDEV-30009~~.

Attachments

Issue Links

blocks

MDEV-11658 Simpler, faster IMPORT of InnoDB tables

Open

MDEV-32042 Special handling of crash recovery in buf_page_get_gen() may cause overhead

Closed

causes

MDEV-30895 Assertion `btr_cur->rtr_info->thr || !btr_cur->index()->is_committed()' failed in rtr_ins_enlarge_mbr after ALTER table force

Closed

MDEV-31443 assert [FATAL] InnoDB: Unable to find charset-collation for 254 after upgrade from 10.11.4 to 11.0

Closed

is blocked by

MDEV-27734 Set innodb_change_buffering=none by default

Closed

MDEV-27735 Deprecate the parameter innodb_change_buffering

Closed

MDEV-30106 InnoDB fails to validate the change buffer on startup

Closed

relates to

MDEV-12700 Allow innodb_read_only startup without prior slow shutdown

Closed

MDEV-23755 safety for upgrades (innodb) and packaging

Closed

MDEV-25267 Reported latching order violation in ibuf_remove_free_page()

Closed

MDEV-30009 InnoDB shutdown hangs when the change buffer is corrupted

Closed

MDEV-30134 buf_page_t::unfix(): Assertion `!((f ^ (f - 1)) & LRU_MASK)' failed

Closed

MDEV-30148 Race condition between non-persistent statistics and purge of InnoDB history

Closed

MDEV-30216 Read-ahead unnecessarily allocates and frees pages when a page is in the buffer pool

Closed

MDEV-31088 Server freeze due to innodb_change_buffering and innodb_file_per_table=0

Closed

MDEV-32044 Mariadb crash after upgrading to 11.0.3: Failing assertion: local_len >= BTR_EXTERN_FIELD_REF_SIZE

Closed

MDEV-32132 DROP INDEX followed by CREATE INDEX may corrupt data

Closed

MDEV-34759 buf_page_get_low() is unnecessarily acquiring exclusive latch on secondary index pages

Closed

MDEV-35628 Lock order inversion between ibuf_mutex and ibuf.index->lock

Open

MDEV-11634 Improve the InnoDB change buffer

Closed

MDEV-24449 Corruption of system tablespace or last recovered page

Closed

MDEV-26917 InnoDB: Clustered record for sec rec not found index

Closed

MDEV-30009 InnoDB shutdown hangs when the change buffer is corrupted

Closed

MDEV-30819 InnoDB fails to start up after downgrading from MariaDB 11.0

Closed

MDEV-35553 InnoDB: ibuf cursor restoration fails error in MariaDB 10.6.20

Closed

(2 is blocked by, 18 relates to)

Activity

Ascending order - Click to sort in descending order

Marko Mäkelä added a comment - 2022-11-30 14:33 - edited

The code to upgrade the MySQL 3.23 or MySQL 4.0 change buffer (before the introduction of innodb_file_per_table) was removed in MySQL 5.6.5 and in MariaDB 10.0.11.

Marko Mäkelä added a comment - 2022-11-30 14:33 - edited The code to upgrade the MySQL 3.23 or MySQL 4.0 change buffer (before the introduction of innodb_file_per_table ) was removed in MySQL 5.6.5 and in MariaDB 10.0.11 .

Marko Mäkelä added a comment - 2022-12-01 13:38

mleich, I only tested the upgrade of an empty change buffer.

Can you please provide a copy of a 10.6 data directory where the change buffer is not empty, for some additional testing by me? Use innodb_change_buffering_debug=1 and update an indexed secondary index field, and shut down normally, without innodb_fast_shutdown=0. The current implementation is missing an upgrade of the redo log format before upgrading the change buffer.

Please pay special attention to testing SPATIAL INDEX, because as part of removing the change buffer, the field btr_cur_t::thr was removed. The innodb_gis test suite passed (including the disabled sporadically failing tests), but I think that we need some more testing there.

Marko Mäkelä added a comment - 2022-12-01 13:38 mleich , I only tested the upgrade of an empty change buffer. Can you please provide a copy of a 10.6 data directory where the change buffer is not empty, for some additional testing by me? Use innodb_change_buffering_debug=1 and update an indexed secondary index field, and shut down normally, without innodb_fast_shutdown=0 . The current implementation is missing an upgrade of the redo log format before upgrading the change buffer. Please pay special attention to testing SPATIAL INDEX , because as part of removing the change buffer, the field btr_cur_t::thr was removed. The innodb_gis test suite passed (including the disabled sporadically failing tests), but I think that we need some more testing there.

Marko Mäkelä added a comment - 2022-12-02 07:12

The upgrade from a nonempty change buffer can be tested with the following patch applied to an earlier version:

diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test

index 9362f8daffa..84854607be7 100644

--- a/mysql-test/suite/innodb/t/ibuf_not_empty.test

+++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test

@@ -76,6 +76,7 @@ EOF

 check table t1;

 --source include/shutdown_mysqld.inc

+stop here;

 # Truncate the file to 5 pages, as if it were empty

 perl;

The test will fail like this:

CURRENT_TEST: innodb.ibuf_not_empty

mysqltest: At line 79: query 'stop here' failed: <Unknown> (2006): Server has gone away

After that, we can find a data directory in mysql-test/var/log/innodb.ibuf_not_empty*/mysqld.1/data on which we can test the upgrade:

…/mariadbd --innodb-page-size=4k --datadir=/…/data

This will actually crash with the current code (trying an upgrade from 10.6):

2022-12-02  8:54:46 0 [Note] InnoDB: Upgrading the change buffer

221202  8:54:46 [ERROR] mysqld got signal 11 ;

The upgrade needs to be revised so that the log will be upgraded to the most recent format (~~MDEV-14425~~) first. An attempt to upgrade from 10.8 (or later) will not crash, but the index will be reported corrupted:

MariaDB [test]> check table t1;

+---------+-------+----------+---------------------------------------------------------+

| Table   | Op    | Msg_type | Msg_text                                                |

+---------+-------+----------+---------------------------------------------------------+

| test.t1 | check | Warning  | InnoDB: Index 'b' contains 990 entries, should be 1024. |

| test.t1 | check | error    | Corrupt                                                 |

+---------+-------+----------+---------------------------------------------------------+

What if we upgraded before corrupting the change buffer bitmap?

diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test

index 9362f8daffa..50fe70d3b8a 100644

--- a/mysql-test/suite/innodb/t/ibuf_not_empty.test

+++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test

@@ -37,6 +37,7 @@ let MYSQLD_DATADIR=`select @@datadir`;

 let PAGE_SIZE=`select @@innodb_page_size`;

 --source include/shutdown_mysqld.inc

+stop already here;

 # Corrupt the change buffer bitmap, to claim that pages are clean

 perl;

Unfortunately, the result is the same. I will fix these issues. This should not block the testing of the bulk of the changes (that everything still works during normal operation).

Marko Mäkelä added a comment - 2022-12-02 07:12 The upgrade from a nonempty change buffer can be tested with the following patch applied to an earlier version: diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test index 9362f8daffa..84854607be7 100644 --- a/mysql-test/suite/innodb/t/ibuf_not_empty.test +++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test @@ -76,6 +76,7 @@ EOF check table t1; --source include/shutdown_mysqld.inc +stop here; # Truncate the file to 5 pages, as if it were empty perl; The test will fail like this: CURRENT_TEST: innodb.ibuf_not_empty mysqltest: At line 79: query 'stop here' failed: <Unknown> (2006): Server has gone away After that, we can find a data directory in mysql-test/var/log/innodb.ibuf_not_empty*/mysqld.1/data on which we can test the upgrade: …/mariadbd --innodb-page-size=4k --datadir=/…/data This will actually crash with the current code (trying an upgrade from 10.6): 2022-12-02 8:54:46 0 [Note] InnoDB: Upgrading the change buffer 221202 8:54:46 [ERROR] mysqld got signal 11 ; The upgrade needs to be revised so that the log will be upgraded to the most recent format ( MDEV-14425 ) first. An attempt to upgrade from 10.8 (or later) will not crash, but the index will be reported corrupted: MariaDB [test]> check table t1; +---------+-------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------+-------+----------+---------------------------------------------------------+ | test.t1 | check | Warning | InnoDB: Index 'b' contains 990 entries, should be 1024. | | test.t1 | check | error | Corrupt | +---------+-------+----------+---------------------------------------------------------+ What if we upgraded before corrupting the change buffer bitmap? diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test index 9362f8daffa..50fe70d3b8a 100644 --- a/mysql-test/suite/innodb/t/ibuf_not_empty.test +++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test @@ -37,6 +37,7 @@ let MYSQLD_DATADIR=`select @@datadir`; let PAGE_SIZE=`select @@innodb_page_size`; --source include/shutdown_mysqld.inc +stop already here; # Corrupt the change buffer bitmap, to claim that pages are clean perl; Unfortunately, the result is the same. I will fix these issues. This should not block the testing of the bulk of the changes (that everything still works during normal operation).

Marko Mäkelä added a comment - 2022-12-02 14:16

The change buffer contains 34 buffered insert operations, all for a single index page. That coincides with the reported mismatch. I fixed a few things, but I still have to add the opening of data files, so that the buffered changes can be merged. I think that we can find the tablespaces by scanning the clustered index of SYS_TABLES. Accessing the secondary indes on SYS_TABLES.ID is out of the question, because there may exist buffered changes for that index. The upgrade must follow a particular order:

If the change buffer does not exist, continue with normal recovery (to not make life harder for ~~MDEV-14481~~).
If the change buffer exists:
1. Apply all redo log records.
2. If the redo log needs to be upgraded, upgrade it.
3. Open the tablespaces for all tables found in SYS_TABLES.
4. For each buffered change, in an atomic mini-transaction, apply the change and delete-mark the change buffer record.
5. Empty the change buffer root page.
6. Free the change buffer pages.
7. Initialize the change buffer root page.
8. Continue with normal recovery.

Marko Mäkelä added a comment - 2022-12-02 14:16 The change buffer contains 34 buffered insert operations, all for a single index page. That coincides with the reported mismatch. I fixed a few things, but I still have to add the opening of data files, so that the buffered changes can be merged. I think that we can find the tablespaces by scanning the clustered index of SYS_TABLES . Accessing the secondary indes on SYS_TABLES.ID is out of the question, because there may exist buffered changes for that index. The upgrade must follow a particular order: If the change buffer does not exist, continue with normal recovery (to not make life harder for MDEV-14481 ). If the change buffer exists: Apply all redo log records. If the redo log needs to be upgraded, upgrade it. Open the tablespaces for all tables found in SYS_TABLES . For each buffered change, in an atomic mini-transaction, apply the change and delete-mark the change buffer record. Empty the change buffer root page. Free the change buffer pages. Initialize the change buffer root page. Continue with normal recovery.

Marko Mäkelä added a comment - 2022-12-02 14:16

~~MDEV-30148~~ was found while testing this.

Marko Mäkelä added a comment - 2022-12-02 14:16 MDEV-30148 was found while testing this.

Marko Mäkelä added a comment - 2022-12-02 15:29

I fixed the upgrade from 10.6 and 10.8 using the above test. It only covers buffered inserts, not delete-mark or purge operations. And there were buffered inserts only to one index page.

Additional testing of the upgrades will be needed.

Marko Mäkelä added a comment - 2022-12-02 15:29 I fixed the upgrade from 10.6 and 10.8 using the above test. It only covers buffered inserts, not delete-mark or purge operations. And there were buffered inserts only to one index page. Additional testing of the upgrades will be needed.

Marko Mäkelä added a comment - 2022-12-05 12:57

Sysbench oltp_update_index did not use the change buffer, no matter what I tried (yes, it included innodb_change_buffering_debug=1. I suppose that the test is searching via the indexed column, which would force change buffer merges.

I was a little more successful with the following test:

diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test

index 9362f8daffa..46cd5bbce99 100644

--- a/mysql-test/suite/innodb/t/ibuf_not_empty.test

+++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test

@@ -33,10 +33,14 @@ SET GLOBAL innodb_change_buffering=all;

 # used for modifying the secondary index page. There must be multiple

 # index pages, because changes to the root page are never buffered.

 INSERT INTO t1 SELECT 0,'x',1 FROM seq_1_to_1024;

+UPDATE t1 SET b='y';

+UPDATE t1 SET b='x';

 let MYSQLD_DATADIR=`select @@datadir`;

 let PAGE_SIZE=`select @@innodb_page_size`;

 --source include/shutdown_mysqld.inc

+stop already here;

 # Corrupt the change buffer bitmap, to claim that pages are clean

 perl;

As far as I can tell, this may not trigger purge buffering, but it still is better than my initial upgrade test, which only covered insert buffering.

Marko Mäkelä added a comment - 2022-12-05 12:57 Sysbench oltp_update_index did not use the change buffer, no matter what I tried (yes, it included innodb_change_buffering_debug=1 . I suppose that the test is searching via the indexed column, which would force change buffer merges. I was a little more successful with the following test: diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test index 9362f8daffa..46cd5bbce99 100644 --- a/mysql-test/suite/innodb/t/ibuf_not_empty.test +++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test @@ -33,10 +33,14 @@ SET GLOBAL innodb_change_buffering=all; # used for modifying the secondary index page. There must be multiple # index pages, because changes to the root page are never buffered. INSERT INTO t1 SELECT 0,'x',1 FROM seq_1_to_1024; +UPDATE t1 SET b='y'; +UPDATE t1 SET b='x'; + let MYSQLD_DATADIR=`select @@datadir`; let PAGE_SIZE=`select @@innodb_page_size`; --source include/shutdown_mysqld.inc +stop already here; # Corrupt the change buffer bitmap, to claim that pages are clean perl; As far as I can tell, this may not trigger purge buffering, but it still is better than my initial upgrade test, which only covered insert buffering.

Marko Mäkelä added a comment - 2022-12-05 13:32

The second UPDATE appears to trigger a change buffer merge. With the following, I got 79 INSERT operations and 3700 delete-mark operations buffered for 90 pages:

diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test

index 9362f8daffa..00d766cbe72 100644

--- a/mysql-test/suite/innodb/t/ibuf_not_empty.test

+++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test

@@ -32,11 +32,14 @@ SET GLOBAL innodb_change_buffering=all;

 # Create enough rows for the table, so that the change buffer will be

 # used for modifying the secondary index page. There must be multiple

 # index pages, because changes to the root page are never buffered.

-INSERT INTO t1 SELECT 0,'x',1 FROM seq_1_to_1024;

+INSERT INTO t1 SELECT 0,'x',1 FROM seq_1_to_4096;

+UPDATE t1 SET b='y';

 let MYSQLD_DATADIR=`select @@datadir`;

 let PAGE_SIZE=`select @@innodb_page_size`;

 --source include/shutdown_mysqld.inc

+stop already here;

 # Corrupt the change buffer bitmap, to claim that pages are clean

 perl;

The upgrade was successful, and CHECK TABLE did not report any errors. One more test with 65,536 rows: 193 pages, 33,426 delete-mark operations, 7 insert operations, no purge operations.

Marko Mäkelä added a comment - 2022-12-05 13:32 The second UPDATE appears to trigger a change buffer merge. With the following, I got 79 INSERT operations and 3700 delete-mark operations buffered for 90 pages: diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test index 9362f8daffa..00d766cbe72 100644 --- a/mysql-test/suite/innodb/t/ibuf_not_empty.test +++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test @@ -32,11 +32,14 @@ SET GLOBAL innodb_change_buffering=all; # Create enough rows for the table, so that the change buffer will be # used for modifying the secondary index page. There must be multiple # index pages, because changes to the root page are never buffered. -INSERT INTO t1 SELECT 0,'x',1 FROM seq_1_to_1024; +INSERT INTO t1 SELECT 0,'x',1 FROM seq_1_to_4096; +UPDATE t1 SET b='y'; + let MYSQLD_DATADIR=`select @@datadir`; let PAGE_SIZE=`select @@innodb_page_size`; --source include/shutdown_mysqld.inc +stop already here; # Corrupt the change buffer bitmap, to claim that pages are clean perl; The upgrade was successful, and CHECK TABLE did not report any errors. One more test with 65,536 rows: 193 pages, 33,426 delete-mark operations, 7 insert operations, no purge operations.

Matthias Leich added a comment - 2022-12-06 10:13

Scenario:

1. Start the server with the combination of innodb_fast_shutdown and innodb_change_buffering values mentioned.

2. Create a InnoDB table and fill it with records

3. Shutdown+Restart

4. CHECK TABLE test.t1 EXTENDED

OLD_MARIADB: origin/10.6 commit 6d40274f65b8d145fbf496e9b1b1d46f258de227 (HEAD, origin/10.6)

NEW_MARIADB: origin/bb-10.11-MDEV-29694 commit acc6ac8f6fc460df3d551f61fcdff5f5d0276cac

FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'inserts'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'none'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'deletes'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'purges'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'changes'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'all'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'inserts'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'none'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'deletes'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'purges'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'changes'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'all'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'inserts'

Table   Op  Msg_type    Msg_text

test.t1 check   Error   Unknown storage engine 'InnoDB'

test.t1 check   error   Corrupt

2022-12-05 11:15:01 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.

2022-12-05 11:15:01 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error

2022-12-05 11:15:01 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-12-05 11:15:01 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'none'

Table   Op  Msg_type    Msg_text

test.t1 check   Error   Unknown storage engine 'InnoDB'

test.t1 check   error   Corrupt

2022-12-05 11:15:09 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.

2022-12-05 11:15:09 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error

2022-12-05 11:15:09 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-12-05 11:15:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'deletes'

Table   Op  Msg_type    Msg_text

test.t1 check   Error   Unknown storage engine 'InnoDB'

test.t1 check   error   Corrupt

2022-12-05 11:15:17 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.

2022-12-05 11:15:17 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error

2022-12-05 11:15:17 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-12-05 11:15:17 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'purges'

Table   Op  Msg_type    Msg_text

test.t1 check   Error   Unknown storage engine 'InnoDB'

test.t1 check   error   Corrupt

2022-12-05 11:15:25 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.

2022-12-05 11:15:25 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error

2022-12-05 11:15:25 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-12-05 11:15:25 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'changes'

Table   Op  Msg_type    Msg_text

test.t1 check   Error   Unknown storage engine 'InnoDB'

test.t1 check   error   Corrupt

2022-12-05 11:15:33 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.

2022-12-05 11:15:33 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error

2022-12-05 11:15:33 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-12-05 11:15:33 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'all'

Table   Op  Msg_type    Msg_text

test.t1 check   Error   Unknown storage engine 'InnoDB'

test.t1 check   error   Corrupt

2022-12-05 11:15:41 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.

2022-12-05 11:15:41 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error

2022-12-05 11:15:41 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-12-05 11:15:41 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'inserts'

able   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'none'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'deletes'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'purges'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'changes'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'all'

Table   Op  Msg_type    Msg_text

test.t1 check   status  OK

Matthias Leich added a comment - 2022-12-06 10:13 Scenario: 1. Start the server with the combination of innodb_fast_shutdown and innodb_change_buffering values mentioned. 2. Create a InnoDB table and fill it with records 3. Shutdown+Restart 4. CHECK TABLE test.t1 EXTENDED OLD_MARIADB: origin/10.6 commit 6d40274f65b8d145fbf496e9b1b1d46f258de227 (HEAD, origin/10.6) NEW_MARIADB: origin/bb-10.11-MDEV-29694 commit acc6ac8f6fc460df3d551f61fcdff5f5d0276cac FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'inserts' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'none' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'deletes' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'purges' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'changes' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 0 -- CHANGE_BUFFERING: 'all' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'inserts' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'none' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'deletes' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'purges' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'changes' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 1 -- CHANGE_BUFFERING: 'all' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'inserts' Table Op Msg_type Msg_text test.t1 check Error Unknown storage engine 'InnoDB' test.t1 check error Corrupt 2022-12-05 11:15:01 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier. 2022-12-05 11:15:01 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error 2022-12-05 11:15:01 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-12-05 11:15:01 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'none' Table Op Msg_type Msg_text test.t1 check Error Unknown storage engine 'InnoDB' test.t1 check error Corrupt 2022-12-05 11:15:09 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier. 2022-12-05 11:15:09 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error 2022-12-05 11:15:09 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-12-05 11:15:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'deletes' Table Op Msg_type Msg_text test.t1 check Error Unknown storage engine 'InnoDB' test.t1 check error Corrupt 2022-12-05 11:15:17 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier. 2022-12-05 11:15:17 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error 2022-12-05 11:15:17 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-12-05 11:15:17 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'purges' Table Op Msg_type Msg_text test.t1 check Error Unknown storage engine 'InnoDB' test.t1 check error Corrupt 2022-12-05 11:15:25 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier. 2022-12-05 11:15:25 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error 2022-12-05 11:15:25 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-12-05 11:15:25 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'changes' Table Op Msg_type Msg_text test.t1 check Error Unknown storage engine 'InnoDB' test.t1 check error Corrupt 2022-12-05 11:15:33 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier. 2022-12-05 11:15:33 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error 2022-12-05 11:15:33 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-12-05 11:15:33 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. FAST_SHUTDOWN: 2 -- CHANGE_BUFFERING: 'all' Table Op Msg_type Msg_text test.t1 check Error Unknown storage engine 'InnoDB' test.t1 check error Corrupt 2022-12-05 11:15:41 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier. 2022-12-05 11:15:41 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1562] with error Generic error 2022-12-05 11:15:41 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-12-05 11:15:41 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'inserts' able Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'none' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'deletes' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'purges' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'changes' Table Op Msg_type Msg_text test.t1 check status OK FAST_SHUTDOWN: 3 -- CHANGE_BUFFERING: 'all' Table Op Msg_type Msg_text test.t1 check status OK

Matthias Leich added a comment - 2022-12-13 10:00

origin/bb-10.11-new-innodb-defaults c434f870a346fb11a6b5932dbdf860ed7e2d2f74 2022-12-12T10:05:22+02:00
which contains ~~MDEV-29986~~, ~~MDEV-19506~~, ~~MDEV-29694~~, ~~MDEV-30136~~, ~~MDEV-29983~~
performed well in RQG testing. No new problems

Matthias Leich added a comment - 2022-12-13 10:00 origin/bb-10.11-new-innodb-defaults c434f870a346fb11a6b5932dbdf860ed7e2d2f74 2022-12-12T10:05:22+02:00 which contains MDEV-29986 , MDEV-19506 , MDEV-29694 , MDEV-30136 , MDEV-29983 performed well in RQG testing. No new problems

Marko Mäkelä added a comment - 2023-01-25 09:20 - edited

It is worth noting that starting with ~~MDEV-30400~~, the test innodb.ibuf_not_empty no longer actually uses the change buffer, because the debug parameter innodb_change_buffering_debug lost many ‘teeth’ because of the locking changes. I retested an upgrade from 10.8 right before the merge to 11.0 with MDEV-30400, using my above patch to the 10.8 innodb.ibuf_not_empty test:

2023-01-25  9:58:03 0 [Note] InnoDB: Upgrading the change buffer

2023-01-25  9:58:03 0 [Note] InnoDB: Upgraded the change buffer: 1 tablespaces, 14 pages

Marko Mäkelä added a comment - 2023-01-25 09:20 - edited It is worth noting that starting with MDEV-30400 , the test innodb.ibuf_not_empty no longer actually uses the change buffer, because the debug parameter innodb_change_buffering_debug lost many ‘teeth’ because of the locking changes. I retested an upgrade from 10.8 right before the merge to 11.0 with MDEV-30400 , using my above patch to the 10.8 innodb.ibuf_not_empty test: 2023-01-25 9:58:03 0 [Note] InnoDB: Upgrading the change buffer 2023-01-25 9:58:03 0 [Note] InnoDB: Upgraded the change buffer: 1 tablespaces, 14 pages

Marko Mäkelä added a comment - 2023-09-08 07:54

Users should beware of dormant corruption, such as that one caused by ~~MDEV-32132~~ a.k.a. ~~MDEV-30009~~. We got a case of that in ~~MDEV-32044~~.

greenman, I think that we must amend the recommended upgrade procedure from 10.x to 11.x along the following lines:

If the change buffer had been enabled in the past (~~MDEV-27734~~ disabled it by default in 10.5.15, 10.6.7), and
if DROP INDEX and CREATE INDEX had been executed such that buffered changes existed for the index that had been dropped,
it would best to execute a shutdown with innodb_fast_shutdown=0 before upgrading, to prevent corruption of data.

The upgrade procedure ignores the "change buffer bitmaps" and trusts that all records in the change buffer are valid. The slow shutdown procedure of 10.4 (~~MDEV-20934~~; revised in ~~MDEV-30009~~ for 10.5 and later) will discard bogus entries for pages to which there are exist buffered changes according to the change buffer bitmap.

We might choose to revise the upgrade logic in 11.0 to work in a similar fashion. Until or unless that is done, I think that we must document this.

Marko Mäkelä added a comment - 2023-09-08 07:54 Users should beware of dormant corruption, such as that one caused by MDEV-32132 a.k.a. MDEV-30009 . We got a case of that in MDEV-32044 . greenman , I think that we must amend the recommended upgrade procedure from 10.x to 11.x along the following lines: If the change buffer had been enabled in the past ( MDEV-27734 disabled it by default in 10.5.15, 10.6.7), and if DROP INDEX and CREATE INDEX had been executed such that buffered changes existed for the index that had been dropped, it would best to execute a shutdown with innodb_fast_shutdown=0 before upgrading, to prevent corruption of data. The upgrade procedure ignores the "change buffer bitmaps" and trusts that all records in the change buffer are valid. The slow shutdown procedure of 10.4 ( MDEV-20934 ; revised in MDEV-30009 for 10.5 and later) will discard bogus entries for pages to which there are exist buffered changes according to the change buffer bitmap. We might choose to revise the upgrade logic in 11.0 to work in a similar fashion. Until or unless that is done, I think that we must document this.

Marko Mäkelä added a comment - 2023-09-19 14:40

My plan is to revise the upgrade logic in ~~MDEV-32044~~.

Marko Mäkelä added a comment - 2023-09-19 14:40 My plan is to revise the upgrade logic in MDEV-32044 .

People

Assignee:: Marko Mäkelä

Reporter:: Marko Mäkelä

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2022-10-04 06:38

Updated:: 2024-12-11 15:34

Resolved:: 2023-01-11 16:11

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.