Summary of Aug 14th (10.3 revision 347d9456e4eb)
10.1.20 => 10.3
Failed on the 2nd step, old server restart with ifr=3 after crash
Unrelated to upgrade. I assume it's a 10.1.20 problem fixed in recent 10.1
# 2017-08-14T09:33:12 [43513] | 49 InnoDB: Error: trying to access page number 1 in space 10,
|
# 2017-08-14T09:33:12 [43513] | InnoDB: space name test/t5,
|
# 2017-08-14T09:33:12 [43513] | InnoDB: which is outside the tablespace bounds.
|
# 2017-08-14T09:33:12 [43513] | InnoDB: Byte offset 0, len 65536, i/o type 10.
|
# 2017-08-14T09:33:12 [43513] | InnoDB: If you get this error at mysqld startup, please check that
|
# 2017-08-14T09:33:12 [43513] | InnoDB: your my.cnf matches the ibdata files that you have in the
|
# 2017-08-14T09:33:12 [43513] | InnoDB: MySQL server.
|
# 2017-08-14T09:33:12 [43513] | 2017-08-14 09:33:11 7f82c9cc1780 InnoDB: Assertion failure in thread 140199708071808 in file fil0fil.cc line 5931
|
10.1 => 10.3
Failed on the 2nd step, old server restart with ifr=3 after crash
Unrelated to upgrade. Need to make a bug report about it for 10.1
# 2017-08-14T09:38:35 [43731] | 2017-08-14 9:38:33 139695152883584 [Note] InnoDB: !!! innodb_force_recovery is set to 3 !!!
|
# 2017-08-14T09:38:35 [43731] | 2017-08-14 09:38:34 7f0d4ff6e780 InnoDB: Assertion failure in thread 139695152883584 in file os0sync.cc line 339
|
# 2017-08-14T09:38:35 [43731] | InnoDB: Failing assertion: event
|
10.1.20 => 10.3, 10.1 => 10.3
Failed on the 3rd step, new server start with read-only
As discussed, the read-only upgrade is not applicable to 10.1 => 10.3. Need to modify the scenario
# 2017-08-14T09:49:30 [44160] | 2017-08-14 9:49:29 140171267110720 [Note] InnoDB: Completed initialization of buffer pool
|
# 2017-08-14T09:49:30 [44160] | 2017-08-14 9:49:30 140171267110720 [ERROR] InnoDB: Cannot create system tables in read-only mode
|
# 2017-08-14T09:49:30 [44160] | 2017-08-14 9:49:30 140171267110720 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2580] with error Read only transaction
|
# 2017-08-14T09:49:30 [44160] | 2017-08-14 9:49:30 140171267110720 [Note] InnoDB: Starting shutdown...
|
10.2 => 10.3, 10.3.0 => 10.3
Failed after the 4th step
Unrelated to upgrade. MDEV-13512
2017-08-14 11:40:30 0x7fd2fdc8a700 InnoDB: Assertion failure in file /home/elenst/git/10.3/storage/innobase/page/page0zip.cc line 3075
|
InnoDB: Failing assertion: !memcmp(FIL_PAGE_TYPE + page, FIL_PAGE_TYPE + page_zip->data, PAGE_HEADER - FIL_PAGE_TYPE)
|
10.1.20 => 10.3
Failed on 2nd step
Unrelated to upgrade. Might be 10.1.20 failure only, need to try to get it on recent trees.
# 2017-08-14T05:45:50 [31943] | InnoDB: Warning: database page corruption or a failed
|
# 2017-08-14T05:45:50 [31943] | InnoDB: file read of space 0 page 624.
|
# 2017-08-14T05:45:50 [31943] | InnoDB: Trying to recover it from the doublewrite buffer.
|
# 2017-08-14T05:45:50 [31943] | InnoDB: Dump of the page:
|
# 2017-08-14T05:45:50 [31943] | 2017-08-14 05:45:49 7f54eb5ab780 InnoDB: Page dump in ascii and hex (8192 bytes):
|
...
|
# 2017-08-14T05:45:50 [31943] | 2017-08-14 05:45:49 7f54eb5ab780 InnoDB: uncompressed page, stored checksum in field1 3887984507, calculated checksums for field1: crc32 3550797666, innodb 1228661961, none 3735928559, stored checksum in field2 44434364, calculated checksums for field2: crc32 3550797666, innodb 44434364, none 3735928559, page LSN 0 10495857, low 4 bytes of LSN at page end 10495857, page number (if stored to page already) 624, space id (if created with >= MySQL-4.1.1 and stored already) 0
|
# 2017-08-14T05:45:50 [31943] | InnoDB: page type 2 meaning UNDO LOG
|
# 2017-08-14T05:45:50 [31943] | InnoDB: Also the page in the doublewrite buffer is corrupt.
|
# 2017-08-14T05:45:50 [31943] | InnoDB: Cannot continue operation.
|
# 2017-08-14T05:45:50 [31943] | InnoDB: You can try to recover the database with the my.cnf
|
# 2017-08-14T05:45:50 [31943] | InnoDB: option:
|
# 2017-08-14T05:45:50 [31943] | InnoDB: innodb_force_recovery=6
|
# 2017-08-14T05:45:50 [31943] | 2017-08-14 05:45:49 7f54eb5ab780 InnoDB: Assertion failure in thread 140002702571392 in file buf0dblwr.cc line 601
|
10.3 => 10.3
Failed on 2nd step
Unrelated to upgrade. Something wrong with crash recovery.
mysqld: /home/elenst/git/10.3/storage/innobase/btr/btr0cur.cc:3800: dberr_t btr_cur_optimistic_update(ulint, btr_cur_t*, ulint**, mem_heap_t**, const upd_t*, ulint, que_thr_t*, trx_id_t, mtr_t*): Assertion `trx_id > 0 || (flags & BTR_KEEP_SYS_FLAG)' failed.
|
170814 5:50:45 [ERROR] mysqld got signal 6 ;
|
10.3 => 10.3
Failed on 2nd step
Unrelated to upgrade. MDEV-13103
# 2017-08-14T08:01:50 [39117] | 2017-08-14 8:01:48 139991503341376 [Note] InnoDB: Starting crash recovery from checkpoint LSN=9840311
|
# 2017-08-14T08:01:50 [39117] | 2017-08-14 8:01:49 139991503341376 [ERROR] InnoDB: Corruption: Page is marked as compressed space: 40 name: test/t043 but uncompress failed with error: -3 size: 4920 len: 8192 compression method: ZLIB.
|
# 2017-08-14T08:01:50 [39117] | 2017-08-14 8:01:49 139991503341376 [Note] InnoDB: Page dump in ascii and hex (8192 bytes):
|
...
|
# 2017-08-14T08:01:50 [39117] | mysqld: /home/elenst/git/10.3/storage/innobase/buf/buf0buf.cc:1355: void buf_page_print(const byte*, const page_
|
size_t&, ulint): Assertion `flags & BUF_PAGE_PRINT_NO_CRASH' failed.
|
|
|
Undo upgrade tests have been set up for running regularly in travis, in 10.x-innodb-upgrade branches (along with other kinds of upgrade and recovery tests):
https://travis-ci.org/elenst/mariadb-tests/branches
The algorithm is pretty much the same as in the initial description, with one change – instead of running with innodb-read-only, the server on the step 3 is just made unavailable for client connections, it seems to be a more realistic scenario and does not set unnecessary limitations.
The scenario is implemented via RQG as upgrade-test=undo, which triggers UpgradeUndoLogs reporter. The code is in github.com/elenst/rqg, branch experimental (which will be merged into master with time).
As of now (Sep 8th, 2017), the current combinations of undo tests are running:
10.0 branch
- upgrade from the latest released 10.0, page sizes 4K, 8K, 16K;
- upgrade from 10.0.18, page sizes 4K, 8K, 16K;
- upgrade from the latest released MySQL 5.6, page sizes 4K, 8K, 16K
10.1 branch
- upgrade from the latest released 10.1, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from 10.1.22, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from the latest released 10.0, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
- upgrade from 10.0.18, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
- upgrade from the latest released MySQL 5.6, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
10.2 branch
- upgrade from the latest released 10.2, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from 10.2.6, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from the latest released 10.1, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from 10.1.22, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from the latest released 10.0, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
- upgrade from 10.0.18, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
- upgrade from the latest released MySQL 5.7, page sizes 4K, 8K, 16K, 32K, 64K, encryption turn-on*/OFF;
- upgrade from the latest released MySQL 5.6, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
10.3 branch
- upgrade from the latest released 10.3, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from the latest released 10.2, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from 10.2.6, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from the latest released 10.1, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from 10.1.22, page sizes 4K, 8K, 16K, 32K, 64K, encryption ON/OFF, compression none/zlib;
- upgrade from the latest released 10.0, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
- upgrade from 10.0.18, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
- upgrade from the latest released MySQL 5.7, page sizes 4K, 8K, 16K, 32K, 64K, encryption turn-on*/OFF;
- upgrade from the latest released MySQL 5.6, page sizes 4K, 8K, 16K, encryption turn-on*/OFF;
—
* encryption turn-on means that encryption was off (or unavailable) on the old server, but ON on the new server
Tests are currently affected by MDEV-13103, MDEV-13112, MDEV-13512, MDEV-13165, and MySQL 5.6 sporadically hanging on shutdown.
Further improvements and additions to the tests will be made outside the scope of this task.
|