Debian Jessie (MariaDB 10.0) / Debian Stretch (MariaDB 10.1)

FYI, I just tested vanilla 10.1.28+maria-1~stretch instead of Debian's 10.1.26-0+deb9u1 but the error still persists. Thus it's not a Debian-only issue.

Thanks for the report.
Apparently, the fix in MDEV-11623 did not cover upgrade from 5.5. Reproducible easily:

• bootstrap current 5.5 server with --innodb-page-size=8192, otherwise defaults;
• start the server with --innodb-page-size=8192;
• create a table (innodb_file_per_table value does not matter);
• shut down the server properly;
• try to start any current 10.x on the same directory with --innodb-page-size=8192 --innodb_data_file_path=ibdata1:10M:autoextend, otherwise defaults (given innodb_data_file_path is the default value of 5.5, it is here to get rid of the auto-extending error which happens upon upgrade from 5.5)

171030 23:46:57 [ERROR] InnoDB: innodb-page-size mismatch in data file ./ibdata1

171030 23:46:57 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!

171030 23:46:57 [ERROR] Plugin 'InnoDB' init function returned error.

171030 23:46:57 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

171030 23:46:57 [ERROR] Unknown/unsupported storage engine: InnoDB

171030 23:46:57 [ERROR] Aborting

2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Data file './ibdata1' uses page size 16384, but the innodb_page_size start-up parameter is 8192

2017-10-30 23:51:11 140576015199104 [Note] InnoDB: Restoring page [page id: space=0, page number=0] of datafile './ibdata1' from the doublewrite buffer. Writing 16384 bytes into file './ibdata1'

2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Data file './ibdata1' uses page size 16384, but the innodb_page_size start-up parameter is 8192

2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1948] with error Generic error

5.5 datadir pre-created in the described way is attached.

This message caught my attention:

171106 15:23:49 [Note] InnoDB: Restoring page 0 of tablespace 0

In the attached dataset, the doublewrite buffer contains a copy for the first page of the data file.
The doublewrite buffer comprises pages 128 to 256:

0000bf40: 1432 1fff bd5f 0000 0080 0000 0100 1fff  .2..._..........

0000bf50: bd5f 0000 0080 0000 0100 6a50 73aa 0000  ._........jPs...

There are quite a few doublewrite entries for the system tablespace:

od -Ax -t x1 -j 0x100000 -N 0x100000 data/ibdata1 |grep -B 2 '[02468ace]020 .. .. 00 00 00 00'

There is only one copy of the first page:

104000 45 b3 f8 be 00 00 00 00 00 00 00 00 00 00 00 00

104010 00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 00

104020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Note the FIL_PAGE_LSN=0x0dba60.
The page contents differ a little:

2,3c2,3

<  00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 0d

<  ba 67 00 00 00 00 00 00 00 00 00 00 00 00 00 00

---

>  00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 00

>  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Same FIL_PAGE_LSN, but the FIL_PAGE_FILE_FLUSH_LSN differs, because when it is written at shutdown, the doublewrite buffer is being bypassed.
But this does not explain the failure.

The failure seems to be worse. Let us look at fil_user_tablespace_restore_page():

        flags = mach_read_from_4(FSP_HEADER_OFFSET + FSP_SPACE_FLAGS + page);

	zip_size = fsp_flags_get_zip_size(flags);

	page_size = fsp_flags_get_page_size(flags);

The field flags=0 indicates that the tablespace is in the old innodb_page_size=16k format.

An XtraDB start-up parameter innodb_page_size was introduced in MariaDB 5.1.47, 5.2.1, 5.3.0, 5.5.20.
When the parameter innodb_page_size was introduced in MySQL 5.6, the FSP_SPACE_FLAGS in the tablespace header would indicate the page size of each data file. But, in the XtraDB implementation, the flags would be 0, so it would be hard to determine the actual page size of the files (and outright impossible if the server was killed soon after creating some .ibd file, with all pages except the first one being filled with zero bytes).

Theoretically, an upgrade from 5.5 with non-default innodb_page_size is possible if the tablespace flags at byte offset 54 are adjusted. In this case, I believe that the correct flags would be 256, which corresponds to 4<<6, for 512<<4 = 8192.
Indeed, if I change the byte at offset 56 from 0 to 1 (changing the 32-bit big endian flags at offset 54 from 0 to 0x100), and if I also change the 32-bit big endian checksum fields at offset 0 and 0x1ff8 to the magic value 0xdeadbeef, I will be able to start up the 10.0 server:

171106 16:22:33 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.2 started; log sequence number 899687

I wonder how upstream supports such upgrades. Maybe not at all? For another compatibility problem that XtraDB 5.5 introduced (innodb_fast_checksum) there exists a work-around of rewriting all page checksums by using the MariaDB 10.2 innochecksum tool.

A possible fix would be to introduce an innochecksum option for updating the tablespace flags.

According to the MariaDB documentation on InnoDB and XtraDB parameters, Percona deprecated the parameter innodb_page_size in XtraDB 5.5.30-30.2. This seemed to ‘fix’ the upgrade problem for them.

It is technically possible to implement an option in the innochecksum tool that is included in MariaDB 10.2 or later, to set the tablespace flags in the first page of each InnoDB tablespace. But given that the option was marked experimental and also deprecated in XtraDB 5.5, I would prefer to not do that unless requested by a customer.