Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14169

MariaDB 10.0/10.1 fail to start with datadir created by MariaDB 5.5 using innodb-page-size != 16K

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Won't Fix
    • 10.1.26, 10.0.32, 10.0(EOL), 10.1(EOL), 10.2(EOL)
    • N/A
    • Debian Jessie (MariaDB 10.0) / Debian Stretch (MariaDB 10.1)
    • 10.1.29

    Description

      I'm trying to migrate from MariaDB 5.5 to 10.1 via 10.0 with innodb-page-size = 8192.

      I first got this error when I migrated to 10.0.32 and only then came across the original bug report. Now I still get the same known error, despite having upgraded to 10.1.26 and the original error said to be fixed in 10.1.21:

      2017-10-27 11:35:23 139861476192832 [Note] InnoDB:  innodb-page-size has been changed from the default value 16384 to 8192 .
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: The InnoDB memory heap is disabled
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: Compressed tables use zlib 1.2.8
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: Using Linux native AIO
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: Using SSE crc32 instructions
      2017-10-27 11:35:23 139861476192832 [Note] InnoDB: Initializing buffer pool, size = 192.0G
      2017-10-27 11:35:34 139861476192832 [Note] InnoDB: Completed initialization of buffer pool
      InnoDB: Error: Current page size 8192 !=  page size on page 16384
      2017-10-27 11:35:36 139861476192832 [Note] InnoDB: Restoring page 0 of tablespace 0
      2017-10-27 11:35:36 139861476192832 [Warning] InnoDB: Doublewrite does not have page_no=0 of space: 0
      2017-10-27 11:35:36 139861476192832 [ERROR] InnoDB: innodb-page-size mismatch in data file /db/data/ibdata1
      2017-10-27 11:35:36 139861476192832 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
      2017-10-27 11:35:36 139861476192832 [ERROR] Plugin 'InnoDB' init function returned error.
      2017-10-27 11:35:36 139861476192832 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      

      Attachments

        1. data.tar.gz
          427 kB
          Elena Stepanova

        Issue Links

          Activity

            ligoob Oliver Bock added a comment -

            FYI, I just tested vanilla 10.1.28+maria-1~stretch instead of Debian's 10.1.26-0+deb9u1 but the error still persists. Thus it's not a Debian-only issue.

            ligoob Oliver Bock added a comment - FYI, I just tested vanilla 10.1.28+maria-1~stretch instead of Debian's 10.1.26-0+deb9u1 but the error still persists. Thus it's not a Debian-only issue.

            Thanks for the report.
            Apparently, the fix in MDEV-11623 did not cover upgrade from 5.5. Reproducible easily:

            • bootstrap current 5.5 server with --innodb-page-size=8192, otherwise defaults;
            • start the server with --innodb-page-size=8192;
            • create a table (innodb_file_per_table value does not matter);
            • shut down the server properly;
            • try to start any current 10.x on the same directory with --innodb-page-size=8192 --innodb_data_file_path=ibdata1:10M:autoextend, otherwise defaults (given innodb_data_file_path is the default value of 5.5, it is here to get rid of the auto-extending error which happens upon upgrade from 5.5)

            10.0 c3592ca7b says

            171030 23:46:57 [ERROR] InnoDB: innodb-page-size mismatch in data file ./ibdata1
            171030 23:46:57 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
            171030 23:46:57 [ERROR] Plugin 'InnoDB' init function returned error.
            171030 23:46:57 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
            171030 23:46:57 [ERROR] Unknown/unsupported storage engine: InnoDB
            171030 23:46:57 [ERROR] Aborting
            

            10.2 7cca0df0d says

            2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Data file './ibdata1' uses page size 16384, but the innodb_page_size start-up parameter is 8192
            2017-10-30 23:51:11 140576015199104 [Note] InnoDB: Restoring page [page id: space=0, page number=0] of datafile './ibdata1' from the doublewrite buffer. Writing 16384 bytes into file './ibdata1'
            2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Data file './ibdata1' uses page size 16384, but the innodb_page_size start-up parameter is 8192
            2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1948] with error Generic error
            

            5.5 datadir pre-created in the described way is attached.

            elenst Elena Stepanova added a comment - Thanks for the report. Apparently, the fix in MDEV-11623 did not cover upgrade from 5.5. Reproducible easily: bootstrap current 5.5 server with --innodb-page-size=8192 , otherwise defaults; start the server with --innodb-page-size=8192 ; create a table ( innodb_file_per_table value does not matter); shut down the server properly; try to start any current 10.x on the same directory with --innodb-page-size=8192 --innodb_data_file_path=ibdata1:10M:autoextend , otherwise defaults (given innodb_data_file_path is the default value of 5.5, it is here to get rid of the auto-extending error which happens upon upgrade from 5.5) 10.0 c3592ca7b says 171030 23:46:57 [ERROR] InnoDB: innodb-page-size mismatch in data file ./ibdata1 171030 23:46:57 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 171030 23:46:57 [ERROR] Plugin 'InnoDB' init function returned error. 171030 23:46:57 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 171030 23:46:57 [ERROR] Unknown/unsupported storage engine: InnoDB 171030 23:46:57 [ERROR] Aborting 10.2 7cca0df0d says 2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Data file './ibdata1' uses page size 16384, but the innodb_page_size start-up parameter is 8192 2017-10-30 23:51:11 140576015199104 [Note] InnoDB: Restoring page [page id: space=0, page number=0] of datafile './ibdata1' from the doublewrite buffer. Writing 16384 bytes into file './ibdata1' 2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Data file './ibdata1' uses page size 16384, but the innodb_page_size start-up parameter is 8192 2017-10-30 23:51:11 140576015199104 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1948] with error Generic error 5.5 datadir pre-created in the described way is attached.

            This message caught my attention:

            171106 15:23:49 [Note] InnoDB: Restoring page 0 of tablespace 0
            

            In the attached dataset, the doublewrite buffer contains a copy for the first page of the data file.
            The doublewrite buffer comprises pages 128 to 256:

            0000bf40: 1432 1fff bd5f 0000 0080 0000 0100 1fff  .2..._..........
            0000bf50: bd5f 0000 0080 0000 0100 6a50 73aa 0000  ._........jPs...
            

            There are quite a few doublewrite entries for the system tablespace:

            od -Ax -t x1 -j 0x100000 -N 0x100000 data/ibdata1 |grep -B 2 '[02468ace]020 .. .. 00 00 00 00'
            

            There is only one copy of the first page:

            104000 45 b3 f8 be 00 00 00 00 00 00 00 00 00 00 00 00
            104010 00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 00
            104020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
            

            Note the FIL_PAGE_LSN=0x0dba60.
            The page contents differ a little:

            diff <(od -An -t x1 -N 8192 data/ibdata) <(od -An -t x1 -N 8192 -j 0x104000 data/ibdata1)

            2,3c2,3
            <  00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 0d
            <  ba 67 00 00 00 00 00 00 00 00 00 00 00 00 00 00
            ---
            >  00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 00
            >  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
            

            Same FIL_PAGE_LSN, but the FIL_PAGE_FILE_FLUSH_LSN differs, because when it is written at shutdown, the doublewrite buffer is being bypassed.
            But this does not explain the failure.

            The failure seems to be worse. Let us look at fil_user_tablespace_restore_page():

                    flags = mach_read_from_4(FSP_HEADER_OFFSET + FSP_SPACE_FLAGS + page);
            	zip_size = fsp_flags_get_zip_size(flags);
            	page_size = fsp_flags_get_page_size(flags);
            

            The field flags=0 indicates that the tablespace is in the old innodb_page_size=16k format.

            An XtraDB start-up parameter innodb_page_size was introduced in MariaDB 5.1.47, 5.2.1, 5.3.0, 5.5.20.
            When the parameter innodb_page_size was introduced in MySQL 5.6, the FSP_SPACE_FLAGS in the tablespace header would indicate the page size of each data file. But, in the XtraDB implementation, the flags would be 0, so it would be hard to determine the actual page size of the files (and outright impossible if the server was killed soon after creating some .ibd file, with all pages except the first one being filled with zero bytes).

            Theoretically, an upgrade from 5.5 with non-default innodb_page_size is possible if the tablespace flags at byte offset 54 are adjusted. In this case, I believe that the correct flags would be 256, which corresponds to 4<<6, for 512<<4 = 8192.
            Indeed, if I change the byte at offset 56 from 0 to 1 (changing the 32-bit big endian flags at offset 54 from 0 to 0x100), and if I also change the 32-bit big endian checksum fields at offset 0 and 0x1ff8 to the magic value 0xdeadbeef, I will be able to start up the 10.0 server:

            171106 16:22:33 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.2 started; log sequence number 899687
            

            I wonder how upstream supports such upgrades. Maybe not at all? For another compatibility problem that XtraDB 5.5 introduced (innodb_fast_checksum) there exists a work-around of rewriting all page checksums by using the MariaDB 10.2 innochecksum tool.

            A possible fix would be to introduce an innochecksum option for updating the tablespace flags.

            marko Marko Mäkelä added a comment - This message caught my attention: 171106 15:23:49 [Note] InnoDB: Restoring page 0 of tablespace 0 In the attached dataset, the doublewrite buffer contains a copy for the first page of the data file. The doublewrite buffer comprises pages 128 to 256: 0000bf40: 1432 1fff bd5f 0000 0080 0000 0100 1fff .2..._.......... 0000bf50: bd5f 0000 0080 0000 0100 6a50 73aa 0000 ._........jPs... There are quite a few doublewrite entries for the system tablespace: od -Ax -t x1 -j 0x100000 -N 0x100000 data/ibdata1 |grep -B 2 '[02468ace]020 .. .. 00 00 00 00' There is only one copy of the first page: 104000 45 b3 f8 be 00 00 00 00 00 00 00 00 00 00 00 00 104010 00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 00 104020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Note the FIL_PAGE_LSN=0x0dba60. The page contents differ a little: diff <(od -An -t x1 -N 8192 data/ibdata) <(od -An -t x1 -N 8192 -j 0x104000 data/ibdata1) 2,3c2,3 < 00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 0d < ba 67 00 00 00 00 00 00 00 00 00 00 00 00 00 00 --- > 00 00 00 00 00 0d ba 60 00 08 00 00 00 00 00 00 > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Same FIL_PAGE_LSN, but the FIL_PAGE_FILE_FLUSH_LSN differs, because when it is written at shutdown, the doublewrite buffer is being bypassed. But this does not explain the failure. The failure seems to be worse. Let us look at fil_user_tablespace_restore_page(): flags = mach_read_from_4(FSP_HEADER_OFFSET + FSP_SPACE_FLAGS + page); zip_size = fsp_flags_get_zip_size(flags); page_size = fsp_flags_get_page_size(flags); The field flags=0 indicates that the tablespace is in the old innodb_page_size=16k format. An XtraDB start-up parameter innodb_page_size was introduced in MariaDB 5.1.47, 5.2.1, 5.3.0, 5.5.20. When the parameter innodb_page_size was introduced in MySQL 5.6, the FSP_SPACE_FLAGS in the tablespace header would indicate the page size of each data file. But, in the XtraDB implementation, the flags would be 0, so it would be hard to determine the actual page size of the files (and outright impossible if the server was killed soon after creating some .ibd file, with all pages except the first one being filled with zero bytes). Theoretically, an upgrade from 5.5 with non-default innodb_page_size is possible if the tablespace flags at byte offset 54 are adjusted. In this case, I believe that the correct flags would be 256, which corresponds to 4<<6, for 512<<4 = 8192. Indeed, if I change the byte at offset 56 from 0 to 1 (changing the 32-bit big endian flags at offset 54 from 0 to 0x100), and if I also change the 32-bit big endian checksum fields at offset 0 and 0x1ff8 to the magic value 0xdeadbeef, I will be able to start up the 10.0 server: 171106 16:22:33 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.2 started; log sequence number 899687 I wonder how upstream supports such upgrades. Maybe not at all? For another compatibility problem that XtraDB 5.5 introduced (innodb_fast_checksum) there exists a work-around of rewriting all page checksums by using the MariaDB 10.2 innochecksum tool. A possible fix would be to introduce an innochecksum option for updating the tablespace flags.

            According to the MariaDB documentation on InnoDB and XtraDB parameters, Percona deprecated the parameter innodb_page_size in XtraDB 5.5.30-30.2. This seemed to ‘fix’ the upgrade problem for them.

            It is technically possible to implement an option in the innochecksum tool that is included in MariaDB 10.2 or later, to set the tablespace flags in the first page of each InnoDB tablespace. But given that the option was marked experimental and also deprecated in XtraDB 5.5, I would prefer to not do that unless requested by a customer.

            marko Marko Mäkelä added a comment - According to the MariaDB documentation on InnoDB and XtraDB parameters , Percona deprecated the parameter innodb_page_size in XtraDB 5.5.30-30.2. This seemed to ‘fix’ the upgrade problem for them. It is technically possible to implement an option in the innochecksum tool that is included in MariaDB 10.2 or later, to set the tablespace flags in the first page of each InnoDB tablespace. But given that the option was marked experimental and also deprecated in XtraDB 5.5, I would prefer to not do that unless requested by a customer.

            People

              marko Marko Mäkelä
              ligoob Oliver Bock
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.