Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12600

crash during install_db with innodb_page_size=32K and ibdata1=3M;

Details

    Description

      May be related to MDEV-12113 , just this is without encryption.
      When trying to install_db in 10.1 with :
      innodb_page_size=32k
      innodb_data_file_path=ibdata1:3M;ibdata2:10M:autoextend

      I am not getting stack resolved - let me know if this is required to troubleshoot:

      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(my_print_stacktrace+0x2e)[0xc0b60e]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(handle_fatal_signal+0x4bf)[0x76902f]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11630)[0x7fd86e014630]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x9f)[0x7fd86cd9a7ef]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fd86cd9c3ea]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0xa9b877]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0xa49a8f]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0xa4f75e]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x94e3a3]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x951768]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x9e2232]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x908304]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5e)[0x76eede]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x5c6008]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(_Z11plugin_initPiPPci+0xbc0)[0x5c6fd0]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x5204ad]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(_Z11mysqld_mainiPPc+0x8c3)[0x5216f3]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fd86cd853f1]
      /home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x516a49]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      

      https://travis-ci.org/AndriiNikitin/mariadb-environs-MDEV-12113/builds/226281024

      Only these configurations do show crash:

      VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      

      the rest pass:

      VERSION=10.1.22 PAGE_SIZE=16k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=16k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend' REST=configure_rest_encryption.sh=1
      VERSION=10.1.22 PAGE_SIZE=8k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=64k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:1M;ibdata2:10M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:11M;ibdata2:10M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:31M;ibdata2:10M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:3M:autoextend'
      VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:1M;ibdata2:3M:autoextend'
      VERSION=10.0.30 PAGE_SIZE=8k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      VERSION=5.5.55 PAGE_SIZE=8k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      

      10.2 reports an (expected) error that 3Mb is too small :
      InnoDB: The innodb_system data file 'ibdata1' must be at least 6 MB

      VERSION=10.2.5 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
      

      Attachments

        Issue Links

          Activity

            After all it looks that several more jobs at the link actually had crash, just mysql_install_db did report 'success', so job marked as green.

            anikitin Andrii Nikitin (Inactive) added a comment - After all it looks that several more jobs at the link actually had crash, just mysql_install_db did report 'success', so job marked as green.

            10.1 935a1c676e1fbfea79

            scripts/mysql_install_db --no-defaults --innodb-data-file-path="ibdata1:3M;ibdata2:10M:autoextend" --innodb-page-size=32K
            ...
            2017-05-01  4:03:37 139673798764416 [Note] InnoDB: Waiting for purge to start
            2017-05-01  4:03:37 139673798764416 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 0
            2017-05-01 04:03:37 7f085728e780  InnoDB: Assertion failure in thread 139673798764416 in file fil0fil.cc line 6430
            InnoDB: Failing assertion: node->size - block_offset >= ((byte_offset + len + (UNIV_PAGE_SIZE - 1)) / UNIV_PAGE_SIZE)
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            170501  4:03:37 [ERROR] mysqld got signal 6 ;
            

            #4  0x00007f590b16c448 in __GI_abort () at abort.c:89
            #5  0x00007f590e014bbe in _fil_io (type=11, sync=true, space_id=0, zip_size=0, block_offset=64, byte_offset=0, len=2097152, buf=0x7f58e6c00000, message=0x0, write_size=0x0, trx=0x0) at /data/src/10.1/storage/xtradb/fil/fil0fil.cc:6428
            #6  0x00007f590dfa4d88 in buf_dblwr_flush_buffered_writes () at /data/src/10.1/storage/xtradb/buf/buf0dblwr.cc:974
            #7  0x00007f590dfadf1b in buf_flush_common (flush_type=BUF_FLUSH_LIST, page_count=100) at /data/src/10.1/storage/xtradb/buf/buf0flu.cc:1860
            #8  0x00007f590dfae7b7 in buf_flush_list (min_n=18446744073709551614, lsn_limit=18446744073709551615, n_processed=0x7fff17f8bb68) at /data/src/10.1/storage/xtradb/buf/buf0flu.cc:2143
            #9  0x00007f590de14861 in log_preflush_pool_modified_pages (new_oldest=18446744073709551615) at /data/src/10.1/storage/xtradb/log/log0log.cc:1893
            #10 0x00007f590de15b49 in log_make_checkpoint_at (lsn=18446744073709551615, write_always=1) at /data/src/10.1/storage/xtradb/log/log0log.cc:2377
            #11 0x00007f590df0a072 in innobase_start_or_create_for_mysql () at /data/src/10.1/storage/xtradb/srv/srv0start.cc:3015
            #12 0x00007f590dd9959d in innobase_init (p=0x7f590a824a70) at /data/src/10.1/storage/xtradb/handler/ha_innodb.cc:4454
            #13 0x00007f590dba8791 in ha_initialize_handlerton (plugin=0x7f590ab12770) at /data/src/10.1/sql/handler.cc:513
            #14 0x00007f590d985af5 in plugin_initialize (tmp_root=0x7fff17f8f880, plugin=0x7f590ab12770, argc=0x7f590ec00770 <remaining_argc>, argv=0x7f590a821428, options_only=false) at /data/src/10.1/sql/sql_plugin.cc:1400
            #15 0x00007f590d9866de in plugin_init (argc=0x7f590ec00770 <remaining_argc>, argv=0x7f590a821428, flags=2) at /data/src/10.1/sql/sql_plugin.cc:1678
            #16 0x00007f590d8a59f2 in init_server_components () at /data/src/10.1/sql/mysqld.cc:5145
            #17 0x00007f590d8a6a5d in mysqld_main (argc=12, argv=0x7f590a821428) at /data/src/10.1/sql/mysqld.cc:5732
            #18 0x00007f590d89c1f0 in main (argc=12, argv=0x7fff17f90618) at /data/src/10.1/sql/main.cc:25
            

            elenst Elena Stepanova added a comment - 10.1 935a1c676e1fbfea79 scripts/mysql_install_db --no-defaults --innodb-data-file-path="ibdata1:3M;ibdata2:10M:autoextend" --innodb-page-size=32K ... 2017-05-01 4:03:37 139673798764416 [Note] InnoDB: Waiting for purge to start 2017-05-01 4:03:37 139673798764416 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 0 2017-05-01 04:03:37 7f085728e780 InnoDB: Assertion failure in thread 139673798764416 in file fil0fil.cc line 6430 InnoDB: Failing assertion: node->size - block_offset >= ((byte_offset + len + (UNIV_PAGE_SIZE - 1)) / UNIV_PAGE_SIZE) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 170501 4:03:37 [ERROR] mysqld got signal 6 ; #4 0x00007f590b16c448 in __GI_abort () at abort.c:89 #5 0x00007f590e014bbe in _fil_io (type=11, sync=true, space_id=0, zip_size=0, block_offset=64, byte_offset=0, len=2097152, buf=0x7f58e6c00000, message=0x0, write_size=0x0, trx=0x0) at /data/src/10.1/storage/xtradb/fil/fil0fil.cc:6428 #6 0x00007f590dfa4d88 in buf_dblwr_flush_buffered_writes () at /data/src/10.1/storage/xtradb/buf/buf0dblwr.cc:974 #7 0x00007f590dfadf1b in buf_flush_common (flush_type=BUF_FLUSH_LIST, page_count=100) at /data/src/10.1/storage/xtradb/buf/buf0flu.cc:1860 #8 0x00007f590dfae7b7 in buf_flush_list (min_n=18446744073709551614, lsn_limit=18446744073709551615, n_processed=0x7fff17f8bb68) at /data/src/10.1/storage/xtradb/buf/buf0flu.cc:2143 #9 0x00007f590de14861 in log_preflush_pool_modified_pages (new_oldest=18446744073709551615) at /data/src/10.1/storage/xtradb/log/log0log.cc:1893 #10 0x00007f590de15b49 in log_make_checkpoint_at (lsn=18446744073709551615, write_always=1) at /data/src/10.1/storage/xtradb/log/log0log.cc:2377 #11 0x00007f590df0a072 in innobase_start_or_create_for_mysql () at /data/src/10.1/storage/xtradb/srv/srv0start.cc:3015 #12 0x00007f590dd9959d in innobase_init (p=0x7f590a824a70) at /data/src/10.1/storage/xtradb/handler/ha_innodb.cc:4454 #13 0x00007f590dba8791 in ha_initialize_handlerton (plugin=0x7f590ab12770) at /data/src/10.1/sql/handler.cc:513 #14 0x00007f590d985af5 in plugin_initialize (tmp_root=0x7fff17f8f880, plugin=0x7f590ab12770, argc=0x7f590ec00770 <remaining_argc>, argv=0x7f590a821428, options_only=false) at /data/src/10.1/sql/sql_plugin.cc:1400 #15 0x00007f590d9866de in plugin_init (argc=0x7f590ec00770 <remaining_argc>, argv=0x7f590a821428, flags=2) at /data/src/10.1/sql/sql_plugin.cc:1678 #16 0x00007f590d8a59f2 in init_server_components () at /data/src/10.1/sql/mysqld.cc:5145 #17 0x00007f590d8a6a5d in mysqld_main (argc=12, argv=0x7f590a821428) at /data/src/10.1/sql/mysqld.cc:5732 #18 0x00007f590d89c1f0 in main (argc=12, argv=0x7fff17f90618) at /data/src/10.1/sql/main.cc:25

            Includes the two other related fixes as you can't test this without fixing the first two first.

            https://github.com/MariaDB/server/tree/bb-10.1-MDEV-12113

            Manual testing (see attached output).

            jplindst Jan Lindström (Inactive) added a comment - Includes the two other related fixes as you can't test this without fixing the first two first. https://github.com/MariaDB/server/tree/bb-10.1-MDEV-12113 Manual testing (see attached output).

            MDEV-12600: crash during install_db with innodb_page_size=32K and ibdata1=3M;

            Problem was that all doublewrite buffer pages must fit to first
            system datafile.

            Ported commit 27a34df7882b1f8ed283f22bf83e8bfc523cbfde
            Author: Shaohua Wang <shaohua.wang@oracle.com>
            Date: Wed Aug 12 15:55:19 2015 +0800

            BUG#21551464 - SEGFAULT WHILE INITIALIZING DATABASE WHEN
            INNODB_DATA_FILE SIZE IS SMALL

            To 10.1 (with extended error printout).

            btr_create(): If ibuf header page allocation fails report error and
            return FIL_NULL. Similarly if root page allocation fails return a error.

            dict_build_table_def_step: If fsp_header_init fails return
            error code.

            fsp_header_init: returns true if header initialization succeeds
            and false if not.

            fseg_create_general: report error if segment or page allocation fails.

            innobase_init: If first datafile is smaller than 3M and could not
            contain all doublewrite buffer pages report error and fail to
            initialize InnoDB plugin.

            row_truncate_table_for_mysql: report error if fsp header init
            fails.

            srv_init_abort: New function to report database initialization errors.

            srv_undo_tablespaces_init, innobase_start_or_create_for_mysql: If
            database initialization fails report error and abort.

            trx_rseg_create: If segment header creation fails return.

            jplindst Jan Lindström (Inactive) added a comment - MDEV-12600 : crash during install_db with innodb_page_size=32K and ibdata1=3M; Problem was that all doublewrite buffer pages must fit to first system datafile. Ported commit 27a34df7882b1f8ed283f22bf83e8bfc523cbfde Author: Shaohua Wang <shaohua.wang@oracle.com> Date: Wed Aug 12 15:55:19 2015 +0800 BUG#21551464 - SEGFAULT WHILE INITIALIZING DATABASE WHEN INNODB_DATA_FILE SIZE IS SMALL To 10.1 (with extended error printout). btr_create(): If ibuf header page allocation fails report error and return FIL_NULL. Similarly if root page allocation fails return a error. dict_build_table_def_step: If fsp_header_init fails return error code. fsp_header_init: returns true if header initialization succeeds and false if not. fseg_create_general: report error if segment or page allocation fails. innobase_init: If first datafile is smaller than 3M and could not contain all doublewrite buffer pages report error and fail to initialize InnoDB plugin. row_truncate_table_for_mysql: report error if fsp header init fails. srv_init_abort: New function to report database initialization errors. srv_undo_tablespaces_init, innobase_start_or_create_for_mysql: If database initialization fails report error and abort. trx_rseg_create: If segment header creation fails return.

            I think that more work is needed. The check was added at a too high level.

            marko Marko Mäkelä added a comment - I think that more work is needed. The check was added at a too high level.

            I pushed a revision to bb-10.1-marko and merged to bb-10.2-marko. In the merge, I removed a similar too-high-level check from SysTablespace::check_file_spec(), now that buf_dblwr_create() does a more accurate check.

            I am not sure if it is because of MDEV-12042 by elenst, but with the following patch on top of my fix I am getting a new type of failure:

            diff --git a/mysql-test/suite/innodb/t/log_data_file_size.opt b/mysql-test/suite/innodb/t/log_data_file_size.opt
            index d9a364a3287..ca4c26e3bb0 100644
            --- a/mysql-test/suite/innodb/t/log_data_file_size.opt
            +++ b/mysql-test/suite/innodb/t/log_data_file_size.opt
            @@ -1,2 +1,2 @@
             --loose-innodb-sys-indexes
            ---innodb-data-file-path=ibdata1:1M:autoextend
            +--innodb-data-file-path=ibdata1:11M;ibdata2:1M:autoextend
            

            ./mtr innodb.log_data_file_size,32k
            

            innodb.log_data_file_size '32k,innodb_plugin' [ fail ]  Found warnings/errors in server log file!
                    Test ended at 2017-06-06 18:27:57
            line
            2017-06-06 18:27:56 140650424006080 [ERROR] InnoDB: Data file ./ibdata1 is of a different size 323 pages (rounded down to MB) than specified in the .cnf file 352 pages!
            

            The 352 pages is correct, and the first data file indeed is of wrong size. Sometimes (with different sizes for ibdata1 specified, and with different page size) the first file can even be bigger than specified.

            I had trouble debugging this. How can I get to debug the bootstrap with innodb_page_size=32k? The initial bootstrap seemed to take place with the default innodb_page_size=16k.

            marko Marko Mäkelä added a comment - I pushed a revision to bb-10.1-marko and merged to bb-10.2-marko . In the merge, I removed a similar too-high-level check from SysTablespace::check_file_spec(), now that buf_dblwr_create() does a more accurate check. I am not sure if it is because of MDEV-12042 by elenst , but with the following patch on top of my fix I am getting a new type of failure: diff --git a/mysql-test/suite/innodb/t/log_data_file_size.opt b/mysql-test/suite/innodb/t/log_data_file_size.opt index d9a364a3287..ca4c26e3bb0 100644 --- a/mysql-test/suite/innodb/t/log_data_file_size.opt +++ b/mysql-test/suite/innodb/t/log_data_file_size.opt @@ -1,2 +1,2 @@ --loose-innodb-sys-indexes ---innodb-data-file-path=ibdata1:1M:autoextend +--innodb-data-file-path=ibdata1:11M;ibdata2:1M:autoextend ./mtr innodb.log_data_file_size,32k innodb.log_data_file_size '32k,innodb_plugin' [ fail ] Found warnings/errors in server log file! Test ended at 2017-06-06 18:27:57 line 2017-06-06 18:27:56 140650424006080 [ERROR] InnoDB: Data file ./ibdata1 is of a different size 323 pages (rounded down to MB) than specified in the .cnf file 352 pages! The 352 pages is correct, and the first data file indeed is of wrong size. Sometimes (with different sizes for ibdata1 specified, and with different page size) the first file can even be bigger than specified. I had trouble debugging this. How can I get to debug the bootstrap with innodb_page_size=32k? The initial bootstrap seemed to take place with the default innodb_page_size=16k.

            MDEV-13013 is a related problem that makes the shared temporary tablespace unnecessarily large in MariaDB 10.2.

            marko Marko Mäkelä added a comment - MDEV-13013 is a related problem that makes the shared temporary tablespace unnecessarily large in MariaDB 10.2.

            ok to push.

            jplindst Jan Lindström (Inactive) added a comment - ok to push.

            People

              marko Marko Mäkelä
              anikitin Andrii Nikitin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.