[MDEV-12600] crash during install_db with innodb_page_size=32K and ibdata1=3M; Created: 2017-04-27  Updated: 2017-08-11  Resolved: 2017-07-04

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.1.22
Fix Version/s: 10.1.25, 10.2.7

Type: Bug Priority: Major
Reporter: Andrii Nikitin (Inactive) Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Attachments: HTML File out    
Issue Links:
Relates
relates to MDEV-12042 Re-bootstrap the server if InnoDB opt... Closed
relates to MDEV-13013 InnoDB unnecessarily extends data files Closed
relates to MDEV-12905 InnoDB occasionally skips the doublew... Closed

 Description   

May be related to MDEV-12113 , just this is without encryption.
When trying to install_db in 10.1 with :
innodb_page_size=32k
innodb_data_file_path=ibdata1:3M;ibdata2:10M:autoextend

I am not getting stack resolved - let me know if this is required to troubleshoot:

/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(my_print_stacktrace+0x2e)[0xc0b60e]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(handle_fatal_signal+0x4bf)[0x76902f]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11630)[0x7fd86e014630]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x9f)[0x7fd86cd9a7ef]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fd86cd9c3ea]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0xa9b877]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0xa49a8f]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0xa4f75e]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x94e3a3]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x951768]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x9e2232]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x908304]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5e)[0x76eede]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x5c6008]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(_Z11plugin_initPiPPci+0xbc0)[0x5c6fd0]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x5204ad]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld(_Z11mysqld_mainiPPc+0x8c3)[0x5216f3]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fd86cd853f1]
 
/home/a/mariadb-environs/m9-10.1.22/../_depot/m-tar/10.1.22/bin/mysqld[0x516a49]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

https://travis-ci.org/AndriiNikitin/mariadb-environs-MDEV-12113/builds/226281024

Only these configurations do show crash:

VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'

the rest pass:

VERSION=10.1.22 PAGE_SIZE=16k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=16k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend' REST=configure_rest_encryption.sh=1
 
VERSION=10.1.22 PAGE_SIZE=8k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=64k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:1M;ibdata2:10M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:11M;ibdata2:10M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:31M;ibdata2:10M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:3M:autoextend'
 
VERSION=10.1.22 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:1M;ibdata2:3M:autoextend'
 
VERSION=10.0.30 PAGE_SIZE=8k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'
 
VERSION=5.5.55 PAGE_SIZE=8k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'

10.2 reports an (expected) error that 3Mb is too small :
InnoDB: The innodb_system data file 'ibdata1' must be at least 6 MB

VERSION=10.2.5 PAGE_SIZE=32k DATA_FILE_PATH='ibdata1:3M;ibdata2:10M:autoextend'



 Comments   
Comment by Andrii Nikitin (Inactive) [ 2017-04-27 ]

After all it looks that several more jobs at the link actually had crash, just mysql_install_db did report 'success', so job marked as green.

Comment by Elena Stepanova [ 2017-05-01 ]

10.1 935a1c676e1fbfea79

 
scripts/mysql_install_db --no-defaults --innodb-data-file-path="ibdata1:3M;ibdata2:10M:autoextend" --innodb-page-size=32K
 
...
 
2017-05-01  4:03:37 139673798764416 [Note] InnoDB: Waiting for purge to start
 
2017-05-01  4:03:37 139673798764416 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 0
 
2017-05-01 04:03:37 7f085728e780  InnoDB: Assertion failure in thread 139673798764416 in file fil0fil.cc line 6430
 
InnoDB: Failing assertion: node->size - block_offset >= ((byte_offset + len + (UNIV_PAGE_SIZE - 1)) / UNIV_PAGE_SIZE)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: about forcing recovery.
 
170501  4:03:37 [ERROR] mysqld got signal 6 ;


#4  0x00007f590b16c448 in __GI_abort () at abort.c:89
 
#5  0x00007f590e014bbe in _fil_io (type=11, sync=true, space_id=0, zip_size=0, block_offset=64, byte_offset=0, len=2097152, buf=0x7f58e6c00000, message=0x0, write_size=0x0, trx=0x0) at /data/src/10.1/storage/xtradb/fil/fil0fil.cc:6428
 
#6  0x00007f590dfa4d88 in buf_dblwr_flush_buffered_writes () at /data/src/10.1/storage/xtradb/buf/buf0dblwr.cc:974
 
#7  0x00007f590dfadf1b in buf_flush_common (flush_type=BUF_FLUSH_LIST, page_count=100) at /data/src/10.1/storage/xtradb/buf/buf0flu.cc:1860
 
#8  0x00007f590dfae7b7 in buf_flush_list (min_n=18446744073709551614, lsn_limit=18446744073709551615, n_processed=0x7fff17f8bb68) at /data/src/10.1/storage/xtradb/buf/buf0flu.cc:2143
 
#9  0x00007f590de14861 in log_preflush_pool_modified_pages (new_oldest=18446744073709551615) at /data/src/10.1/storage/xtradb/log/log0log.cc:1893
 
#10 0x00007f590de15b49 in log_make_checkpoint_at (lsn=18446744073709551615, write_always=1) at /data/src/10.1/storage/xtradb/log/log0log.cc:2377
 
#11 0x00007f590df0a072 in innobase_start_or_create_for_mysql () at /data/src/10.1/storage/xtradb/srv/srv0start.cc:3015
 
#12 0x00007f590dd9959d in innobase_init (p=0x7f590a824a70) at /data/src/10.1/storage/xtradb/handler/ha_innodb.cc:4454
 
#13 0x00007f590dba8791 in ha_initialize_handlerton (plugin=0x7f590ab12770) at /data/src/10.1/sql/handler.cc:513
 
#14 0x00007f590d985af5 in plugin_initialize (tmp_root=0x7fff17f8f880, plugin=0x7f590ab12770, argc=0x7f590ec00770 <remaining_argc>, argv=0x7f590a821428, options_only=false) at /data/src/10.1/sql/sql_plugin.cc:1400
 
#15 0x00007f590d9866de in plugin_init (argc=0x7f590ec00770 <remaining_argc>, argv=0x7f590a821428, flags=2) at /data/src/10.1/sql/sql_plugin.cc:1678
 
#16 0x00007f590d8a59f2 in init_server_components () at /data/src/10.1/sql/mysqld.cc:5145
 
#17 0x00007f590d8a6a5d in mysqld_main (argc=12, argv=0x7f590a821428) at /data/src/10.1/sql/mysqld.cc:5732
 
#18 0x00007f590d89c1f0 in main (argc=12, argv=0x7fff17f90618) at /data/src/10.1/sql/main.cc:25

Comment by Jan Lindström (Inactive) [ 2017-05-28 ]

Includes the two other related fixes as you can't test this without fixing the first two first.

https://github.com/MariaDB/server/tree/bb-10.1-MDEV-12113

Manual testing (see attached output).

Comment by Jan Lindström (Inactive) [ 2017-06-02 ]

MDEV-12600: crash during install_db with innodb_page_size=32K and ibdata1=3M;

Problem was that all doublewrite buffer pages must fit to first
system datafile.

Ported commit 27a34df7882b1f8ed283f22bf83e8bfc523cbfde
Author: Shaohua Wang <shaohua.wang@oracle.com>
Date: Wed Aug 12 15:55:19 2015 +0800

BUG#21551464 - SEGFAULT WHILE INITIALIZING DATABASE WHEN
INNODB_DATA_FILE SIZE IS SMALL

To 10.1 (with extended error printout).

btr_create(): If ibuf header page allocation fails report error and
return FIL_NULL. Similarly if root page allocation fails return a error.

dict_build_table_def_step: If fsp_header_init fails return
error code.

fsp_header_init: returns true if header initialization succeeds
and false if not.

fseg_create_general: report error if segment or page allocation fails.

innobase_init: If first datafile is smaller than 3M and could not
contain all doublewrite buffer pages report error and fail to
initialize InnoDB plugin.

row_truncate_table_for_mysql: report error if fsp header init
fails.

srv_init_abort: New function to report database initialization errors.

srv_undo_tablespaces_init, innobase_start_or_create_for_mysql: If
database initialization fails report error and abort.

trx_rseg_create: If segment header creation fails return.

Comment by Marko Mäkelä [ 2017-06-06 ]

I think that more work is needed. The check was added at a too high level.

Comment by Marko Mäkelä [ 2017-06-06 ]

I pushed a revision to bb-10.1-marko and merged to bb-10.2-marko. In the merge, I removed a similar too-high-level check from SysTablespace::check_file_spec(), now that buf_dblwr_create() does a more accurate check.

I am not sure if it is because of MDEV-12042 by elenst, but with the following patch on top of my fix I am getting a new type of failure:

diff --git a/mysql-test/suite/innodb/t/log_data_file_size.opt b/mysql-test/suite/innodb/t/log_data_file_size.opt
 
index d9a364a3287..ca4c26e3bb0 100644
 
--- a/mysql-test/suite/innodb/t/log_data_file_size.opt
 
+++ b/mysql-test/suite/innodb/t/log_data_file_size.opt
 
@@ -1,2 +1,2 @@
 
 --loose-innodb-sys-indexes
 
---innodb-data-file-path=ibdata1:1M:autoextend
 
+--innodb-data-file-path=ibdata1:11M;ibdata2:1M:autoextend


./mtr innodb.log_data_file_size,32k


innodb.log_data_file_size '32k,innodb_plugin' [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2017-06-06 18:27:57
 
line
 
2017-06-06 18:27:56 140650424006080 [ERROR] InnoDB: Data file ./ibdata1 is of a different size 323 pages (rounded down to MB) than specified in the .cnf file 352 pages!

The 352 pages is correct, and the first data file indeed is of wrong size. Sometimes (with different sizes for ibdata1 specified, and with different page size) the first file can even be bigger than specified.

I had trouble debugging this. How can I get to debug the bootstrap with innodb_page_size=32k? The initial bootstrap seemed to take place with the default innodb_page_size=16k.

Comment by Marko Mäkelä [ 2017-06-07 ]

MDEV-13013 is a related problem that makes the shared temporary tablespace unnecessarily large in MariaDB 10.2.

Comment by Jan Lindström (Inactive) [ 2017-06-08 ]

ok to push.

Generated at Thu Feb 08 07:58:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.