Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.1.17, 10.1.18
-
Centos 6.5, Mariadb 10.1.18 and Mariadb 10.1.17 (and also on all older)
Linode virtual server 48GB RAM
-
10.1.20
Description
Adding fulltext index on bigger table crash server
2016-11-04 07:41:26 7f7149bfe700 InnoDB: Assertion failure in thread 140124545345280 in file row0merge.cc line 890 |
InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE |
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to http://bugs.mysql.com. |
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html |
InnoDB: about forcing recovery.
|
161104 7:41:26 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
|
Server version: 10.1.17-MariaDB |
key_buffer_size=134217728 |
read_buffer_size=131072 |
max_used_connections=54 |
max_threads=902 |
thread_count=8 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2112329 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
|
Thread pointer: 0x0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x48400 |
2016-11-04 07:41:27 7f714a3ff700 InnoDB: Assertion failure in thread 140124553737984 in file row0merge.cc line 890 |
InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE |
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to http://bugs.mysql.com. |
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html |
InnoDB: about forcing recovery.
|
161104 07:41:29 mysqld_safe Number of processes running now: 0 |
161104 07:41:29 mysqld_safe mysqld restarted |
2016-11-04 7:41:29 140601780914208 [Note] /usr/sbin/mysqld (mysqld 10.1.17-MariaDB) starting as process 25262 ... |
2016-11-04 7:41:29 140601780914208 [Warning] Although a path was specified for the --log-slow-queries option, log tables are used. To enable logging to files use the --log-output=file option. |
2016-11-04 7:41:29 140601780914208 [Warning] option 'innodb-ft-cache-size': unsigned value 83886080 adjusted to 80000000 |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using mutexes to ref count buffer pool pages |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: The InnoDB memory heap is disabled |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Compressed tables use zlib 1.2.3 |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using Linux native AIO |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using SSE crc32 instructions |
2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Initializing buffer pool, size = 14.0G |
2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Completed initialization of buffer pool |
2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Highest supported file format is Barracuda. |
2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Log scan progressed past the checkpoint lsn 5089194166241 |
2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Database was not shutdown normally! |
2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Starting crash recovery. |
2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Reading tablespace information from the .ibd files... |
2016-11-04 7:41:46 140601780914208 [Note] InnoDB: Processed 81433 .ibd/.isl files |
2016-11-04 7:42:02 140601780914208 [Note] InnoDB: Processed 165583 .ibd/.isl files |
2016-11-04 7:42:18 140601780914208 [Note] InnoDB: Processed 248732 .ibd/.isl files |
2016-11-04 7:42:34 140601780914208 [Note] InnoDB: Processed 331595 .ibd/.isl files |
2016-11-04 7:42:50 140601780914208 [Note] InnoDB: Processed 410190 .ibd/.isl files |
2016-11-04 7:43:06 140601780914208 [Note] InnoDB: Processed 490369 .ibd/.isl files |
2016-11-04 7:43:22 140601780914208 [Note] InnoDB: Processed 571032 .ibd/.isl files |
2016-11-04 7:43:38 140601780914208 [Note] InnoDB: Processed 653103 .ibd/.isl files |
2016-11-04 7:43:54 140601780914208 [Note] InnoDB: Processed 733414 .ibd/.isl files |
2016-11-04 7:44:10 140601780914208 [Note] InnoDB: Processed 811624 .ibd/.isl files |
2016-11-04 7:44:26 140601780914208 [Note] InnoDB: Processed 891341 .ibd/.isl files |
2016-11-04 7:44:42 140601780914208 [Note] InnoDB: Processed 969320 .ibd/.isl files |
2016-11-04 7:44:58 140601780914208 [Note] InnoDB: Processed 1047838 .ibd/.isl files |
2016-11-04 7:45:14 140601780914208 [Note] InnoDB: Processed 1123078 .ibd/.isl files |
2016-11-04 7:45:25 140601780914208 [Note] InnoDB: Restoring possible half-written data pages |
2016-11-04 7:45:25 140601780914208 [Note] InnoDB: from the doublewrite buffer... |
InnoDB: Doing recovery: scanned up to log sequence number 5089199408640 |
InnoDB: Doing recovery: scanned up to log sequence number 5089204651520 |
InnoDB: Doing recovery: scanned up to log sequence number 5089209894400 |
InnoDB: Doing recovery: scanned up to log sequence number 5089215137280 |
InnoDB: Doing recovery: scanned up to log sequence number 5089220380160 |
InnoDB: Doing recovery: scanned up to log sequence number 5089225623040 |
InnoDB: Doing recovery: scanned up to log sequence number 5089230865920 |
.... etc
|
Problem was started by SQL:
ALTER TABLE `qu_la_contacts` ADD FULLTEXT `qu_la_contacts_ftidx` (`firstname` ,`lastname` ,`emails` ,`system_name`)
on table, which has about 200k rows
CREATE TABLE IF NOT EXISTS `qu_la_contacts` (
|
`contactid` CHAR(8) NOT NULL, |
`parent_contactid` CHAR(8) NULL, |
`company` CHAR(1) NOT NULL DEFAULT 'N', |
`job_position` VARCHAR(255) NULL, |
`emails` TEXT NULL,
|
`firstname` VARCHAR(100) NULL, |
`lastname` VARCHAR(100) NULL, |
`system_name` VARCHAR(100) NULL, |
`description` VARCHAR(255) NULL, |
`rtype` CHAR(1) NOT NULL, |
`status` CHAR(1) NOT NULL DEFAULT 'A' COMMENT 'A=Active, X=Deleted', |
`datecreated` DATETIME NULL,
|
`avatar_url` TEXT NULL,
|
`city` VARCHAR(255) NULL, |
`countrycode` VARCHAR(2) NULL, |
`note` TEXT NULL,
|
`time_offset` INT NULL,
|
`language` VARCHAR(10) NULL, |
`gender` CHAR(1) NOT NULL DEFAULT 'M', |
`levelid` CHAR(8) NULL, |
`ldap_id` VARCHAR(255) NULL, |
`groups` VARCHAR(255) NULL, |
`ip` VARCHAR(39) NULL, |
`useragent` TEXT NULL,
|
`screen` VARCHAR(12) NULL, |
`latitude` FLOAT NULL,
|
`longitude` FLOAT NULL,
|
`last_action_type` CHAR(1) NULL, |
`last_action_date` DATETIME NULL,
|
`last_action_meta` VARCHAR(80) NULL, |
PRIMARY KEY (`contactid`),
|
INDEX `fk_qu_la_contacts_qu_la_levels1_idx` (`levelid` ASC),
|
INDEX `fk_parent_contactid` (`parent_contactid` ASC),
|
INDEX `qu_la_contacts_datecreated` (`datecreated` ASC),
|
INDEX `qu_la_contacts_firstname` (`firstname` ASC),
|
INDEX `qu_la_contacts_lastname` (`lastname` ASC),
|
INDEX `qu_la_contacts_system_name` (`system_name` ASC),
|
INDEX `qu_la_contacts_company` (`company`)
|
)
|
ENGINE = InnoDB;
|
Server settings:
[mysqld]
|
server-id=14 |
datadir=/opt/mysql/
|
socket=/var/lib/mysql/mysql.sock
|
tmpdir=/tmp/
|
|
log_bin=mysql-bin
|
expire_logs_days=6 |
|
wait_timeout=60 |
|
long_query_time=1 |
slow_query_log=1 |
slow_query_log_file=/opt/log/slow.log
|
log_output=TABLE
|
|
userstat = 1 |
|
user=root
|
symbolic-links=0 |
binlog_format=STATEMENT
|
|
default_storage_engine=InnoDB
|
|
slave_skip_errors=1062,1396,1690 |
|
innodb_autoinc_lock_mode=2 |
|
innodb_buffer_pool_size=14G
|
innodb_buffer_pool_instances=10 |
innodb_log_file_size=1G
|
innodb_log_buffer_size=196M
|
innodb_flush_log_at_trx_commit=1 |
innodb_thread_concurrency=24 |
innodb_file_per_table
|
innodb_write_io_threads=24 |
innodb_read_io_threads=24 |
innodb_sched_priority_cleaner=39 |
innodb_adaptive_flushing=1 |
innodb_purge_threads=5 |
#transaction-isolation=READ-COMMITTED
|
innodb_adaptive_hash_index_partitions=64 |
innodb_flush_neighbors=0 |
innodb_flush_method=O_DIRECT
|
innodb_io_capacity=5000 |
innodb_io_capacity_max=8000 |
innodb_lru_scan_depth=1024 |
innodb_sort_buffer_size=32M
|
innodb_ft_cache_size=80M
|
innodb_ft_total_cache_size=1G
|
|
|
|
slave_parallel_threads=10 |
log_slave_updates=on
|
|
performance_schema=off
|
|
skip-name-resolve
|
|
max_allowed_packet = 512M
|
|
query_cache_type=1 |
query_cache_size = 0 |
query_cache_limit = 1M
|
query_cache_min_res_unit=1K
|
max_connections = 900 |
|
table_open_cache=64K
|
innodb_open_files=64K
|
open_files_limit=1020000 |
collation-server = utf8_general_ci
|
character-set-server = utf8
|
|
log-error=/opt/log/error.log
|
|
[mysqld_safe]
|
log-error=/opt/log/error.log
|
pid-file=/var/run/mysqld/mysqld.pid
|
malloc-lib=/usr/lib64/libjemalloc.so.1 |
|
Exactly the same problem happen on another table when adding fulltext index (table size about 150MB) (reported in additional comments here: https://jira.mariadb.org/browse/MDEV-9129):
ALTER TABLE qu_g_mails ADD FULLTEXT ft_qu_g_mails1 (subject, body_text, body_html);
|
CREATE TABLE `qu_g_mails` (
|
`mailid` int(10) unsigned NOT NULL AUTO_INCREMENT, |
`hdr_message_id` varchar(255) DEFAULT NULL, |
`unique_message_id` varchar(255) NOT NULL, |
`subject` text,
|
`headers` text,
|
`body_text` longtext,
|
`body_html` longtext,
|
`created` datetime NOT NULL,
|
`delivered` datetime DEFAULT NULL,
|
`from_mail` text,
|
`to_recipients` text,
|
`cc_recipients` text,
|
`bcc_recipients` text,
|
`accountuserid` char(8) DEFAULT NULL, |
`reply_to` text,
|
PRIMARY KEY (`mailid`),
|
KEY `IDX_qu_g_mails_1` (`accountuserid`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;
|
Attachments
Issue Links
- is duplicated by
-
MDEV-11918 CREATE FULLTEXT INDEX with a token longer than 127 bytes crashes server (10.0 version)
-
- Closed
-
- relates to
-
MDEV-11241 Certain combining marks cause MariaDB to crash when doing Full-Text searches
-
- Closed
-
Activity
Field | Original Value | New Value |
---|---|---|
Description |
Adding fulltext index on bigger table crash server
{code:java} 2016-11-04 07:41:26 7f7149bfe700 InnoDB: Assertion failure in thread 140124545345280 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 7:41:26 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.17-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=54 max_threads=902 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2112329 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 2016-11-04 07:41:27 7f714a3ff700 InnoDB: Assertion failure in thread 140124553737984 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 07:41:29 mysqld_safe Number of processes running now: 0 161104 07:41:29 mysqld_safe mysqld restarted 2016-11-04 7:41:29 140601780914208 [Note] /usr/sbin/mysqld (mysqld 10.1.17-MariaDB) starting as process 25262 ... 2016-11-04 7:41:29 140601780914208 [Warning] Although a path was specified for the --log-slow-queries option, log tables are used. To enable logging to files use the --log-output=file option. 2016-11-04 7:41:29 140601780914208 [Warning] option 'innodb-ft-cache-size': unsigned value 83886080 adjusted to 80000000 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: The InnoDB memory heap is disabled 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using Linux native AIO 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using SSE crc32 instructions 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Initializing buffer pool, size = 14.0G 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Completed initialization of buffer pool 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Highest supported file format is Barracuda. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Log scan progressed past the checkpoint lsn 5089194166241 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Database was not shutdown normally! 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Starting crash recovery. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-11-04 7:41:46 140601780914208 [Note] InnoDB: Processed 81433 .ibd/.isl files 2016-11-04 7:42:02 140601780914208 [Note] InnoDB: Processed 165583 .ibd/.isl files 2016-11-04 7:42:18 140601780914208 [Note] InnoDB: Processed 248732 .ibd/.isl files 2016-11-04 7:42:34 140601780914208 [Note] InnoDB: Processed 331595 .ibd/.isl files 2016-11-04 7:42:50 140601780914208 [Note] InnoDB: Processed 410190 .ibd/.isl files 2016-11-04 7:43:06 140601780914208 [Note] InnoDB: Processed 490369 .ibd/.isl files 2016-11-04 7:43:22 140601780914208 [Note] InnoDB: Processed 571032 .ibd/.isl files 2016-11-04 7:43:38 140601780914208 [Note] InnoDB: Processed 653103 .ibd/.isl files 2016-11-04 7:43:54 140601780914208 [Note] InnoDB: Processed 733414 .ibd/.isl files 2016-11-04 7:44:10 140601780914208 [Note] InnoDB: Processed 811624 .ibd/.isl files 2016-11-04 7:44:26 140601780914208 [Note] InnoDB: Processed 891341 .ibd/.isl files 2016-11-04 7:44:42 140601780914208 [Note] InnoDB: Processed 969320 .ibd/.isl files 2016-11-04 7:44:58 140601780914208 [Note] InnoDB: Processed 1047838 .ibd/.isl files 2016-11-04 7:45:14 140601780914208 [Note] InnoDB: Processed 1123078 .ibd/.isl files 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: Restoring possible half-written data pages 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 5089199408640 InnoDB: Doing recovery: scanned up to log sequence number 5089204651520 InnoDB: Doing recovery: scanned up to log sequence number 5089209894400 InnoDB: Doing recovery: scanned up to log sequence number 5089215137280 InnoDB: Doing recovery: scanned up to log sequence number 5089220380160 InnoDB: Doing recovery: scanned up to log sequence number 5089225623040 InnoDB: Doing recovery: scanned up to log sequence number 5089230865920 .... etc {code} Problem was started by SQL: ALTER TABLE `qu_la_contacts` ADD FULLTEXT `qu_la_contacts_ftidx` (`firstname` ,`lastname` ,`emails` ,`system_name`) on table {code:java} CREATE TABLE IF NOT EXISTS `qu_la_contacts` ( `contactid` CHAR(8) NOT NULL, `parent_contactid` CHAR(8) NULL, `company` CHAR(1) NOT NULL DEFAULT 'N', `job_position` VARCHAR(255) NULL, `emails` TEXT NULL, `firstname` VARCHAR(100) NULL, `lastname` VARCHAR(100) NULL, `system_name` VARCHAR(100) NULL, `description` VARCHAR(255) NULL, `rtype` CHAR(1) NOT NULL, `status` CHAR(1) NOT NULL DEFAULT 'A' COMMENT 'A=Active, X=Deleted', `datecreated` DATETIME NULL, `avatar_url` TEXT NULL, `city` VARCHAR(255) NULL, `countrycode` VARCHAR(2) NULL, `note` TEXT NULL, `time_offset` INT NULL, `language` VARCHAR(10) NULL, `gender` CHAR(1) NOT NULL DEFAULT 'M', `levelid` CHAR(8) NULL, `ldap_id` VARCHAR(255) NULL, `groups` VARCHAR(255) NULL, `ip` VARCHAR(39) NULL, `useragent` TEXT NULL, `screen` VARCHAR(12) NULL, `latitude` FLOAT NULL, `longitude` FLOAT NULL, `last_action_type` CHAR(1) NULL, `last_action_date` DATETIME NULL, `last_action_meta` VARCHAR(80) NULL, PRIMARY KEY (`contactid`), INDEX `fk_qu_la_contacts_qu_la_levels1_idx` (`levelid` ASC), INDEX `fk_parent_contactid` (`parent_contactid` ASC), INDEX `qu_la_contacts_datecreated` (`datecreated` ASC), INDEX `qu_la_contacts_firstname` (`firstname` ASC), INDEX `qu_la_contacts_lastname` (`lastname` ASC), INDEX `qu_la_contacts_system_name` (`system_name` ASC), INDEX `qu_la_contacts_company` (`company`) ) ENGINE = InnoDB; {code} Server settings: {code:java} [mysqld] server-id=14 datadir=/opt/mysql/ socket=/var/lib/mysql/mysql.sock tmpdir=/tmp/ log_bin=mysql-bin expire_logs_days=6 wait_timeout=60 long_query_time=1 slow_query_log=1 slow_query_log_file=/opt/log/slow.log log_output=TABLE userstat = 1 user=root symbolic-links=0 binlog_format=STATEMENT default_storage_engine=InnoDB slave_skip_errors=1062,1396,1690 innodb_autoinc_lock_mode=2 innodb_buffer_pool_size=14G innodb_buffer_pool_instances=10 innodb_log_file_size=1G innodb_log_buffer_size=196M innodb_flush_log_at_trx_commit=1 innodb_thread_concurrency=24 innodb_file_per_table innodb_write_io_threads=24 innodb_read_io_threads=24 innodb_sched_priority_cleaner=39 innodb_adaptive_flushing=1 innodb_purge_threads=5 #transaction-isolation=READ-COMMITTED innodb_adaptive_hash_index_partitions=64 innodb_flush_neighbors=0 innodb_flush_method=O_DIRECT innodb_io_capacity=5000 innodb_io_capacity_max=8000 innodb_lru_scan_depth=1024 innodb_sort_buffer_size=32M innodb_ft_cache_size=80M innodb_ft_total_cache_size=1G slave_parallel_threads=10 log_slave_updates=on performance_schema=off skip-name-resolve max_allowed_packet = 512M query_cache_type=1 query_cache_size = 0 query_cache_limit = 1M query_cache_min_res_unit=1K max_connections = 900 table_open_cache=64K innodb_open_files=64K open_files_limit=1020000 collation-server = utf8_general_ci character-set-server = utf8 log-error=/opt/log/error.log [mysqld_safe] log-error=/opt/log/error.log pid-file=/var/run/mysqld/mysqld.pid malloc-lib=/usr/lib64/libjemalloc.so.1 {code} |
Adding fulltext index on bigger table crash server
{code:java} 2016-11-04 07:41:26 7f7149bfe700 InnoDB: Assertion failure in thread 140124545345280 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 7:41:26 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.17-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=54 max_threads=902 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2112329 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 2016-11-04 07:41:27 7f714a3ff700 InnoDB: Assertion failure in thread 140124553737984 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 07:41:29 mysqld_safe Number of processes running now: 0 161104 07:41:29 mysqld_safe mysqld restarted 2016-11-04 7:41:29 140601780914208 [Note] /usr/sbin/mysqld (mysqld 10.1.17-MariaDB) starting as process 25262 ... 2016-11-04 7:41:29 140601780914208 [Warning] Although a path was specified for the --log-slow-queries option, log tables are used. To enable logging to files use the --log-output=file option. 2016-11-04 7:41:29 140601780914208 [Warning] option 'innodb-ft-cache-size': unsigned value 83886080 adjusted to 80000000 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: The InnoDB memory heap is disabled 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using Linux native AIO 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using SSE crc32 instructions 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Initializing buffer pool, size = 14.0G 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Completed initialization of buffer pool 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Highest supported file format is Barracuda. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Log scan progressed past the checkpoint lsn 5089194166241 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Database was not shutdown normally! 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Starting crash recovery. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-11-04 7:41:46 140601780914208 [Note] InnoDB: Processed 81433 .ibd/.isl files 2016-11-04 7:42:02 140601780914208 [Note] InnoDB: Processed 165583 .ibd/.isl files 2016-11-04 7:42:18 140601780914208 [Note] InnoDB: Processed 248732 .ibd/.isl files 2016-11-04 7:42:34 140601780914208 [Note] InnoDB: Processed 331595 .ibd/.isl files 2016-11-04 7:42:50 140601780914208 [Note] InnoDB: Processed 410190 .ibd/.isl files 2016-11-04 7:43:06 140601780914208 [Note] InnoDB: Processed 490369 .ibd/.isl files 2016-11-04 7:43:22 140601780914208 [Note] InnoDB: Processed 571032 .ibd/.isl files 2016-11-04 7:43:38 140601780914208 [Note] InnoDB: Processed 653103 .ibd/.isl files 2016-11-04 7:43:54 140601780914208 [Note] InnoDB: Processed 733414 .ibd/.isl files 2016-11-04 7:44:10 140601780914208 [Note] InnoDB: Processed 811624 .ibd/.isl files 2016-11-04 7:44:26 140601780914208 [Note] InnoDB: Processed 891341 .ibd/.isl files 2016-11-04 7:44:42 140601780914208 [Note] InnoDB: Processed 969320 .ibd/.isl files 2016-11-04 7:44:58 140601780914208 [Note] InnoDB: Processed 1047838 .ibd/.isl files 2016-11-04 7:45:14 140601780914208 [Note] InnoDB: Processed 1123078 .ibd/.isl files 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: Restoring possible half-written data pages 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 5089199408640 InnoDB: Doing recovery: scanned up to log sequence number 5089204651520 InnoDB: Doing recovery: scanned up to log sequence number 5089209894400 InnoDB: Doing recovery: scanned up to log sequence number 5089215137280 InnoDB: Doing recovery: scanned up to log sequence number 5089220380160 InnoDB: Doing recovery: scanned up to log sequence number 5089225623040 InnoDB: Doing recovery: scanned up to log sequence number 5089230865920 .... etc {code} Problem was started by SQL: ALTER TABLE `qu_la_contacts` ADD FULLTEXT `qu_la_contacts_ftidx` (`firstname` ,`lastname` ,`emails` ,`system_name`) on table, which has about 200k rows {code:java} CREATE TABLE IF NOT EXISTS `qu_la_contacts` ( `contactid` CHAR(8) NOT NULL, `parent_contactid` CHAR(8) NULL, `company` CHAR(1) NOT NULL DEFAULT 'N', `job_position` VARCHAR(255) NULL, `emails` TEXT NULL, `firstname` VARCHAR(100) NULL, `lastname` VARCHAR(100) NULL, `system_name` VARCHAR(100) NULL, `description` VARCHAR(255) NULL, `rtype` CHAR(1) NOT NULL, `status` CHAR(1) NOT NULL DEFAULT 'A' COMMENT 'A=Active, X=Deleted', `datecreated` DATETIME NULL, `avatar_url` TEXT NULL, `city` VARCHAR(255) NULL, `countrycode` VARCHAR(2) NULL, `note` TEXT NULL, `time_offset` INT NULL, `language` VARCHAR(10) NULL, `gender` CHAR(1) NOT NULL DEFAULT 'M', `levelid` CHAR(8) NULL, `ldap_id` VARCHAR(255) NULL, `groups` VARCHAR(255) NULL, `ip` VARCHAR(39) NULL, `useragent` TEXT NULL, `screen` VARCHAR(12) NULL, `latitude` FLOAT NULL, `longitude` FLOAT NULL, `last_action_type` CHAR(1) NULL, `last_action_date` DATETIME NULL, `last_action_meta` VARCHAR(80) NULL, PRIMARY KEY (`contactid`), INDEX `fk_qu_la_contacts_qu_la_levels1_idx` (`levelid` ASC), INDEX `fk_parent_contactid` (`parent_contactid` ASC), INDEX `qu_la_contacts_datecreated` (`datecreated` ASC), INDEX `qu_la_contacts_firstname` (`firstname` ASC), INDEX `qu_la_contacts_lastname` (`lastname` ASC), INDEX `qu_la_contacts_system_name` (`system_name` ASC), INDEX `qu_la_contacts_company` (`company`) ) ENGINE = InnoDB; {code} Server settings: {code:java} [mysqld] server-id=14 datadir=/opt/mysql/ socket=/var/lib/mysql/mysql.sock tmpdir=/tmp/ log_bin=mysql-bin expire_logs_days=6 wait_timeout=60 long_query_time=1 slow_query_log=1 slow_query_log_file=/opt/log/slow.log log_output=TABLE userstat = 1 user=root symbolic-links=0 binlog_format=STATEMENT default_storage_engine=InnoDB slave_skip_errors=1062,1396,1690 innodb_autoinc_lock_mode=2 innodb_buffer_pool_size=14G innodb_buffer_pool_instances=10 innodb_log_file_size=1G innodb_log_buffer_size=196M innodb_flush_log_at_trx_commit=1 innodb_thread_concurrency=24 innodb_file_per_table innodb_write_io_threads=24 innodb_read_io_threads=24 innodb_sched_priority_cleaner=39 innodb_adaptive_flushing=1 innodb_purge_threads=5 #transaction-isolation=READ-COMMITTED innodb_adaptive_hash_index_partitions=64 innodb_flush_neighbors=0 innodb_flush_method=O_DIRECT innodb_io_capacity=5000 innodb_io_capacity_max=8000 innodb_lru_scan_depth=1024 innodb_sort_buffer_size=32M innodb_ft_cache_size=80M innodb_ft_total_cache_size=1G slave_parallel_threads=10 log_slave_updates=on performance_schema=off skip-name-resolve max_allowed_packet = 512M query_cache_type=1 query_cache_size = 0 query_cache_limit = 1M query_cache_min_res_unit=1K max_connections = 900 table_open_cache=64K innodb_open_files=64K open_files_limit=1020000 collation-server = utf8_general_ci character-set-server = utf8 log-error=/opt/log/error.log [mysqld_safe] log-error=/opt/log/error.log pid-file=/var/run/mysqld/mysqld.pid malloc-lib=/usr/lib64/libjemalloc.so.1 {code} Exactly the same problem happen on another table when adding fulltext index (table size about 150MB): {code:java} ALTER TABLE qu_g_mails ADD FULLTEXT ft_qu_g_mails1 (subject, body_text, body_html); {code} {code:java} CREATE TABLE `qu_g_mails` ( `mailid` int(10) unsigned NOT NULL AUTO_INCREMENT, `hdr_message_id` varchar(255) DEFAULT NULL, `unique_message_id` varchar(255) NOT NULL, `subject` text, `headers` text, `body_text` longtext, `body_html` longtext, `created` datetime NOT NULL, `delivered` datetime DEFAULT NULL, `from_mail` text, `to_recipients` text, `cc_recipients` text, `bcc_recipients` text, `accountuserid` char(8) DEFAULT NULL, `reply_to` text, PRIMARY KEY (`mailid`), KEY `IDX_qu_g_mails_1` (`accountuserid`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED; {code} |
Description |
Adding fulltext index on bigger table crash server
{code:java} 2016-11-04 07:41:26 7f7149bfe700 InnoDB: Assertion failure in thread 140124545345280 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 7:41:26 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.17-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=54 max_threads=902 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2112329 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 2016-11-04 07:41:27 7f714a3ff700 InnoDB: Assertion failure in thread 140124553737984 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 07:41:29 mysqld_safe Number of processes running now: 0 161104 07:41:29 mysqld_safe mysqld restarted 2016-11-04 7:41:29 140601780914208 [Note] /usr/sbin/mysqld (mysqld 10.1.17-MariaDB) starting as process 25262 ... 2016-11-04 7:41:29 140601780914208 [Warning] Although a path was specified for the --log-slow-queries option, log tables are used. To enable logging to files use the --log-output=file option. 2016-11-04 7:41:29 140601780914208 [Warning] option 'innodb-ft-cache-size': unsigned value 83886080 adjusted to 80000000 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: The InnoDB memory heap is disabled 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using Linux native AIO 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using SSE crc32 instructions 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Initializing buffer pool, size = 14.0G 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Completed initialization of buffer pool 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Highest supported file format is Barracuda. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Log scan progressed past the checkpoint lsn 5089194166241 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Database was not shutdown normally! 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Starting crash recovery. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-11-04 7:41:46 140601780914208 [Note] InnoDB: Processed 81433 .ibd/.isl files 2016-11-04 7:42:02 140601780914208 [Note] InnoDB: Processed 165583 .ibd/.isl files 2016-11-04 7:42:18 140601780914208 [Note] InnoDB: Processed 248732 .ibd/.isl files 2016-11-04 7:42:34 140601780914208 [Note] InnoDB: Processed 331595 .ibd/.isl files 2016-11-04 7:42:50 140601780914208 [Note] InnoDB: Processed 410190 .ibd/.isl files 2016-11-04 7:43:06 140601780914208 [Note] InnoDB: Processed 490369 .ibd/.isl files 2016-11-04 7:43:22 140601780914208 [Note] InnoDB: Processed 571032 .ibd/.isl files 2016-11-04 7:43:38 140601780914208 [Note] InnoDB: Processed 653103 .ibd/.isl files 2016-11-04 7:43:54 140601780914208 [Note] InnoDB: Processed 733414 .ibd/.isl files 2016-11-04 7:44:10 140601780914208 [Note] InnoDB: Processed 811624 .ibd/.isl files 2016-11-04 7:44:26 140601780914208 [Note] InnoDB: Processed 891341 .ibd/.isl files 2016-11-04 7:44:42 140601780914208 [Note] InnoDB: Processed 969320 .ibd/.isl files 2016-11-04 7:44:58 140601780914208 [Note] InnoDB: Processed 1047838 .ibd/.isl files 2016-11-04 7:45:14 140601780914208 [Note] InnoDB: Processed 1123078 .ibd/.isl files 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: Restoring possible half-written data pages 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 5089199408640 InnoDB: Doing recovery: scanned up to log sequence number 5089204651520 InnoDB: Doing recovery: scanned up to log sequence number 5089209894400 InnoDB: Doing recovery: scanned up to log sequence number 5089215137280 InnoDB: Doing recovery: scanned up to log sequence number 5089220380160 InnoDB: Doing recovery: scanned up to log sequence number 5089225623040 InnoDB: Doing recovery: scanned up to log sequence number 5089230865920 .... etc {code} Problem was started by SQL: ALTER TABLE `qu_la_contacts` ADD FULLTEXT `qu_la_contacts_ftidx` (`firstname` ,`lastname` ,`emails` ,`system_name`) on table, which has about 200k rows {code:java} CREATE TABLE IF NOT EXISTS `qu_la_contacts` ( `contactid` CHAR(8) NOT NULL, `parent_contactid` CHAR(8) NULL, `company` CHAR(1) NOT NULL DEFAULT 'N', `job_position` VARCHAR(255) NULL, `emails` TEXT NULL, `firstname` VARCHAR(100) NULL, `lastname` VARCHAR(100) NULL, `system_name` VARCHAR(100) NULL, `description` VARCHAR(255) NULL, `rtype` CHAR(1) NOT NULL, `status` CHAR(1) NOT NULL DEFAULT 'A' COMMENT 'A=Active, X=Deleted', `datecreated` DATETIME NULL, `avatar_url` TEXT NULL, `city` VARCHAR(255) NULL, `countrycode` VARCHAR(2) NULL, `note` TEXT NULL, `time_offset` INT NULL, `language` VARCHAR(10) NULL, `gender` CHAR(1) NOT NULL DEFAULT 'M', `levelid` CHAR(8) NULL, `ldap_id` VARCHAR(255) NULL, `groups` VARCHAR(255) NULL, `ip` VARCHAR(39) NULL, `useragent` TEXT NULL, `screen` VARCHAR(12) NULL, `latitude` FLOAT NULL, `longitude` FLOAT NULL, `last_action_type` CHAR(1) NULL, `last_action_date` DATETIME NULL, `last_action_meta` VARCHAR(80) NULL, PRIMARY KEY (`contactid`), INDEX `fk_qu_la_contacts_qu_la_levels1_idx` (`levelid` ASC), INDEX `fk_parent_contactid` (`parent_contactid` ASC), INDEX `qu_la_contacts_datecreated` (`datecreated` ASC), INDEX `qu_la_contacts_firstname` (`firstname` ASC), INDEX `qu_la_contacts_lastname` (`lastname` ASC), INDEX `qu_la_contacts_system_name` (`system_name` ASC), INDEX `qu_la_contacts_company` (`company`) ) ENGINE = InnoDB; {code} Server settings: {code:java} [mysqld] server-id=14 datadir=/opt/mysql/ socket=/var/lib/mysql/mysql.sock tmpdir=/tmp/ log_bin=mysql-bin expire_logs_days=6 wait_timeout=60 long_query_time=1 slow_query_log=1 slow_query_log_file=/opt/log/slow.log log_output=TABLE userstat = 1 user=root symbolic-links=0 binlog_format=STATEMENT default_storage_engine=InnoDB slave_skip_errors=1062,1396,1690 innodb_autoinc_lock_mode=2 innodb_buffer_pool_size=14G innodb_buffer_pool_instances=10 innodb_log_file_size=1G innodb_log_buffer_size=196M innodb_flush_log_at_trx_commit=1 innodb_thread_concurrency=24 innodb_file_per_table innodb_write_io_threads=24 innodb_read_io_threads=24 innodb_sched_priority_cleaner=39 innodb_adaptive_flushing=1 innodb_purge_threads=5 #transaction-isolation=READ-COMMITTED innodb_adaptive_hash_index_partitions=64 innodb_flush_neighbors=0 innodb_flush_method=O_DIRECT innodb_io_capacity=5000 innodb_io_capacity_max=8000 innodb_lru_scan_depth=1024 innodb_sort_buffer_size=32M innodb_ft_cache_size=80M innodb_ft_total_cache_size=1G slave_parallel_threads=10 log_slave_updates=on performance_schema=off skip-name-resolve max_allowed_packet = 512M query_cache_type=1 query_cache_size = 0 query_cache_limit = 1M query_cache_min_res_unit=1K max_connections = 900 table_open_cache=64K innodb_open_files=64K open_files_limit=1020000 collation-server = utf8_general_ci character-set-server = utf8 log-error=/opt/log/error.log [mysqld_safe] log-error=/opt/log/error.log pid-file=/var/run/mysqld/mysqld.pid malloc-lib=/usr/lib64/libjemalloc.so.1 {code} Exactly the same problem happen on another table when adding fulltext index (table size about 150MB): {code:java} ALTER TABLE qu_g_mails ADD FULLTEXT ft_qu_g_mails1 (subject, body_text, body_html); {code} {code:java} CREATE TABLE `qu_g_mails` ( `mailid` int(10) unsigned NOT NULL AUTO_INCREMENT, `hdr_message_id` varchar(255) DEFAULT NULL, `unique_message_id` varchar(255) NOT NULL, `subject` text, `headers` text, `body_text` longtext, `body_html` longtext, `created` datetime NOT NULL, `delivered` datetime DEFAULT NULL, `from_mail` text, `to_recipients` text, `cc_recipients` text, `bcc_recipients` text, `accountuserid` char(8) DEFAULT NULL, `reply_to` text, PRIMARY KEY (`mailid`), KEY `IDX_qu_g_mails_1` (`accountuserid`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED; {code} |
Adding fulltext index on bigger table crash server
{code:java} 2016-11-04 07:41:26 7f7149bfe700 InnoDB: Assertion failure in thread 140124545345280 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 7:41:26 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.17-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=54 max_threads=902 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2112329 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 2016-11-04 07:41:27 7f714a3ff700 InnoDB: Assertion failure in thread 140124553737984 in file row0merge.cc line 890 InnoDB: Failing assertion: b == &block[0] + buf->total_size + ROW_MERGE_RESERVE_SIZE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 161104 07:41:29 mysqld_safe Number of processes running now: 0 161104 07:41:29 mysqld_safe mysqld restarted 2016-11-04 7:41:29 140601780914208 [Note] /usr/sbin/mysqld (mysqld 10.1.17-MariaDB) starting as process 25262 ... 2016-11-04 7:41:29 140601780914208 [Warning] Although a path was specified for the --log-slow-queries option, log tables are used. To enable logging to files use the --log-output=file option. 2016-11-04 7:41:29 140601780914208 [Warning] option 'innodb-ft-cache-size': unsigned value 83886080 adjusted to 80000000 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: The InnoDB memory heap is disabled 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using Linux native AIO 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Using SSE crc32 instructions 2016-11-04 7:41:29 140601780914208 [Note] InnoDB: Initializing buffer pool, size = 14.0G 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Completed initialization of buffer pool 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Highest supported file format is Barracuda. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Log scan progressed past the checkpoint lsn 5089194166241 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Database was not shutdown normally! 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Starting crash recovery. 2016-11-04 7:41:30 140601780914208 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-11-04 7:41:46 140601780914208 [Note] InnoDB: Processed 81433 .ibd/.isl files 2016-11-04 7:42:02 140601780914208 [Note] InnoDB: Processed 165583 .ibd/.isl files 2016-11-04 7:42:18 140601780914208 [Note] InnoDB: Processed 248732 .ibd/.isl files 2016-11-04 7:42:34 140601780914208 [Note] InnoDB: Processed 331595 .ibd/.isl files 2016-11-04 7:42:50 140601780914208 [Note] InnoDB: Processed 410190 .ibd/.isl files 2016-11-04 7:43:06 140601780914208 [Note] InnoDB: Processed 490369 .ibd/.isl files 2016-11-04 7:43:22 140601780914208 [Note] InnoDB: Processed 571032 .ibd/.isl files 2016-11-04 7:43:38 140601780914208 [Note] InnoDB: Processed 653103 .ibd/.isl files 2016-11-04 7:43:54 140601780914208 [Note] InnoDB: Processed 733414 .ibd/.isl files 2016-11-04 7:44:10 140601780914208 [Note] InnoDB: Processed 811624 .ibd/.isl files 2016-11-04 7:44:26 140601780914208 [Note] InnoDB: Processed 891341 .ibd/.isl files 2016-11-04 7:44:42 140601780914208 [Note] InnoDB: Processed 969320 .ibd/.isl files 2016-11-04 7:44:58 140601780914208 [Note] InnoDB: Processed 1047838 .ibd/.isl files 2016-11-04 7:45:14 140601780914208 [Note] InnoDB: Processed 1123078 .ibd/.isl files 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: Restoring possible half-written data pages 2016-11-04 7:45:25 140601780914208 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 5089199408640 InnoDB: Doing recovery: scanned up to log sequence number 5089204651520 InnoDB: Doing recovery: scanned up to log sequence number 5089209894400 InnoDB: Doing recovery: scanned up to log sequence number 5089215137280 InnoDB: Doing recovery: scanned up to log sequence number 5089220380160 InnoDB: Doing recovery: scanned up to log sequence number 5089225623040 InnoDB: Doing recovery: scanned up to log sequence number 5089230865920 .... etc {code} Problem was started by SQL: ALTER TABLE `qu_la_contacts` ADD FULLTEXT `qu_la_contacts_ftidx` (`firstname` ,`lastname` ,`emails` ,`system_name`) on table, which has about 200k rows {code:java} CREATE TABLE IF NOT EXISTS `qu_la_contacts` ( `contactid` CHAR(8) NOT NULL, `parent_contactid` CHAR(8) NULL, `company` CHAR(1) NOT NULL DEFAULT 'N', `job_position` VARCHAR(255) NULL, `emails` TEXT NULL, `firstname` VARCHAR(100) NULL, `lastname` VARCHAR(100) NULL, `system_name` VARCHAR(100) NULL, `description` VARCHAR(255) NULL, `rtype` CHAR(1) NOT NULL, `status` CHAR(1) NOT NULL DEFAULT 'A' COMMENT 'A=Active, X=Deleted', `datecreated` DATETIME NULL, `avatar_url` TEXT NULL, `city` VARCHAR(255) NULL, `countrycode` VARCHAR(2) NULL, `note` TEXT NULL, `time_offset` INT NULL, `language` VARCHAR(10) NULL, `gender` CHAR(1) NOT NULL DEFAULT 'M', `levelid` CHAR(8) NULL, `ldap_id` VARCHAR(255) NULL, `groups` VARCHAR(255) NULL, `ip` VARCHAR(39) NULL, `useragent` TEXT NULL, `screen` VARCHAR(12) NULL, `latitude` FLOAT NULL, `longitude` FLOAT NULL, `last_action_type` CHAR(1) NULL, `last_action_date` DATETIME NULL, `last_action_meta` VARCHAR(80) NULL, PRIMARY KEY (`contactid`), INDEX `fk_qu_la_contacts_qu_la_levels1_idx` (`levelid` ASC), INDEX `fk_parent_contactid` (`parent_contactid` ASC), INDEX `qu_la_contacts_datecreated` (`datecreated` ASC), INDEX `qu_la_contacts_firstname` (`firstname` ASC), INDEX `qu_la_contacts_lastname` (`lastname` ASC), INDEX `qu_la_contacts_system_name` (`system_name` ASC), INDEX `qu_la_contacts_company` (`company`) ) ENGINE = InnoDB; {code} Server settings: {code:java} [mysqld] server-id=14 datadir=/opt/mysql/ socket=/var/lib/mysql/mysql.sock tmpdir=/tmp/ log_bin=mysql-bin expire_logs_days=6 wait_timeout=60 long_query_time=1 slow_query_log=1 slow_query_log_file=/opt/log/slow.log log_output=TABLE userstat = 1 user=root symbolic-links=0 binlog_format=STATEMENT default_storage_engine=InnoDB slave_skip_errors=1062,1396,1690 innodb_autoinc_lock_mode=2 innodb_buffer_pool_size=14G innodb_buffer_pool_instances=10 innodb_log_file_size=1G innodb_log_buffer_size=196M innodb_flush_log_at_trx_commit=1 innodb_thread_concurrency=24 innodb_file_per_table innodb_write_io_threads=24 innodb_read_io_threads=24 innodb_sched_priority_cleaner=39 innodb_adaptive_flushing=1 innodb_purge_threads=5 #transaction-isolation=READ-COMMITTED innodb_adaptive_hash_index_partitions=64 innodb_flush_neighbors=0 innodb_flush_method=O_DIRECT innodb_io_capacity=5000 innodb_io_capacity_max=8000 innodb_lru_scan_depth=1024 innodb_sort_buffer_size=32M innodb_ft_cache_size=80M innodb_ft_total_cache_size=1G slave_parallel_threads=10 log_slave_updates=on performance_schema=off skip-name-resolve max_allowed_packet = 512M query_cache_type=1 query_cache_size = 0 query_cache_limit = 1M query_cache_min_res_unit=1K max_connections = 900 table_open_cache=64K innodb_open_files=64K open_files_limit=1020000 collation-server = utf8_general_ci character-set-server = utf8 log-error=/opt/log/error.log [mysqld_safe] log-error=/opt/log/error.log pid-file=/var/run/mysqld/mysqld.pid malloc-lib=/usr/lib64/libjemalloc.so.1 {code} Exactly the same problem happen on another table when adding fulltext index (table size about 150MB) (reported in additional comments here: https://jira.mariadb.org/browse/MDEV-9129): {code:java} ALTER TABLE qu_g_mails ADD FULLTEXT ft_qu_g_mails1 (subject, body_text, body_html); {code} {code:java} CREATE TABLE `qu_g_mails` ( `mailid` int(10) unsigned NOT NULL AUTO_INCREMENT, `hdr_message_id` varchar(255) DEFAULT NULL, `unique_message_id` varchar(255) NOT NULL, `subject` text, `headers` text, `body_text` longtext, `body_html` longtext, `created` datetime NOT NULL, `delivered` datetime DEFAULT NULL, `from_mail` text, `to_recipients` text, `cc_recipients` text, `bcc_recipients` text, `accountuserid` char(8) DEFAULT NULL, `reply_to` text, PRIMARY KEY (`mailid`), KEY `IDX_qu_g_mails_1` (`accountuserid`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED; {code} |
Labels | index innodb | index innodb need_feedback |
Labels | index innodb need_feedback | index innodb |
Assignee | Marko Mäkelä [ marko ] |
Fix Version/s | 10.1 [ 16100 ] |
Sprint | 10.1.20 [ 119 ] |
Rank | Ranked lower |
Link |
This issue relates to |
Summary | ADDING FULLTEXT INDEX crash server | CREATE FULLTEXT INDEX with a token longer than 127 bytes crashes server |
Status | Open [ 1 ] | In Progress [ 3 ] |
Assignee | Marko Mäkelä [ marko ] | Jan Lindström [ jplindst ] |
Status | In Progress [ 3 ] | In Review [ 10002 ] |
Assignee | Jan Lindström [ jplindst ] | Marko Mäkelä [ marko ] |
Status | In Review [ 10002 ] | Stalled [ 10000 ] |
issue.field.resolutiondate | 2016-12-05 14:33:23.0 | 2016-12-05 14:33:23.097 |
Component/s | Storage Engine - InnoDB [ 10129 ] | |
Fix Version/s | 10.1.20 [ 22112 ] | |
Fix Version/s | 10.2.3 [ 22115 ] | |
Fix Version/s | 10.1 [ 16100 ] | |
Resolution | Fixed [ 1 ] | |
Status | Stalled [ 10000 ] | Closed [ 6 ] |
Resolution | Fixed [ 1 ] | |
Status | Closed [ 6 ] | Stalled [ 10000 ] |
Fix Version/s | 10.0.30 [ 22313 ] | |
Resolution | Fixed [ 1 ] | |
Status | Stalled [ 10000 ] | Closed [ 6 ] |
Link |
This issue is duplicated by |
Workflow | MariaDB v3 [ 78206 ] | MariaDB v4 [ 151186 ] |