[MDEV-444] Crash in ALTER TABLE ADD PRIMARY KEY ..., ADD INDEX Created: 2012-08-10  Updated: 2012-09-05  Resolved: 2012-09-05

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 5.5.25
Fix Version/s: 5.5.27

Type: Bug Priority: Major
Reporter: Timour Katchaounov (Inactive) Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None
Environment:

OpenSuse 11.2 64-bit


Attachments: File mdev-444.data.gz    

 Description   

When attempting to add indexes to a smaller version of table lineitem from DBT3, the ALTER TABLE crashes the server. How to reproduce:

  • Use DBT3 scale 10
  • Run the following statements:

create table t_ps_partkey as
(select distinct ps_partkey, ps_suppkey
from partsupp
where ps_partkey in (select p_partkey from part where p_name like 'forest%'));

alter table t_ps_partkey add index i_ps_partkey(ps_partkey, ps_suppkey);

create table lineitem_small as
(select distinct lineitem.* from lineitem, t_ps_partkey
where l_partkey = ps_partkey
and l_suppkey = ps_suppkey
and l_shipdate >= date('1994-01-01')
and l_shipdate < date('1994-01-01') + interval '1' year );

ALTER TABLE lineitem_small
ADD PRIMARY KEY (l_orderkey, l_linenumber),
ADD INDEX i_l_shipdate(l_shipdate),
ADD INDEX i_l_suppkey_partkey (l_partkey, l_suppkey),
ADD INDEX i_l_partkey (l_partkey),
ADD INDEX i_l_suppkey (l_suppkey),
ADD INDEX i_l_receiptdate (l_receiptdate),
ADD INDEX i_l_orderkey (l_orderkey),
ADD INDEX i_l_orderkey_quantity (l_orderkey, l_quantity),
ADD INDEX i_l_commitdate (l_commitdate);

The last statement results in a crash:

InnoDB: DEBUG: update_statistics for dbt3/lineitem_small.
120809 23:51:18 InnoDB: Assertion failure in thread 140347972777728 in file buf0buf.ic line 515
InnoDB: Failing assertion: mutex_own(&buf_pool->LRU_list_mutex)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.

The stack trace is:

#0 0x00007fa5b074ad95 in raise () from /lib64/libc.so.6
#1 0x00007fa5b074c2ab in abort () from /lib64/libc.so.6
#2 0x0000000000b365f3 in buf_page_set_sticky (bpage=0x7fa55b773cc0) at /home/tsk/mprog/src/5.5/storage/xtradb/include/buf0buf.ic:515
#3 0x0000000000b494aa in buf_flush_yield (buf_pool=0x4bedf08, bpage=0x7fa55b773cc0) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:399
#4 0x0000000000b495a9 in buf_flush_try_yield (buf_pool=0x4bedf08, bpage=0x7fa55b773cc0, processed=1024) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:445
#5 0x0000000000b498b9 in buf_flush_or_remove_pages (buf_pool=0x4bedf08, id=30) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:569
#6 0x0000000000b4992c in buf_flush_dirty_pages (buf_pool=0x4bedf08, id=30) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:600
#7 0x0000000000b49e2f in buf_LRU_flush_or_remove_pages (id=30, buf_remove=BUF_REMOVE_FLUSH_NO_WRITE) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:789
#8 0x0000000000b74ada in fil_delete_tablespace (id=30, evict_all=0) at /home/tsk/mprog/src/5.5/storage/xtradb/fil/fil0fil.c:2383
#9 0x0000000000add415 in row_drop_table_for_mysql (name=0x7fa5200f0098 "dbt3/lineitem_small@00232", trx=0x7fa520063ac8, drop_db=0) at /home/tsk/mprog/src/5.5/storage/xtradb/row/row0mysql.c:3565
#10 0x0000000000ad73e9 in row_merge_drop_table (trx=0x7fa520063ac8, table=0x7fa5200683d8) at /home/tsk/mprog/src/5.5/storage/xtradb/row/row0merge.c:2634
#11 0x0000000000ac2c9a in ha_innobase::final_add_index (this=0x7fa5200deee8, add_arg=0x7fa5200da970, commit=true) at /home/tsk/mprog/src/5.5/storage/xtradb/handler/handler0alter.cc:1023
#12 0x00000000006c51c6 in mysql_alter_table (thd=0x7e38870, new_db=0x7fa520006cb8 "dbt3", new_name=0x7fa5200066a8 "lineitem_small", create_info=0x7fa54f0fdd00, table_list=0x7fa5200066f0, alter_info=0x7fa54f0fde00, order_num=0, order=0x0, ignore=false, require_online=false) at /home/tsk/mprog/src/5.5/sql/sql_table.cc:6946
#13 0x000000000093f508 in Alter_table_statement::execute (this=0x7fa5200074b8, thd=0x7e38870) at /home/tsk/mprog/src/5.5/sql/sql_alter.cc:106
#14 0x000000000062f6ee in mysql_execute_command (thd=0x7e38870) at /home/tsk/mprog/src/5.5/sql/sql_parse.cc:4459
#15 0x000000000063272a in mysql_parse (thd=0x7e38870, rawbuf=0x7fa5200062c8 "ALTER TABLE lineitem_small\n ADD PRIMARY KEY (l_orderkey, l_linenumber),\n ADD INDEX i_l_shipdate(l_shipdate),\n ADD INDEX i_l_suppkey_partkey (l_partkey, l_suppkey),\n ADD INDEX i_l_partkey (l_pa"..., length=435, parser_state=0x7fa54f0fe660) at /home/tsk/mprog/src/5.5/sql/sql_parse.cc:5736

The crash is in the function: buf_page_set_sticky(), on the line:
ut_ad(mutex_own(&buf_pool->LRU_list_mutex));

p buf_pool->LRU_list_mutex
$5 = {
event = 0x4bee840,
lock_word = 0 '\000',
waiters = 0,
list =

{ prev = 0x4bee150, next = 0x4bedf08 }

,
cfile_name = 0xe0b8f0 "/home/src/5.5/storage/xtradb/buf/buf0buf.c",
cline = 1267,
thread_id = 18446744073709551615,
magic_n = 979585,
count_os_wait = 0,
count_using = 3578904,
count_spin_loop = 0,
count_spin_rounds = 0,
count_os_yield = 0,
lspent_time = 0,
lmax_spent_time = 0,
mutex_type = 0,
cmutex_name = 0xe0c1ce "&buf_pool->LRU_list_mutex",
pfs_psi = 0x0
}



 Comments   
Comment by Timour Katchaounov (Inactive) [ 2012-08-10 ]

p mutex_own(&buf_pool->LRU_list_mutex)
$9 = 0

Comment by Timour Katchaounov (Inactive) [ 2012-08-10 ]

In order to reproduce the crash, it is sufficient to run only this reduced statement:

ALTER TABLE lineitem_small ADD PRIMARY KEY (l_orderkey, l_linenumber);

Comment by Elena Stepanova [ 2012-08-11 ]

I haven't got the crash so far. Here is what I did in my latest (cleanest) attempt:

  • start server with innodb-file-per-table=1 and innodb-file-format=Barracuda, tmpdir pointing at a disk with more space, the rest is default;
  • feed mariadb-tools/dbt3_benchmark/dbt3_mysql/make-dbt3-db_pre_create_PK.sql to the server. No modifications were done to the script, apart from pointing it to SF10 data;
  • stop server (to backup the database);
  • start server with the same exact parameters as before;
  • run create statements table t_ps_partkey, alter table t_ps_partkey, create table lineitem_small as given in the description;
  • run ALTER TABLE lineitem_small ADD PRIMARY KEY as given in the comment;

The last ALTER TABLE finished fairly fast (in ~30 seconds), no problems or errors in the log file.
After that I tried to drop the PK and run the big ALTER TABLE lineitem_small from the description, adding many indexes, which also went all right.

I was running it on perro, which is openSUSE 11.3 x86_64 8 Gb.

A couple observations:

First, despite what was discussed on IRC, it seems to depend on tmpdir after all. In previous dirty attempts I had been having a problem with disk space on alter table until I switched tmpdir from /tmp (which had about 2 Gb of free space) to the main storage which had about 17 Gb.
Second, when I was having the disk problems, I was getting an error "wrong key", which was preceded by the server complaining into the error log about not enough disk space (error 28). I'm wondering if in certain cases it asserts instead of aborting the operation.

Timour, you were going to check the tmpdir when you were reproducing the issue. Did you have enough space in there?
Also, could you please provide the entire command line for the server startup (or your cnf config if you are using one). I am wondering if it behaves differently for me because I'm using a different set of parameters.

Comment by Timour Katchaounov (Inactive) [ 2012-08-13 ]

I changed the temporary directory to my home dir, where there was ~25 GB free space.
Still, just executing the statement that adds a PK results in a crash.

The server is a debug build, built with ./BUILD/compile-amd64-debug-max-no-ndb

The server was started from the source tree with the following command line:

./sql/mysqld --no-defaults --skip-networking --datadir=/home/<user>/dbdata/dbt3s10-mdb-5.5-a --tmpdir=/home/<user>/tmp/db --socket=/home/<user>/dbdata/dbt3s10-mdb-5.5-a/mysql.sock --lc-messages-dir=/home/<user>/mprog/src/5.5/sql/share --character-sets-dir=/home/<user>/mprog/src/5.5/sql/share/charsets --lower-case-table-names=1 --key_buffer_size=64M --table_cache=256 --sort_buffer_size=64M --join_buffer_size=512M --read_buffer_size=16M --innodb_buffer_pool_size=1024M --innodb_additional_mem_pool_size=128M --innodb_log_buffer_size=64M --innodb_log_file_size=256M --innodb_file_per_table --innodb_data_file_path=ibdata1:128M:autoextend

Comment by Elena Stepanova [ 2012-08-15 ]

Attached mdev-444.data – the load file for the provided test case. ~12 Mb uncompressed, contains 100K rows.

Comment by Elena Stepanova [ 2012-08-15 ]

Reproducible on the current MariaDB 5.5 revno 3492 with XtraDB, percona-server revno 288.
Not reproducible on MariaDB 5.5 with InnoDB plugin, current MySQL 5.5, 5.6, MariaDB 5.3.
No crash on a release build.

  1. Test case
  2. (please note that you need to download and extract
  3. the attached file mdev-444.data, put it in <basedir>/mysql-test).
  4. run as
  5. perl ./mtr -mysqld=innodb_log_buffer_size=64M --mysqld=innodb_log_file_size=64M --mysqld=-innodb_buffer_pool_size=128M main.mdev-444
  6. where t/mdev-444.test is the test file

--source include/have_innodb.inc

SET GLOBAL innodb_file_per_table=ON;

USE test;
DROP TABLE IF EXISTS t1;

CREATE TABLE t1 (
`l_orderkey` int(11) NOT NULL DEFAULT '0',
`l_partkey` int(11) DEFAULT NULL,
`l_suppkey` int(11) DEFAULT NULL,
`l_linenumber` int(11) NOT NULL DEFAULT '0',
`l_quantity` double DEFAULT NULL,
`l_extendedprice` double DEFAULT NULL,
`l_discount` double DEFAULT NULL,
`l_tax` double DEFAULT NULL,
`l_returnflag` char(1) DEFAULT NULL,
`l_linestatus` char(1) DEFAULT NULL,
`l_shipDATE` date DEFAULT NULL,
`l_commitDATE` date DEFAULT NULL,
`l_receiptDATE` date DEFAULT NULL,
`l_shipinstruct` char(25) DEFAULT NULL,
`l_shipmode` char(10) DEFAULT NULL,
`l_comment` varchar(44) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

LOAD DATA LOCAL INFILE 'mdev-444.data' INTO TABLE t1;

--enable_reconnect
--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
restart
EOF
--shutdown_server 60
--source include/wait_until_connected_again.inc

ALTER TABLE t1 ADD PRIMARY KEY (l_orderkey, l_linenumber);

  1. Cleanup
    DROP TABLE t1;
  1. End of test case
Comment by Elena Stepanova [ 2012-08-17 ]

Also filed for Percona Server as https://bugs.launchpad.net/percona-server/+bug/1038225

Comment by Vladislav Vaintroub [ 2012-08-30 ]

After conversation with Laurinas on IRC, found out that fix for the bug will be included into the next Percona Server, which should be released ASAP. It could make sense to wait and merge the new XtraDB before MariaDB release, since it has at least one more critical bugfix

Comment by Sergei Golubchik [ 2012-09-05 ]

merged with XtraDB from Percona-Server-5.5.27-rel28.1

Generated at Thu Feb 08 06:28:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.