[MDEV-7374] Losing connection to MySQL while running ALTER TABLE Created: 2014-12-25  Updated: 2015-02-17  Due: 2014-12-31  Resolved: 2015-01-01

Status: Closed
Project: MariaDB Server
Component/s: wsrep
Affects Version/s: 10.1.2
Fix Version/s: 5.5.42-galera, 10.0.16-galera, 10.1.3

Type: Bug Priority: Critical
Reporter: Koustubh Sinkar Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 0
Labels: alter, wsrep
Environment:

OS: Fedora 19
Architecture: x86_64
Downloaded from http://yum.mariadb.org/10.1/fedora20-amd64

Attachments: Zip Archive mdev_7373.zip    
Issue Links:
Relates
relates to MDEV-7373 Error while running mysql_upgrade Closed

 Description   

While running the following command:

ALTER TABLE `feed_properties` CHANGE `label_1` `label_1` varchar(1
 
00) DEFAULT NULL

MySQL(MariaDB) goes away.

ERROR_LOG:

2014-12-25 20:02:33 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).    [91/483]
 
2014-12-25 20:02:33 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."tr8n_languages" but the required system tables mysql.innodb_table_stats and
 
 mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:34 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:34 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."tr8n_translation_keys" but the required system tables mysql.innodb_table_st
 
ats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:34 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:34 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."users" but the required system tables mysql.innodb_table_stats and mysql.in
 
nodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:35 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:35 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."feed_settings" but the required system tables mysql.innodb_table_stats and 
mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:35 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:35 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."global_settings" but the required system tables mysql.innodb_table_stats an
 
d mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:37 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:37 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."tasks" but the required system tables mysql.innodb_table_stats and mysql.in
 
nodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:37 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:37 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."events" but the required system tables mysql.innodb_table_stats and mysql.i
 
nnodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:37 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:37 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."site_pages" but the required system tables mysql.innodb_table_stats and mys
 
ql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:38 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:38 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."wikis" but the required system tables mysql.innodb_table_stats and mysql.in
 
nodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:38 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:38 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."todos" but the required system tables mysql.innodb_table_stats and mysql.in
 
nodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:38 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:38 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."domains" but the required system tables mysql.innodb_table_stats and mysql.
 
innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:39 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:39 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."presence_options" but the required system tables mysql.innodb_table_stats a
 
nd mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:43 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:43 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."schema_migrations" but the required system tables mysql.innodb_table_stats 
and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
2014-12-25 20:02:43 7f14fee87700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:43 7f14fee87700 InnoDB: Error: Fetch of persistent statistics requested for table "mangoapps_dev"."feed_properties" but the required system tables mysql.innodb_table_stats an
 
d mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
141225 20:02:45 [ERROR] mysqld got signal 11 ;                                                                                                                                         [45/483]
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.2-MariaDB-wsrep
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=1
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467237 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f14e8ff3008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f14fee86cf0 thread_stack 0x48000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f14ff93887e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x37d)[0x7f14ff49df2d]
 
/lib64/libpthread.so.0(+0x308960f6d0)[0x7f14feaf56d0]
 
/usr/sbin/mysqld(thd_get_ha_data+0xa)[0x7f14ff2fd32a]
 
/usr/sbin/mysqld(_Z13get_trans_logP3THD+0x31)[0x7f14ff55ee71]
 
/usr/sbin/mysqld(wsrep_run_wsrep_commit+0x62c)[0x7f14ff43cfec]
 
/usr/sbin/mysqld(+0x71790a)[0x7f14ff63c90a]
 
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x3bf)[0x7f14ff4a7caf]
 
/usr/sbin/mysqld(+0x3555bb)[0x7f14ff27a5bb]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x3501)[0x7f14ff3b4f91]
 
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x65a)[0x7f14ff3f832a]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x182b)[0x7f14ff32dbfb]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20b)[0x7f14ff334d5b]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x21a5)[0x7f14ff337cc5]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x134)[0x7f14ff3383f4]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f14ff3f551a]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7f14ff3f56e0]                                                                                                                            [3/483]
 
/lib64/libpthread.so.0(+0x3089607ee5)[0x7f14feaedee5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f14fcf0db8d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f14d1823020): is an invalid pointer
 
Connection ID (thread ID): 2
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pus
 
hdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cach
 
e=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=of
 
f,table_elimination=on,extended_keys=on,exists_to_in=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
141225 20:02:45 mysqld_safe Number of processes running now: 0
 
141225 20:02:45 mysqld_safe mysqld restarted
 
141225 20:02:46 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.bjEOXw' --pid-file='/var/lib/mysql/c4.pune.mangopulse.com-recover.pid'
 
141225 20:02:50 mysqld_safe WSREP: Position recovery skipped
 
141225 20:02:50 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
141225 20:02:50 [Note] InnoDB: The InnoDB memory heap is disabled
 
141225 20:02:50 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
141225 20:02:50 [Note] InnoDB: Memory barrier is not used
 
141225 20:02:50 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
141225 20:02:50 [Note] InnoDB: Using Linux native AIO
 
141225 20:02:50 [Note] InnoDB: Using CPU crc32 instructions
 
141225 20:02:50 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 
141225 20:02:50 [Note] InnoDB: Completed initialization of buffer pool
 
141225 20:02:50 [Note] InnoDB: Highest supported file format is Barracuda.
 
141225 20:02:50 [Note] InnoDB: 128 rollback segment(s) are active.
 
141225 20:02:50 [Note] InnoDB: Waiting for purge to start
 
141225 20:02:50 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.21-70.0 started; log sequence number 95625746288
 
141225 20:02:50 [Note] Plugin 'FEEDBACK' is disabled.
 
141225 20:02:50 [Note] Server socket created on IP: '::'.
 
141225 20:02:50 [Note] Event Scheduler: Loaded 0 events
 
141225 20:02:50 [Note] WSREP: Read nil XID from storage engines, skipping position init
 
141225 20:02:50 [Note] WSREP: wsrep_load(): loading provider library 'none'
 
2014-12-25 20:02:50 7fbdff6e2700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).
 
2014-12-25 20:02:50 7fbdff6e2700 InnoDB: Error: Fetch of persistent statistics requested for table "mysql"."gtid_slave_pos" but the required system tables mysql.innodb_table_stats and mysql.i
 
nnodb_index_stats are not present or have unexpected structure. Using transient stats instead.
 
141225 20:02:50 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.1.2-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r4123



 Comments   
Comment by Elena Stepanova [ 2014-12-25 ]

Please attach your cnf file(s) and/or the output of SHOW GLOBAL VARIABLES.

If you have wsrep_on=1, please set it to 0 and try again.

Comment by Koustubh Sinkar [ 2014-12-25 ]

I have a attached a file with all the requisite settings now.

Comment by Elena Stepanova [ 2014-12-26 ]

With the priceless help from Daniel Black, here is the test case:

--source include/have_innodb.inc
 
 
 
create table t1 (i int) engine=InnoDB;
 
insert into t1 values (1),(2),(3),(4),(5),(6),(7),(8);
 
insert into t1 select a.* from t1 a, t1 b, t1 c, t1 d, t1 e;
 
alter table t1 force, algorithm=copy;

Stack trace from 10.1 rev ea01fff5053acc6526bc995c2a3c85474c844508

 
#3  <signal handler called>
 
#4  0x00007fd4fe90c3eb in thd_get_ha_data (thd=0x7fd4f4dc8070, hton=0x7fd4f6fc2270) at 10.1/sql/sql_class.cc:637
 
#5  0x00007fd4fec608bd in get_trans_log (thd=0x7fd4f4dc8070) at 10.1/sql/log.cc:9731
 
#6  0x00007fd4feaea5be in wsrep_run_wsrep_commit (thd=0x7fd4f4dc8070, hton=0x0, all=true) at 10.1/sql/wsrep_hton.cc:390
 
#7  0x00007fd4f371b4f7 in ha_innodb::write_row (this=0x7fd4e7c82088, record=0x7fd4e7c2b188 "\375\001") at 10.1/storage/innobase/handler/ha_innodb.cc:7668
 
#8  0x00007fd4feb84b67 in handler::ha_write_row (this=0x7fd4e7c82088, buf=0x7fd4e7c2b188 "\375\001") at 10.1/sql/handler.cc:5864
 
#9  0x00007fd4fea110a1 in copy_data_between_tables (thd=0x7fd4f4dc8070, from=0x7fd4e7ca4870, to=0x7fd4e7d1e070, create=..., ignore=false, order_num=0, order=0x0, copied=0x7fd4fe4045b8, deleted=0x7fd4fe4045c0, keys_onoff=Alter_info::LEAVE_AS_IS, alter_ctx=0x7fd4fe405240) at 10.1/sql/sql_table.cc:9393
 
#10 0x00007fd4fea0f84d in mysql_alter_table (thd=0x7fd4f4dc8070, new_db=0x7fd4e7df4758 "test", new_name=0x0, create_info=0x7fd4fe405e40, table_list=0x7fd4e7df4180, alter_info=0x7fd4fe405db0, order_num=0, order=0x0, ignore=false) at 10.1/sql/sql_table.cc:8904
 
#11 0x00007fd4fea7e5a0 in Sql_cmd_alter_table::execute (this=0x7fd4e7df4768, thd=0x7fd4f4dc8070) at 10.1/sql/sql_alter.cc:329
 
#12 0x00007fd4fe957ce7 in mysql_execute_command (thd=0x7fd4f4dc8070) at 10.1/sql/sql_parse.cc:5521
 
#13 0x00007fd4fe95bb94 in mysql_parse (thd=0x7fd4f4dc8070, rawbuf=0x7fd4e7df4088 "alter table t1 force, algorithm=copy", length=36, parser_state=0x7fd4fe4071c0) at 10.1/sql/sql_parse.cc:7024
 
#14 0x00007fd4fe94b431 in dispatch_command (command=COM_QUERY, thd=0x7fd4f4dc8070, packet=0x7fd4f4ea0071 "alter table t1 force, algorithm=copy", packet_length=36) at 10.1/sql/sql_parse.cc:1458
 
#15 0x00007fd4fe94a254 in do_command (thd=0x7fd4f4dc8070) at 10.1/sql/sql_parse.cc:1089
 
#16 0x00007fd4fea7986c in do_handle_one_connection (thd_arg=0x7fd4f4dc8070) at 10.1/sql/sql_connect.cc:1347
 
#17 0x00007fd4fea795b1 in handle_one_connection (arg=0x7fd4f4dc8070) at 10.1/sql/sql_connect.cc:1258
 
#18 0x00007fd4ff017bb6 in pfs_spawn_thread (arg=0x7fd4f4e379f0) at 10.1/storage/perfschema/pfs.cc:1860
 
#19 0x00007fd4fe048b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
 
#20 0x00007fd4fc0df20d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

Comment by Elena Stepanova [ 2014-12-26 ]

ksinkar,

answering your question about the workaround, in this case (when you are dealing with your own tables), you can try the following:

CREATE TABLE feed_properties_new LIKE feed_properties;
 
ALTER TABLE feed_properties_new CHANGE `label_1` `label_1` varchar(100) DEFAULT NULL;
 
INSERT INTO feed_properties_new SELECT * FROM feed_properties;
 
RENAME TABLE feed_properties_new INTO feed_properties;

If there is traffic on the server that you cannot control and it can use this table as well, you might want to do it under a lock.

Comment by Koustubh Sinkar [ 2014-12-29 ]

Thanks for the help elenst.

For users who are using a Rails application and are facing similar problem with their MariaDB servers can apply the following solution (it is a variation from Elena's solution)

mysql> RENAME TABLE feed_properties TO feed_properties_old;
mysql> CREATE TABLE feed_properties LIKE feed_properties_old;
$ bundle exec rake db:migrate
mysql> INSERT INTO feed_properties SELECT * FROM feed_properties_old;

You can optionally drop your old tables in case you are sure about your data integrity.

This will not work if you are also ALTERing/UPDATEing values in other columns of the same table based on certain other properties in your rails migration. In that case you will have to add the above statements in your migration file around the ALTER statement.

Comment by Nirbhay Choubey (Inactive) [ 2014-12-31 ]

http://lists.askmonty.org/pipermail/commits/2014-December/007224.html

Comment by Jan Lindström (Inactive) [ 2014-12-31 ]

ok to push.

Comment by Nirbhay Choubey (Inactive) [ 2015-01-01 ]

https://github.com/MariaDB/server/commit/6f4f8c5f8ae55f6fe05990e047c10309d598eb28

Comment by Nirbhay Choubey (Inactive) [ 2015-01-25 ]

http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3555
http://bazaar.launchpad.net/~maria-captains/maria/maria-10.0-galera/revision/3916

Generated at Thu Feb 08 07:19:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.