[MDEV-6651] MariaDB galera cluster crashes Created: 2014-08-27  Updated: 2014-09-03  Resolved: 2014-09-03

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.0.12-galera
Fix Version/s: 10.0.13-galera

Type: Bug Priority: Major
Reporter: Michael Spector Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 1
Labels: galera

Attachments: File datafile.dat     File fk.test    

 Description   

Version: '10.0.12-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r4002
WSREP: FK key len exceeded 45 5 45
140827 10:09:45 [ERROR] WSREP: FK key set failed: 11 (1 1), index: PRIMARY quartz/QRTZ5_JOB_DETAILS, INSERT INTO QRTZ5_TRIGGERS (TRIGGER_NAME, TRIGGER_GROUP, JOB_NAME, JOB_GROUP, IS_VOLATILE, DESCRIPTION, NEXT_FIRE_TIME, PREV_FIRE_TIME, TRIGGER_STATE, TRIGGER_TYPE, START_TIME, END_TIME, CALENDAR_NAME, MISFIRE_INSTR, JOB_DATA, PRIORITY)  VALUES('admin        PentahoSystemVersionCheck       1409134185226', 'admin', 'admin PentahoSystemVersionCheck       1409134185226', 'admin', 0, NULL, 1409134185224, -1, 'WAITING', 'SIMPLE', 1409134185224, 0, NULL, 4, _binary '', 5)
InnoDB: unknown error code 11
2014-08-27 10:09:45 7f7996f96700  InnoDB: Assertion failure in thread 140160200697600 in file row0mysql.cc line 684
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
140827 10:09:45 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.0.12-MariaDB-wsrep
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=9
max_threads=153
thread_count=11
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467215 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x7f79789d9008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f7996f95ce0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb914fb]
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x743318]
/lib64/libpthread.so.0(+0xf710)[0x7f79a3ff2710]
/lib64/libc.so.6(gsignal+0x35)[0x7f79a2653925]
/lib64/libc.so.6(abort+0x175)[0x7f79a2655105]
/usr/sbin/mysqld[0x930f0b]
/usr/sbin/mysqld[0x932100]
/usr/sbin/mysqld[0x89e567]
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f)[0x74ca6f]
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x62)[0x5c1392]
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0xc1b)[0x5c662b]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x161f)[0x5d9dff]
/usr/sbin/mysqld[0x5e0412]
/usr/sbin/mysqld[0x5e0dcb]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x16d0)[0x5e28a0]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x132)[0x5e3072]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x54b)[0x6a190b]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6a1a02]
/lib64/libpthread.so.0(+0x79d1)[0x7f79a3fea9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f79a2709b6d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f799287d020): is an invalid pointer
Connection ID (thread ID): 277
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.



 Comments   
Comment by Johnny Antonsen [ 2014-09-02 ]

I'm having a very similar problem on MariaDB 10.0.12. Here are the details:

galera 25.3.5-wheezy
mysqld Ver 10.0.12-MariaDB-1~trusty-wsrep-log for debian-linux-gnu on x86_64 (mariadb.org binary distribution, wsrep_25.10.r4002)
Ubuntu 14.04.1 LTS

WSREP: FK key len exceeded 8 36 16
140828 0:44:51 [ERROR] WSREP: FK key set failed: 11 (0 0), index: PRIMARY neutron/ipallocations, DELETE FROM ports WHERE ports.id = 'f37aa3fe-ab99-4d0f-a566-6cd3169d7516'
WSREP: foreign key append failed: 11
InnoDB: unknown error code 11
2014-08-28 00:44:51 7f7f1ddff700 InnoDB: Assertion failure in thread 140183938791168 in file row0mysql.cc line 684
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
140828 0:44:51 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.12-MariaDB-1~trusty-wsrep-log
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=57
max_threads=302
thread_count=32
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1992928 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f7e9657c008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f7f1ddfedf0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f7f62ce293e]
/usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f7f62821d17]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f7f60fb0340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f7f605fef89]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f7f60602398]
/usr/sbin/mysqld(+0x89aed2)[0x7f7f62b32ed2]
/usr/sbin/mysqld(+0x7fc5f7)[0x7f7f62a945f7]
/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0x3e7)[0x7f7f6282ce37]
/usr/sbin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyyP13select_result+0xf31)[0x7f7f629643f1]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2b37)[0x7f7f6269c337]
/usr/sbin/mysqld(+0x40aba0)[0x7f7f626a2ba0]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c21)[0x7f7f626a4d31]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x158)[0x7f7f626a5938]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1fb)[0x7f7f627725db]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7f7f627727b0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f7f60fa8182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7f606c338d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f7f5e856020): is an invalid pointer
Connection ID (thread ID): 6037
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

Hi,

Is it possible to get full unedited error log and show create table for table neutron/ipallocations and if it has foreign keys, table where it references or table that references to this table.

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

Hmm, I could not repeat with 10.0.13-galera, let me see if this is repeatable with 10.0.12-galera.

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

No, it does not repeat.

Comment by Johnny Antonsen [ 2014-09-03 ]

You are trying both 10.0.13 and 10.0.12 galera with replication? It works fine on a single node, but fails once we run with replication.

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

Both runs with 2-node galera cluster.

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

Hmm, that key used on delete does not exists on database, that could be the why it does not fail, I would need database before the delete.

Comment by Johnny Antonsen [ 2014-09-03 ]

I'm a bit uncertain on how to produce that data for you. Is there any more I can provide you with to help out in the investigation? This is a development system we are working on, so there are no backups run of the databases, which means I can't produce a database from prior to the delete query.

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

Ok, using the key on query, I generated data and I was able to repeat the problem on 10.0.12-galera

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

File fk.test should be copied to mysql-test/suite/galera/t directory and datafile.dat to mysql-test/std_data directory

Comment by Jan Lindström (Inactive) [ 2014-09-03 ]

Problem is already fixed on 10.0.13-galera.

Generated at Thu Feb 08 07:13:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.