[MDEV-9530] Galera Cluster crashed after some hours of usage Created: 2016-02-08  Updated: 2021-11-25  Resolved: 2021-11-25

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.1.11
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Madarassy Laszlo Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: debian, galera, xtrabackup
Environment:

Debian Jessie

Attachments: Text File syslog.txt    
Issue Links:
Duplicate
is duplicated by MDEV-9663 InnoDB assertion failure: *cursor->in... Closed
Relates
relates to MDEV-16759 InnoDB: Assertion failure in thread 1... Closed

 Description   

I have set up a two node Galera cluster with xtrabackup sync method.
I had to install xtrabackup package (2.3.3) from percona repo, because the version shipped with mariadb debian repo didn't support Mariadb 10.1.

The setup was successful, the cluster worked fine for 18 hours without problems.
After 16 hours the mysqld crashed, when I restarted it the cluster won't started anymore, so I have to disable cluster to use my system again.

Feb  5 14:25:02 master mysqld: 2016-02-05 14:25:02 7f0284c44700  InnoDB: Assertion failure in thread 139648794117888 in file row0ins.cc line 283
 
Feb  5 14:25:02 master mysqld: InnoDB: Failing assertion: *cursor->index->name == TEMP_INDEX_PREFIX
 
Feb  5 14:25:02 master mysqld: InnoDB: We intentionally generate a memory trap.
 
Feb  5 14:25:02 master mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
 
Feb  5 14:25:02 master mysqld: InnoDB: If you get repeated assertion failures or crashes, even
 
Feb  5 14:25:02 master mysqld: InnoDB: immediately after the mysqld startup, there may be
 
Feb  5 14:25:02 master mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
 
Feb  5 14:25:02 master mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
Feb  5 14:25:02 master mysqld: InnoDB: about forcing recovery.
 
Feb  5 14:25:02 master mysqld: 160205 14:25:02 [ERROR] mysqld got signal 6 ;
 
Feb  5 14:25:02 master mysqld: This could be because you hit a bug. It is also possible that this binary
 
Feb  5 14:25:02 master mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
Feb  5 14:25:02 master mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Feb  5 14:25:02 master mysqld: 
Feb  5 14:25:02 master mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
Feb  5 14:25:02 master mysqld: 
Feb  5 14:25:02 master mysqld: We will try our best to scrape up some info that will hopefully help
 
Feb  5 14:25:02 master mysqld: diagnose the problem, but since we have already crashed, 
Feb  5 14:25:02 master mysqld: something is definitely wrong and this may fail.
 
Feb  5 14:25:02 master mysqld: 
Feb  5 14:25:02 master mysqld: Server version: 10.1.11-MariaDB-1~jessie
 
Feb  5 14:25:02 master mysqld: key_buffer_size=134217728
 
Feb  5 14:25:02 master mysqld: read_buffer_size=8388608
 
Feb  5 14:25:02 master mysqld: max_used_connections=34
 
Feb  5 14:25:02 master mysqld: max_threads=72
 
Feb  5 14:25:02 master mysqld: thread_count=14
 
Feb  5 14:25:02 master mysqld: It is possible that mysqld could use up to 
Feb  5 14:25:02 master mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6620588 K  bytes of memory
 
Feb  5 14:25:02 master mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
Feb  5 14:25:02 master mysqld: 
Feb  5 14:25:02 master mysqld: Thread pointer: 0x0x7f01d0239008
 
Feb  5 14:25:02 master mysqld: Attempting backtrace. You can use the following information to find out
 
Feb  5 14:25:02 master mysqld: where mysqld died. If you see no messages after this, something went
 
Feb  5 14:25:02 master mysqld: terribly wrong...
 
Feb  5 14:25:02 master mysqld: stack_bottom = 0x7f0284c43df8 thread_stack 0x40000
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f02887a432e]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x34d)[0x7f02882e68fd]
 
Feb  5 14:25:02 master mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f02879158d0]
 
Feb  5 14:25:02 master mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f0285a07147]
 
Feb  5 14:25:02 master mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f0285a08528]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x7a5803)[0x7f02884eb803]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x7a7a22)[0x7f02884eda22]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x7d17db)[0x7f02885177db]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x7d5f8a)[0x7f028851bf8a]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x7b5ca5)[0x7f02884fbca5]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x704923)[0x7f028844a923]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0x462)[0x7f02882f0fe2]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x1639)[0x7f02881fe6a9]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3d57)[0x7f0288160317]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20e)[0x7f0288165a6e]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(+0x420269)[0x7f0288166269]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x181c)[0x7f02881680bc]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x16e)[0x7f0288168d9e]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x182)[0x7f0288231fc2]
 
Feb  5 14:25:02 master mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f0288232180]
 
Feb  5 14:25:02 master mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f028790e0a4]
 
Feb  5 14:25:03 master mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0285aba9cd]
 
Feb  5 14:25:03 master mysqld: 
Feb  5 14:25:03 master mysqld: Trying to get some variables.
 
Feb  5 14:25:03 master mysqld: Some pointers may be invalid and cause the dump to abort.
 
Feb  5 14:25:03 master mysqld: Query (0x7f01b22d5020): is an invalid pointer
 
Feb  5 14:25:03 master mysqld: Connection ID (thread ID): 71882
 
Feb  5 14:25:03 master mysqld: Status: NOT_KILLED



 Comments   
Comment by Elena Stepanova [ 2016-02-10 ]

nirbhay_c, jplindst,

do you think it might be Galera-specific because of the way it treats DDL vs DML, or does it look like a genuine InnoDB problem?

Comment by Madarassy Laszlo [ 2016-02-10 ]

Hi,

Today I had this bug again, but without galera enabled, so you are right, it's not a galera related issue.
How can we investigate it? Now I have binlogs (previously it wasn't enabled).

Laszlo

Comment by Nirbhay Choubey (Inactive) [ 2016-04-01 ]

https://bugs.launchpad.net/percona-server/+bug/1195614

Comment by Marko Mäkelä [ 2018-07-31 ]

If innodb_adaptive_hash_index=ON and the workload is using any DDL (especially TRUNCATE), this could be a duplicate of MDEV-16759.

Comment by Elena Stepanova [ 2018-07-31 ]

The version doesn't fit, though. MDEV-16759 seems to fail in the range of 10.1.29 - 10.1.34, while this is for 10.1.11.

Comment by Elena Stepanova [ 2021-11-25 ]

Closing as 10.1 is now EOL, and we don't have information of higher versions being affected.

Generated at Thu Feb 08 07:35:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.