Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6443

Server crashed with assertaion failure in file ha_innodb.cc line 8473

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.0.11
    • Fix Version/s: 10.0.13
    • Component/s: None
    • Environment:
      Linux, kernel-3.10.25-gentoo, 10.0.11-MariaDB

      Description

      Hi, guys.

      I've got server crashes with InnoDB assertation failure 3 times for the last 2 month, when I'm starting to use MariaDB-10.0.11.

      First, in 1 Jun 2014

      2014-06-01 07:32:05 7f6b22403700  InnoDB: Assertion failure in thread 140098112861952 in file ha_innodb.cc line 8473
      InnoDB: Failing assertion: ut_strcmp(index->name, key->name) == 0
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      140601  7:32:05 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.0.11-MariaDB-log
      key_buffer_size=16777216
      read_buffer_size=262144
      max_used_connections=213
      max_threads=502
      thread_count=135
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 412284 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x1621bd28
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f6b22402e48 thread_stack 0x48000
      (my_addr_resolve failure: fork)
      /usr/sbin/mysqld(my_print_stacktrace+0x26) [0xb0d036]
      /usr/sbin/mysqld(handle_fatal_signal+0x398) [0x6ef3d8]
      /lib64/libpthread.so.0(+0x10c20) [0x7f6df6bbdc20]
      /lib64/libc.so.6(gsignal+0x35) [0x7f6df57d9265]
      /lib64/libc.so.6(abort+0x17b) [0x7f6df57da6db]
      /usr/sbin/mysqld() [0x9626af]
      /usr/sbin/mysqld() [0x962a92]
      /usr/sbin/mysqld() [0x96b564]
      /usr/sbin/mysqld(handler::ha_open(TABLE*, char const*, int, unsigned int)+0x3f) [0x6f2fff]
      /usr/sbin/mysqld() [0xaebedd]
      /usr/sbin/mysqld(handler::ha_open(TABLE*, char const*, int, unsigned int)+0x3f) [0x6f2fff]
      /usr/sbin/mysqld(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool)+0x86c) [0x64d20c]
      /usr/sbin/mysqld(open_table(THD*, TABLE_LIST*, st_mem_root*, Open_table_context*)+0x786) [0x5891f6]
      /usr/sbin/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*)+0xd36) [0x58a326]
      /usr/sbin/mysqld(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*)+0x49) [0x58a509]
      /usr/sbin/mysqld() [0x5b95cd]
      /usr/sbin/mysqld(mysql_execute_command(THD*)+0x5088) [0x5c42b8]
      /usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1fc) [0x5c6b5c]
      /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1530) [0x5c80e0]
      /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x19d) [0x66fa4d]
      /usr/sbin/mysqld(handle_one_connection+0x40) [0x66fb10]
      /usr/sbin/mysqld() [0x89bc04]
      /lib64/libpthread.so.0(+0x8eb7) [0x7f6df6bb5eb7]
      /lib64/libc.so.6(clone+0x6d) [0x7f6df5890a7d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f6ade043660): is an invalid pointer
      Connection ID (thread ID): 48545
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      140601  9:18:31 [Note] CONNECT: Version 1.02.0002 March 16, 2014
      2014-06-01 09:18:31 7f80f8ea6740 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
      140601  9:18:31 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      140601  9:18:31 [Note] InnoDB: The InnoDB memory heap is disabled
      140601  9:18:31 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      140601  9:18:31 [Note] InnoDB: Compressed tables use zlib 1.2.8
      140601  9:18:31 [Note] InnoDB: Using Linux native AIO
      140601  9:18:31 [Note] InnoDB: Using CPU crc32 instructions
      140601  9:18:31 [Note] InnoDB: Initializing buffer pool, size = 10.0G
      140601  9:18:32 [Note] InnoDB: Completed initialization of buffer pool
      140601  9:18:32 [Note] InnoDB: Highest supported file format is Barracuda.
      140601  9:18:32 [Note] InnoDB: Log scan progressed past the checkpoint lsn 495548503459
      140601  9:18:32 [Note] InnoDB: Database was not shutdown normally!
      140601  9:18:32 [Note] InnoDB: Starting crash recovery.
      140601  9:18:32 [Note] InnoDB: Reading tablespace information from the .ibd files...
      140601  9:18:39 [Note] InnoDB: Restoring possible half-written data pages 
      140601  9:18:39 [Note] InnoDB: from the doublewrite buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 495553745920
      ....
      InnoDB: Doing recovery: scanned up to log sequence number 495862328921
      140601  9:19:02 [Note] InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      InnoDB: Last MySQL binlog file position 0 674273136, file name ./emsow-bin.000123
      140601  9:19:12 [Note] InnoDB: 128 rollback segment(s) are active.
      140601  9:20:00 [Note] InnoDB: Waiting for purge to start
      140601  9:20:00 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 495862328921
      140601  9:20:00 [Note] Recovering after a crash using emsow-bin
      140601  9:20:02 [Note] Starting crash recovery...
      140601  9:20:02 [Note] Crash recovery finished.
      140601  9:20:02 [Note] Server socket created on IP: '0.0.0.0'.
      140601  9:20:02 [Note] Event Scheduler: Loaded 0 events
      140601  9:20:02 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '10.0.11-MariaDB-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Source distribution

      Then on Jun, 8

      2014-06-08 07:40:59 7f110d35e700  InnoDB: Assertion failure in thread 139711212807936 in file ha_innodb.cc line 8473
      InnoDB: Failing assertion: ut_strcmp(index->name, key->name) == 0
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      140608  7:40:59 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.0.11-MariaDB-log
      key_buffer_size=16777216
      read_buffer_size=262144
      max_used_connections=215
      max_threads=502
      thread_count=140
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 412284 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x1658f4f8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f110d35de48 thread_stack 0x48000
      (my_addr_resolve failure: fork)
      /usr/sbin/mysqld(my_print_stacktrace+0x26) [0xb0d036]
      /usr/sbin/mysqld(handle_fatal_signal+0x398) [0x6ef3d8]
      /lib64/libpthread.so.0(+0x10c20) [0x7f13e4e08c20]
      /lib64/libc.so.6(gsignal+0x35) [0x7f13e3a24265]
      /lib64/libc.so.6(abort+0x17b) [0x7f13e3a256db]
      /usr/sbin/mysqld() [0x9626af]
      /usr/sbin/mysqld() [0x962a92]
      /usr/sbin/mysqld() [0x96b564]
      /usr/sbin/mysqld(handler::ha_open(TABLE*, char const*, int, unsigned int)+0x3f) [0x6f2fff]
      /usr/sbin/mysqld() [0xaebedd]
      /usr/sbin/mysqld(handler::ha_open(TABLE*, char const*, int, unsigned int)+0x3f) [0x6f2fff]
      /usr/sbin/mysqld(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool)+0x86c) [0x64d20c]
      /usr/sbin/mysqld(open_table(THD*, TABLE_LIST*, st_mem_root*, Open_table_context*)+0x786) [0x5891f6]
      /usr/sbin/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*)+0xd36) [0x58a326]
      /usr/sbin/mysqld(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*)+0x49) [0x58a509]
      /usr/sbin/mysqld() [0x5b95cd]
      /usr/sbin/mysqld(mysql_execute_command(THD*)+0x5088) [0x5c42b8]
      /usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1fc) [0x5c6b5c]
      /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1530) [0x5c80e0]
      /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x19d) [0x66fa4d]
      /usr/sbin/mysqld(handle_one_connection+0x40) [0x66fb10]
      /usr/sbin/mysqld() [0x89bc04]
      /lib64/libpthread.so.0(+0x8eb7) [0x7f13e4e00eb7]
      /lib64/libc.so.6(clone+0x6d) [0x7f13e3adba7d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f10cd5a1c20): is an invalid pointer
      Connection ID (thread ID): 158842
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      140608 10:38:16 [Note] CONNECT: Version 1.02.0002 March 16, 2014
      2014-06-08 10:38:16 7f6adec14740 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
      140608 10:38:16 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      140608 10:38:16 [Note] InnoDB: The InnoDB memory heap is disabled
      140608 10:38:16 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      140608 10:38:16 [Note] InnoDB: Compressed tables use zlib 1.2.8
      140608 10:38:16 [Note] InnoDB: Using Linux native AIO
      140608 10:38:16 [Note] InnoDB: Using CPU crc32 instructions
      140608 10:38:16 [Note] InnoDB: Initializing buffer pool, size = 10.0G
      140608 10:38:18 [Note] InnoDB: Completed initialization of buffer pool
      140608 10:38:18 [Note] InnoDB: Highest supported file format is Barracuda.
      140608 10:38:18 [Note] InnoDB: Log scan progressed past the checkpoint lsn 547302896106
      140608 10:38:18 [Note] InnoDB: Database was not shutdown normally!
      140608 10:38:18 [Note] InnoDB: Starting crash recovery.
      140608 10:38:18 [Note] InnoDB: Reading tablespace information from the .ibd files...
      140608 10:38:24 [Note] InnoDB: Restoring possible half-written data pages 
      140608 10:38:24 [Note] InnoDB: from the doublewrite buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 547308138496
      ...
      InnoDB: Doing recovery: scanned up to log sequence number 547571025915
      140608 10:38:45 [Note] InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      InnoDB: Last MySQL binlog file position 0 81382035, file name ./emsow-bin.000131
      140608 10:38:54 [Note] InnoDB: 128 rollback segment(s) are active.
      140608 10:39:29 [Note] InnoDB: Waiting for purge to start
      140608 10:39:29 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 547571025915
      140608 10:39:29 [Note] Recovering after a crash using emsow-bin
      140608 10:39:29 [Note] Starting crash recovery...
      140608 10:39:29 [Note] Crash recovery finished.
      140608 10:39:30 [Note] Server socket created on IP: '0.0.0.0'.
      140608 10:39:30 [Note] Event Scheduler: Loaded 0 events
      140608 10:39:30 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '10.0.11-MariaDB-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Source distribution

      Then, I've got an error as described in MDEV-6424, but in another line of file ha_innodb.cc (line 11652).

      And, I've got the an error yesterday (Jul, 13)

      2014-07-13 09:57:57 7f6b4bfff700 InnoDB: FTS Optimize Removing table emsow_demo4/vendor_files_text
      2014-07-13 10:10:49 7f6b58054700  InnoDB: Assertion failure in thread 140099014969088 in file ha_innodb.cc line 8473
      InnoDB: Failing assertion: ut_strcmp(index->name, key->name) == 0
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      140713 10:10:49 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.0.11-MariaDB-log
      key_buffer_size=16777216
      read_buffer_size=262144
      max_used_connections=240
      max_threads=502
      thread_count=158
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 412284 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x12e2f498
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f6b58053e48 thread_stack 0x48000
      /usr/sbin/mysqld(my_print_stacktrace+0x26)[0xb0d036]
      /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6ef3d8]
      /lib64/libpthread.so.0(+0x10c20)[0x7f6da1a1bc20]
      /lib64/libc.so.6(gsignal+0x35)[0x7f6da0637265]
      /lib64/libc.so.6(abort+0x17b)[0x7f6da06386db]
      /usr/sbin/mysqld[0x9626af]
      /usr/sbin/mysqld[0x962a92]
      /usr/sbin/mysqld[0x96b564]
      /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcij+0x3f)[0x6f2fff]
      /usr/sbin/mysqld[0xaebedd]
      /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcij+0x3f)[0x6f2fff]
      /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x86c)[0x64d20c]
      /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0x786)[0x5891f6]
      /usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0xd36)[0x58a326]
      /usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0x49)[0x58a509]
      /usr/sbin/mysqld[0x5b95cd]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5088)[0x5c42b8]
      /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1fc)[0x5c6b5c]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1530)[0x5c80e0]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x19d)[0x66fa4d]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x66fb10]
      /usr/sbin/mysqld[0x89bc04]
      /lib64/libpthread.so.0(+0x8eb7)[0x7f6da1a13eb7]
      /lib64/libc.so.6(clone+0x6d)[0x7f6da06eea7d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f6a97bbcbe0): is an invalid pointer
      Connection ID (thread ID): 434313
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      140713 12:57:40 [Note] CONNECT: Version 1.02.0002 March 16, 2014
      2014-07-13 12:57:40 7f7a1ef59740 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
      140713 12:57:40 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      140713 12:57:40 [Note] InnoDB: The InnoDB memory heap is disabled
      140713 12:57:40 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      140713 12:57:40 [Note] InnoDB: Compressed tables use zlib 1.2.8
      140713 12:57:40 [Note] InnoDB: Using Linux native AIO
      140713 12:57:40 [Note] InnoDB: Using CPU crc32 instructions
      140713 12:57:40 [Note] InnoDB: Initializing buffer pool, size = 8.0G
      140713 12:57:41 [Note] InnoDB: Completed initialization of buffer pool
      140713 12:57:41 [Note] InnoDB: Highest supported file format is Barracuda.
      140713 12:57:41 [Note] InnoDB: Log scan progressed past the checkpoint lsn 795580159620
      140713 12:57:41 [Note] InnoDB: Database was not shutdown normally!
      140713 12:57:41 [Note] InnoDB: Starting crash recovery.
      140713 12:57:41 [Note] InnoDB: Reading tablespace information from the .ibd files...
      140713 12:57:48 [Note] InnoDB: Restoring possible half-written data pages 
      140713 12:57:48 [Note] InnoDB: from the doublewrite buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 795585402368
      ...
      InnoDB: Doing recovery: scanned up to log sequence number 795810796772
      140713 12:58:05 [Note] InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percent: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      InnoDB: Last MySQL binlog file position 0 5130472, file name ./emsow-bin.000017
      140713 12:58:13 [Note] InnoDB: 128 rollback segment(s) are active.
      140713 12:58:32 [Note] InnoDB: Waiting for purge to start
      140713 12:58:32 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 795810796772
      140713 12:58:33 [Note] Recovering after a crash using emsow-bin
      140713 12:58:33 [Note] Starting crash recovery...
      140713 12:58:33 [Note] Crash recovery finished.
      140713 12:58:33 [Note] Server socket created on IP: '0.0.0.0'.
      140713 12:58:33 [Note] Event Scheduler: Loaded 0 events
      140713 12:58:33 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '10.0.11-MariaDB-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Source distribution

      I detect, that error appear when I've rewrite one DB (demo_db) to another DB (real_db). Total size of real_db about 15GB, all tables are InnoDB engine.

      mysqldump real_db | mysql demo_db

      But, sometimes rewriting finishes good (without error), sometimes it's cause an error.

      The error may be related with FULLTEXT index in updating tables:

      CREATE TABLE `files_text` (
      	`file_id` INT(11) NOT NULL AUTO_INCREMENT,
      	`file_text` LONGTEXT NOT NULL,
      	PRIMARY KEY (`file_id`),
      	FULLTEXT INDEX `file_text` (`file_text`)
      )
      COLLATE='utf8_general_ci'
      ENGINE=InnoDB;
       
      CREATE TABLE `inventory_files_text` (
      	`inventory_file_id` INT(11) NOT NULL AUTO_INCREMENT,
      	`inventory_file_text` LONGTEXT NOT NULL,
      	PRIMARY KEY (`inventory_file_id`),
      	FULLTEXT INDEX `inventory_file_text` (`inventory_file_text`)
      )
      COLLATE='utf8_general_ci'
      ENGINE=InnoDB;
       
      CREATE TABLE `vendor_files_text` (
      	`vid` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
      	`cts` DATETIME NULL DEFAULT NULL,
      	`dts` DATETIME NULL DEFAULT NULL,
      	`cusr` INT(10) UNSIGNED NULL DEFAULT NULL,
      	`dusr` INT(10) UNSIGNED NULL DEFAULT NULL,
      	`del` TINYINT(1) NOT NULL DEFAULT '0',
      	`vendor_file_id` INT(10) UNSIGNED NOT NULL,
      	`vendor_file_text` TEXT NULL,
      	PRIMARY KEY (`vid`),
      	INDEX `cts` (`cts`),
      	INDEX `dts` (`dts`),
      	INDEX `cusr` (`cusr`),
      	INDEX `dusr` (`dusr`),
      	INDEX `vendor_file_id` (`vendor_file_id`),
      	FULLTEXT INDEX `vendor_file_text` (`vendor_file_text`)
      )
      COLLATE='utf8_general_ci'
      ENGINE=InnoDB;

      Because, in some minites before crashing, I've got there messages:

      2014-07-13 06:19:07 7f6b4bfff700 InnoDB: FTS Optimize Removing table demo_db/files_text
      2014-07-13 07:22:58 7f6b4bfff700 InnoDB: FTS Optimize Removing table demo_db/inventory_files_text
      2014-07-13 09:57:57 7f6b4bfff700 InnoDB: FTS Optimize Removing table demo_db/vendor_files_text

      And I found, that table vendor_files_text has a partiotion by del (in mysql official site this fact is a restriction for fulltext index, see http://dev.mysql.com/doc/refman/5.6/en/fulltext-restrictions.html).

      My server:

      # uname -a 
      Linux db 3.10.25-gentoo #1 SMP PREEMPT Wed Jan 29 19:50:21 EST 2014 x86_64 Intel(R) Xeon(R) CPU E5504 @ 2.00GHz GenuineIntel GNU/Linux
      #  free -m
                   total       used       free     shared    buffers     cached
      Mem:         22137      21089       1047          0        689       6065
      -/+ buffers/cache:      14334       7802
      Swap:         3814         54       3760
      # mysql -V 
      mysql  Ver 15.1 Distrib 10.0.11-MariaDB, for Linux (x86_64) using readline 6.2
      # echo "show variables" | mysql
      Variable_name   Value
      aria_block_size 8192
      aria_checkpoint_interval        30
      aria_checkpoint_log_activity    1048576
      aria_force_start_after_recovery_failures        0
      aria_group_commit       none
      aria_group_commit_interval      0
      aria_log_file_size      1073741824
      aria_log_purge_type     immediate
      aria_max_sort_file_size 9223372036853727232
      aria_page_checksum      ON
      aria_pagecache_age_threshold    300
      aria_pagecache_buffer_size      134217728
      aria_pagecache_division_limit   100
      aria_recover    NORMAL
      aria_repair_threads     1
      aria_sort_buffer_size   268434432
      aria_stats_method       nulls_unequal
      aria_sync_log_dir       NEWFILE
      aria_used_for_temp_tables       ON
      auto_increment_increment        1
      auto_increment_offset   1
      autocommit      ON
      automatic_sp_privileges ON
      back_log        150
      basedir /usr
      big_tables      OFF
      binlog_annotate_row_events      OFF
      binlog_cache_size       32768
      binlog_checksum NONE
      binlog_commit_wait_count        0
      binlog_commit_wait_usec 100000
      binlog_direct_non_transactional_updates OFF
      binlog_format   MIXED
      binlog_optimize_thread_scheduling       ON
      binlog_stmt_cache_size  32768
      bulk_insert_buffer_size 8388608
      character_set_client    utf8
      character_set_connection        utf8
      character_set_database  utf8
      character_set_filesystem        binary
      character_set_results   utf8
      character_set_server    utf8
      character_set_system    utf8
      character_sets_dir      /usr/share/mysql/charsets/
      collation_connection    utf8_general_ci
      collation_database      utf8_general_ci
      collation_server        utf8_general_ci
      completion_type NO_CHAIN
      concurrent_insert       AUTO
      connect_conv_size       8192
      connect_indx_map        OFF
      connect_timeout 10
      connect_type_conv       NO
      connect_work_size       67108864
      connect_xtrace  0
      datadir /var/lib/mysql/
      date_format     %Y-%m-%d
      datetime_format %Y-%m-%d %H:%i:%s
      deadlock_search_depth_long      15
      deadlock_search_depth_short     4
      deadlock_timeout_long   50000000
      deadlock_timeout_short  10000
      debug_no_thread_alarm   OFF
      default_master_connection
      default_regex_flags
      default_storage_engine  InnoDB
      default_week_format     0
      delay_key_write ON
      delayed_insert_limit    100
      delayed_insert_timeout  300
      delayed_queue_size      1000
      div_precision_increment 4
      error_count     0
      event_scheduler OFF
      expensive_subquery_limit        100
      expire_logs_days        7
      external_user
      extra_max_connections   1
      extra_port      0
      flush   OFF
      flush_time      0
      foreign_key_checks      ON
      ft_boolean_syntax       + -><()~*:""&|
      ft_max_word_len 84
      ft_min_word_len 4
      ft_query_expansion_limit        20
      ft_stopword_file        (built-in)
      general_log     OFF
      general_log_file        masys.log
      group_concat_max_len    2097152
      gtid_binlog_pos 0-1-1254091
      gtid_binlog_state       0-1-1254091
      gtid_current_pos        0-1-1254091
      gtid_domain_id  0
      gtid_ignore_duplicates  OFF
      gtid_seq_no     0
      gtid_slave_pos
      gtid_strict_mode        OFF
      have_compress   YES
      have_crypt      YES
      have_dynamic_loading    YES
      have_geometry   YES
      have_openssl    YES
      have_profiling  YES
      have_query_cache        YES
      have_rtree_keys YES
      have_ssl        DISABLED
      have_symlink    YES
      histogram_size  0
      histogram_type  SINGLE_PREC_HB
      host_cache_size 128
      hostname        masys
      identity        0
      ignore_builtin_innodb   OFF
      ignore_db_dirs
      in_transaction  0
      init_connect
      init_file
      init_slave
      innodb_adaptive_flushing        ON
      innodb_adaptive_flushing_lwm    10
      innodb_adaptive_hash_index      ON
      innodb_adaptive_hash_index_partitions   1
      innodb_adaptive_max_sleep_delay 150000
      innodb_additional_mem_pool_size 2097152
      innodb_api_bk_commit_interval   5
      innodb_api_disable_rowlock      OFF
      innodb_api_enable_binlog        OFF
      innodb_api_enable_mdl   OFF
      innodb_api_trx_level    0
      innodb_autoextend_increment     64
      innodb_autoinc_lock_mode        1
      innodb_buffer_pool_dump_at_shutdown     OFF
      innodb_buffer_pool_dump_now     OFF
      innodb_buffer_pool_filename     ib_buffer_pool
      innodb_buffer_pool_instances    8
      innodb_buffer_pool_load_abort   OFF
      innodb_buffer_pool_load_at_startup      OFF
      innodb_buffer_pool_load_now     OFF
      innodb_buffer_pool_populate     OFF
      innodb_buffer_pool_size 8589934592
      innodb_change_buffer_max_size   25
      innodb_change_buffering all
      innodb_checksum_algorithm       innodb
      innodb_checksums        ON
      innodb_cleaner_lsn_age_factor   high_checkpoint
      innodb_cmp_per_index_enabled    OFF
      innodb_commit_concurrency       0
      innodb_compression_failure_threshold_pct        5
      innodb_compression_level        6
      innodb_compression_pad_pct_max  50
      innodb_concurrency_tickets      5000
      innodb_corrupt_table_action     assert
      innodb_data_file_path   ibdata1:128M:autoextend
      innodb_data_home_dir
      innodb_disable_sort_file_cache  OFF
      innodb_doublewrite      ON
      innodb_empty_free_list_algorithm        backoff
      innodb_fake_changes     OFF
      innodb_fast_shutdown    1
      innodb_file_format      Antelope
      innodb_file_format_check        ON
      innodb_file_format_max  Antelope
      innodb_file_per_table   ON
      innodb_flush_log_at_timeout     1
      innodb_flush_log_at_trx_commit  1
      innodb_flush_method
      innodb_flush_neighbors  1
      innodb_flushing_avg_loops       30
      innodb_force_load_corrupted     OFF
      innodb_force_recovery   0
      innodb_foreground_preflush      exponential_backoff
      innodb_ft_aux_table
      innodb_ft_cache_size    8000000
      innodb_ft_enable_diag_print     OFF
      innodb_ft_enable_stopword       ON
      innodb_ft_max_token_size        84
      innodb_ft_min_token_size        3
      innodb_ft_num_word_optimize     2000
      innodb_ft_result_cache_limit    2000000000
      innodb_ft_server_stopword_table
      innodb_ft_sort_pll_degree       2
      innodb_ft_total_cache_size      640000000
      innodb_ft_user_stopword_table
      innodb_io_capacity      200
      innodb_io_capacity_max  2000
      innodb_kill_idle_transaction    0
      innodb_large_prefix     OFF
      innodb_lock_wait_timeout        50
      innodb_locking_fake_changes     ON
      innodb_locks_unsafe_for_binlog  OFF
      innodb_log_arch_dir     ./
      innodb_log_arch_expire_sec      0
      innodb_log_archive      OFF
      innodb_log_block_size   512
      innodb_log_buffer_size  8388608
      innodb_log_checksum_algorithm   innodb
      innodb_log_compressed_pages     ON
      innodb_log_file_size    536870912
      innodb_log_files_in_group       2
      innodb_log_group_home_dir       ./
      innodb_lru_scan_depth   1024
      innodb_max_bitmap_file_size     104857600
      innodb_max_changed_pages        1000000
      innodb_max_dirty_pages_pct      75
      innodb_max_dirty_pages_pct_lwm  0
      innodb_max_purge_lag    0
      innodb_max_purge_lag_delay      0
      innodb_mirrored_log_groups      1
      innodb_monitor_disable
      innodb_monitor_enable
      innodb_monitor_reset
      innodb_monitor_reset_all
      innodb_old_blocks_pct   37
      innodb_old_blocks_time  1000
      innodb_online_alter_log_max_size        134217728
      innodb_open_files       300
      innodb_optimize_fulltext_only   OFF
      innodb_page_size        16384
      innodb_print_all_deadlocks      OFF
      innodb_purge_batch_size 300
      innodb_purge_threads    1
      innodb_random_read_ahead        OFF
      innodb_read_ahead_threshold     56
      innodb_read_io_threads  4
      innodb_read_only        OFF
      innodb_replication_delay        0
      innodb_rollback_on_timeout      OFF
      innodb_rollback_segments        128
      innodb_sched_priority_cleaner   19
      innodb_show_locks_held  10
      innodb_show_verbose_locks       0
      innodb_sort_buffer_size 1048576
      innodb_spin_wait_delay  6
      innodb_stats_auto_recalc        ON
      innodb_stats_method     nulls_equal
      innodb_stats_on_metadata        OFF
      innodb_stats_persistent ON
      innodb_stats_persistent_sample_pages    20
      innodb_stats_sample_pages       8
      innodb_stats_transient_sample_pages     8
      innodb_status_output    OFF
      innodb_status_output_locks      OFF
      innodb_strict_mode      OFF
      innodb_support_xa       ON
      innodb_sync_array_size  1
      innodb_sync_spin_loops  30
      innodb_table_locks      ON
      innodb_thread_concurrency       0
      innodb_thread_sleep_delay       10000
      innodb_track_changed_pages      OFF
      innodb_undo_directory   .
      innodb_undo_logs        128
      innodb_undo_tablespaces 0
      innodb_use_atomic_writes        OFF
      innodb_use_fallocate    OFF
      innodb_use_global_flush_log_at_trx_commit       ON
      innodb_use_native_aio   ON
      innodb_use_stacktrace   OFF
      innodb_use_sys_malloc   ON
      innodb_version  5.6.17-65.0
      innodb_write_io_threads 4
      insert_id       0
      interactive_timeout     28800
      join_buffer_size        131072
      join_buffer_space_limit 2097152
      join_cache_level        2
      keep_files_on_create    OFF
      key_buffer_size 16777216
      key_cache_age_threshold 300
      key_cache_block_size    1024
      key_cache_division_limit        100
      key_cache_segments      0
      large_files_support     ON
      large_page_size 0
      large_pages     OFF
      last_gtid
      last_insert_id  0
      lc_messages     en_US
      lc_messages_dir /usr/share/mysql
      lc_time_names   en_US
      license GPL
      local_infile    ON
      lock_wait_timeout       31536000
      locked_in_memory        OFF
      log_bin ON
      log_bin_trust_function_creators ON
      log_error       /var/log/mysql/mysqld.err
      log_output      FILE
      log_queries_not_using_indexes   OFF
      log_slave_updates       OFF
      log_slow_filter admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk
      log_slow_rate_limit     1
      log_slow_verbosity
      log_warnings    1
      long_query_time 30.000000
      low_priority_updates    OFF
      lower_case_file_system  OFF
      lower_case_table_names  0
      master_verify_checksum  OFF
      max_allowed_packet      2097152
      max_binlog_cache_size   18446744073709547520
      max_binlog_size 1073741824
      max_binlog_stmt_cache_size      18446744073709547520
      max_connect_errors      100
      max_connections 500
      max_delayed_threads     20
      max_error_count 64
      max_heap_table_size     16777216
      max_insert_delayed_threads      20
      max_join_size   18446744073709551615
      max_length_for_sort_data        1024
      max_long_data_size      2097152
      max_prepared_stmt_count 16382
      max_relay_log_size      1073741824
      max_seeks_for_key       4294967295
      max_sort_length 1024
      max_sp_recursion_depth  0
      max_tmp_tables  32
      max_user_connections    0
      max_write_lock_count    4294967295
      metadata_locks_cache_size       1024
      metadata_locks_hash_instances   8
      min_examined_row_limit  0
      mrr_buffer_size 262144
      multi_range_count       256
      myisam_block_size       1024
      myisam_data_pointer_size        6
      myisam_max_sort_file_size       9223372036853727232
      myisam_mmap_size        18446744073709551615
      myisam_recover_options  DEFAULT
      myisam_repair_threads   1
      myisam_sort_buffer_size 8388608
      myisam_stats_method     nulls_unequal
      myisam_use_mmap OFF
      net_buffer_length       8192
      net_read_timeout        30
      net_retry_count 10
      net_write_timeout       60
      old     OFF
      old_alter_table OFF
      old_mode
      old_passwords   OFF
      open_files_limit        2505
      optimizer_prune_level   1
      optimizer_search_depth  62
      optimizer_selectivity_sampling_limit    100
      optimizer_switch        index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
      optimizer_use_condition_selectivity     1
      performance_schema      ON
      performance_schema_accounts_size        100
      performance_schema_digests_size 10000
      performance_schema_events_stages_history_long_size      10000
      performance_schema_events_stages_history_size   10
      performance_schema_events_statements_history_long_size  10000
      performance_schema_events_statements_history_size       10
      performance_schema_events_waits_history_long_size       10000
      performance_schema_events_waits_history_size    10
      performance_schema_hosts_size   100
      performance_schema_max_cond_classes     80
      performance_schema_max_cond_instances   2900
      performance_schema_max_file_classes     50
      performance_schema_max_file_handles     32768
      performance_schema_max_file_instances   2154
      performance_schema_max_mutex_classes    200
      performance_schema_max_mutex_instances  8000
      performance_schema_max_rwlock_classes   40
      performance_schema_max_rwlock_instances 3800
      performance_schema_max_socket_classes   10
      performance_schema_max_socket_instances 1020
      performance_schema_max_stage_classes    150
      performance_schema_max_statement_classes        180
      performance_schema_max_table_handles    800
      performance_schema_max_table_instances  12500
      performance_schema_max_thread_classes   50
      performance_schema_max_thread_instances 1100
      performance_schema_session_connect_attrs_size   512
      performance_schema_setup_actors_size    100
      performance_schema_setup_objects_size   100
      performance_schema_users_size   100
      pid_file        /var/run/mysqld/mysqld.pid
      plugin_dir      /usr/lib64/mysql/plugin/
      plugin_maturity unknown
      port    3306
      preload_buffer_size     32768
      profiling       OFF
      profiling_history_size  15
      progress_report_time    5
      protocol_version        10
      proxy_user
      pseudo_slave_mode       OFF
      pseudo_thread_id        17151
      query_alloc_block_size  8192
      query_cache_limit       1048576
      query_cache_min_res_unit        4096
      query_cache_size        0
      query_cache_strip_comments      OFF
      query_cache_type        ON
      query_cache_wlock_invalidate    OFF
      query_prealloc_size     8192
      rand_seed1      0
      rand_seed2      0
      range_alloc_block_size  4096
      read_buffer_size        262144
      read_only       OFF
      read_rnd_buffer_size    524288
      relay_log
      relay_log_index
      relay_log_info_file     relay-log.info
      relay_log_purge ON
      relay_log_recovery      OFF
      relay_log_space_limit   0
      replicate_annotate_row_events   OFF
      replicate_do_db
      replicate_do_table
      replicate_events_marked_for_skip        replicate
      replicate_ignore_db
      replicate_ignore_table
      replicate_wild_do_table
      replicate_wild_ignore_table
      report_host
      report_password
      report_port     3306
      report_user
      rowid_merge_buff_size   8388608
      rpl_recovery_rank       0
      secure_auth     OFF
      secure_file_priv
      server_id       1
      skip_external_locking   ON
      skip_name_resolve       OFF
      skip_networking OFF
      skip_replication        OFF
      skip_show_database      OFF
      slave_compressed_protocol       OFF
      slave_ddl_exec_mode     IDEMPOTENT
      slave_domain_parallel_threads   0
      slave_exec_mode STRICT
      slave_load_tmpdir       /tmp/
      slave_max_allowed_packet        1073741824
      slave_net_timeout       3600
      slave_parallel_max_queued       131072
      slave_parallel_threads  0
      slave_skip_errors       OFF
      slave_sql_verify_checksum       ON
      slave_transaction_retries       10
      slave_type_conversions
      slow_launch_time        2
      slow_query_log  ON
      slow_query_log_file     /var/log/mysql/slow.log
      socket  /var/run/mysqld/mysqld.sock
      sort_buffer_size        524288
      sql_auto_is_null        OFF
      sql_big_selects ON
      sql_buffer_result       OFF
      sql_log_bin     ON
      sql_log_off     OFF
      sql_mode
      sql_notes       ON
      sql_quote_show_create   ON
      sql_safe_updates        OFF
      sql_select_limit        18446744073709551615
      sql_slave_skip_counter  0
      sql_warnings    OFF
      ssl_ca
      ssl_capath
      ssl_cert
      ssl_cipher
      ssl_crl
      ssl_crlpath
      ssl_key
      storage_engine  InnoDB
      stored_program_cache    256
      sync_binlog     0
      sync_frm        ON
      sync_master_info        0
      sync_relay_log  0
      sync_relay_log_info     0
      system_time_zone        EDT
      table_definition_cache  400
      table_open_cache        64
      thread_cache_size       0
      thread_concurrency      10
      thread_handling one-thread-per-connection
      thread_pool_idle_timeout        60
      thread_pool_max_threads 500
      thread_pool_oversubscribe       3
      thread_pool_size        8
      thread_pool_stall_limit 500
      thread_stack    294912
      time_format     %H:%i:%s
      time_zone       SYSTEM
      timed_mutexes   OFF
      timestamp       1405324688.886423
      tmp_table_size  16777216
      tmpdir  /tmp/
      transaction_alloc_block_size    8192
      transaction_prealloc_size       4096
      tx_isolation    REPEATABLE-READ
      tx_read_only    OFF
      unique_checks   ON
      updatable_views_with_limit      YES
      use_stat_tables NEVER
      userstat        OFF
      version 10.0.11-MariaDB-log
      version_comment Source distribution
      version_compile_machine x86_64
      version_compile_os      Linux
      version_malloc_library  system
      wait_timeout    28800
      warning_count   0

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jplindst Jan Lindström
                Reporter:
                kirill.komlev Kirill Komlev
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: