Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6288

Innodb causes server crash after disk full, then can't ALTER TABLE any more

    XMLWordPrintable

    Details

      Description

      Tried to do an ALTER TABLE to add a column, which ran out of disk space. No big deal, but that crashed the entire mysqld. However, the #sql tempfile was not removed, and now, I get this error when I try to repeat the ALTER TABLE:

      MariaDB [willbryant_production]> alter table comments add testcol int;
      ERROR 1050 (42S01): Table 'willbryant_production/#sql-ib133' already exists

      Can't find a way to fix that - have tried removing the file and restarting mysqld, but still getting the same error.

      Here's the original crash report (note also that it refers to the mysql 5.6 website - is that expected?):

      Jun  2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700 InnoDB: Error: Write to file ./willbryant_production/#sql-ib133.ibd failed at offset 59768832.
      Jun  2 17:58:45 loaf mysqld: InnoDB: 1048576 bytes should have been written, only 0 were written.
      Jun  2 17:58:45 loaf mysqld: InnoDB: Operating system error number 28.
      Jun  2 17:58:45 loaf mysqld: InnoDB: Check that your OS and file system support files of this size.
      Jun  2 17:58:45 loaf mysqld: InnoDB: Check also that the disk is not full or a disk quota exceeded.
      Jun  2 17:58:45 loaf mysqld: InnoDB: Error number 28 means 'No space left on device'.
      Jun  2 17:58:45 loaf mysqld: InnoDB: Some operating system error numbers are described at
      Jun  2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
      Jun  2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700  InnoDB: Assertion failure in thread 140022000006912 in file os0file.cc line 4738
      Jun  2 17:58:45 loaf mysqld: InnoDB: Failing assertion: ret
      Jun  2 17:58:45 loaf mysqld: InnoDB: We intentionally generate a memory trap.
      Jun  2 17:58:45 loaf mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      Jun  2 17:58:45 loaf mysqld: InnoDB: If you get repeated assertion failures or crashes, even
      Jun  2 17:58:45 loaf mysqld: InnoDB: immediately after the mysqld startup, there may be
      Jun  2 17:58:45 loaf mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
      Jun  2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      Jun  2 17:58:45 loaf mysqld: InnoDB: about forcing recovery.
      Jun  2 17:58:45 loaf mysqld: 140602 17:58:45 [ERROR] mysqld got signal 6 ;
      Jun  2 17:58:45 loaf mysqld: This could be because you hit a bug. It is also possible that this binary
      Jun  2 17:58:45 loaf mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Jun  2 17:58:45 loaf mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: We will try our best to scrape up some info that will hopefully help
      Jun  2 17:58:45 loaf mysqld: diagnose the problem, but since we have already crashed, 
      Jun  2 17:58:45 loaf mysqld: something is definitely wrong and this may fail.
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: Server version: 10.0.11-MariaDB-1~saucy
      Jun  2 17:58:45 loaf mysqld: key_buffer_size=134217728
      Jun  2 17:58:45 loaf mysqld: read_buffer_size=2097152
      Jun  2 17:58:45 loaf mysqld: max_used_connections=5
      Jun  2 17:58:45 loaf mysqld: max_threads=102
      Jun  2 17:58:45 loaf mysqld: thread_count=4
      Jun  2 17:58:45 loaf mysqld: It is possible that mysqld could use up to 
      Jun  2 17:58:45 loaf mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759849 K  bytes of memory
      Jun  2 17:58:45 loaf mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: Thread pointer: 0x0x7f5936a93008
      Jun  2 17:58:45 loaf mysqld: Attempting backtrace. You can use the following information to find out
      Jun  2 17:58:45 loaf mysqld: where mysqld died. If you see no messages after this, something went
      Jun  2 17:58:45 loaf mysqld: terribly wrong...
      Jun  2 17:58:45 loaf mysqld: 140602 17:58:45 [Warning] Aborted connection 48 to db: 'willbryant_production' user: 'willbryant' host: 'localhost' (Unknown error)
      Jun  2 17:58:45 loaf mysqld: stack_bottom = 0x7f5969922da0 thread_stack 0x48000
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f596a489b9e]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f5969fd4147]
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbb0)[0x7f5968feebb0]
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f5967e07f77]
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f5967e0b5e8]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x810320)[0x7f596a285320]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9382f5)[0x7f596a3ad2f5]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93d9aa)[0x7f596a3b29aa]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93e327)[0x7f596a3b3327]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93eac8)[0x7f596a3b3ac8]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93ee21)[0x7f596a3b3e21]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9426a9)[0x7f596a3b76a9]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x944a09)[0x7f596a3b9a09]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8b8399)[0x7f596a32d399]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8c6276)[0x7f596a33b276]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84c8a8)[0x7f596a2c18a8]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84f78f)[0x7f596a2c478f]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x7c376d)[0x7f596a23876d]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2c47)[0x7f5969ef3187]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x55d)[0x7f5969f3be1d]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1939)[0x7f5969e64e49]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x27a)[0x7f5969e6ad6a]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x191b)[0x7f5969e6cc4b]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1e3)[0x7f5969f38263]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f5969f38350]
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x6fdb6d)[0x7f596a172b6d]
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f6e)[0x7f5968fe6f6e]
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5967ecb9cd]
      Jun  2 17:58:46 loaf mysqld: 
      Jun  2 17:58:46 loaf mysqld: Trying to get some variables.
      Jun  2 17:58:46 loaf mysqld: Some pointers may be invalid and cause the dump to abort.
      Jun  2 17:58:46 loaf mysqld: Query (0x7f592fc1f020): is an invalid pointer
      Jun  2 17:58:46 loaf mysqld: Connection ID (thread ID): 50
      Jun  2 17:58:46 loaf mysqld: Status: NOT_KILLED
      Jun  2 17:58:46 loaf mysqld: 
      Jun  2 17:58:46 loaf mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
      Jun  2 17:58:46 loaf mysqld: 
      Jun  2 17:58:46 loaf mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Jun  2 17:58:46 loaf mysqld: information that should help you find out what is causing the crash.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              will.bryant Will Bryant
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: