Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6288

Innodb causes server crash after disk full, then can't ALTER TABLE any more

Details

    Description

      Tried to do an ALTER TABLE to add a column, which ran out of disk space. No big deal, but that crashed the entire mysqld. However, the #sql tempfile was not removed, and now, I get this error when I try to repeat the ALTER TABLE:

      MariaDB [willbryant_production]> alter table comments add testcol int;
      		 
      ERROR 1050 (42S01): Table 'willbryant_production/#sql-ib133' already exists

      Can't find a way to fix that - have tried removing the file and restarting mysqld, but still getting the same error.

      Here's the original crash report (note also that it refers to the mysql 5.6 website - is that expected?):

      Jun  2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700 InnoDB: Error: Write to file ./willbryant_production/#sql-ib133.ibd failed at offset 59768832.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: 1048576 bytes should have been written, only 0 were written.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Operating system error number 28.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Check that your OS and file system support files of this size.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Check also that the disk is not full or a disk quota exceeded.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Error number 28 means 'No space left on device'.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Some operating system error numbers are described at
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
      		 
      Jun  2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700  InnoDB: Assertion failure in thread 140022000006912 in file os0file.cc line 4738
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Failing assertion: ret
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: We intentionally generate a memory trap.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: If you get repeated assertion failures or crashes, even
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: immediately after the mysqld startup, there may be
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      		 
      Jun  2 17:58:45 loaf mysqld: InnoDB: about forcing recovery.
      		 
      Jun  2 17:58:45 loaf mysqld: 140602 17:58:45 [ERROR] mysqld got signal 6 ;
      		 
      Jun  2 17:58:45 loaf mysqld: This could be because you hit a bug. It is also possible that this binary
      		 
      Jun  2 17:58:45 loaf mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      		 
      Jun  2 17:58:45 loaf mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
      		 
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: We will try our best to scrape up some info that will hopefully help
      		 
      Jun  2 17:58:45 loaf mysqld: diagnose the problem, but since we have already crashed, 
      Jun  2 17:58:45 loaf mysqld: something is definitely wrong and this may fail.
      		 
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: Server version: 10.0.11-MariaDB-1~saucy
      		 
      Jun  2 17:58:45 loaf mysqld: key_buffer_size=134217728
      		 
      Jun  2 17:58:45 loaf mysqld: read_buffer_size=2097152
      		 
      Jun  2 17:58:45 loaf mysqld: max_used_connections=5
      		 
      Jun  2 17:58:45 loaf mysqld: max_threads=102
      		 
      Jun  2 17:58:45 loaf mysqld: thread_count=4
      		 
      Jun  2 17:58:45 loaf mysqld: It is possible that mysqld could use up to 
      Jun  2 17:58:45 loaf mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759849 K  bytes of memory
      		 
      Jun  2 17:58:45 loaf mysqld: Hope that's ok; if not, decrease some variables in the equation.
      		 
      Jun  2 17:58:45 loaf mysqld: 
      Jun  2 17:58:45 loaf mysqld: Thread pointer: 0x0x7f5936a93008
      		 
      Jun  2 17:58:45 loaf mysqld: Attempting backtrace. You can use the following information to find out
      		 
      Jun  2 17:58:45 loaf mysqld: where mysqld died. If you see no messages after this, something went
      		 
      Jun  2 17:58:45 loaf mysqld: terribly wrong...
      		 
      Jun  2 17:58:45 loaf mysqld: 140602 17:58:45 [Warning] Aborted connection 48 to db: 'willbryant_production' user: 'willbryant' host: 'localhost' (Unknown error)
      		 
      Jun  2 17:58:45 loaf mysqld: stack_bottom = 0x7f5969922da0 thread_stack 0x48000
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f596a489b9e]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f5969fd4147]
      		 
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbb0)[0x7f5968feebb0]
      		 
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f5967e07f77]
      		 
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f5967e0b5e8]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x810320)[0x7f596a285320]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9382f5)[0x7f596a3ad2f5]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93d9aa)[0x7f596a3b29aa]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93e327)[0x7f596a3b3327]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93eac8)[0x7f596a3b3ac8]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93ee21)[0x7f596a3b3e21]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9426a9)[0x7f596a3b76a9]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x944a09)[0x7f596a3b9a09]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8b8399)[0x7f596a32d399]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8c6276)[0x7f596a33b276]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84c8a8)[0x7f596a2c18a8]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84f78f)[0x7f596a2c478f]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x7c376d)[0x7f596a23876d]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2c47)[0x7f5969ef3187]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x55d)[0x7f5969f3be1d]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1939)[0x7f5969e64e49]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x27a)[0x7f5969e6ad6a]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x191b)[0x7f5969e6cc4b]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1e3)[0x7f5969f38263]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f5969f38350]
      		 
      Jun  2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x6fdb6d)[0x7f596a172b6d]
      		 
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f6e)[0x7f5968fe6f6e]
      		 
      Jun  2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5967ecb9cd]
      		 
      Jun  2 17:58:46 loaf mysqld: 
      Jun  2 17:58:46 loaf mysqld: Trying to get some variables.
      		 
      Jun  2 17:58:46 loaf mysqld: Some pointers may be invalid and cause the dump to abort.
      		 
      Jun  2 17:58:46 loaf mysqld: Query (0x7f592fc1f020): is an invalid pointer
      		 
      Jun  2 17:58:46 loaf mysqld: Connection ID (thread ID): 50
      		 
      Jun  2 17:58:46 loaf mysqld: Status: NOT_KILLED
      		 
      Jun  2 17:58:46 loaf mysqld: 
      Jun  2 17:58:46 loaf mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
      		 
      Jun  2 17:58:46 loaf mysqld: 
      Jun  2 17:58:46 loaf mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      		 
      Jun  2 17:58:46 loaf mysqld: information that should help you find out what is causing the crash.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-8442 InnoDB: Assertion failure in file handler0alter.cc line 131

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            will.bryant Will Bryant created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Assignee Elena Stepanova [ elenst ]
            serg Sergei Golubchik made changes -
            Description Tried to do an ALTER TABLE to add a column, which ran out of disk space. No big deal, but that crashed the entire mysqld. However, the #sql tempfile was not removed, and now, I get this error when I try to repeat the ALTER TABLE:

            MariaDB [willbryant_production]> alter table comments add testcol int;
            ERROR 1050 (42S01): Table 'willbryant_production/#sql-ib133' already exists

            Can't find a way to fix that - have tried removing the file and restarting mysqld, but still getting the same error.

            Here's the original crash report (note also that it refers to the mysql 5.6 website - is that expected?):

            Jun 2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700 InnoDB: Error: Write to file ./willbryant_production/#sql-ib133.ibd failed at offset 59768832.
            Jun 2 17:58:45 loaf mysqld: InnoDB: 1048576 bytes should have been written, only 0 were written.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Operating system error number 28.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Check that your OS and file system support files of this size.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Check also that the disk is not full or a disk quota exceeded.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Error number 28 means 'No space left on device'.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Some operating system error numbers are described at
            Jun 2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
            Jun 2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700 InnoDB: Assertion failure in thread 140022000006912 in file os0file.cc line 4738
            Jun 2 17:58:45 loaf mysqld: InnoDB: Failing assertion: ret
            Jun 2 17:58:45 loaf mysqld: InnoDB: We intentionally generate a memory trap.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            Jun 2 17:58:45 loaf mysqld: InnoDB: If you get repeated assertion failures or crashes, even
            Jun 2 17:58:45 loaf mysqld: InnoDB: immediately after the mysqld startup, there may be
            Jun 2 17:58:45 loaf mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Jun 2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            Jun 2 17:58:45 loaf mysqld: InnoDB: about forcing recovery.
            Jun 2 17:58:45 loaf mysqld: 140602 17:58:45 [ERROR] mysqld got signal 6 ;
            Jun 2 17:58:45 loaf mysqld: This could be because you hit a bug. It is also possible that this binary
            Jun 2 17:58:45 loaf mysqld: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 2 17:58:45 loaf mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: We will try our best to scrape up some info that will hopefully help
            Jun 2 17:58:45 loaf mysqld: diagnose the problem, but since we have already crashed,
            Jun 2 17:58:45 loaf mysqld: something is definitely wrong and this may fail.
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: Server version: 10.0.11-MariaDB-1~saucy
            Jun 2 17:58:45 loaf mysqld: key_buffer_size=134217728
            Jun 2 17:58:45 loaf mysqld: read_buffer_size=2097152
            Jun 2 17:58:45 loaf mysqld: max_used_connections=5
            Jun 2 17:58:45 loaf mysqld: max_threads=102
            Jun 2 17:58:45 loaf mysqld: thread_count=4
            Jun 2 17:58:45 loaf mysqld: It is possible that mysqld could use up to
            Jun 2 17:58:45 loaf mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759849 K bytes of memory
            Jun 2 17:58:45 loaf mysqld: Hope that's ok; if not, decrease some variables in the equation.
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: Thread pointer: 0x0x7f5936a93008
            Jun 2 17:58:45 loaf mysqld: Attempting backtrace. You can use the following information to find out
            Jun 2 17:58:45 loaf mysqld: where mysqld died. If you see no messages after this, something went
            Jun 2 17:58:45 loaf mysqld: terribly wrong...
            Jun 2 17:58:45 loaf mysqld: 140602 17:58:45 [Warning] Aborted connection 48 to db: 'willbryant_production' user: 'willbryant' host: 'localhost' (Unknown error)
            Jun 2 17:58:45 loaf mysqld: stack_bottom = 0x7f5969922da0 thread_stack 0x48000
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f596a489b9e]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f5969fd4147]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbb0)[0x7f5968feebb0]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f5967e07f77]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f5967e0b5e8]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x810320)[0x7f596a285320]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9382f5)[0x7f596a3ad2f5]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93d9aa)[0x7f596a3b29aa]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93e327)[0x7f596a3b3327]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93eac8)[0x7f596a3b3ac8]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93ee21)[0x7f596a3b3e21]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9426a9)[0x7f596a3b76a9]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x944a09)[0x7f596a3b9a09]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8b8399)[0x7f596a32d399]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8c6276)[0x7f596a33b276]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84c8a8)[0x7f596a2c18a8]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84f78f)[0x7f596a2c478f]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x7c376d)[0x7f596a23876d]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2c47)[0x7f5969ef3187]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x55d)[0x7f5969f3be1d]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1939)[0x7f5969e64e49]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x27a)[0x7f5969e6ad6a]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x191b)[0x7f5969e6cc4b]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1e3)[0x7f5969f38263]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f5969f38350]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x6fdb6d)[0x7f596a172b6d]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f6e)[0x7f5968fe6f6e]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5967ecb9cd]
            Jun 2 17:58:46 loaf mysqld:
            Jun 2 17:58:46 loaf mysqld: Trying to get some variables.
            Jun 2 17:58:46 loaf mysqld: Some pointers may be invalid and cause the dump to abort.
            Jun 2 17:58:46 loaf mysqld: Query (0x7f592fc1f020): is an invalid pointer
            Jun 2 17:58:46 loaf mysqld: Connection ID (thread ID): 50
            Jun 2 17:58:46 loaf mysqld: Status: NOT_KILLED
            Jun 2 17:58:46 loaf mysqld:
            Jun 2 17:58:46 loaf mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
            Jun 2 17:58:46 loaf mysqld:
            Jun 2 17:58:46 loaf mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 2 17:58:46 loaf mysqld: information that should help you find out what is causing the crash.
            		 Tried to do an ALTER TABLE to add a column, which ran out of disk space. No big deal, but that crashed the entire mysqld. However, the #sql tempfile was not removed, and now, I get this error when I try to repeat the ALTER TABLE:
            {noformat}
            MariaDB [willbryant_production]> alter table comments add testcol int;
            ERROR 1050 (42S01): Table 'willbryant_production/#sql-ib133' already exists
            {noformat}
            Can't find a way to fix that - have tried removing the file and restarting mysqld, but still getting the same error.

            Here's the original crash report (note also that it refers to the mysql 5.6 website - is that expected?):
            {noformat}
            Jun 2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700 InnoDB: Error: Write to file ./willbryant_production/#sql-ib133.ibd failed at offset 59768832.
            Jun 2 17:58:45 loaf mysqld: InnoDB: 1048576 bytes should have been written, only 0 were written.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Operating system error number 28.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Check that your OS and file system support files of this size.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Check also that the disk is not full or a disk quota exceeded.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Error number 28 means 'No space left on device'.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Some operating system error numbers are described at
            Jun 2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
            Jun 2 17:58:45 loaf mysqld: 2014-06-02 17:58:45 7f5969923700 InnoDB: Assertion failure in thread 140022000006912 in file os0file.cc line 4738
            Jun 2 17:58:45 loaf mysqld: InnoDB: Failing assertion: ret
            Jun 2 17:58:45 loaf mysqld: InnoDB: We intentionally generate a memory trap.
            Jun 2 17:58:45 loaf mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            Jun 2 17:58:45 loaf mysqld: InnoDB: If you get repeated assertion failures or crashes, even
            Jun 2 17:58:45 loaf mysqld: InnoDB: immediately after the mysqld startup, there may be
            Jun 2 17:58:45 loaf mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
            Jun 2 17:58:45 loaf mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            Jun 2 17:58:45 loaf mysqld: InnoDB: about forcing recovery.
            Jun 2 17:58:45 loaf mysqld: 140602 17:58:45 [ERROR] mysqld got signal 6 ;
            Jun 2 17:58:45 loaf mysqld: This could be because you hit a bug. It is also possible that this binary
            Jun 2 17:58:45 loaf mysqld: or one of the libraries it was linked against is corrupt, improperly built,
            Jun 2 17:58:45 loaf mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: We will try our best to scrape up some info that will hopefully help
            Jun 2 17:58:45 loaf mysqld: diagnose the problem, but since we have already crashed,
            Jun 2 17:58:45 loaf mysqld: something is definitely wrong and this may fail.
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: Server version: 10.0.11-MariaDB-1~saucy
            Jun 2 17:58:45 loaf mysqld: key_buffer_size=134217728
            Jun 2 17:58:45 loaf mysqld: read_buffer_size=2097152
            Jun 2 17:58:45 loaf mysqld: max_used_connections=5
            Jun 2 17:58:45 loaf mysqld: max_threads=102
            Jun 2 17:58:45 loaf mysqld: thread_count=4
            Jun 2 17:58:45 loaf mysqld: It is possible that mysqld could use up to
            Jun 2 17:58:45 loaf mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759849 K bytes of memory
            Jun 2 17:58:45 loaf mysqld: Hope that's ok; if not, decrease some variables in the equation.
            Jun 2 17:58:45 loaf mysqld:
            Jun 2 17:58:45 loaf mysqld: Thread pointer: 0x0x7f5936a93008
            Jun 2 17:58:45 loaf mysqld: Attempting backtrace. You can use the following information to find out
            Jun 2 17:58:45 loaf mysqld: where mysqld died. If you see no messages after this, something went
            Jun 2 17:58:45 loaf mysqld: terribly wrong...
            Jun 2 17:58:45 loaf mysqld: 140602 17:58:45 [Warning] Aborted connection 48 to db: 'willbryant_production' user: 'willbryant' host: 'localhost' (Unknown error)
            Jun 2 17:58:45 loaf mysqld: stack_bottom = 0x7f5969922da0 thread_stack 0x48000
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f596a489b9e]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f5969fd4147]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbb0)[0x7f5968feebb0]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f5967e07f77]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f5967e0b5e8]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x810320)[0x7f596a285320]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9382f5)[0x7f596a3ad2f5]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93d9aa)[0x7f596a3b29aa]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93e327)[0x7f596a3b3327]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93eac8)[0x7f596a3b3ac8]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x93ee21)[0x7f596a3b3e21]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x9426a9)[0x7f596a3b76a9]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x944a09)[0x7f596a3b9a09]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8b8399)[0x7f596a32d399]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x8c6276)[0x7f596a33b276]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84c8a8)[0x7f596a2c18a8]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x84f78f)[0x7f596a2c478f]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x7c376d)[0x7f596a23876d]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2c47)[0x7f5969ef3187]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x55d)[0x7f5969f3be1d]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1939)[0x7f5969e64e49]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x27a)[0x7f5969e6ad6a]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x191b)[0x7f5969e6cc4b]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1e3)[0x7f5969f38263]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f5969f38350]
            Jun 2 17:58:46 loaf mysqld: /usr/sbin/mysqld(+0x6fdb6d)[0x7f596a172b6d]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7f6e)[0x7f5968fe6f6e]
            Jun 2 17:58:46 loaf mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5967ecb9cd]
            Jun 2 17:58:46 loaf mysqld:
            Jun 2 17:58:46 loaf mysqld: Trying to get some variables.
            Jun 2 17:58:46 loaf mysqld: Some pointers may be invalid and cause the dump to abort.
            Jun 2 17:58:46 loaf mysqld: Query (0x7f592fc1f020): is an invalid pointer
            Jun 2 17:58:46 loaf mysqld: Connection ID (thread ID): 50
            Jun 2 17:58:46 loaf mysqld: Status: NOT_KILLED
            Jun 2 17:58:46 loaf mysqld:
            Jun 2 17:58:46 loaf mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=off
            Jun 2 17:58:46 loaf mysqld:
            Jun 2 17:58:46 loaf mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            Jun 2 17:58:46 loaf mysqld: information that should help you find out what is causing the crash.
            {noformat}
            elenst Elena Stepanova made changes -
            Labels alter crash disk full innodb table alter crash disk full innodb table upstream
            elenst Elena Stepanova added a comment -

            I could reproduce it both on MariaDB and MySQL (I did it with OPTIMIZE, but I suppose it doesn't matter).

            Also, it seems to be a known problem for a while, see e.g. http://grokbase.com/t/mysql/mysql/136kx5qp1w/how-can-i-drop-a-table-that-is-named-logs-sql-ib203-and-appeared-after-mysql-crash

            I'm not sure if there is a relatively easy fix for this, assigning to Jan to decide.

            elenst Elena Stepanova added a comment - I could reproduce it both on MariaDB and MySQL (I did it with OPTIMIZE, but I suppose it doesn't matter). Also, it seems to be a known problem for a while, see e.g. http://grokbase.com/t/mysql/mysql/136kx5qp1w/how-can-i-drop-a-table-that-is-named-logs-sql-ib203-and-appeared-after-mysql-crash I'm not sure if there is a relatively easy fix for this, assigning to Jan to decide.
            elenst Elena Stepanova made changes -
            Assignee Elena Stepanova [ elenst ] Jan Lindström [ jplindst ]
            serg Sergei Golubchik made changes -
            Workflow defaullt [ 41702 ] MariaDB v2 [ 43469 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) added a comment -

            revno: 4270
            committer: Jan Lindström <jplindst@mariadb.org>
            branch nick: 10.0-innodb
            timestamp: Thu 2014-07-03 14:55:03 +0300
            message:
            MDEV-6288: Innodb causes server crash after disk full, then can't
            ALTER TABLE any more.

            jplindst Jan Lindström (Inactive) added a comment - revno: 4270 committer: Jan Lindström <jplindst@mariadb.org> branch nick: 10.0-innodb timestamp: Thu 2014-07-03 14:55:03 +0300 message: MDEV-6288 : Innodb causes server crash after disk full, then can't ALTER TABLE any more.
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s 10.0.13 [ 16300 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            jplindst Jan Lindström (Inactive) added a comment -

            revno: 4271
            committer: Jan Lindström <jplindst@mariadb.org>
            branch nick: 10.0-innodb
            timestamp: Fri 2014-07-04 06:31:48 +0300
            message:
            MDEV-6288: Innodb causes server crash after disk full,
            then can't ALTER TABLE any more.

            Fix for InnoDB storage engine.

            jplindst Jan Lindström (Inactive) added a comment - revno: 4271 committer: Jan Lindström <jplindst@mariadb.org> branch nick: 10.0-innodb timestamp: Fri 2014-07-04 06:31:48 +0300 message: MDEV-6288 : Innodb causes server crash after disk full, then can't ALTER TABLE any more. Fix for InnoDB storage engine.
            jplindst Jan Lindström (Inactive) added a comment -

            Additional crash cases found

            jplindst Jan Lindström (Inactive) added a comment - Additional crash cases found
            jplindst Jan Lindström (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Stalled [ 10000 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) added a comment -

            revno: 4601
            committer: Jan Lindström <jplindst@mariadb.org>
            branch nick: 10.0
            timestamp: Fri 2015-02-13 11:49:31 +0200
            message:
            MDEV-6288 :Innodb causes server crash after disk full, then can't ALTER TABLE any more

            Fix try to avoid unnecessary crashes when disk full situation is reached
            on alter table.

            jplindst Jan Lindström (Inactive) added a comment - revno: 4601 committer: Jan Lindström <jplindst@mariadb.org> branch nick: 10.0 timestamp: Fri 2015-02-13 11:49:31 +0200 message: MDEV-6288 :Innodb causes server crash after disk full, then can't ALTER TABLE any more Fix try to avoid unnecessary crashes when disk full situation is reached on alter table.
            jplindst Jan Lindström (Inactive) made changes -
            Component/s Storage Engine - InnoDB [ 10129 ]
            Component/s Storage Engine - XtraDB [ 10135 ]
            Fix Version/s 10.0.17 [ 18300 ]
            Fix Version/s 10.0.13 [ 16300 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Workflow MariaDB v2 [ 43469 ] MariaDB v3 [ 62586 ]
            nirbhay_c Nirbhay Choubey (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 62586 ] MariaDB v4 [ 147923 ]

            People

              jplindst Jan Lindström (Inactive)
              will.bryant Will Bryant
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.