Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7685

MariaDB - server crashes when inserting more rows than available space on disk

Details

    Attachments

      Issue Links

        Activity

          btr_page_split_and_insert() assumes that the operation of this
          function must always succeed, we cannot reverse it: therefore enough
          free disk space (2 pages) must be guaranteed to be available before
          this function is called.

          jplindst Jan Lindström (Inactive) added a comment - btr_page_split_and_insert() assumes that the operation of this function must always succeed, we cannot reverse it: therefore enough free disk space (2 pages) must be guaranteed to be available before this function is called.

          commit a7fd11b31d52b62ef7b61783bb83a5e62271307b
          Author: Jan Lindström <jan.lindstrom@mariadb.com>
          Date: Mon Mar 9 18:21:48 2015 +0200

          MDEV-7685: MariaDB - server crashes when inserting more rows than
          available space on disk

          Add error handling when disk full situation happens and
          intentionally bring server down with stacktrace because
          on all cases InnoDB can't continue anyway.

          jplindst Jan Lindström (Inactive) added a comment - commit a7fd11b31d52b62ef7b61783bb83a5e62271307b Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Mon Mar 9 18:21:48 2015 +0200 MDEV-7685 : MariaDB - server crashes when inserting more rows than available space on disk Add error handling when disk full situation happens and intentionally bring server down with stacktrace because on all cases InnoDB can't continue anyway.
          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Patch : https://github.com/MariaDB/server/commit/a7fd11b31d52b62ef7b61783bb83a5e62271307b

          10.0.20 still seem to be affected.

          See : https://bugzilla.suse.com/show_bug.cgi?id=919053

          MariaDB [test]> create table t1(a int,b int) engine=innodb data directory='/data1';
          Query OK, 0 rows affected (0.41 sec)
           
          MariaDB [test]> \! ls -lah /data1/test
          total 99K
          drwxrwx--- 2 mysql mysql 1.0K Jul  8 17:26 .
          drwxrwxrwx 4 root  root  1.0K Jul  8 17:26 ..
          -rw-rw---- 1 mysql mysql  96K Jul  8 17:26 t1.ibd
          MariaDB [test]> call innodb_insert_proc(14700);
          ERROR 2013 (HY000): Lost connection to MySQL server during query

          Coredump:

          [Thread debugging using libthread_db enabled]
          Using host libthread_db library "/lib64/libthread_db.so.1".
          Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib6'.
          Program terminated with signal SIGSEGV, Segmentation fault.
          #0  0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0
          Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-20.3.x86_64 libaio1-debuginfo-0.3.109-17.15.x86_64 libgcc_s1-debuginfo-4.8.3+r212056-11.2.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-25.1.x86_64 libstdc++6-debuginfo-4.8.3+r212056-11.2.x86_64 libz1-debuginfo-1.2.8-5.1.x86_64
          (gdb) bt
          #0  0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0
          #1  0x0000000000720184 in handle_fatal_signal (sig=11)
              at /usr/src/debug/mariadb-10.0.20/sql/signal_handler.cc:262
          #2  <signal handler called>
          #3  0x0000000000a80686 in btr_page_split_and_insert (flags=flags@entry=0, 
              cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, 
              heap=heap@entry=0x7fba9c39a930, tuple=tuple@entry=0x7fba7403c928, n_ext=<optimized out>, 
              mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0btr.cc:3101
          #4  0x0000000000a8da44 in btr_cur_pessimistic_insert (flags=flags@entry=0, 
              cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, 
              heap=heap@entry=0x7fba9c39a930, entry=entry@entry=0x7fba7403c928, rec=rec@entry=0x7fba9c39a9f0, 
              big_rec=big_rec@entry=0x7fba9c39a920, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8, 
              mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0cur.cc:1731
          #5  0x0000000000a0dfdf in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, 
              mode@entry=33, index=index@entry=0x7fba7403ef98, n_uniq=n_uniq@entry=0, 
              entry=entry@entry=0x7fba7403c928, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8)
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2512
          #6  0x0000000000a0ea39 in row_ins_clust_index_entry (index=index@entry=0x7fba7403ef98, 
              entry=entry@entry=0x7fba7403c928, thr=thr@entry=0x7fba74043ec8, n_ext=n_ext@entry=0)
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2920
          #7  0x0000000000a0efe2 in row_ins_index_entry (thr=0x7fba74043ec8, entry=0x7fba7403c928, 
              index=0x7fba7403ef98) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2997
          #8  row_ins_index_entry_step (thr=0x7fba74043ec8, node=<optimized out>)
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3074
          #9  row_ins (thr=<optimized out>, node=<optimized out>)
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3214
          #10 row_ins_step (thr=thr@entry=0x7fba74043ec8)
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3339
          #11 0x0000000000a1a608 in row_insert_for_mysql (
              mysql_rec=mysql_rec@entry=0x7fba74031bf0 "\371\335\067", prebuilt=0x7fba740437b8)
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0mysql.cc:1369
          #12 0x0000000000983a41 in ha_innobase::write_row (this=0x7fba74031480, 
              record=0x7fba74031bf0 "\371\335\067")
              at /usr/src/debug/mariadb-10.0.20/storage/xtradb/handler/ha_innodb.cc:7593
          ---Type <return> to continue, or q <return> to quit---
          #13 0x0000000000729459 in handler::ha_write_row (this=0x7fba74031480, 
              buf=0x7fba74031bf0 "\371\335\067") at /usr/src/debug/mariadb-10.0.20/sql/handler.cc:5984
          #14 0x00000000005c0c5f in write_record (thd=thd@entry=0x4519c08, table=table@entry=0x7fba74030ac8, 
              info=info@entry=0x7fba9c39b640) at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:1848
          #15 0x00000000005c429a in mysql_insert (thd=thd@entry=0x4519c08, table_list=0x7fba7402a270, 
              fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false)
              at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:965
          #16 0x00000000005dc2d9 in mysql_execute_command (thd=0x4519c08)
              at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:3449
          #17 0x0000000000813d54 in sp_instr_stmt::exec_core (this=0x7fba7402c0a0, thd=<optimized out>, 
              nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3204
          #18 0x0000000000819add in sp_lex_keeper::reset_lex_and_exec_core (this=this@entry=0x7fba7402c0e0, 
              thd=thd@entry=0x4519c08, nextp=nextp@entry=0x7fba9c39ce40, open_tables=open_tables@entry=false, 
              instr=instr@entry=0x7fba7402c0a0) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2971
          #19 0x0000000000819fb4 in sp_instr_stmt::execute (this=0x7fba7402c0a0, thd=0x4519c08, 
              nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3120
          #20 0x00000000008169af in sp_head::execute (this=this@entry=0x7fba74028ae0, thd=thd@entry=0x4519c08, 
              merge_da_on_success=merge_da_on_success@entry=true)
              at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:1371
          #21 0x0000000000817e17 in sp_head::execute_procedure (this=this@entry=0x7fba74028ae0, 
              thd=thd@entry=0x4519c08, args=args@entry=0x451e0d0)
              at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2159
          #22 0x00000000005d91e1 in mysql_execute_command (thd=thd@entry=0x4519c08)
              at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:4701
          #23 0x00000000005df668 in mysql_parse (thd=0x4519c08, rawbuf=<optimized out>, length=30, 
              parser_state=0x7fba9c39e730) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:6529
          #24 0x00000000005e16c7 in dispatch_command (command=COM_QUERY, thd=0x4519c08, packet=<optimized out>, 
              packet_length=<optimized out>) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:1308
          #25 0x00000000005e21c0 in do_command (thd=<optimized out>)
              at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:999
          #26 0x0000000000697742 in do_handle_one_connection (thd_arg=thd_arg@entry=0x4519c08)
              at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1378
          #27 0x00000000006977c7 in handle_one_connection (arg=0x4519c08)
              at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1293
          ---Type <return> to continue, or q <return> to quit---
          #28 0x00007fbaafffd0a4 in start_thread () from /lib64/libpthread.so.0
          #29 0x00007fbaaeeca08d in clone () from /lib64/libc.so.6
          (gdb) quit

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - 10.0.20 still seem to be affected. See : https://bugzilla.suse.com/show_bug.cgi?id=919053 MariaDB [test]> create table t1(a int,b int) engine=innodb data directory='/data1'; Query OK, 0 rows affected (0.41 sec)   MariaDB [test]> \! ls -lah /data1/test total 99K drwxrwx--- 2 mysql mysql 1.0K Jul 8 17:26 . drwxrwxrwx 4 root root 1.0K Jul 8 17:26 .. -rw-rw---- 1 mysql mysql 96K Jul 8 17:26 t1.ibd MariaDB [test]> call innodb_insert_proc(14700); ERROR 2013 (HY000): Lost connection to MySQL server during query Coredump: [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib6'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0 Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-20.3.x86_64 libaio1-debuginfo-0.3.109-17.15.x86_64 libgcc_s1-debuginfo-4.8.3+r212056-11.2.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-25.1.x86_64 libstdc++6-debuginfo-4.8.3+r212056-11.2.x86_64 libz1-debuginfo-1.2.8-5.1.x86_64 (gdb) bt #0 0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000000000720184 in handle_fatal_signal (sig=11) at /usr/src/debug/mariadb-10.0.20/sql/signal_handler.cc:262 #2 <signal handler called> #3 0x0000000000a80686 in btr_page_split_and_insert (flags=flags@entry=0, cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, heap=heap@entry=0x7fba9c39a930, tuple=tuple@entry=0x7fba7403c928, n_ext=<optimized out>, mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0btr.cc:3101 #4 0x0000000000a8da44 in btr_cur_pessimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, heap=heap@entry=0x7fba9c39a930, entry=entry@entry=0x7fba7403c928, rec=rec@entry=0x7fba9c39a9f0, big_rec=big_rec@entry=0x7fba9c39a920, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8, mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0cur.cc:1731 #5 0x0000000000a0dfdf in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, mode@entry=33, index=index@entry=0x7fba7403ef98, n_uniq=n_uniq@entry=0, entry=entry@entry=0x7fba7403c928, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2512 #6 0x0000000000a0ea39 in row_ins_clust_index_entry (index=index@entry=0x7fba7403ef98, entry=entry@entry=0x7fba7403c928, thr=thr@entry=0x7fba74043ec8, n_ext=n_ext@entry=0) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2920 #7 0x0000000000a0efe2 in row_ins_index_entry (thr=0x7fba74043ec8, entry=0x7fba7403c928, index=0x7fba7403ef98) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2997 #8 row_ins_index_entry_step (thr=0x7fba74043ec8, node=<optimized out>) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3074 #9 row_ins (thr=<optimized out>, node=<optimized out>) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3214 #10 row_ins_step (thr=thr@entry=0x7fba74043ec8) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3339 #11 0x0000000000a1a608 in row_insert_for_mysql ( mysql_rec=mysql_rec@entry=0x7fba74031bf0 "\371\335\067", prebuilt=0x7fba740437b8) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0mysql.cc:1369 #12 0x0000000000983a41 in ha_innobase::write_row (this=0x7fba74031480, record=0x7fba74031bf0 "\371\335\067") at /usr/src/debug/mariadb-10.0.20/storage/xtradb/handler/ha_innodb.cc:7593 ---Type <return> to continue, or q <return> to quit--- #13 0x0000000000729459 in handler::ha_write_row (this=0x7fba74031480, buf=0x7fba74031bf0 "\371\335\067") at /usr/src/debug/mariadb-10.0.20/sql/handler.cc:5984 #14 0x00000000005c0c5f in write_record (thd=thd@entry=0x4519c08, table=table@entry=0x7fba74030ac8, info=info@entry=0x7fba9c39b640) at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:1848 #15 0x00000000005c429a in mysql_insert (thd=thd@entry=0x4519c08, table_list=0x7fba7402a270, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:965 #16 0x00000000005dc2d9 in mysql_execute_command (thd=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:3449 #17 0x0000000000813d54 in sp_instr_stmt::exec_core (this=0x7fba7402c0a0, thd=<optimized out>, nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3204 #18 0x0000000000819add in sp_lex_keeper::reset_lex_and_exec_core (this=this@entry=0x7fba7402c0e0, thd=thd@entry=0x4519c08, nextp=nextp@entry=0x7fba9c39ce40, open_tables=open_tables@entry=false, instr=instr@entry=0x7fba7402c0a0) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2971 #19 0x0000000000819fb4 in sp_instr_stmt::execute (this=0x7fba7402c0a0, thd=0x4519c08, nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3120 #20 0x00000000008169af in sp_head::execute (this=this@entry=0x7fba74028ae0, thd=thd@entry=0x4519c08, merge_da_on_success=merge_da_on_success@entry=true) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:1371 #21 0x0000000000817e17 in sp_head::execute_procedure (this=this@entry=0x7fba74028ae0, thd=thd@entry=0x4519c08, args=args@entry=0x451e0d0) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2159 #22 0x00000000005d91e1 in mysql_execute_command (thd=thd@entry=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:4701 #23 0x00000000005df668 in mysql_parse (thd=0x4519c08, rawbuf=<optimized out>, length=30, parser_state=0x7fba9c39e730) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:6529 #24 0x00000000005e16c7 in dispatch_command (command=COM_QUERY, thd=0x4519c08, packet=<optimized out>, packet_length=<optimized out>) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:1308 #25 0x00000000005e21c0 in do_command (thd=<optimized out>) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:999 #26 0x0000000000697742 in do_handle_one_connection (thd_arg=thd_arg@entry=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1378 #27 0x00000000006977c7 in handle_one_connection (arg=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1293 ---Type <return> to continue, or q <return> to quit--- #28 0x00007fbaafffd0a4 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fbaaeeca08d in clone () from /lib64/libc.so.6 (gdb) quit

          Reopening for further investigation.

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Reopening for further investigation.

          Tested with 10.0.20 and I got (/mnt/jan is about 7M):

          MariaDB Version 10.0.20-MariaDB-debug
           - SSL connections supported
           - binaries are debug compiled
          Collecting tests...
          Installing system database...
           
          ==============================================================================
           
          TEST                                      RESULT   TIME (ms) or COMMENT
          --------------------------------------------------------------------------
           
          worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
          create table t1(a int,b int) engine=innodb data directory='/mnt/jan';
          create procedure innodb_insert_proc (repeat_count int)
          begin
          declare current_num int;
          set current_num = 0;
          while current_num < repeat_count do
          insert into t1 values(current_num, current_num);
          set current_num = current_num + 1;
          end while;
          end//
          commit;
          set autocommit=0;
          call innodb_insert_proc(14700);
          call innodb_insert_proc(14700);
          innodb.innodb-out-of-disk 'innodb_plugin' [ fail ]
                  Test ended at 2015-07-21 13:45:40
           
          CURRENT_TEST: innodb.innodb-out-of-disk
          mysqltest: At line 18: query 'call innodb_insert_proc(14700)' failed: 1114: The table 't1' is full

          jplindst Jan Lindström (Inactive) added a comment - Tested with 10.0.20 and I got (/mnt/jan is about 7M): MariaDB Version 10.0.20-MariaDB-debug - SSL connections supported - binaries are debug compiled Collecting tests... Installing system database...   ==============================================================================   TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 create table t1(a int,b int) engine=innodb data directory='/mnt/jan'; create procedure innodb_insert_proc (repeat_count int) begin declare current_num int; set current_num = 0; while current_num < repeat_count do insert into t1 values(current_num, current_num); set current_num = current_num + 1; end while; end// commit; set autocommit=0; call innodb_insert_proc(14700); call innodb_insert_proc(14700); innodb.innodb-out-of-disk 'innodb_plugin' [ fail ] Test ended at 2015-07-21 13:45:40   CURRENT_TEST: innodb.innodb-out-of-disk mysqltest: At line 18: query 'call innodb_insert_proc(14700)' failed: 1114: The table 't1' is full

          Note there are some historical cases where InnoDB assumes some operations to succeed. In these cases there is no rollback, thus it will crash.

          jplindst Jan Lindström (Inactive) added a comment - Note there are some historical cases where InnoDB assumes some operations to succeed. In these cases there is no rollback, thus it will crash.

          People

            jplindst Jan Lindström (Inactive)
            jplindst Jan Lindström (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.