Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27593

InnoDB handle AIO errors with message rather than assertion

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: In Progress (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.5.13, 10.8.1
    • Fix Version/s: 10.5
    • Labels:
      None
    • Environment:
      fc35 + uring

      Description

      In a case of filling up a disk with space an assertion was generated.

      Like MDEV-25760 fatal error with code would be better.

      2022-01-24  8:21:51 0 [Note] Plugin 'FEEDBACK' is disabled.
      2022-01-24  8:21:51 0 [Note] InnoDB: Loading buffer pool(s) from /mnt/dd/ib_buffer_pool
      2022-01-24  8:21:51 0 [Note] InnoDB: Buffer pool(s) load completed at 220124  8:21:51
      2022-01-24  8:21:51 0 [Note] sql/mysqld: ready for connections.
      Version: '10.8.0-MariaDB'  socket: '/tmp/mysql.sock'  port: 0  Source distribution
      2022-01-24 12:04:01 4 [Warning] Aborted connection 4 to db: 'test' user: 'dan' host: 'localhost' (Got an error writing communication packets)
      2022-01-24 12:09:33 0x7f7a8dffb640  InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc line 3596
      InnoDB: Failing assertion: cb->m_err == DB_SUCCESS
      InnoDB: We intentionally generate a memory trap.
      

      backtrace

      $ coredumpctl debug
                 PID: 16931 (mysqld)
                 UID: 1000 (dan)
                 GID: 1000 (dan)
              Signal: 6 (ABRT)
           Timestamp: Mon 2022-01-24 12:09:33 AEDT (4min 3s ago)
        Command Line: sql/mysqld --no-defaults --skip-networking --datadir=/mnt/dd --verbose
          Executable: /home/dan/repos/build-mariadb-server-10.8/sql/mariadbd
       
      Core was generated by `sql/mysqld --no-defaults --skip-networking --datadir=/mnt/dd --verbose'.
      Program terminated with signal SIGABRT, Aborted.
       
      gdb) bt full
      #0  0x00007f7ad93df84c in __pthread_kill_implementation () from /lib64/libc.so.6
      No symbol table info available.
      #1  0x00007f7ad93926a6 in raise () from /lib64/libc.so.6
      No symbol table info available.
      #2  0x00007f7ad937c865 in abort () from /lib64/libc.so.6
      No symbol table info available.
      #3  0x0000000000c9ecfb in ut_dbg_assertion_failed (expr=0x1152a4f "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3596) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
      No locals.
      #4  0x0000000000c037d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3596
              request = <optimized out>
      #5  0x0000000000d6588d in tpool::task_group::execute (this=0x2348930, t=0x23613d0) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
              lk = {_M_device = 0x2348960, _M_owns = false}
      #6  0x0000000000d63976 in tpool::thread_pool_generic::worker_main (this=0x22dd700, thread_var=0x22ed2a0) at /home/dan/repos/mariadb-server-10.8/tpool/tpool_generic.cc:549
              task = 0x4223
      #7  0x00007f7ad964e5c4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
      No symbol table info available.
      #8  0x00007f7ad93dda87 in start_thread () from /lib64/libc.so.6
      No symbol table info available.
      #9  0x00007f7ad9462640 in clone3 () from /lib64/libc.so.6
      No symbol table info available.
      (gdb) up
      #1  0x00007f7ad93926a6 in raise () from /lib64/libc.so.6
      (gdb) up
      #2  0x00007f7ad937c865 in abort () from /lib64/libc.so.6
      (gdb) up
      #3  0x0000000000c9ecfb in ut_dbg_assertion_failed (expr=0x1152a4f "cb->m_err == DB_SUCCESS", file=<optimized out>, line=<optimized out>, line@entry=3596) at /home/dan/repos/mariadb-server-10.8/storage/innobase/ut/ut0dbg.cc:60
      60		abort();
      (gdb) up
      #4  0x0000000000c037d0 in io_callback (cb=<optimized out>) at /home/dan/repos/mariadb-server-10.8/storage/innobase/os/os0file.cc:3596
      3596	  ut_a(cb->m_err == DB_SUCCESS);
      (gdb) p cb
      $1 = <optimized out>
      (gdb) p cb->m_err
      value has been optimized out
      (gdb) list
      3591	
      3592	extern void fil_aio_callback(const IORequest &request);
      3593	
      3594	static void io_callback(tpool::aiocb *cb)
      3595	{
      3596	  ut_a(cb->m_err == DB_SUCCESS);
      3597	  const IORequest &request= *static_cast<const IORequest*>
      3598	    (static_cast<const void*>(cb->m_userdata));
      3599	
      3600	  /* Return cb back to cache*/
      (gdb) up
      #5  0x0000000000d6588d in tpool::task_group::execute (this=0x2348930, t=0x23613d0) at /home/dan/repos/mariadb-server-10.8/tpool/task_group.cc:55
      55	        t->m_func(t->m_arg);
      (gdb) p *t
      $2 = (tpool::task) {_vptr$task = 0x1174848 <vtable for tpool::task+16>, m_func = 0xc03740 <io_callback(tpool::aiocb*)>, m_arg = 0x2361378, m_group = 0x2348930}
      (gdb) p (tpool::aiocb *) t->m_arg
      $3 = (tpool::aiocb *) 0x2361378
      (gdb) p *((tpool::aiocb *) t->m_arg)
      $4 = {<iovec> = {iov_base = 0x7f7ab47f8000, iov_len = 2097152}, m_fh = 12, m_opcode = tpool::aio_opcode::AIO_PWRITE, m_offset = 1048576, m_buffer = 0x7f7ab47f8000, m_len = 2097152, m_callback = 0xc03740 <io_callback(tpool::aiocb*)>, m_group = 0x2348930, m_ret_len = 0, m_err = 28, m_internal = 0x0, m_internal_task = {_vptr$task = 0x1174848 <vtable for tpool::task+16>, m_func = 0xc03740 <io_callback(tpool::aiocb*)>, m_arg = 0x2361378, m_group = 0x2348930}, m_userdata = '\000' <repeats 16 times>, "\310\065V\002\000\000\000\000\031\000\000\000\000\000\000"}
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              danblack Daniel Black
              Reporter:
              danblack Daniel Black
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.