Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13975

[out of disk space assert] I was uploading 100 million records to a rocsdb table and Mariadb crashed

    XMLWordPrintable

Details

    Description

      The drive was out of space, but it should not crash/

      2017-10-01 13:25:08 139821712402176 [ERROR] LibRocksDB:[/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.9/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1257] Waiting after background flush error: IO error: Error allocating space to file : ./.rocksdb/004708.sstError : No space left on deviceAccumulated background error counts: 1
      2017-10-01 13:25:08 139821934552832 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: Error allocating space to file : ./.rocksdb/004708.sstError : No space left on device
      2017-10-01 13:25:08 139821934552832 [ERROR] MyRocks: aborting on WAL write error.
      2017-10-01 13:25:08 139821934053120 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: Error allocating space to file : ./.rocksdb/004708.sstError : No space left on device
      2017-10-01 13:25:08 139821934053120 [ERROR] MyRocks: aborting on WAL write error.
      171001 13:25:08 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      2017-10-01 13:25:17 140164155345024 [Note] RocksDB: 2 column families found
      2017-10-01 13:25:17 140164155345024 [Note] RocksDB: Column Families at start:
      2017-10-01 13:25:17 140164155345024 [Note]   cf=default
      2017-10-01 13:25:17 140164155345024 [Note]     write_buffer_size=134217728
      2017-10-01 13:25:17 140164155345024 [Note]     target_file_size_base=67108864
      2017-10-01 13:25:17 140164155345024 [Note]   cf=__system__
      2017-10-01 13:25:17 140164155345024 [Note]     write_buffer_size=134217728
      2017-10-01 13:25:17 140164155345024 [Note]     target_file_size_base=67108864
      mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.9/storage/rocksdb/rocksdb/env/posix_logger.h:153: virtual void rocksdb::PosixLogger::Logv(const char*, __va_list_tag*): Assertion `sz == write_size' failed.
      171001 13:25:17 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.9-MariaDB
      key_buffer_size=16384
      read_buffer_size=262144
      max_used_connections=0
      max_threads=65537
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 553658729 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x3c000
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5560a1f043ee]
      /usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x5560a19816fd]
      /lib64/libpthread.so.0(+0xf5e0)[0x7f7a8271d5e0]
      /lib64/libc.so.6(gsignal+0x37)[0x7f7a80c2a1f7]
      /lib64/libc.so.6(abort+0x148)[0x7f7a80c2b8e8]
      /lib64/libc.so.6(+0x2e266)[0x7f7a80c23266]
      /lib64/libc.so.6(+0x2e312)[0x7f7a80c23312]
      /usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb11PosixLogger4LogvEPKcP13__va_list_tag+0x3e4)[0x7f7a73ca86f4]
      /usr/lib64/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6Logger4LogvENS_12In[root@demo mysql]# 
      

      Attachments

        Issue Links

          Activity

            People

              psergei Sergei Petrunia
              philip_38 Philip orleans
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.