Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-36368

RocksDB + Swapoff -a => compilation of crash

Details

    • Bug
    • Status: Open (View Workflow)
    • Blocker
    • Resolution: Unresolved
    • 10.11.11
    • None
    • Server
    • None

    Description

      I can reproduce really easely now :

      terminate called after throwing an instance of 'std::bad_alloc'
        what():  std::bad_alloc
      250324 17:41:46 [ERROR] /usr/sbin/mariadbd got signal 6 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
      a bug on https://jira.mariadb.org/.
       
      Please include the information from the server start above, to the end of the
      information below.
       
      Server version: 10.11.11-MariaDB-deb12-log source revision: e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
       
      The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
      contains instructions to obtain a better version of the backtrace below.
      Following these instructions will help MariaDB developers provide a fix quicker.
       
      Attempting backtrace. Include this in the bug report.
      (note: Retrieving this information may fail)
       
      Thread pointer: 0x0
      stack_bottom = 0x0 thread_stack 0x49000
      Can't start addr2line
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55e77f50200e]
      /usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x55e77efe0e89]
      /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7fd928e5b050]
      /lib/x86_64-linux-gnu/libc.so.6(+0x8aebc)[0x7fd928ea9ebc]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7fd928e5afb2]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fd928e45472]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9d919)[0x7fd92909d919]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa8e1a)[0x7fd9290a8e1a]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa8e85)[0x7fd9290a8e85]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa90d8)[0x7fd9290a90d8]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9d524)[0x7fd92909d524]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb18FilePrefetchBuffer8PrefetchEPNS_22RandomAccessFileReaderEmmb+0x1e7)[0x7fd8bcef7db7]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb18FilePrefetchBuffer16TryReadFromCacheEmmPNS_5SliceEb+0x94)[0x7fd8bcef7f34]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb12BlockFetcher17ReadBlockContentsEv+0x8a)[0x7fd8bce090aa]
      /usr/lib/mysql/plugin/ha_rocksdb.so(+0x3d86e6)[0x7fd8bcdd86e6]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZNK7rocksdb15BlockBasedTable13RetrieveBlockINS_5BlockEEENS_6StatusEPNS_18FilePrefetchBufferERKNS_11ReadOptionsERKNS_11BlockHandleERKNS_17UncompressionDictEPNS_13CachableEntryIT_EENS_9BlockTypeEPNS_10GetContextEPNS_23BlockCacheLookupContextEbb+0x1bc)[0x7fd8bcdf13ac]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZNK7rocksdb15BlockBasedTable20NewDataBlockIteratorINS_13DataBlockIterEEEPT_RKNS_11ReadOptionsERKNS_11BlockHandleES4_NS_9BlockTypeEPNS_10GetContextEPNS_23BlockCacheLookupContextENS_6StatusEPNS_18FilePrefetchBufferEb+0x490)[0x7fd8bcdf3970]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb23BlockBasedTableIteratorINS_13DataBlockIterENS_5SliceEE13InitDataBlockEv+0xc9)[0x7fd8bcdfbf59]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb23BlockBasedTableIteratorINS_13DataBlockIterENS_5SliceEE11SeekToFirstEv+0x1c8)[0x7fd8bcdff348]
      /usr/lib/mysql/plugin/ha_rocksdb.so(+0x33c5e7)[0x7fd8bcd3c5e7]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13CompactionJob25ProcessKeyValueCompactionEPNS0_18SubcompactionStateE+0x396)[0x7fd8bcedcc06]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb13CompactionJob3RunEv+0xaa5)[0x7fd8bcedeb85]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl20BackgroundCompactionEPbPNS_10JobContextEPNS_9LogBufferEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0xeb9)[0x7fd8bcc8e2c9]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl24BackgroundCallCompactionEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0xf3)[0x7fd8bcc92f23]
      2025-03-24 17:41:46 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl16BGWorkCompactionEPv+0x3a)[0x7fd8bcc9398a]
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm+0x260)[0x7fd8bce4e5f0]
      2025-03-24 17:41:46 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 2
      2025-03-24 17:41:46 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply
      /usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv+0x5f)[0x7fd8bce4e7af]
      2025-03-24 17:41:46 0 [Note] Event Scheduler: Stopped
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7fd9290d44a3]
      /lib/x86_64-linux-gnu/libc.so.6(+0x891c4)[0x7fd928ea81c4]
      /lib/x86_64-linux-gnu/libc.so.6(+0x10985c)[0x7fd928f2885c]
      Writing a core file...
      Working directory at /srv/mysql/data
      Resource Limits (excludes unlimited resources):
      Limit                     Soft Limit           Hard Limit           Units     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max processes             63785                63785                processes 
      Max open files            32768                32768                files     
      Max locked memory         524288               524288               bytes     
      Max pending signals       63785                63785                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Core pattern: core
       
      Kernel version: Linux version 6.1.0-31-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07)
      

      my.cnf for RocksDB options

      rocksdb_flush_log_at_trx-commit=2
      rocksdb_allow_concurrent_memtable_write=1
      rocksdb_allow_to_start_after_corruption=1
      rocksdb_db_write_buffer_size=256M
      rocksdb_block_cache_size=1G
      rocksdb_deadlock_detect=1
      rocksdb_delayed_write_rate=167772160
      rocksdb_table_cache_numshardbits=6
      rocksdb_print_snapshot_conflict_queries=ON
      rocksdb_two_write_queues=ON
      rocksdb_use_direct_io_for_flush_and_compaction=ON
      rocksdb_wal_recovery_mode=2
      rocksdb_max_total_wal_size = 1G
      rocksdb_compaction_readahead_size=1G
      rocksdb_max_background_jobs=4
      rocksdb_compaction_readahead_size=16M  # Optimisation pour éviter des lectures trop petites
      rocksdb_use_direct_reads=ON
      rocksdb_use_direct_io_for_flush_and_compaction=ON
      rocksdb_skip_fill_cache=ON
      

      it's complain about lack of memory : (the server got RAM 10Go free)

      [ERROR] mariadbd: Out of memory (Needed 33554400 bytes)
      ...
      

      But more than 10Go available on server, when swap is activated RocksDB can took memory as defined if not complain about lack of memory.

      read : https://jira.mariadb.org/browse/MDEV-36252?filter=-2 for more informations

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Aurelien_LEQUOY Aurélien LEQUOY
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.