Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16523

Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles

Details

    Description

      https://travis-ci.org/elenst/travis-tests/jobs/393093617

      10.2 b8514c94f61f37757a24df1c6b9a7dd530b3de12

      mysqld: /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143: void rocksdb::{anonymous}::LevelCompactionBuilder::SetupInitialFiles(): Assertion `!level_and_file.second->being_compacted' failed.
      180616 19:22:08 [ERROR] mysqld got signal 6 ;
       
      #7  0x00007f9379b21ca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
      #8  0x00007f93718f4214 in rocksdb::(anonymous namespace)::LevelCompactionBuilder::SetupInitialFiles (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143
      #9  0x00007f93718f46dd in rocksdb::(anonymous namespace)::LevelCompactionBuilder::PickCompaction (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1232
      #10 0x00007f93718f5497 in rocksdb::LevelCompactionPicker::PickCompaction (this=0x55ab0393e100, cf_name="default", mutable_cf_options=..., vstorage=0x7f9358034290, log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1431
      #11 0x00007f93718c534a in rocksdb::ColumnFamilyData::PickCompaction (this=0x55ab0393bb10, mutable_options=..., log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/column_family.cc:838
      #12 0x00007f9371605f96 in rocksdb::DBImpl::BackgroundCompaction (this=0x55ab0392e6f0, made_progress=0x7f936575706f, job_context=0x7f93657570c0, log_buffer=0x7f93657573a0, prepicked_compaction=0x0) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1541
      #13 0x00007f937160519f in rocksdb::DBImpl::BackgroundCallCompaction (this=0x55ab0392e6f0, prepicked_compaction=0x0, bg_thread_pri=rocksdb::Env::LOW) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1358
      #14 0x00007f93716042fd in rocksdb::DBImpl::BGWorkCompaction (arg=0x7f9358008b80) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1177
      #15 0x00007f937195f96c in rocksdb::ThreadPoolImpl::__lambda0::operator() (__closure=0x7f93580096b0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:441
      #16 0x00007f937195fdfa in std::_Function_handler<void(), rocksdb::ThreadPoolImpl::Schedule(void (*)(void*), void*, void*, void (*)(void*))::__lambda0>::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/4.8/functional:2071
      #17 0x00007f9371859b96 in std::function<void ()>::operator()() const (this=0x7f9365757db0) at /usr/include/c++/4.8/functional:2471
      #18 0x00007f937195ed54 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55ab036c4b90, thread_id=0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:240
      #19 0x00007f937195eeb7 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55ab0392db00) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:278
      #20 0x00007f9371967cde in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1732
      #21 0x00007f9371967be9 in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::operator()() (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1720
      #22 0x00007f9371967b82 in std::thread::_Impl<std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)> >::_M_run() (this=0x55ab0392db38) at /usr/include/c++/4.8/thread:115
      #23 0x00007f937a488a60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #24 0x00007f937a6e3184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #25 0x00007f9379befffd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      

      travis-workarounds c1710043b35269d4bae46f892bfccc02cd7cf2e2

      perl /home/travis/rqg/runall-new.pl --basedir=/home/travis/server --vardir=/home/travis/logs/vardir --engine=RocksDB --duration=350 --threads=6 --seed=1529176799 --reporters=Backtrace,ErrorLog,Deadlock --validators=TransformerNoComparator --views --redefine=conf/mariadb/alter_table.yy --redefine=conf/mariadb/bulk_insert.yy --mysqld=--log_output=FILE --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --mysqld=--default-storage-engine=RocksDB --mysqld=--plugin-load-add=ha_rocksdb --grammar=conf/runtime/performance_schema.yy --mysqld=--performance-schema --ps-protocol --transformers=ExecuteAsCTE,ExecuteAsDeleteReturning,ExecuteAsInsertSelect,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,ExecuteAsSPTwice
      

      Not reproducible so far.
      Stack traces are attached.
      Coredump, datadir etc. are available.

      Attachments

        1. threads
          74 kB
        2. threads_full
          408 kB

        Issue Links

          Activity

            elenst Elena Stepanova created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Attachment threads [ 45776 ]
            Attachment threads_full [ 45777 ]
            elenst Elena Stepanova added a comment - Still happens: https://travis-ci.org/elenst/travis-tests/jobs/409140012
            tomwang tom wang added a comment -

            I had this same problem, but I don't know why.

            tomwang tom wang added a comment - I had this same problem, but I don't know why.
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 87919 ] MariaDB v4 [ 140819 ]
            elenst Elena Stepanova made changes -
            allen.lee@mariadb.com Allen Lee (Inactive) made changes -
            Affects Version/s 10.4.13 [ 24223 ]
            elenst Elena Stepanova added a comment - - edited

            The test from the description fails fairly easily still, but it fails with MDEV-15164 instead of the failure reported here; MDEV-15164 is another assertion failure in compaction, which is apparently more probably to happen.

            If we want to get to analyzing the failure described here, MDEV-15164 needs to be fixed first. Either it will fix both at once, or at least will remove the obstacle on the way to reproduce and debug.

            MDEV-15164 needs a new test case though, something must have changed on the locking side since it was filed, so the test case from there does not work the same way as it did before.

            elenst Elena Stepanova added a comment - - edited The test from the description fails fairly easily still, but it fails with MDEV-15164 instead of the failure reported here; MDEV-15164 is another assertion failure in compaction, which is apparently more probably to happen. If we want to get to analyzing the failure described here, MDEV-15164 needs to be fixed first. Either it will fix both at once, or at least will remove the obstacle on the way to reproduce and debug. MDEV-15164 needs a new test case though, something must have changed on the locking side since it was filed, so the test case from there does not work the same way as it did before.
            elenst Elena Stepanova made changes -
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.4 [ 22408 ]
            elenst Elena Stepanova made changes -
            Fix Version/s 10.2 [ 14601 ]
            elenst Elena Stepanova made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.4 [ 22408 ]
            elenst Elena Stepanova made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s N/A [ 14700 ]
            Affects Version/s 10.4 [ 22408 ]
            Affects Version/s 10.2 [ 14601 ]
            Affects Version/s 10.4.13 [ 24223 ]
            Assignee Elena Stepanova [ elenst ]
            Summary [Draft] Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            Assignee Sergei Petrunia [ psergey ]
            Richard Richard Stracke added a comment - https://review.spdk.io/gerrit/plugins/gitiles/spdk/rocksdb/+/ab8f6c01a6c48fd7b8c752a3ef0ef8640065dd48
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            serg Sergei Golubchik made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 126144
            valerii Valerii Kravchuk made changes -
            Affects Version/s 10.11.9 [ 29834 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.11 [ 27614 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.4(EOL) [ 22408 ]
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            julien.fritsch Julien Fritsch made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]

            let's upgrade RocksDB to 6.29

            serg Sergei Golubchik added a comment - let's upgrade RocksDB to 6.29
            serg Sergei Golubchik made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            psergei Sergei Petrunia made changes -
            Status Confirmed [ 10101 ] Open [ 1 ]

            Right. Updating to RocksDB 6.29 isn't "just update the submodule" - I can already see APIs changed and files moved, but it is still easier than update to RocksDB 8.x.

            psergei Sergei Petrunia added a comment - Right. Updating to RocksDB 6.29 isn't "just update the submodule" - I can already see APIs changed and files moved, but it is still easier than update to RocksDB 8.x.

            https://github.com/MariaDB/server/tree/bb-10.11-mdev16253-update-rocksdb-6.29fb - this updates RocksDB to https://github.com/facebook/rocksdb/tree/6.29.fb .
            There are some test failures still (I also had some before the RocksDB update)

            psergei Sergei Petrunia added a comment - https://github.com/MariaDB/server/tree/bb-10.11-mdev16253-update-rocksdb-6.29fb - this updates RocksDB to https://github.com/facebook/rocksdb/tree/6.29.fb . There are some test failures still (I also had some before the RocksDB update)
            psergei Sergei Petrunia made changes -
            psergei Sergei Petrunia added a comment -

            Ok, the commit

             
            commit 281dc05bb764c737d35b51c9da25b7ddb636d51e (HEAD -> bb-10.11.11-mdev16253-rocksdb-6.29fb, origin/bb-10.11.11-mdev16253-rocksdb-6.29fb)
            Author: Sergei Petrunia <sergey@mariadb.com>
            Date:   Fri Mar 21 11:43:31 2025 +0200
             
                MDEV-16523, In MariaDB 10.11.11, update RocksDB to 6.29fb. (try3)
            

            seems to be good. Waiting for https://buildbot.mariadb.org/#/grid?branch=bb-10.11.11-mdev16253-rocksdb-6.29fb to finish for exotic platforms.

            psergei Sergei Petrunia added a comment - Ok, the commit   commit 281dc05bb764c737d35b51c9da25b7ddb636d51e (HEAD -> bb-10.11.11-mdev16253-rocksdb-6.29fb, origin/bb-10.11.11-mdev16253-rocksdb-6.29fb) Author: Sergei Petrunia <sergey@mariadb.com> Date: Fri Mar 21 11:43:31 2025 +0200   MDEV-16523, In MariaDB 10.11.11, update RocksDB to 6.29fb. (try3) seems to be good. Waiting for https://buildbot.mariadb.org/#/grid?branch=bb-10.11.11-mdev16253-rocksdb-6.29fb to finish for exotic platforms.
            psergei Sergei Petrunia added a comment -

            ... it has finished and it looks Ok. We're ready to create a custom build from this.

            psergei Sergei Petrunia added a comment - ... it has finished and it looks Ok. We're ready to create a custom build from this.
            psergei Sergei Petrunia made changes -

            People

              psergei Sergei Petrunia
              elenst Elena Stepanova
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.