Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16523

Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles

Details

    Description

      https://travis-ci.org/elenst/travis-tests/jobs/393093617

      10.2 b8514c94f61f37757a24df1c6b9a7dd530b3de12

      mysqld: /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143: void rocksdb::{anonymous}::LevelCompactionBuilder::SetupInitialFiles(): Assertion `!level_and_file.second->being_compacted' failed.
      180616 19:22:08 [ERROR] mysqld got signal 6 ;
       
      #7  0x00007f9379b21ca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
      #8  0x00007f93718f4214 in rocksdb::(anonymous namespace)::LevelCompactionBuilder::SetupInitialFiles (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143
      #9  0x00007f93718f46dd in rocksdb::(anonymous namespace)::LevelCompactionBuilder::PickCompaction (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1232
      #10 0x00007f93718f5497 in rocksdb::LevelCompactionPicker::PickCompaction (this=0x55ab0393e100, cf_name="default", mutable_cf_options=..., vstorage=0x7f9358034290, log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1431
      #11 0x00007f93718c534a in rocksdb::ColumnFamilyData::PickCompaction (this=0x55ab0393bb10, mutable_options=..., log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/column_family.cc:838
      #12 0x00007f9371605f96 in rocksdb::DBImpl::BackgroundCompaction (this=0x55ab0392e6f0, made_progress=0x7f936575706f, job_context=0x7f93657570c0, log_buffer=0x7f93657573a0, prepicked_compaction=0x0) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1541
      #13 0x00007f937160519f in rocksdb::DBImpl::BackgroundCallCompaction (this=0x55ab0392e6f0, prepicked_compaction=0x0, bg_thread_pri=rocksdb::Env::LOW) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1358
      #14 0x00007f93716042fd in rocksdb::DBImpl::BGWorkCompaction (arg=0x7f9358008b80) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1177
      #15 0x00007f937195f96c in rocksdb::ThreadPoolImpl::__lambda0::operator() (__closure=0x7f93580096b0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:441
      #16 0x00007f937195fdfa in std::_Function_handler<void(), rocksdb::ThreadPoolImpl::Schedule(void (*)(void*), void*, void*, void (*)(void*))::__lambda0>::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/4.8/functional:2071
      #17 0x00007f9371859b96 in std::function<void ()>::operator()() const (this=0x7f9365757db0) at /usr/include/c++/4.8/functional:2471
      #18 0x00007f937195ed54 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55ab036c4b90, thread_id=0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:240
      #19 0x00007f937195eeb7 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55ab0392db00) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:278
      #20 0x00007f9371967cde in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1732
      #21 0x00007f9371967be9 in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::operator()() (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1720
      #22 0x00007f9371967b82 in std::thread::_Impl<std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)> >::_M_run() (this=0x55ab0392db38) at /usr/include/c++/4.8/thread:115
      #23 0x00007f937a488a60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #24 0x00007f937a6e3184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #25 0x00007f9379befffd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      

      travis-workarounds c1710043b35269d4bae46f892bfccc02cd7cf2e2

      perl /home/travis/rqg/runall-new.pl --basedir=/home/travis/server --vardir=/home/travis/logs/vardir --engine=RocksDB --duration=350 --threads=6 --seed=1529176799 --reporters=Backtrace,ErrorLog,Deadlock --validators=TransformerNoComparator --views --redefine=conf/mariadb/alter_table.yy --redefine=conf/mariadb/bulk_insert.yy --mysqld=--log_output=FILE --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --mysqld=--default-storage-engine=RocksDB --mysqld=--plugin-load-add=ha_rocksdb --grammar=conf/runtime/performance_schema.yy --mysqld=--performance-schema --ps-protocol --transformers=ExecuteAsCTE,ExecuteAsDeleteReturning,ExecuteAsInsertSelect,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,ExecuteAsSPTwice
      

      Not reproducible so far.
      Stack traces are attached.
      Coredump, datadir etc. are available.

      Attachments

        1. threads
          74 kB
          Elena Stepanova
        2. threads_full
          408 kB
          Elena Stepanova

        Issue Links

          Activity

            elenst Elena Stepanova created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Attachment threads [ 45776 ]
            Attachment threads_full [ 45777 ]
            elenst Elena Stepanova added a comment - Still happens: https://travis-ci.org/elenst/travis-tests/jobs/409140012
            tomwang tom wang added a comment -

            I had this same problem, but I don't know why.

            tomwang tom wang added a comment - I had this same problem, but I don't know why.
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 87919 ] MariaDB v4 [ 140819 ]
            elenst Elena Stepanova made changes -
            allen.lee@mariadb.com Allen Lee (Inactive) made changes -
            Affects Version/s 10.4.13 [ 24223 ]
            elenst Elena Stepanova added a comment - - edited

            The test from the description fails fairly easily still, but it fails with MDEV-15164 instead of the failure reported here; MDEV-15164 is another assertion failure in compaction, which is apparently more probably to happen.

            If we want to get to analyzing the failure described here, MDEV-15164 needs to be fixed first. Either it will fix both at once, or at least will remove the obstacle on the way to reproduce and debug.

            MDEV-15164 needs a new test case though, something must have changed on the locking side since it was filed, so the test case from there does not work the same way as it did before.

            elenst Elena Stepanova added a comment - - edited The test from the description fails fairly easily still, but it fails with MDEV-15164 instead of the failure reported here; MDEV-15164 is another assertion failure in compaction, which is apparently more probably to happen. If we want to get to analyzing the failure described here, MDEV-15164 needs to be fixed first. Either it will fix both at once, or at least will remove the obstacle on the way to reproduce and debug. MDEV-15164 needs a new test case though, something must have changed on the locking side since it was filed, so the test case from there does not work the same way as it did before.
            elenst Elena Stepanova made changes -
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.4 [ 22408 ]
            elenst Elena Stepanova made changes -
            Fix Version/s 10.2 [ 14601 ]
            elenst Elena Stepanova made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.4 [ 22408 ]
            elenst Elena Stepanova made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s N/A [ 14700 ]
            Affects Version/s 10.4 [ 22408 ]
            Affects Version/s 10.2 [ 14601 ]
            Affects Version/s 10.4.13 [ 24223 ]
            Assignee Elena Stepanova [ elenst ]
            Summary [Draft] Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            Assignee Sergei Petrunia [ psergey ]
            Richard Richard Stracke added a comment - https://review.spdk.io/gerrit/plugins/gitiles/spdk/rocksdb/+/ab8f6c01a6c48fd7b8c752a3ef0ef8640065dd48
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            serg Sergei Golubchik made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 126144
            valerii Valerii Kravchuk made changes -
            Affects Version/s 10.11.9 [ 29834 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.11 [ 27614 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.4(EOL) [ 22408 ]
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            julien.fritsch Julien Fritsch made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]

            let's upgrade RocksDB to 6.29

            serg Sergei Golubchik added a comment - let's upgrade RocksDB to 6.29
            serg Sergei Golubchik made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            psergei Sergei Petrunia made changes -
            Status Confirmed [ 10101 ] Open [ 1 ]

            Right. Updating to RocksDB 6.29 isn't "just update the submodule" - I can already see APIs changed and files moved, but it is still easier than update to RocksDB 8.x.

            psergei Sergei Petrunia added a comment - Right. Updating to RocksDB 6.29 isn't "just update the submodule" - I can already see APIs changed and files moved, but it is still easier than update to RocksDB 8.x.

            https://github.com/MariaDB/server/tree/bb-10.11-mdev16253-update-rocksdb-6.29fb - this updates RocksDB to https://github.com/facebook/rocksdb/tree/6.29.fb .
            There are some test failures still (I also had some before the RocksDB update)

            psergei Sergei Petrunia added a comment - https://github.com/MariaDB/server/tree/bb-10.11-mdev16253-update-rocksdb-6.29fb - this updates RocksDB to https://github.com/facebook/rocksdb/tree/6.29.fb . There are some test failures still (I also had some before the RocksDB update)
            psergei Sergei Petrunia made changes -

            Ok, the commit

             
            commit 281dc05bb764c737d35b51c9da25b7ddb636d51e (HEAD -> bb-10.11.11-mdev16253-rocksdb-6.29fb, origin/bb-10.11.11-mdev16253-rocksdb-6.29fb)
            Author: Sergei Petrunia <sergey@mariadb.com>
            Date:   Fri Mar 21 11:43:31 2025 +0200
             
                MDEV-16523, In MariaDB 10.11.11, update RocksDB to 6.29fb. (try3)
            

            seems to be good. Waiting for https://buildbot.mariadb.org/#/grid?branch=bb-10.11.11-mdev16253-rocksdb-6.29fb to finish for exotic platforms.

            psergei Sergei Petrunia added a comment - Ok, the commit   commit 281dc05bb764c737d35b51c9da25b7ddb636d51e (HEAD -> bb-10.11.11-mdev16253-rocksdb-6.29fb, origin/bb-10.11.11-mdev16253-rocksdb-6.29fb) Author: Sergei Petrunia <sergey@mariadb.com> Date: Fri Mar 21 11:43:31 2025 +0200   MDEV-16523, In MariaDB 10.11.11, update RocksDB to 6.29fb. (try3) seems to be good. Waiting for https://buildbot.mariadb.org/#/grid?branch=bb-10.11.11-mdev16253-rocksdb-6.29fb to finish for exotic platforms.

            ... it has finished and it looks Ok. We're ready to create a custom build from this.

            psergei Sergei Petrunia added a comment - ... it has finished and it looks Ok. We're ready to create a custom build from this.
            psergei Sergei Petrunia made changes -
            julien.fritsch Julien Fritsch made changes -
            Sprint Server 12.1 dev sprint [ 793 ]
            serg Sergei Golubchik made changes -
            Sprint Server 12.1 dev sprint [ 793 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Petrunia [ psergey ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Status In Progress [ 3 ] In Testing [ 10301 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Roel Van de Paar [ roel ]

            Running a generic/broad RocksDB test on bb-10.11.11-mdev16253-rocksdb-6.29fb as discussed.

            Roel Roel Van de Paar added a comment - Running a generic/broad RocksDB test on bb-10.11.11-mdev16253-rocksdb-6.29fb as discussed.
            serg Sergei Golubchik made changes -
            Priority Blocker [ 1 ] Critical [ 2 ]
            Roel Roel Van de Paar added a comment - - edited

            Non-actionable, FYI
            Created MDEV-36611 (SIGSEGV & UBASAN null-pointer-use in myrocks::Rdb_transaction_impl::get_write_batch): also present in current versions
            Created MDEV-36624 (UBSAN: invalid-shift-exponent: shift exponent 32 is too large in myrocks::ha_rocksdb::info): also present in current versions
            Created MDEV-36632 (Assertion `check_field_pointers(table)' failed and SIGSEGV): also present in current versions
            Created MDEV-36640 (RocksDB Corruption/errors/assert/SIGSEGV upon CHECK TABLE): also present in current versions
            Created MENT-2277 (ES 10.6 Optimized/Release has significant differences when it comes to RocksDB stability): no connection with this 6.29fb branch
            Various other pre-existing RocksDB bugs also seen in bb-10.11.11-mdev16253-rocksdb-6.29fb

            Actionable
            None

            Roel Roel Van de Paar added a comment - - edited Non-actionable, FYI Created MDEV-36611 (SIGSEGV & UBASAN null-pointer-use in myrocks::Rdb_transaction_impl::get_write_batch): also present in current versions Created MDEV-36624 (UBSAN: invalid-shift-exponent: shift exponent 32 is too large in myrocks::ha_rocksdb::info): also present in current versions Created MDEV-36632 (Assertion `check_field_pointers(table)' failed and SIGSEGV): also present in current versions Created MDEV-36640 (RocksDB Corruption/errors/assert/SIGSEGV upon CHECK TABLE): also present in current versions Created MENT-2277 (ES 10.6 Optimized/Release has significant differences when it comes to RocksDB stability): no connection with this 6.29fb branch Various other pre-existing RocksDB bugs also seen in bb-10.11.11-mdev16253-rocksdb-6.29fb Actionable None
            Roel Roel Van de Paar added a comment - - edited

            I am still working through the test results, and there are a few RocksDB issues remaining, however they are either low frequency, look benign or look related to other [now] known issues. Otherwise the summary is as per the last comment; all new bugs are also present in current versions.

            I also tested MDEV-15164, and it is considered fixed in bb-10.11.11-mdev16253-rocksdb-6.29fb. As for this bug (MDEV-16523), no occurences of being_compacted were seen.

            If the push of bb-10.11.11-mdev16253-rocksdb-6.29fb is urgent, I believe we can proceed now.

            In any case, I will roundup the runs.

            Update: runs complete. All other remaining RocksDB issues were present in base versions as well. Fine to push, which already happened.

            Roel Roel Van de Paar added a comment - - edited I am still working through the test results , and there are a few RocksDB issues remaining, however they are either low frequency, look benign or look related to other [now] known issues. Otherwise the summary is as per the last comment; all new bugs are also present in current versions. I also tested MDEV-15164 , and it is considered fixed in bb-10.11.11-mdev16253-rocksdb-6.29fb . As for this bug ( MDEV-16523 ), no occurences of being_compacted were seen. If the push of bb-10.11.11-mdev16253-rocksdb-6.29fb is urgent, I believe we can proceed now. In any case, I will roundup the runs. Update: runs complete. All other remaining RocksDB issues were present in base versions as well. Fine to push, which already happened.
            Roel Roel Van de Paar made changes -
            Assignee Roel Van de Paar [ roel ] Sergei Golubchik [ serg ]
            Status In Testing [ 10301 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Sergei Petrunia [ psergey ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Petrunia [ psergey ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.11.12 [ 29998 ]
            Fix Version/s 10.11 [ 27614 ]
            Assignee Sergei Golubchik [ serg ] Sergei Petrunia [ psergey ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.4.6 [ 29999 ]
            Fix Version/s 11.8.2 [ 30001 ]

            People

              psergei Sergei Petrunia
              elenst Elena Stepanova
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.