Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9658

Make MyRocks in MariaDB stable

Details

    • 10.2.3-2, 10.2.4-4, 10.2.4-5, 10.2.4-1, 10.2.4-2, 10.2.5-1, 10.2.6-2, 10.2.6-3

    Description

      This is the umbrella task for including MyRocks Storage Engine from https://github.com/facebook/mysql-5.6 ("FB tree") into MariaDB.

      Inclusion itself

      See MDEV-10965: Add MyRocks into MariaDB repository

      Packaging

      See MDEV-10966 - Packaging for MariaRocks

      Required changes at the SQL layer

      A list of changes in the FB tree that touch the SQL layer (based on git log output):
      65501b5 Disable unique checks when replica is lagging
      dd7eeae Issue#250: MyRocks/Innodb different output from query with order ...
      f0a2ded Per database uuid
      9b439ee Add option to exclude tables from gap lock check
      5bcb50c Per database uuid
      e9ef099 Improve code style for classes RDBSE_TABLE_DEF and RDBSE_KEYDEF
      5a82f20 Add gcc function attributes to MyRocks code with a sprinkle of assertions
      6078c85 Preventing to write "Got error 122.." to err log on Gap Lock errors
      782cc5a Optionally block/log queries relying on Gap Locks
      c2dbe68 Add lock information to lock timeout error message
      c1f1f0b Issue #108: Index-only scans do not work for partitioned tables and extended keys
      3c0802f Improve singled thread replication performance
      55622f1 Issue #75: Prefix bloom filter is not used for LinkBench style range scan
      ee00797 Compile rocksdb library with mysql compiler settings
      8098b78 Add support for reporting keys/deletes skipped in the extra slow query log
      bcd7646 Supporting START TRANSACTION WITH CONSISTENT [ROCKSDB] SNAPSHOT
      ff91c80 Crash safe slave and master in RocksDB

      • also check if da1d92fd (set_end_range) is ported from mysql-5.6 because MyRocks' bloom prefix filter uses that.
      • MariaDB has just got NO_PAD collations: MDEV-9711. MyRocks actually had NO_PAD behaviour for PAD collations, and this was a bug
        https://github.com/facebook/mysql-5.6/issues/257. MyRocks will have to provide PAD/NO_PAD behaviour, accordingly to what charset is used.
      • rdb_perf_context.cc uses my_io_perf_sum_atomic_helper() the function is not in MySQL or MariaDB, it comes from bc4145e7
      • MDEV-10975: Merging of @@rocksdb_skip_unique_check
      • MDEV-10976: Port MyRocks' Read Free Replication to MariaRocks

        Group Commit with Binlog (and Optimistic Parallel Replication).

      MyRocks (actually, RocksDB and MyRocks together) need support for high-priority transactions to get MariaDB's optimistic Parallel Replication to work.

      Other issues

      • mysql-test-run should not require any additional parameters when running MyRocks tests.
      • packaging for myrocks_hotbackup
      • Do we include MyRocks' RQG tests? It has some MTR tests which actually just run the bundled RQG with a special grammar.
      • TODO what is missing?

      Attachments

        Issue Links

          includes

          Task - A task that needs to be done. MDEV-10965 Add MyRocks into MariaDB repository

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-10966 Packaging for MariaRocks

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-10975 Merging of @@rocksdb_skip_unique_check

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-10976 Port MyRocks' Read Free Replication to MariaRocks

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-11124 MariaRocks: rocksdb.rocksdb fails with a duplicate key error

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11194 MariaRocks: rocksdb.rocksdb_range test failure

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11318 MariaRocks: rocksdb.rocksdb test fails

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11320 MariaRocks: rocksdb.type_text_indexes fails

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11321 MariaRocks: type_binary_indexes, type_blob_indexes fail due to different index statistics

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11329 MariaRocks: rocksdb.add_index_inplace fails

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11462 MariaRocks: rocksdb.type_float_indexes fails

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11477 MariaRocks: rocksdb.type_varchar failure

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11480 MariaRocks port: rocksdb.slow_query_log test (DISABLED)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11481 MariaRocks port: rocksdb.select_for_update_skip_locked_nowait (DISABLED)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11735 MyRocks: Gap Lock detector support

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-11901 MariaDB Rocks on Windows

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-11934 MariaRocks: Group Commit with binlog

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-11946 MariaRocks: review the changes on SQL layer

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-12189 MariaRocks packaging: disable x86 builds

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12196 MariaRocks: compile failure on Windows

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12249 MariaRocks: find a way to handle paths to scripts/includes in MTR

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-12254 MariaRocks: check whether it gets into packages

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-12277 rocksdb.rocksdb fails with Sort Aborted error in server stderr

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12279 rocksdb.tbl_opt_data_index_dir fails, wrong error code

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12280 rocksdb.collation test started to fail with a crash

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-12285 MariaRocks: [ERROR] mysqld: Deadlock found when trying to get lock messages in server stderr

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12286 rocksdb.blind_delete_without_tx_api test fails

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-12316 MariaRocks: dont package ldb binary

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12341 MariaRocks: rocksdb.persistent_cache test fails on some platforms

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Task - A task that needs to be done. MDEV-12380 MariaRocks: repo, tests, and packaging for myrocks_hotbackup

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12424 binlog_encryption.encrypted_* tests fail with Can't locate autodie.pm error

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12425 MariaRocks: Tests require too much disk space

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12427 rocksdb.write_sync fails with "SyncWAL() is not supported for this implementation of WAL file"

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12431 MariaRocks unittest does not compile

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12433 MariaRocks: rocksdb.collation crashes because of gcc bug.

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12439 MariaRocks produces numerous (spurious?) valgrind failures

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12445 Rocksdb does not shutdown worker threads and aborts in memleak check on server shutdown

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-12528 Run the engine-agnostic test suite on MyRocks, too

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12663 rocksdb.compact_deletes times out and causes other tests to fail

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-14103 Testing for group commit in MyRocks

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14181 rocksdb.rocksdb fails: line 1117: query 'reap' succeeded - should have failed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14892 Porting Rdb_binlog_manager to MariaDB

          • Major - Major loss of function.
          • Open
          relates to

          Task - A task that needs to be done. MDEV-14106 MyRocks: port rocksdb.rpl_gtid_crash_safe test

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-14109 MyRocks port: do we need manual update of slave_gtid_info?

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-14372 Fix and enable rocksdb.information_schema test

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11172 EXPLAIN shows non-sensical value for key_len with type=index

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-14351 MyRocks as an embedded library

          • Major - Major loss of function.
          • Open

          Activity

            Ascending order - Click to sort in descending order
            View 34 older comments
            psergei Sergei Petrunia added a comment -

            obissick, currently there are no plans for Galera support. Galera integrates tightly with the storage engine, and at the moment the only engine it has integration with is InnoDB/XtraDB.

            psergei Sergei Petrunia added a comment - obissick , currently there are no plans for Galera support. Galera integrates tightly with the storage engine, and at the moment the only engine it has integration with is InnoDB/XtraDB.
            psergei Sergei Petrunia added a comment -

            Current status is:

            • Upstream is merged up to 

                  commit 184a4a2d82f4f6f3cbcb1015bcdb32bebe73315c
              		 
                  Author: Abhinav Sharma <abhinavsharma@fb.com>
              		 
                  Date:   Thu Sep 14 11:40:08 2017 -0700

            • Most of the features are in the 10.2 tree
            • The candidate code for "Group Commit with Binlog" (MDEV-11934) feature is in bb-10.2-mariarocks tree.
            psergei Sergei Petrunia added a comment - Current status is: Upstream is merged up to commit 184a4a2d82f4f6f3cbcb1015bcdb32bebe73315c Author: Abhinav Sharma <abhinavsharma@fb.com> Date: Thu Sep 14 11:40:08 2017 -0700 Most of the features are in the 10.2 tree The candidate code for "Group Commit with Binlog" ( MDEV-11934 ) feature is in bb-10.2-mariarocks tree.
            psergei Sergei Petrunia added a comment -

            Filed MDEV-14103, it is on elenst's plate now.

            psergei Sergei Petrunia added a comment - Filed MDEV-14103 , it is on elenst 's plate now.
            elenst Elena Stepanova added a comment -

            psergey,
            The previous comment is ambiguous, could you please specify what exactly is on my plate – MDEV-14103 or "making MyRocks in MariaDB stable"?
            I acknowledge receiving of MDEV-14103 (Testing for group commit in MyRocks), but it cannot possibly be the only obstacle between the current state and declaring MyRocks stable.
            We have 7 open buildbot bugs for RocksDB which need to be closed before declaring MyRocks stable, and quite a few other open bugs/tasks, priority of which needs to be revised, because it was initially set based on the fact that MyRocks was alpha/beta.

            elenst Elena Stepanova added a comment - psergey , The previous comment is ambiguous, could you please specify what exactly is on my plate – MDEV-14103 or "making MyRocks in MariaDB stable"? I acknowledge receiving of MDEV-14103 (Testing for group commit in MyRocks), but it cannot possibly be the only obstacle between the current state and declaring MyRocks stable. We have 7 open buildbot bugs for RocksDB which need to be closed before declaring MyRocks stable, and quite a few other open bugs/tasks, priority of which needs to be revised, because it was initially set based on the fact that MyRocks was alpha/beta.
            psergei Sergei Petrunia added a comment -

            elenst, it is only MDEV-14103 that is on your plate. Sorry for the confusion.

            psergei Sergei Petrunia added a comment - elenst , it is only MDEV-14103 that is on your plate. Sorry for the confusion.

            People

              psergei Sergei Petrunia
              colin Colin Charles
              Votes:
              16 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.