Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14290

Mariadb 10.2 RocksDB crash on binlog switch to new file

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.2(EOL), 10.3(EOL)
    • 10.2.12
    • None
    • Debian 9 x64, Mariadb 10.2

    Description

          04 23:04:33 mon-db-st mysqld[1180]: 2017-11-04 23:04:33 139846979504320 [Note] Reading of all Master_info entries succeded
      Nov 04 23:04:33 mon-db-st mysqld[1180]: 2017-11-04 23:04:33 139846979504320 [Note] Added new Master_info '' to hash table
      Nov 04 23:04:33 mon-db-st mysqld[1180]: 2017-11-04 23:04:33 139846979504320 [Note] /usr/sbin/mysqld: ready for connections.
      Nov 04 23:04:33 mon-db-st mysqld[1180]: Version: '10.2.10-MariaDB-10.2.10+maria~stretch-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
      Nov 04 23:04:34 mon-db-st systemd[1]: Started MariaDB database server.
      Nov 04 23:05:20 mon-db-st mysqld[1180]: 2017-11-04 23:05:20 139843793712896 [ERROR] Invalid (old?) table or database name '.rocksdb'
      Nov 04 23:05:20 mon-db-st mysqld[1180]: 2017-11-04 23:05:20 139843793712896 [ERROR] Invalid (old?) table or database name '.binlog'
      Nov 04 23:08:49 mon-db-st mysqld[1180]: 171104 23:08:49 [ERROR] mysqld got signal 11 ;
      Nov 04 23:08:49 mon-db-st mysqld[1180]: This could be because you hit a bug. It is also possible that this binary
      Nov 04 23:08:49 mon-db-st mysqld[1180]: or one of the libraries it was linked against is corrupt, improperly built,
      Nov 04 23:08:49 mon-db-st mysqld[1180]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Nov 04 23:08:49 mon-db-st mysqld[1180]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Nov 04 23:08:49 mon-db-st mysqld[1180]: We will try our best to scrape up some info that will hopefully help
      Nov 04 23:08:49 mon-db-st mysqld[1180]: diagnose the problem, but since we have already crashed,
      Nov 04 23:08:49 mon-db-st mysqld[1180]: something is definitely wrong and this may fail.
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Server version: 10.2.10-MariaDB-10.2.10+maria~stretch-log
      Nov 04 23:08:49 mon-db-st mysqld[1180]: key_buffer_size=33554432
      Nov 04 23:08:49 mon-db-st mysqld[1180]: read_buffer_size=1048576
      Nov 04 23:08:49 mon-db-st mysqld[1180]: max_used_connections=6
      Nov 04 23:08:49 mon-db-st mysqld[1180]: max_threads=302
      Nov 04 23:08:49 mon-db-st mysqld[1180]: thread_count=13
      Nov 04 23:08:49 mon-db-st mysqld[1180]: It is possible that mysqld could use up to
      Nov 04 23:08:49 mon-db-st mysqld[1180]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 966889 K  bytes of memory
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Hope that's ok; if not, decrease some variables in the equation.
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Thread pointer: 0x7f2fc00009a8
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Attempting backtrace. You can use the following information to find out
      Nov 04 23:08:49 mon-db-st mysqld[1180]: where mysqld died. If you see no messages after this, something went
      Nov 04 23:08:49 mon-db-st mysqld[1180]: terribly wrong...
      Nov 04 23:08:49 mon-db-st mysqld[1180]: stack_bottom = 0x7f308c1ded18 thread_stack 0x49000
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56063106bc2e]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /usr/sbin/mysqld(handle_fatal_signal+0x3cd)[0x560630b0a8dd]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f30a947c0c0]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x97)[0x560630bd8e97]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /usr/sbin/mysqld(binlog_background_thread+0x323)[0x560630bd9573]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7f30a9472494]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f30a7743aff]
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Trying to get some variables.
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Some pointers may be invalid and cause the dump to abort.
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Query (0x0):
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Connection ID (thread ID): 6
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Status: NOT_KILLED
      Nov 04 23:08:49 mon-db-st mysqld[1180]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
      Nov 04 23:08:49 mon-db-st mysqld[1180]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Nov 04 23:08:49 mon-db-st mysqld[1180]: information that should help you find out what is causing the crash.
      Nov 04 23:08:50 mon-db-st systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
      

      I just try move some data from table to table,
      src table TokuDB, dst table RocksDB
      When binlog reach maximum size and new binlog must be created, mariadb just crash with above info.

      Thanks

      Attachments

        Issue Links

          Activity

            Nefelim4ag Timofey added a comment -

            I use following in script for data migration:
            REPLACE INTO $newtable SELECT * FROM $oldtable WHERE itemid = $itemid; DELETE FROM $oldtable WHERE itemid = $itemid;

            If i add "SET sql_log_bin=0;" to above statement, mariadb don't crush, as binlog don't touched.

            Nefelim4ag Timofey added a comment - I use following in script for data migration: REPLACE INTO $newtable SELECT * FROM $oldtable WHERE itemid = $itemid; DELETE FROM $oldtable WHERE itemid = $itemid; If i add "SET sql_log_bin=0;" to above statement, mariadb don't crush, as binlog don't touched.

            Thank you for report.
            Problem happens without TokuDB - basicallly any binlog rotation with RocksDB installed shows the problem:

            Nov 17 10:11:37 UBINTI mysqld[6004]: Version: '10.2.10-MariaDB-10.2.10+maria~zesty-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
            Nov 17 10:11:38 UBINTI systemd[1]: Started MariaDB database server.
            Nov 17 10:12:47 UBINTI mysqld[6004]: 2017-11-17 10:12:47 139650531075840 [ERROR] Invalid (old?) table or database name '.rocksdb'
            Nov 17 10:13:30 UBINTI mysqld[6004]: 171117 10:13:30 [ERROR] mysqld got signal 11 ;
            Nov 17 10:13:30 UBINTI mysqld[6004]: This could be because you hit a bug. It is also possible that this binary
            Nov 17 10:13:30 UBINTI mysqld[6004]: or one of the libraries it was linked against is corrupt, improperly built,
            Nov 17 10:13:30 UBINTI mysqld[6004]: or misconfigured. This error can also be caused by malfunctioning hardware.
            Nov 17 10:13:30 UBINTI mysqld[6004]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            Nov 17 10:13:30 UBINTI mysqld[6004]: We will try our best to scrape up some info that will hopefully help
            Nov 17 10:13:30 UBINTI mysqld[6004]: diagnose the problem, but since we have already crashed,
            Nov 17 10:13:30 UBINTI mysqld[6004]: something is definitely wrong and this may fail.
            Nov 17 10:13:30 UBINTI mysqld[6004]: Server version: 10.2.10-MariaDB-10.2.10+maria~zesty-log
            Nov 17 10:13:30 UBINTI mysqld[6004]: key_buffer_size=134217728
            Nov 17 10:13:30 UBINTI mysqld[6004]: read_buffer_size=2097152
            Nov 17 10:13:30 UBINTI mysqld[6004]: max_used_connections=1
            Nov 17 10:13:30 UBINTI mysqld[6004]: max_threads=102
            Nov 17 10:13:30 UBINTI mysqld[6004]: thread_count=8
            Nov 17 10:13:30 UBINTI mysqld[6004]: It is possible that mysqld could use up to
            Nov 17 10:13:30 UBINTI mysqld[6004]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759899 K  bytes of memory
            Nov 17 10:13:30 UBINTI mysqld[6004]: Hope that's ok; if not, decrease some variables in the equation.
            Nov 17 10:13:30 UBINTI mysqld[6004]: Thread pointer: 0x7f02780009a8
            Nov 17 10:13:30 UBINTI mysqld[6004]: Attempting backtrace. You can use the following information to find out
            Nov 17 10:13:30 UBINTI mysqld[6004]: where mysqld died. If you see no messages after this, something went
            Nov 17 10:13:30 UBINTI mysqld[6004]: terribly wrong...
            Nov 17 10:13:30 UBINTI mysqld[6004]: stack_bottom = 0x7f02ee1a0d18 thread_stack 0x49000
            Nov 17 10:13:30 UBINTI mysqld[6004]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555b8c8d022e]
            Nov 17 10:13:31 UBINTI mysqld[6004]: /usr/sbin/mysqld(handle_fatal_signal+0x3cd)[0x555b8c36ed9d]
            Nov 17 10:13:31 UBINTI mysqld[6004]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f0308cf8670]
            Nov 17 10:13:31 UBINTI mysqld[6004]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x97)[0x555b8c43d357]
            Nov 17 10:13:31 UBINTI mysqld[6004]: /usr/sbin/mysqld(binlog_background_thread+0x323)[0x555b8c43da33]
            Nov 17 10:13:31 UBINTI mysqld[6004]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f0308cee6da]
            Nov 17 10:13:31 UBINTI mysqld[6004]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f0308397d7f]
            Nov 17 10:13:31 UBINTI mysqld[6004]: Trying to get some variables.
            Nov 17 10:13:31 UBINTI mysqld[6004]: Some pointers may be invalid and cause the dump to abort.
            Nov 17 10:13:31 UBINTI mysqld[6004]: Query (0x0):
            Nov 17 10:13:31 UBINTI mysqld[6004]: Connection ID (thread ID): 6
            Nov 17 10:13:31 UBINTI mysqld[6004]: Status: NOT_KILLED
            

            MariaDB [(none)]> show plugins;
            +-------------------------------+----------+--------------------+---------------+---------+
            | Name                          | Status   | Type               | Library       | License |
            +-------------------------------+----------+--------------------+---------------+---------+
            | binlog                        | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | mysql_native_password         | ACTIVE   | AUTHENTICATION     | NULL          | GPL     |
            | mysql_old_password            | ACTIVE   | AUTHENTICATION     | NULL          | GPL     |
            | wsrep                         | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | CSV                           | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | MEMORY                        | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | MyISAM                        | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | MRG_MyISAM                    | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | CLIENT_STATISTICS             | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INDEX_STATISTICS              | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | TABLE_STATISTICS              | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | USER_STATISTICS               | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | InnoDB                        | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | INNODB_TRX                    | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_LOCKS                  | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_LOCK_WAITS             | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_CMP                    | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_CMP_RESET              | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_CMPMEM                 | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_CMPMEM_RESET           | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_CMP_PER_INDEX          | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_CMP_PER_INDEX_RESET    | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_BUFFER_PAGE            | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_BUFFER_PAGE_LRU        | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_BUFFER_POOL_STATS      | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_METRICS                | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_FT_DEFAULT_STOPWORD    | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_FT_DELETED             | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_FT_BEING_DELETED       | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_FT_CONFIG              | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_FT_INDEX_CACHE         | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_FT_INDEX_TABLE         | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_TABLES             | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_TABLESTATS         | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_INDEXES            | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_COLUMNS            | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_FIELDS             | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_FOREIGN            | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_FOREIGN_COLS       | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_TABLESPACES        | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_DATAFILES          | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_VIRTUAL            | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_MUTEXES                | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_SYS_SEMAPHORE_WAITS    | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | INNODB_TABLESPACES_ENCRYPTION | ACTIVE   | INFORMATION SCHEMA | NULL          | BSD     |
            | INNODB_TABLESPACES_SCRUBBING  | ACTIVE   | INFORMATION SCHEMA | NULL          | BSD     |
            | Aria                          | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | PERFORMANCE_SCHEMA            | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | SEQUENCE                      | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | FEEDBACK                      | DISABLED | INFORMATION SCHEMA | NULL          | GPL     |
            | user_variables                | ACTIVE   | INFORMATION SCHEMA | NULL          | GPL     |
            | partition                     | ACTIVE   | STORAGE ENGINE     | NULL          | GPL     |
            | ROCKSDB                       | ACTIVE   | STORAGE ENGINE     | ha_rocksdb.so | GPL     |
            | ROCKSDB_CFSTATS               | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_DBSTATS               | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_PERF_CONTEXT          | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_PERF_CONTEXT_GLOBAL   | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_CF_OPTIONS            | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_COMPACTION_STATS      | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_GLOBAL_INFO           | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_DDL                   | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_INDEX_FILE_MAP        | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_LOCKS                 | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            | ROCKSDB_TRX                   | ACTIVE   | INFORMATION SCHEMA | ha_rocksdb.so | GPL     |
            +-------------------------------+----------+--------------------+---------------+---------+
            64 rows in set (0.01 sec)
             
            MariaDB [(none)]> use test
            Reading table information for completion of table and column names
            You can turn off this feature to get a quicker startup with -A
             
            Database changed
             
            MariaDB [test]> flush binary logs;
            Query OK, 0 rows affected (0.15 sec)
             
            MariaDB [test]> select 1;
            ERROR 2006 (HY000): MySQL server has gone away
            No connection. Trying to reconnect...
            ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
            ERROR: Can't connect to the server
             
            unknown [test]> flush binary logs;
            No connection. Trying to reconnect...
            Connection id:    9
            Current database: test
             
            Query OK, 0 rows affected (0.15 sec)
             
            MariaDB [test]> select 1;
            ERROR 2006 (HY000): MySQL server has gone away
            No connection. Trying to reconnect...
            ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
            ERROR: Can't connect to the server
             
            unknown [test]> 
            unknown [test]> 
            unknown [test]> uninstall soname 'ha_rocksdb';
            No connection. Trying to reconnect...
            Connection id:    9
            Current database: test
             
            Query OK, 0 rows affected (0.09 sec)
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> flush binary logs;
            Query OK, 0 rows affected (0.15 sec)
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> install soname 'ha_rocksdb';
            ERROR 2013 (HY000): Lost connection to MySQL server during query
            MariaDB [test]> install soname 'ha_rocksdb';
            ERROR 2006 (HY000): MySQL server has gone away
            No connection. Trying to reconnect...
            ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
            ERROR: Can't connect to the server
             
            unknown [test]> install soname 'ha_rocksdb';
            No connection. Trying to reconnect...
            Connection id:    9
            Current database: test
             
            Query OK, 0 rows affected (0.00 sec)
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> flush binary logs;
            Query OK, 0 rows affected (0.16 sec)
             
            MariaDB [test]> select 1;
            ERROR 2006 (HY000): MySQL server has gone away
            No connection. Trying to reconnect...
            ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
            ERROR: Can't connect to the server
             
            unknown [test]> uninstall soname 'ha_rocksdb';
            No connection. Trying to reconnect...
            Connection id:    9
            Current database: test
             
            Query OK, 0 rows affected (0.08 sec)
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> flush binary logs;
            Query OK, 0 rows affected (0.14 sec)
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> flush binary logs;
            Query OK, 0 rows affected (0.18 sec)
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> install soname 'ha_rocksdb';
            ERROR 2013 (HY000): Lost connection to MySQL server during query
            MariaDB [test]> install soname 'ha_rocksdb';
            ERROR 2006 (HY000): MySQL server has gone away
            No connection. Trying to reconnect...
            Connection id:    9
            Current database: test
             
            Query OK, 0 rows affected (0.01 sec)
             
            MariaDB [test]> create table tt engine=rocksdb select 1;
            Query OK, 1 row affected (0.14 sec)
            Records: 1  Duplicates: 0  Warnings: 0
             
            MariaDB [test]> select 1;
            +---+
            | 1 |
            +---+
            | 1 |
            +---+
            1 row in set (0.00 sec)
             
            MariaDB [test]> flush binary logs;
            Query OK, 0 rows affected (0.16 sec)
             
            MariaDB [test]> select 1;
            ERROR 2006 (HY000): MySQL server has gone away
            No connection. Trying to reconnect...
            ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
            ERROR: Can't connect to the server
            

            anikitin Andrii Nikitin (Inactive) added a comment - Thank you for report. Problem happens without TokuDB - basicallly any binlog rotation with RocksDB installed shows the problem: Nov 17 10:11:37 UBINTI mysqld[6004]: Version: '10.2.10-MariaDB-10.2.10+maria~zesty-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution Nov 17 10:11:38 UBINTI systemd[1]: Started MariaDB database server. Nov 17 10:12:47 UBINTI mysqld[6004]: 2017-11-17 10:12:47 139650531075840 [ERROR] Invalid (old?) table or database name '.rocksdb' Nov 17 10:13:30 UBINTI mysqld[6004]: 171117 10:13:30 [ERROR] mysqld got signal 11 ; Nov 17 10:13:30 UBINTI mysqld[6004]: This could be because you hit a bug. It is also possible that this binary Nov 17 10:13:30 UBINTI mysqld[6004]: or one of the libraries it was linked against is corrupt, improperly built, Nov 17 10:13:30 UBINTI mysqld[6004]: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 17 10:13:30 UBINTI mysqld[6004]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 17 10:13:30 UBINTI mysqld[6004]: We will try our best to scrape up some info that will hopefully help Nov 17 10:13:30 UBINTI mysqld[6004]: diagnose the problem, but since we have already crashed, Nov 17 10:13:30 UBINTI mysqld[6004]: something is definitely wrong and this may fail. Nov 17 10:13:30 UBINTI mysqld[6004]: Server version: 10.2.10-MariaDB-10.2.10+maria~zesty-log Nov 17 10:13:30 UBINTI mysqld[6004]: key_buffer_size=134217728 Nov 17 10:13:30 UBINTI mysqld[6004]: read_buffer_size=2097152 Nov 17 10:13:30 UBINTI mysqld[6004]: max_used_connections=1 Nov 17 10:13:30 UBINTI mysqld[6004]: max_threads=102 Nov 17 10:13:30 UBINTI mysqld[6004]: thread_count=8 Nov 17 10:13:30 UBINTI mysqld[6004]: It is possible that mysqld could use up to Nov 17 10:13:30 UBINTI mysqld[6004]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759899 K bytes of memory Nov 17 10:13:30 UBINTI mysqld[6004]: Hope that's ok; if not, decrease some variables in the equation. Nov 17 10:13:30 UBINTI mysqld[6004]: Thread pointer: 0x7f02780009a8 Nov 17 10:13:30 UBINTI mysqld[6004]: Attempting backtrace. You can use the following information to find out Nov 17 10:13:30 UBINTI mysqld[6004]: where mysqld died. If you see no messages after this, something went Nov 17 10:13:30 UBINTI mysqld[6004]: terribly wrong... Nov 17 10:13:30 UBINTI mysqld[6004]: stack_bottom = 0x7f02ee1a0d18 thread_stack 0x49000 Nov 17 10:13:30 UBINTI mysqld[6004]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555b8c8d022e] Nov 17 10:13:31 UBINTI mysqld[6004]: /usr/sbin/mysqld(handle_fatal_signal+0x3cd)[0x555b8c36ed9d] Nov 17 10:13:31 UBINTI mysqld[6004]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f0308cf8670] Nov 17 10:13:31 UBINTI mysqld[6004]: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x97)[0x555b8c43d357] Nov 17 10:13:31 UBINTI mysqld[6004]: /usr/sbin/mysqld(binlog_background_thread+0x323)[0x555b8c43da33] Nov 17 10:13:31 UBINTI mysqld[6004]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7f0308cee6da] Nov 17 10:13:31 UBINTI mysqld[6004]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f0308397d7f] Nov 17 10:13:31 UBINTI mysqld[6004]: Trying to get some variables. Nov 17 10:13:31 UBINTI mysqld[6004]: Some pointers may be invalid and cause the dump to abort. Nov 17 10:13:31 UBINTI mysqld[6004]: Query (0x0): Nov 17 10:13:31 UBINTI mysqld[6004]: Connection ID (thread ID): 6 Nov 17 10:13:31 UBINTI mysqld[6004]: Status: NOT_KILLED MariaDB [(none)]> show plugins; +-------------------------------+----------+--------------------+---------------+---------+ | Name | Status | Type | Library | License | +-------------------------------+----------+--------------------+---------------+---------+ | binlog | ACTIVE | STORAGE ENGINE | NULL | GPL | | mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL | | mysql_old_password | ACTIVE | AUTHENTICATION | NULL | GPL | | wsrep | ACTIVE | STORAGE ENGINE | NULL | GPL | | CSV | ACTIVE | STORAGE ENGINE | NULL | GPL | | MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL | | MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL | | MRG_MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL | | CLIENT_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INDEX_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | TABLE_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | USER_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL | | INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_LOCKS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_LOCK_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_PER_INDEX | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_PER_INDEX_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_METRICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_DEFAULT_STOPWORD | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_BEING_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_CONFIG | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_INDEX_CACHE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_INDEX_TABLE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLESTATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_COLUMNS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FIELDS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FOREIGN | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FOREIGN_COLS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_DATAFILES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_VIRTUAL | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_MUTEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_SEMAPHORE_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_TABLESPACES_ENCRYPTION | ACTIVE | INFORMATION SCHEMA | NULL | BSD | | INNODB_TABLESPACES_SCRUBBING | ACTIVE | INFORMATION SCHEMA | NULL | BSD | | Aria | ACTIVE | STORAGE ENGINE | NULL | GPL | | PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL | | SEQUENCE | ACTIVE | STORAGE ENGINE | NULL | GPL | | FEEDBACK | DISABLED | INFORMATION SCHEMA | NULL | GPL | | user_variables | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | partition | ACTIVE | STORAGE ENGINE | NULL | GPL | | ROCKSDB | ACTIVE | STORAGE ENGINE | ha_rocksdb.so | GPL | | ROCKSDB_CFSTATS | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_DBSTATS | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_PERF_CONTEXT | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_PERF_CONTEXT_GLOBAL | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_CF_OPTIONS | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_COMPACTION_STATS | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_GLOBAL_INFO | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_DDL | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_INDEX_FILE_MAP | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_LOCKS | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | | ROCKSDB_TRX | ACTIVE | INFORMATION SCHEMA | ha_rocksdb.so | GPL | +-------------------------------+----------+--------------------+---------------+---------+ 64 rows in set (0.01 sec)   MariaDB [(none)]> use test Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A   Database changed   MariaDB [ test ]> flush binary logs; Query OK, 0 rows affected (0.15 sec)   MariaDB [ test ]> select 1; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... ERROR 2002 (HY000): Can 't connect to local MySQL server through socket ' /var/run/mysqld/mysqld .sock' (111) ERROR: Can't connect to the server   unknown [ test ]> flush binary logs; No connection. Trying to reconnect... Connection id : 9 Current database: test   Query OK, 0 rows affected (0.15 sec)   MariaDB [ test ]> select 1; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... ERROR 2002 (HY000): Can 't connect to local MySQL server through socket ' /var/run/mysqld/mysqld .sock' (111) ERROR: Can't connect to the server   unknown [ test ]> unknown [ test ]> unknown [ test ]> uninstall soname 'ha_rocksdb' ; No connection. Trying to reconnect... Connection id : 9 Current database: test   Query OK, 0 rows affected (0.09 sec)   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> flush binary logs; Query OK, 0 rows affected (0.15 sec)   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> install soname 'ha_rocksdb' ; ERROR 2013 (HY000): Lost connection to MySQL server during query MariaDB [ test ]> install soname 'ha_rocksdb' ; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... ERROR 2002 (HY000): Can 't connect to local MySQL server through socket ' /var/run/mysqld/mysqld .sock' (111) ERROR: Can't connect to the server   unknown [ test ]> install soname 'ha_rocksdb' ; No connection. Trying to reconnect... Connection id : 9 Current database: test   Query OK, 0 rows affected (0.00 sec)   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> flush binary logs; Query OK, 0 rows affected (0.16 sec)   MariaDB [ test ]> select 1; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... ERROR 2002 (HY000): Can 't connect to local MySQL server through socket ' /var/run/mysqld/mysqld .sock' (111) ERROR: Can't connect to the server   unknown [ test ]> uninstall soname 'ha_rocksdb' ; No connection. Trying to reconnect... Connection id : 9 Current database: test   Query OK, 0 rows affected (0.08 sec)   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> flush binary logs; Query OK, 0 rows affected (0.14 sec)   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> flush binary logs; Query OK, 0 rows affected (0.18 sec)   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> install soname 'ha_rocksdb' ; ERROR 2013 (HY000): Lost connection to MySQL server during query MariaDB [ test ]> install soname 'ha_rocksdb' ; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id : 9 Current database: test   Query OK, 0 rows affected (0.01 sec)   MariaDB [ test ]> create table tt engine=rocksdb select 1; Query OK, 1 row affected (0.14 sec) Records: 1 Duplicates: 0 Warnings: 0   MariaDB [ test ]> select 1; +---+ | 1 | +---+ | 1 | +---+ 1 row in set (0.00 sec)   MariaDB [ test ]> flush binary logs; Query OK, 0 rows affected (0.16 sec)   MariaDB [ test ]> select 1; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... ERROR 2002 (HY000): Can 't connect to local MySQL server through socket ' /var/run/mysqld/mysqld .sock' (111) ERROR: Can't connect to the server
            anikitin Andrii Nikitin (Inactive) added a comment - - edited

            Reproducible in current 10.2 tree with simple `flush binary logs;` command when binlog is enabled and rocksdb plugin loaded

            anikitin Andrii Nikitin (Inactive) added a comment - - edited Reproducible in current 10.2 tree with simple `flush binary logs;` command when binlog is enabled and rocksdb plugin loaded
            anikitin Andrii Nikitin (Inactive) added a comment - - edited

            It looks InnoDB must be enabled for crash to occur. Considering mtr test:

            --source include/have_rocksdb.inc
             
            flush binary logs;
            do sleep(2);
             
            create table t engine=rocksdb select 1;
            flush binary logs;
            do sleep(5);
             
            drop table t;
            flush binary logs;
            select 'passed';
            

            It passes when started with just `--mysqld=--log-bin --mysqld=--binlog-format=row`

            ./mtr binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row
            Logging: /home/a/env1/m8-10.2/src/mysql-test/mysql-test-run.pl  binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row
            Using binlog format 'row'
            vardir: /home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var
            Checking leftover processes...
            Removing old var directory...
            Creating var directory '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var'...
            Checking supported features...
            MariaDB Version 10.2.11-MariaDB
             - SSL connections supported
            Collecting tests...
            Installing system database...
             
            ==============================================================================
             
            TEST                                      RESULT   TIME (ms) or COMMENT
            --------------------------------------------------------------------------
             
            worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
            flush binary logs;
            do sleep(2);
            create table t engine=rocksdb select 1;
            flush binary logs;
            do sleep(5);
            drop table t;
            flush binary logs;
            select 'passed';
            passed
            passed
            rocksdb.binlog_rotate                    [ pass ]   7258
            --------------------------------------------------------------------------
            The servers were restarted 0 times
            Spent 7.258 of 14 seconds executing testcases
             
            Completed: All 1 tests were successful.
            

            And crashes if one extra parameter is used: --mysqld=--innodb

            ./mtr binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row --mysqld=--innodb
            Logging: /home/a/env1/m8-10.2/src/mysql-test/mysql-test-run.pl  binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row --mysqld=--innodb
            Using binlog format 'row'
            vardir: /home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var
            Checking leftover processes...
            Removing old var directory...
            Creating var directory '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var'...
            Checking supported features...
            MariaDB Version 10.2.11-MariaDB
             - SSL connections supported
            Collecting tests...
            Installing system database...
             
            ==============================================================================
             
            TEST                                      RESULT   TIME (ms) or COMMENT
            --------------------------------------------------------------------------
             
            worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
            flush binary logs;
            do sleep(2);
            create table t engine=rocksdb select 1;
            flush binary logs;
            do sleep(5);
            rocksdb.binlog_rotate                    [ fail ]
                    Test ended at 2017-11-17 13:51:57
             
            CURRENT_TEST: rocksdb.binlog_rotate
            mysqltest: At line 8: query 'do sleep(5)' failed: 2013: Lost connection to MySQL server during query
             
             
            Server [mysqld.1 - pid: 5142, winpid: 5142, exit: 256] failed during test run
            Server log from this test:
            ----------SERVER LOG START-----------
            2017-11-17 13:51:54 140295267206976 [Note] /home/a/env1/m8-10.2/bld/sql/mysqld (mysqld 10.2.11-MariaDB-log) starting as process 5143 ...
            2017-11-17 13:51:54 140295267206976 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
            2017-11-17 13:51:54 140295267206976 [Note] RocksDB: Got ENOENT when listing column families
            2017-11-17 13:51:54 140295267206976 [Note] RocksDB:   assuming that we're creating a new database
            2017-11-17 13:51:54 140295267206976 [Note] RocksDB: Column Families at start:
            2017-11-17 13:51:54 140295267206976 [Note]   cf=default
            2017-11-17 13:51:54 140295267206976 [Note]     write_buffer_size=67108864
            2017-11-17 13:51:54 140295267206976 [Note]     target_file_size_base=67108864
            2017-11-17 13:51:54 140295267206976 [Note] RocksDB: creating a column family __system__
            2017-11-17 13:51:54 140295267206976 [Note]     write_buffer_size=67108864
            2017-11-17 13:51:54 140295267206976 [Note]     target_file_size_base=67108864
            2017-11-17 13:51:54 140295267206976 [Note] RocksDB: Table_store: loaded DDL data for 0 tables
            2017-11-17 13:51:54 140295267206976 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
            2017-11-17 13:51:54 140295267206976 [Note] MyRocks storage engine plugin has been successfully initialized.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'SEQUENCE' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'partition' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Uses event mutexes
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Compressed tables use zlib 1.2.11
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Using Linux native AIO
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Number of pools: 1
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Using SSE2 crc32 instructions
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Completed initialization of buffer pool
            2017-11-17 13:51:54 140294695642880 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Highest supported file format is Barracuda.
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: 128 out of 128 rollback segments are active.
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Creating shared tablespace for temporary tables
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
            2017-11-17 13:51:54 140295267206976 [Note] InnoDB: 5.7.20 started; log sequence number 1633766
            2017-11-17 13:51:54 140294527887104 [Note] InnoDB: Loading buffer pool(s) from /home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/mysqld.1/data/ib_buffer_pool
            2017-11-17 13:51:54 140294527887104 [Note] InnoDB: Buffer pool(s) load completed at 171117 13:51:54
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_LOCKS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_MUTEXES' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMPMEM' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_TRX' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_METRICS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'user_variables' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
            2017-11-17 13:51:54 140295267206976 [Note] Plugin 'FEEDBACK' is disabled.
            2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
            2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
            2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-feedback-debug-interval=60'
            2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown option '--loose-pam-debug'
            2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown option '--loose-enable-rocksdb_lock_info'
            2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-debug-sync-timeout=300'
            2017-11-17 13:51:54 140295267206976 [Note] Server socket created on IP: '127.0.0.1'.
            2017-11-17 13:51:54 140295267206976 [Note] /home/a/env1/m8-10.2/bld/sql/mysqld: ready for connections.
            Version: '10.2.11-MariaDB-log'  socket: '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/tmp/mysqld.1.sock'  port: 16000  Source distribution
            2017-11-17 13:51:55 140294768772864 [Note] RocksDB: Manual memtable flush.
            171117 13:51:57 [ERROR] mysqld got signal 11 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.2.11-MariaDB-log
            key_buffer_size=1048576
            read_buffer_size=131072
            max_used_connections=1
            max_threads=153
            thread_count=8
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62996 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7f98a40009a8
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7f99000b8e78 thread_stack 0x49000
            /home/a/env1/m8-10.2/bld/sql/mysqld(my_print_stacktrace+0x29)[0x5627b085bc39]
            /home/a/env1/m8-10.2/bld/sql/mysqld(handle_fatal_signal+0x3bd)[0x5627b039aabd]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f99085ff670]
            /home/a/env1/m8-10.2/bld/sql/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x9f)[0x5627b045929f]
            /home/a/env1/m8-10.2/bld/sql/mysqld(binlog_background_thread+0x30b)[0x5627b045995b]
            /home/a/env1/m8-10.2/bld/sql/mysqld(+0xab32d4)[0x5627b08242d4]
            mysys/stacktrace.c:268(my_print_stacktrace)[0x7f99085f56da]
            sql/log.cc:9462(MYSQL_BIN_LOG::mark_xid_done(unsigned long, bool))[0x7f9907a87d7f]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x0): 
            Connection ID (thread ID): 6
            Status: NOT_KILLED
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
             
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Writing a core file
            ----------SERVER LOG END-------------
             
             
             - saving '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/log/rocksdb.binlog_rotate/' to '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/log/rocksdb.binlog_rotate/'
            --------------------------------------------------------------------------
            The servers were restarted 0 times
            Spent 0.000 of 8 seconds executing testcases
             
            Failure: Failed 1/1 tests, 0.00% were successful.
             
            Failing test(s): rocksdb.binlog_rotate
             
            The log files in var/log may give you some hint of what went wrong.
             
            If you want to report this error, please read first the documentation
            at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html
             
            mysql-test-run: *** ERROR: there were failing test cases
            

            anikitin Andrii Nikitin (Inactive) added a comment - - edited It looks InnoDB must be enabled for crash to occur. Considering mtr test: --source include/have_rocksdb.inc   flush binary logs; do sleep(2);   create table t engine=rocksdb select 1; flush binary logs; do sleep(5);   drop table t; flush binary logs; select 'passed' ; It passes when started with just `--mysqld=--log-bin --mysqld=--binlog-format=row` ./mtr binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row Logging: /home/a/env1/m8-10.2/src/mysql-test/mysql-test-run.pl binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row Using binlog format 'row' vardir: /home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var Checking leftover processes... Removing old var directory... Creating var directory '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var'... Checking supported features... MariaDB Version 10.2.11-MariaDB - SSL connections supported Collecting tests... Installing system database...   ==============================================================================   TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 flush binary logs; do sleep(2); create table t engine=rocksdb select 1; flush binary logs; do sleep(5); drop table t; flush binary logs; select 'passed'; passed passed rocksdb.binlog_rotate [ pass ] 7258 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 7.258 of 14 seconds executing testcases   Completed: All 1 tests were successful. And crashes if one extra parameter is used: --mysqld=--innodb ./mtr binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row --mysqld=--innodb Logging: /home/a/env1/m8-10.2/src/mysql-test/mysql-test-run.pl binlog_rotate --suite=rocksdb --mysqld=--log-bin --mysqld=--binlog-format=row --mysqld=--innodb Using binlog format 'row' vardir: /home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var Checking leftover processes... Removing old var directory... Creating var directory '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var'... Checking supported features... MariaDB Version 10.2.11-MariaDB - SSL connections supported Collecting tests... Installing system database...   ==============================================================================   TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 flush binary logs; do sleep(2); create table t engine=rocksdb select 1; flush binary logs; do sleep(5); rocksdb.binlog_rotate [ fail ] Test ended at 2017-11-17 13:51:57   CURRENT_TEST: rocksdb.binlog_rotate mysqltest: At line 8: query 'do sleep(5)' failed: 2013: Lost connection to MySQL server during query     Server [mysqld.1 - pid: 5142, winpid: 5142, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2017-11-17 13:51:54 140295267206976 [Note] /home/a/env1/m8-10.2/bld/sql/mysqld (mysqld 10.2.11-MariaDB-log) starting as process 5143 ... 2017-11-17 13:51:54 140295267206976 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162) 2017-11-17 13:51:54 140295267206976 [Note] RocksDB: Got ENOENT when listing column families 2017-11-17 13:51:54 140295267206976 [Note] RocksDB: assuming that we're creating a new database 2017-11-17 13:51:54 140295267206976 [Note] RocksDB: Column Families at start: 2017-11-17 13:51:54 140295267206976 [Note] cf=default 2017-11-17 13:51:54 140295267206976 [Note] write_buffer_size=67108864 2017-11-17 13:51:54 140295267206976 [Note] target_file_size_base=67108864 2017-11-17 13:51:54 140295267206976 [Note] RocksDB: creating a column family __system__ 2017-11-17 13:51:54 140295267206976 [Note] write_buffer_size=67108864 2017-11-17 13:51:54 140295267206976 [Note] target_file_size_base=67108864 2017-11-17 13:51:54 140295267206976 [Note] RocksDB: Table_store: loaded DDL data for 0 tables 2017-11-17 13:51:54 140295267206976 [Note] RocksDB: global statistics using get_sched_indexer_t indexer 2017-11-17 13:51:54 140295267206976 [Note] MyRocks storage engine plugin has been successfully initialized. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'SEQUENCE' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'partition' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Uses event mutexes 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Compressed tables use zlib 1.2.11 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Using Linux native AIO 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Number of pools: 1 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Using SSE2 crc32 instructions 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Completed initialization of buffer pool 2017-11-17 13:51:54 140294695642880 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Highest supported file format is Barracuda. 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: 128 out of 128 rollback segments are active. 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Creating shared tablespace for temporary tables 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2017-11-17 13:51:54 140295267206976 [Note] InnoDB: 5.7.20 started; log sequence number 1633766 2017-11-17 13:51:54 140294527887104 [Note] InnoDB: Loading buffer pool(s) from /home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/mysqld.1/data/ib_buffer_pool 2017-11-17 13:51:54 140294527887104 [Note] InnoDB: Buffer pool(s) load completed at 171117 13:51:54 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_LOCKS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMPMEM' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_TRX' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_TABLES' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_METRICS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'user_variables' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled. 2017-11-17 13:51:54 140295267206976 [Note] Plugin 'FEEDBACK' is disabled. 2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20' 2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-feedback-debug-first-interval=60' 2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-feedback-debug-interval=60' 2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown option '--loose-pam-debug' 2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown option '--loose-enable-rocksdb_lock_info' 2017-11-17 13:51:54 140295267206976 [Warning] /home/a/env1/m8-10.2/bld/sql/mysqld: unknown variable 'loose-debug-sync-timeout=300' 2017-11-17 13:51:54 140295267206976 [Note] Server socket created on IP: '127.0.0.1'. 2017-11-17 13:51:54 140295267206976 [Note] /home/a/env1/m8-10.2/bld/sql/mysqld: ready for connections. Version: '10.2.11-MariaDB-log' socket: '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/tmp/mysqld.1.sock' port: 16000 Source distribution 2017-11-17 13:51:55 140294768772864 [Note] RocksDB: Manual memtable flush. 171117 13:51:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.2.11-MariaDB-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=1 max_threads=153 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62996 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f98a40009a8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f99000b8e78 thread_stack 0x49000 /home/a/env1/m8-10.2/bld/sql/mysqld(my_print_stacktrace+0x29)[0x5627b085bc39] /home/a/env1/m8-10.2/bld/sql/mysqld(handle_fatal_signal+0x3bd)[0x5627b039aabd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7f99085ff670] /home/a/env1/m8-10.2/bld/sql/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x9f)[0x5627b045929f] /home/a/env1/m8-10.2/bld/sql/mysqld(binlog_background_thread+0x30b)[0x5627b045995b] /home/a/env1/m8-10.2/bld/sql/mysqld(+0xab32d4)[0x5627b08242d4] mysys/stacktrace.c:268(my_print_stacktrace)[0x7f99085f56da] sql/log.cc:9462(MYSQL_BIN_LOG::mark_xid_done(unsigned long, bool))[0x7f9907a87d7f]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 6 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on   The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file ----------SERVER LOG END-------------     - saving '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/log/rocksdb.binlog_rotate/' to '/home/a/env1/_depot/m-branch/m8-10.2-bld/mysql-test/var/log/rocksdb.binlog_rotate/' -------------------------------------------------------------------------- The servers were restarted 0 times Spent 0.000 of 8 seconds executing testcases   Failure: Failed 1/1 tests, 0.00% were successful.   Failing test(s): rocksdb.binlog_rotate   The log files in var/log may give you some hint of what went wrong.   If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html   mysql-test-run: *** ERROR: there were failing test cases
            Nefelim4ag Timofey added a comment -

            anikitin, JFYI
            yes, i also have innodb enabled, i'm just not report it, because it's not used in query %)

            Nefelim4ag Timofey added a comment - anikitin , JFYI yes, i also have innodb enabled, i'm just not report it, because it's not used in query %)

            10.3 is affected as well

            2017-11-20  9:44:29 0 [Note] /home/a/env1/m7-10.3/bld/sql//mysqld: ready for connections.
            Version: '10.3.3-MariaDB-log'  socket: '/home/a/env1/m7-10.3/dt/my.sock'  port: 3313  Source distribution
            171120  9:44:41 [ERROR] mysqld got signal 11 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.
             
            Server version: 10.3.3-MariaDB-log
            key_buffer_size=134217728
            read_buffer_size=131072
            max_used_connections=1
            max_threads=153
            thread_count=8
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467284 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7fec140009a8
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7fec9013ce98 thread_stack 0x49000
            /home/a/env1/m7-10.3/bld/sql//mysqld(my_print_stacktrace+0x29)[0x5638ea857009]
            /home/a/env1/m7-10.3/bld/sql//mysqld(handle_fatal_signal+0x3bf)[0x5638ea38f08f]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7fec948b7670]
            /home/a/env1/m7-10.3/bld/sql//mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x9f)[0x5638ea44ef5f]
            /home/a/env1/m7-10.3/bld/sql//mysqld(binlog_background_thread+0x323)[0x5638ea44f613]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7fec948ad6da]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7fec93d3fd7f]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x0):
            Connection ID (thread ID): 6
            Status: NOT_KILLED
            

            Cannot reproduce on current master https://github.com/facebook/mysql-5.6

            anikitin Andrii Nikitin (Inactive) added a comment - 10.3 is affected as well 2017-11-20 9:44:29 0 [Note] /home/a/env1/m7-10.3/bld/sql//mysqld: ready for connections. Version: '10.3.3-MariaDB-log' socket: '/home/a/env1/m7-10.3/dt/my.sock' port: 3313 Source distribution 171120 9:44:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.3.3-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=153 thread_count=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467284 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7fec140009a8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fec9013ce98 thread_stack 0x49000 /home/a/env1/m7-10.3/bld/sql//mysqld(my_print_stacktrace+0x29)[0x5638ea857009] /home/a/env1/m7-10.3/bld/sql//mysqld(handle_fatal_signal+0x3bf)[0x5638ea38f08f] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11670)[0x7fec948b7670] /home/a/env1/m7-10.3/bld/sql//mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0x9f)[0x5638ea44ef5f] /home/a/env1/m7-10.3/bld/sql//mysqld(binlog_background_thread+0x323)[0x5638ea44f613] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76da)[0x7fec948ad6da] /lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7fec93d3fd7f]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 6 Status: NOT_KILLED Cannot reproduce on current master https://github.com/facebook/mysql-5.6

            Reproducible on the latest MariaDB 10.2

              mysqld: /home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc:9580: void MYSQL_BIN_LOG::mark_xid_done(ulong, bool): Assertion `b->xid_count >= 0' failed.
            

            (gdb) wher
              #0  0x00007ffff5931267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
              #1  0x00007ffff5932eca in __GI_abort () at abort.c:89
              #2  0x00007ffff592a03d in __assert_fail_base (fmt=0x7ffff5a8c028 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555556694ad0 "b->xid_count >= 0", file=file@entry=0x555556691078 "/home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc", line=line@entry=9580, function=function@entry=0x555556696f80 <MYSQL_BIN_LOG::mark_xid_done(unsigned long, bool)::__PRETTY_FUNCTION__> "void MYSQL_BIN_LOG::mark_xid_done(ulong, bool)") at assert.c:92
              #3  0x00007ffff592a0f2 in __GI___assert_fail (assertion=0x555556694ad0 "b->xid_count >= 0", file=0x555556691078 "/home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc", line=9580, function=0x555556696f80 <MYSQL_BIN_LOG::mark_xid_done(unsigned long, bool)::__PRETTY_FUNCTION__> "void MYSQL_BIN_LOG::mark_xid_done(ulong, bool)") at assert.c:101
              #4  0x0000555555e85681 in MYSQL_BIN_LOG::mark_xid_done (this=0x5555570e4480 <mysql_bin_log>, binlog_id=2, write_checkpoint=true) at /home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc:9580
              #5  0x0000555555e85f1d in binlog_background_thread (arg=0x0) at /home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc:9784
            

            psergei Sergei Petrunia added a comment - Reproducible on the latest MariaDB 10.2 mysqld: /home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc:9580: void MYSQL_BIN_LOG::mark_xid_done(ulong, bool): Assertion `b->xid_count >= 0' failed. (gdb) wher #0 0x00007ffff5931267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007ffff5932eca in __GI_abort () at abort.c:89 #2 0x00007ffff592a03d in __assert_fail_base (fmt=0x7ffff5a8c028 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555556694ad0 "b->xid_count >= 0", file=file@entry=0x555556691078 "/home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc", line=line@entry=9580, function=function@entry=0x555556696f80 <MYSQL_BIN_LOG::mark_xid_done(unsigned long, bool)::__PRETTY_FUNCTION__> "void MYSQL_BIN_LOG::mark_xid_done(ulong, bool)") at assert.c:92 #3 0x00007ffff592a0f2 in __GI___assert_fail (assertion=0x555556694ad0 "b->xid_count >= 0", file=0x555556691078 "/home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc", line=9580, function=0x555556696f80 <MYSQL_BIN_LOG::mark_xid_done(unsigned long, bool)::__PRETTY_FUNCTION__> "void MYSQL_BIN_LOG::mark_xid_done(ulong, bool)") at assert.c:101 #4 0x0000555555e85681 in MYSQL_BIN_LOG::mark_xid_done (this=0x5555570e4480 <mysql_bin_log>, binlog_id=2, write_checkpoint=true) at /home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc:9580 #5 0x0000555555e85f1d in binlog_background_thread (arg=0x0) at /home/psergey/dev-git/10.2-mariarocks-r4/sql/log.cc:9784
            • MyRocks' implementation of rocksdb_checkpoint_request looks trivial.
            • Putting a breakpoint there makes the crash go away
            psergei Sergei Petrunia added a comment - MyRocks' implementation of rocksdb_checkpoint_request looks trivial. Putting a breakpoint there makes the crash go away

            commit 77872e4519a9ebba2cd2ab4f5794faccf0a1f879
            Author: Elena Stepanova <elenst@montyprogram.com>
            Date:   Mon Nov 27 12:04:51 2017 +0200
             
                Additions to the list of unstable tests for 10.2.11
            

            Steps to reproduce:

            [mysqld]
            binlog-format=row
            log_bin=pslp
            sync_binlog=1
            server-id=12
            

            after server start:

            set global max_binlog_size=1024000;
            CREATE TABLE `tslap2` (
              `pk` int(11) NOT NULL AUTO_INCREMENT,
              `a` varchar(255) DEFAULT NULL,
              `b` varchar(255) DEFAULT NULL,
              `c` varchar(255) DEFAULT NULL,
              `d` varchar(255) DEFAULT NULL,
              PRIMARY KEY (`pk`)
            ) ENGINE=ROCKSDB AUTO_INCREMENT=28175 DEFAULT CHARSET=latin1;
            

            file1.pl:

            #!/usr/bin/perl
             
            for ($i =0; $i < 1000*1000; $i++) { 
              print "insert into tslap2 (a,b,c,d) values (repeat(uuid(),7),repeat(uuid(),7),repeat(uuid(),7),repeat(uuid(),7));\n";
            }
            

            Then

            perl file1.pl | mysql -uroot -A --socket=/tmp/mysql4.sock test
            

            psergei Sergei Petrunia added a comment - commit 77872e4519a9ebba2cd2ab4f5794faccf0a1f879 Author: Elena Stepanova <elenst@montyprogram.com> Date: Mon Nov 27 12:04:51 2017 +0200   Additions to the list of unstable tests for 10.2.11 Steps to reproduce: [mysqld] binlog-format=row log_bin=pslp sync_binlog=1 server-id=12 after server start: set global max_binlog_size=1024000; CREATE TABLE `tslap2` ( `pk` int(11) NOT NULL AUTO_INCREMENT, `a` varchar(255) DEFAULT NULL, `b` varchar(255) DEFAULT NULL, `c` varchar(255) DEFAULT NULL, `d` varchar(255) DEFAULT NULL, PRIMARY KEY (`pk`) ) ENGINE=ROCKSDB AUTO_INCREMENT=28175 DEFAULT CHARSET=latin1; file1.pl: #!/usr/bin/perl   for ($i =0; $i < 1000*1000; $i++) { print "insert into tslap2 (a,b,c,d) values (repeat(uuid(),7),repeat(uuid(),7),repeat(uuid(),7),repeat(uuid(),7));\n"; } Then perl file1.pl | mysql -uroot -A --socket=/tmp/mysql4.sock test

            Reproducible on 1d526f31fbb417ecb3e17b1c965050ae18fe3997, too

            psergei Sergei Petrunia added a comment - Reproducible on 1d526f31fbb417ecb3e17b1c965050ae18fe3997, too

            On

            commit 2c1e4d4d7a174c180cfcac5e245840aec8458b77
            Author: Sergei Petrunia <psergey@askmonty.org>
            Date:   Tue Dec 5 16:33:38 2017 +0300
             
                MDEV-14563: Wrong query plan for query with no PK
            

            The failure is reproducible on the tree. But even a patch like this:

            diff --git a/sql/log.cc b/sql/log.cc
            index 93d7a00..52e24b2 100644
            --- a/sql/log.cc
            +++ b/sql/log.cc
            @@ -9535,6 +9535,8 @@ TC_LOG_BINLOG::mark_xids_active(ulong binlog_id, uint xid_count)
               {
                 if (b->binlog_id == binlog_id)
                 {
            +      fprintf(stderr,"TC_LOG_BINLOG::mark_xids_active binlog_id=%ld,  %ld + %ld\n",
            +              binlog_id, b->xid_count, xid_count);
                   b->xid_count += xid_count;
                   break;
                 }
            @@ -9575,7 +9577,9 @@ TC_LOG_BINLOG::mark_xid_done(ulong binlog_id, bool write_checkpoint)
               {
                 if (b->binlog_id == binlog_id)
                 {
            +      //fprintf(stderr,"TC_LOG_BINLOG::mark_xids_active %d %d\n", binlog_id, xid_count);
                   --b->xid_count;
            +      fprintf(stderr, "TC_LOG_BINLOG::mark_xid_done (x-1= %ld)\n", b->xid_count);
             
                   DBUG_ASSERT(b->xid_count >= 0); // catch unmatched (++) decrement
             
            
            

            makes it go away. It seems to be a race condition, and I still have no idea about its cause

            psergei Sergei Petrunia added a comment - On commit 2c1e4d4d7a174c180cfcac5e245840aec8458b77 Author: Sergei Petrunia <psergey@askmonty.org> Date: Tue Dec 5 16:33:38 2017 +0300   MDEV-14563: Wrong query plan for query with no PK The failure is reproducible on the tree. But even a patch like this: diff --git a/sql/log.cc b/sql/log.cc index 93d7a00..52e24b2 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -9535,6 +9535,8 @@ TC_LOG_BINLOG::mark_xids_active(ulong binlog_id, uint xid_count) { if (b->binlog_id == binlog_id) { + fprintf(stderr,"TC_LOG_BINLOG::mark_xids_active binlog_id=%ld, %ld + %ld\n", + binlog_id, b->xid_count, xid_count); b->xid_count += xid_count; break; } @@ -9575,7 +9577,9 @@ TC_LOG_BINLOG::mark_xid_done(ulong binlog_id, bool write_checkpoint) { if (b->binlog_id == binlog_id) { + //fprintf(stderr,"TC_LOG_BINLOG::mark_xids_active %d %d\n", binlog_id, xid_count); --b->xid_count; + fprintf(stderr, "TC_LOG_BINLOG::mark_xid_done (x-1= %ld)\n", b->xid_count); DBUG_ASSERT(b->xid_count >= 0); // catch unmatched (++) decrement makes it go away. It seems to be a race condition, and I still have no idea about its cause

            Didn't find any issues in Andrei's patch. Pushed it to 10.2

            psergei Sergei Petrunia added a comment - Didn't find any issues in Andrei's patch. Pushed it to 10.2

            People

              psergei Sergei Petrunia
              Nefelim4ag Timofey
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.