Details

    • 10.1.9-2

    Description

      We are migrating a bunch of databases from Percona 5.5.33 to MariaDB 10.1.8. Unfortunately, a lot of them crash during startup.

      We do hot backups of the databases with LVM snapshots (flush tables with read lock, make snapshot, tar -cf the snapshot). We move the backup to a different server and extract it.

      MariaDB 5.5.46 works fine with the backup
      MariaDB 10.0.21 works fine with the backup
      MariaDB 10.1.8 crashes

      I've tried going through Percona 5.5 -> MariaDB 5.5.46 -> MariaDB 10.0.21 -> MariaDB 10.1.8 with full mysql_upgrade's and clean shutdowns, but it would still crash on startup with 10.1.8.

      Looks similar to https://mariadb.atlassian.net/browse/MDEV-8672

      Here are the logs from MariaDB:

      2015-10-28 12:09:23 139878086616960 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 37700 ...
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: The InnoDB memory heap is disabled
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Memory barrier is not used
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Compressed tables use zlib 1.2.3
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Using Linux native AIO
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Using CPU crc32 instructions
      2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Initializing buffer pool, size = 10.0G
      2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Completed initialization of buffer pool
      2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Highest supported file format is Barracuda.
      2015-10-28 12:09:27 139878086616960 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164830 in the ib_logfiles!
      2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Database was not shutdown normally!
      2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Starting crash recovery.
      2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Reading tablespace information from the .ibd files...
      2015-10-28 12:09:30 139878086616960 [Note] InnoDB: Restoring possible half-written data pages
      2015-10-28 12:09:30 139878086616960 [Note] InnoDB: from the doublewrite buffer...
      InnoDB: In a MySQL replication slave the last master binlog file
      InnoDB: position 0 1716444, file name mysql-bin.020607
      2015-10-28 12:09:30 139878086616960 [Note] InnoDB: 128 rollback segment(s) are active.
      2015-10-28 12:09:30 139878086616960 [Note] InnoDB: Waiting for purge to start
      2015-10-28 12:09:30 7f348a3fc700  InnoDB: Assertion failure in thread 139863634462464 in file btr0cur.cc line 5341
      InnoDB: Failing assertion: rb_ctx != RB_NONE
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      151028 12:09:30 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.1.8-MariaDB-log
      key_buffer_size=2147483648
      read_buffer_size=4194304  
      max_used_connections=0
      max_threads=502
      thread_count=0
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x40000
       
      015-10-28 12:09:30 139878086616960 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164830
      2015-10-28 12:09:30 139863609284352 [Note] InnoDB: Dumping buffer pool(s) not yet started
      2015-10-28 12:09:30 139878086616960 [Note] Plugin 'FEEDBACK' is disabled.
      2015-10-28 12:09:30 139878086616960 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
      2015-10-28 12:09:30 139878086616960 [Note] Starting crash recovery...
      2015-10-28 12:09:30 139878086616960 [Note] Crash recovery finished.
      2015-10-28 12:09:30 139878086117120 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
      2015-10-28 12:09:30 139878086616960 [Note] Server socket created on IP: '::'.
      2015-10-28 12:09:30 139878086616960 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
      2015-10-28 12:09:30 139878086616960 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
      2015-10-28 12:09:30 139878086616960 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
      2015-10-28 12:09:30 139878085584640 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
      2015-10-28 12:09:30 139878086616960 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
      2015-10-28 12:09:30 139878086616960 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
      Version: '10.1.8-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  MariaDB Server
      /usr/local/mariadb10.1/bin/mysqld(my_print_stacktrace+0x2e)[0xbf55be]
      /usr/local/mariadb10.1/bin/mysqld(handle_fatal_signal+0x464)[0x75e6c4]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f37e76868d0]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f37e627a107]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f37e627b4e8]
      /usr/local/mariadb10.1/bin/mysqld[0xa0e9d3]
      /usr/local/mariadb10.1/bin/mysqld[0x9ae60c]
      /usr/local/mariadb10.1/bin/mysqld[0x979bd4]
      /usr/local/mariadb10.1/bin/mysqld[0x9dba74]
      /usr/local/mariadb10.1/bin/mysqld[0x9c4507]
      /usr/local/mariadb10.1/bin/mysqld[0x9ca5ca]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f37e767f0a4]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f37e632b04d]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      151028 12:09:31 mysqld_safe Number of processes running now: 0
      151028 12:09:31 mysqld_safe mysqld restarted

      Attachments

        Issue Links

          Activity

            Do you know what is the value of rb_ctx in this case ?

            jplindst Jan Lindström (Inactive) added a comment - Do you know what is the value of rb_ctx in this case ?

            Hi,

            No, i don't. But i can compile mysqld with debuging to see if i can get one.

            sysmonk aleksandr stankevic added a comment - Hi, No, i don't. But i can compile mysqld with debuging to see if i can get one.

            MariaDB 10.1.8 compiled with debug gives more info:

            151028 13:53:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
            2015-10-28 13:53:55 140640495765312 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
            2015-10-28 13:53:55 140640495765312 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-debug-log) starting as process 115715 ...
            2015-10-28 13:53:56 140640495765312 [ERROR] mysqld: file-key-management-filename is not set
            2015-10-28 13:53:56 140640495765312 [ERROR] Plugin 'file_key_management' init function returned error.
            2015-10-28 13:53:56 140640495765312 [ERROR] Plugin 'file_key_management' registration as a ENCRYPTION failed.
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Using mutexes to ref count buffer pool pages
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: The InnoDB memory heap is disabled
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Memory barrier is not used
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Compressed tables use zlib 1.2.8
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Using CPU crc32 instructions
            2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Initializing buffer pool, size = 10.0G
            2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Completed initialization of buffer pool
            2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Highest supported file format is Barracuda.
            2015-10-28 13:54:00 140640495765312 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164850 in the ib_logfiles!
            2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Database was not shutdown normally!
            2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Starting crash recovery.
            2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Reading tablespace information from the .ibd files...
            2015-10-28 13:54:20 140640495765312 [Note] InnoDB: Restoring possible half-written data pages 
            2015-10-28 13:54:20 140640495765312 [Note] InnoDB: from the doublewrite buffer...
            InnoDB: In a MySQL replication slave the last master binlog file
            InnoDB: position 0 1716444, file name mysql-bin.020607
            2015-10-28 13:54:22 140640495765312 [Note] InnoDB: 128 rollback segment(s) are active.
            2015-10-28 13:54:22 140640495765312 [Note] InnoDB: Waiting for purge to start
            2015-10-28 13:54:22 140640495765312 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164850
            2015-10-28 13:54:22 140626372826880 [Note] InnoDB: Dumping buffer pool(s) not yet started
            2015-10-28 13:54:22 140640495765312 [Note] CONNECT: Version 1.03.0007 July 05, 2015
            2015-10-28 13:54:22 7fe6227fc700  InnoDB: Assertion failure in thread 140626398004992 in file btr0cur.cc line 5332
            InnoDB: Failing assertion: mtr_memo_contains_page(local_mtr, field_ref, 2)
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            151028 13:54:22 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see http://kb.askmonty.org/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.8-MariaDB-debug-log
            key_buffer_size=2147483648
            read_buffer_size=4194304
            max_used_connections=0
            max_threads=502
            thread_count=0
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6220045 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x40000
            2015-10-28 13:54:22 140640495765312 [Note] Plugin 'FEEDBACK' is disabled.
            2015-10-28 13:54:22 140640495765312 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
            2015-10-28 13:54:22 140640495765312 [Note] Starting crash recovery...
            2015-10-28 13:54:22 140640495765312 [Note] Crash recovery finished.
            2015-10-28 13:54:22 140640252671744 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            2015-10-28 13:54:22 140640495765312 [Note] Server socket created on IP: '::'.
            2015-10-28 13:54:22 140640495765312 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
            2015-10-28 13:54:23 140640495765312 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
            2015-10-28 13:54:23 140640495765312 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
            2015-10-28 13:54:23 140640252139264 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            2015-10-28 13:54:23 140640495765312 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
            2015-10-28 13:54:23 140640495765312 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
            Version: '10.1.8-MariaDB-debug-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  Source distribution
            mysys/stacktrace.c:246(my_print_stacktrace)[0xf7711f]
            sql/signal_handler.cc:162(handle_fatal_signal)[0x88339b]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fe96a88c8d0]
            /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fe969691107]
            /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9696924e8]
            btr/btr0cur.cc:5333(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0xcbd1fe]
            row/row0purge.cc:711(row_purge_upd_exist_or_extern_func)[0xc18134]
            row/row0purge.cc:869(row_purge_record_func)[0xc1859d]
            row/row0purge.cc:912(row_purge)[0xc186e8]
            row/row0purge.cc:994(row_purge_step(que_thr_t*))[0xc18990]
            que/que0que.cc:1089(que_thr_step)[0xbb93e4]
            que/que0que.cc:1151(que_run_threads_low)[0xbb95e8]
            que/que0que.cc:1194(que_run_threads(que_thr_t*))[0xbb9774]
            trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0xc67535]
            srv/srv0srv.cc:3432(srv_do_purge)[0xc4ad31]
            srv/srv0srv.cc:3618(srv_purge_coordinator_thread)[0xc4b445]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fe96a8850a4]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe96974204d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            151028 13:54:23 mysqld_safe Number of processes running now: 0
            151028 13:54:23 mysqld_safe mysqld restarted

            sysmonk aleksandr stankevic added a comment - MariaDB 10.1.8 compiled with debug gives more info: 151028 13:53:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3 2015-10-28 13:53:55 140640495765312 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 2015-10-28 13:53:55 140640495765312 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-debug-log) starting as process 115715 ... 2015-10-28 13:53:56 140640495765312 [ERROR] mysqld: file-key-management-filename is not set 2015-10-28 13:53:56 140640495765312 [ERROR] Plugin 'file_key_management' init function returned error. 2015-10-28 13:53:56 140640495765312 [ERROR] Plugin 'file_key_management' registration as a ENCRYPTION failed. 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!! 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: The InnoDB memory heap is disabled 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Memory barrier is not used 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Compressed tables use zlib 1.2.8 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Using CPU crc32 instructions 2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Initializing buffer pool, size = 10.0G 2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Completed initialization of buffer pool 2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Highest supported file format is Barracuda. 2015-10-28 13:54:00 140640495765312 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164850 in the ib_logfiles! 2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Database was not shutdown normally! 2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Starting crash recovery. 2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Reading tablespace information from the .ibd files... 2015-10-28 13:54:20 140640495765312 [Note] InnoDB: Restoring possible half-written data pages 2015-10-28 13:54:20 140640495765312 [Note] InnoDB: from the doublewrite buffer... InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 1716444, file name mysql-bin.020607 2015-10-28 13:54:22 140640495765312 [Note] InnoDB: 128 rollback segment(s) are active. 2015-10-28 13:54:22 140640495765312 [Note] InnoDB: Waiting for purge to start 2015-10-28 13:54:22 140640495765312 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164850 2015-10-28 13:54:22 140626372826880 [Note] InnoDB: Dumping buffer pool(s) not yet started 2015-10-28 13:54:22 140640495765312 [Note] CONNECT: Version 1.03.0007 July 05, 2015 2015-10-28 13:54:22 7fe6227fc700 InnoDB: Assertion failure in thread 140626398004992 in file btr0cur.cc line 5332 InnoDB: Failing assertion: mtr_memo_contains_page(local_mtr, field_ref, 2) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 151028 13:54:22 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.8-MariaDB-debug-log key_buffer_size=2147483648 read_buffer_size=4194304 max_used_connections=0 max_threads=502 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6220045 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x40000 2015-10-28 13:54:22 140640495765312 [Note] Plugin 'FEEDBACK' is disabled. 2015-10-28 13:54:22 140640495765312 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin 2015-10-28 13:54:22 140640495765312 [Note] Starting crash recovery... 2015-10-28 13:54:22 140640495765312 [Note] Crash recovery finished. 2015-10-28 13:54:22 140640252671744 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2015-10-28 13:54:22 140640495765312 [Note] Server socket created on IP: '::'. 2015-10-28 13:54:22 140640495765312 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it 2015-10-28 13:54:23 140640495765312 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A 2015-10-28 13:54:23 140640495765312 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. 2015-10-28 13:54:23 140640252139264 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2015-10-28 13:54:23 140640495765312 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem. 2015-10-28 13:54:23 140640495765312 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections. Version: '10.1.8-MariaDB-debug-log' socket: '/var/run/mysqld/mysqld44-3.sock' port: 3740 Source distribution mysys/stacktrace.c:246(my_print_stacktrace)[0xf7711f] sql/signal_handler.cc:162(handle_fatal_signal)[0x88339b] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fe96a88c8d0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fe969691107] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9696924e8] btr/btr0cur.cc:5333(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0xcbd1fe] row/row0purge.cc:711(row_purge_upd_exist_or_extern_func)[0xc18134] row/row0purge.cc:869(row_purge_record_func)[0xc1859d] row/row0purge.cc:912(row_purge)[0xc186e8] row/row0purge.cc:994(row_purge_step(que_thr_t*))[0xc18990] que/que0que.cc:1089(que_thr_step)[0xbb93e4] que/que0que.cc:1151(que_run_threads_low)[0xbb95e8] que/que0que.cc:1194(que_run_threads(que_thr_t*))[0xbb9774] trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0xc67535] srv/srv0srv.cc:3432(srv_do_purge)[0xc4ad31] srv/srv0srv.cc:3618(srv_purge_coordinator_thread)[0xc4b445] /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fe96a8850a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe96974204d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 151028 13:54:23 mysqld_safe Number of processes running now: 0 151028 13:54:23 mysqld_safe mysqld restarted

            This is not the same assertion, do you have db that you could share ?

            jplindst Jan Lindström (Inactive) added a comment - This is not the same assertion, do you have db that you could share ?

            The DB is around 150gb and has private customer data so probably can't share it.
            If you give me any steps to help debug this i will do my best to help debug this issue.

            sysmonk aleksandr stankevic added a comment - The DB is around 150gb and has private customer data so probably can't share it. If you give me any steps to help debug this i will do my best to help debug this issue.

            Tried migrate from Percona 5.5 -> MariaDB 5.5.46 -> MariaDB 10.0.21 -> MariaDB 10.1.8 and everything went smoothly, so I need more information. Can you try version from git repository so that you have commit 0fe5eb5cae7cf7da90c8f46997bb331ce2253ac6 compile using normal setup (no debug) and provide me a full unedited error log (do not delete the old one).

            jplindst Jan Lindström (Inactive) added a comment - Tried migrate from Percona 5.5 -> MariaDB 5.5.46 -> MariaDB 10.0.21 -> MariaDB 10.1.8 and everything went smoothly, so I need more information. Can you try version from git repository so that you have commit 0fe5eb5cae7cf7da90c8f46997bb331ce2253ac6 compile using normal setup (no debug) and provide me a full unedited error log (do not delete the old one).

            The error isn't reproducible on every installl. Out of ~10 db's that we tried to migrate, 4 have this error. Others don't.

            If it makes any difference - most of these databases have been started as 4.1 and have been migrated up to 5.5 in years.

            I've already had the error log truncated, so i compiled the git version and started with the mariadb 10.1.8 from package, then started with mariadb10.1.8 from git. Here are the results:

            151029 08:18:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
            2015-10-29  8:18:54 140291706009472 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
            2015-10-29  8:18:54 140291706009472 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 92493 ...
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Using mutexes to ref count buffer pool pages
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: The InnoDB memory heap is disabled
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Memory barrier is not used
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Compressed tables use zlib 1.2.3
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Using Linux native AIO
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Using CPU crc32 instructions
            2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Initializing buffer pool, size = 10.0G
            2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Completed initialization of buffer pool
            2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Highest supported file format is Barracuda.
            2015-10-29  8:18:57 140291706009472 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164950 in the ib_logfiles!
            2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Database was not shutdown normally!
            2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Starting crash recovery.
            2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Reading tablespace information from the .ibd files...
            2015-10-29  8:19:17 140291706009472 [Note] InnoDB: Restoring possible half-written data pages 
            2015-10-29  8:19:17 140291706009472 [Note] InnoDB: from the doublewrite buffer...
            InnoDB: In a MySQL replication slave the last master binlog file
            InnoDB: position 0 1716444, file name mysql-bin.020607
            2015-10-29  8:19:18 140291706009472 [Note] InnoDB: 128 rollback segment(s) are active.
            2015-10-29  8:19:18 140291706009472 [Note] InnoDB: Waiting for purge to start
            2015-10-29 08:19:18 7f94d7ffc700  InnoDB: Assertion failure in thread 140277255751424 in file btr0cur.cc line 5341
            InnoDB: Failing assertion: rb_ctx != RB_NONE
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            151029  8:19:18 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see http://kb.askmonty.org/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.8-MariaDB-log
            key_buffer_size=2147483648
            read_buffer_size=4194304
            max_used_connections=0
            max_threads=502
            thread_count=0
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x40000
            2015-10-29  8:19:18 140291706009472 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164950
            2015-10-29  8:19:18 140277230573312 [Note] InnoDB: Dumping buffer pool(s) not yet started
            2015-10-29  8:19:18 140291706009472 [Note] Plugin 'FEEDBACK' is disabled.
            2015-10-29  8:19:18 140291706009472 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
            2015-10-29  8:19:18 140291706009472 [Note] Starting crash recovery...
            2015-10-29  8:19:18 140291706009472 [Note] Crash recovery finished.
            2015-10-29  8:19:18 140291705509632 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            2015-10-29  8:19:18 140291706009472 [Note] Server socket created on IP: '::'.
            2015-10-29  8:19:18 140291706009472 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
            2015-10-29  8:19:18 140291706009472 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
            2015-10-29  8:19:18 140291706009472 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
            2015-10-29  8:19:18 140291704977152 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            2015-10-29  8:19:18 140291706009472 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
            2015-10-29  8:19:18 140291706009472 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
            Version: '10.1.8-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  MariaDB Server
            mysys/stacktrace.c:247(my_print_stacktrace)[0xbf55be]
            sql/signal_handler.cc:160(handle_fatal_signal)[0x75e6c4]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f98350b88d0]
            /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f9833cac107]
            /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f9833cad4e8]
            btr/btr0cur.cc:5470(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0xa0e9d3]
            row/row0purge.cc:712(row_purge_upd_exist_or_extern_func)[0x9ae60c]
            que/que0que.cc:1089(que_thr_step)[0x979bd4]
            trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0x9dba74]
            srv/srv0srv.cc:3434(srv_do_purge)[0x9c4507]
            srv/srv0srv.cc:3618(srv_purge_coordinator_thread)[0x9ca5ca]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f98350b10a4]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f9833d5d04d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Writing a core file
            151029 08:21:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
            2015-10-29  8:21:40 140708422530880 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
            2015-10-29  8:21:40 140708422530880 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 94154 ...
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Using mutexes to ref count buffer pool pages
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: The InnoDB memory heap is disabled
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Memory barrier is not used
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Compressed tables use zlib 1.2.3
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Using CPU crc32 instructions
            2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Initializing buffer pool, size = 10.0G
            2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Completed initialization of buffer pool
            2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Highest supported file format is Barracuda.
            2015-10-29  8:21:44 140708422530880 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164960 in the ib_logfiles!
            2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Database was not shutdown normally!
            2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Starting crash recovery.
            2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Reading tablespace information from the .ibd files...
            2015-10-29  8:22:03 140708422530880 [Note] InnoDB: Restoring possible half-written data pages 
            2015-10-29  8:22:03 140708422530880 [Note] InnoDB: from the doublewrite buffer...
            InnoDB: In a MySQL replication slave the last master binlog file
            InnoDB: position 0 1716444, file name mysql-bin.020607
            2015-10-29  8:22:04 140708422530880 [Note] InnoDB: 128 rollback segment(s) are active.
            2015-10-29  8:22:04 140708422530880 [Note] InnoDB: Waiting for purge to start
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
            InnoDB: but the tablespace does not exist or is just being dropped.
            InnoDB: Error: Unable to read tablespace 909587505 page no 959523894 into the buffer pool after 100 attempts
            InnoDB: The most probable cause of this error may be that the table has been corrupted.
            InnoDB: You can try to fix this problem by using innodb_force_recovery.
            InnoDB: Please see reference manual for more details.
            InnoDB: Aborting...
            2015-10-29 08:22:04 7ff6048b0700  InnoDB: Assertion failure in thread 140694614902528 in file buf0buf.cc line 2962
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            151029  8:22:04 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see http://kb.askmonty.org/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.8-MariaDB-log
            key_buffer_size=2147483648
            read_buffer_size=4194304
            max_used_connections=0
            max_threads=502
            thread_count=0
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x40000
            2015-10-29  8:22:04 140708422530880 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164960
            2015-10-29  8:22:04 140694589724416 [Note] InnoDB: Dumping buffer pool(s) not yet started
            2015-10-29  8:22:04 140708422530880 [Note] Plugin 'FEEDBACK' is disabled.
            2015-10-29  8:22:04 140708422530880 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
            2015-10-29  8:22:04 140708422530880 [Note] Starting crash recovery...
            2015-10-29  8:22:04 140708422530880 [Note] Crash recovery finished.
            2015-10-29  8:22:04 140694542206720 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            2015-10-29  8:22:04 140708422530880 [Note] Server socket created on IP: '::'.
            2015-10-29  8:22:04 140708422530880 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
            2015-10-29  8:22:04 140708422530880 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
            2015-10-29  8:22:04 140708422530880 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
            2015-10-29  8:22:04 140694541674240 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            2015-10-29  8:22:04 140708422530880 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
            /usr/local/mariadb10.1/bin/mysqld(my_print_stacktrace+0x2e)[0x55e31936cdbe]
            /usr/local/mariadb10.1/bin/mysqld(handle_fatal_signal+0x34d)[0x55e318f2ccfd]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7ff93b48f8d0]
            /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7ff93a4af107]
            /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7ff93a4b04e8]
            /usr/local/mariadb10.1/bin/mysqld(+0x7f9ef7)[0x55e3191acef7]
            /usr/local/mariadb10.1/bin/mysqld(+0x7e1777)[0x55e319194777]
            /usr/local/mariadb10.1/bin/mysqld(+0x77b5c3)[0x55e31912e5c3]
            2015-10-29  8:22:04 140708422530880 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
            Version: '10.1.8-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  Source distribution
            /usr/local/mariadb10.1/bin/mysqld(+0x77c1d4)[0x55e31912f1d4]
            /usr/local/mariadb10.1/bin/mysqld(+0x7489b8)[0x55e3190fb9b8]
            /usr/local/mariadb10.1/bin/mysqld(+0x7a61b6)[0x55e3191591b6]
            /usr/local/mariadb10.1/bin/mysqld(+0x7968f9)[0x55e3191498f9]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7ff93b4880a4]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff93a56004d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Writing a core file

            Looks like a different error. Due to the errors being so different from the initial binary i've used, i also tried compiling MariaDB from the sources available at downloads/mariadb-10.1.8.tar.gz - but i get the same "InnoDB: Failing assertion: rb_ctx != RB_NONE" error out there.

            sysmonk aleksandr stankevic added a comment - The error isn't reproducible on every installl. Out of ~10 db's that we tried to migrate, 4 have this error. Others don't. If it makes any difference - most of these databases have been started as 4.1 and have been migrated up to 5.5 in years. I've already had the error log truncated, so i compiled the git version and started with the mariadb 10.1.8 from package, then started with mariadb10.1.8 from git. Here are the results: 151029 08:18:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3 2015-10-29 8:18:54 140291706009472 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 2015-10-29 8:18:54 140291706009472 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 92493 ... 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: The InnoDB memory heap is disabled 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Memory barrier is not used 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Compressed tables use zlib 1.2.3 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Using Linux native AIO 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Using CPU crc32 instructions 2015-10-29 8:18:54 140291706009472 [Note] InnoDB: Initializing buffer pool, size = 10.0G 2015-10-29 8:18:57 140291706009472 [Note] InnoDB: Completed initialization of buffer pool 2015-10-29 8:18:57 140291706009472 [Note] InnoDB: Highest supported file format is Barracuda. 2015-10-29 8:18:57 140291706009472 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164950 in the ib_logfiles! 2015-10-29 8:18:57 140291706009472 [Note] InnoDB: Database was not shutdown normally! 2015-10-29 8:18:57 140291706009472 [Note] InnoDB: Starting crash recovery. 2015-10-29 8:18:57 140291706009472 [Note] InnoDB: Reading tablespace information from the .ibd files... 2015-10-29 8:19:17 140291706009472 [Note] InnoDB: Restoring possible half-written data pages 2015-10-29 8:19:17 140291706009472 [Note] InnoDB: from the doublewrite buffer... InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 1716444, file name mysql-bin.020607 2015-10-29 8:19:18 140291706009472 [Note] InnoDB: 128 rollback segment(s) are active. 2015-10-29 8:19:18 140291706009472 [Note] InnoDB: Waiting for purge to start 2015-10-29 08:19:18 7f94d7ffc700 InnoDB: Assertion failure in thread 140277255751424 in file btr0cur.cc line 5341 InnoDB: Failing assertion: rb_ctx != RB_NONE InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 151029 8:19:18 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.8-MariaDB-log key_buffer_size=2147483648 read_buffer_size=4194304 max_used_connections=0 max_threads=502 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x40000 2015-10-29 8:19:18 140291706009472 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164950 2015-10-29 8:19:18 140277230573312 [Note] InnoDB: Dumping buffer pool(s) not yet started 2015-10-29 8:19:18 140291706009472 [Note] Plugin 'FEEDBACK' is disabled. 2015-10-29 8:19:18 140291706009472 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin 2015-10-29 8:19:18 140291706009472 [Note] Starting crash recovery... 2015-10-29 8:19:18 140291706009472 [Note] Crash recovery finished. 2015-10-29 8:19:18 140291705509632 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2015-10-29 8:19:18 140291706009472 [Note] Server socket created on IP: '::'. 2015-10-29 8:19:18 140291706009472 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it 2015-10-29 8:19:18 140291706009472 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A 2015-10-29 8:19:18 140291706009472 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. 2015-10-29 8:19:18 140291704977152 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2015-10-29 8:19:18 140291706009472 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem. 2015-10-29 8:19:18 140291706009472 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections. Version: '10.1.8-MariaDB-log' socket: '/var/run/mysqld/mysqld44-3.sock' port: 3740 MariaDB Server mysys/stacktrace.c:247(my_print_stacktrace)[0xbf55be] sql/signal_handler.cc:160(handle_fatal_signal)[0x75e6c4] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f98350b88d0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f9833cac107] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f9833cad4e8] btr/btr0cur.cc:5470(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0xa0e9d3] row/row0purge.cc:712(row_purge_upd_exist_or_extern_func)[0x9ae60c] que/que0que.cc:1089(que_thr_step)[0x979bd4] trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0x9dba74] srv/srv0srv.cc:3434(srv_do_purge)[0x9c4507] srv/srv0srv.cc:3618(srv_purge_coordinator_thread)[0x9ca5ca] /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f98350b10a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f9833d5d04d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file 151029 08:21:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3 2015-10-29 8:21:40 140708422530880 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 2015-10-29 8:21:40 140708422530880 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 94154 ... 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: The InnoDB memory heap is disabled 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: Memory barrier is not used 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: Compressed tables use zlib 1.2.3 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: Using CPU crc32 instructions 2015-10-29 8:21:40 140708422530880 [Note] InnoDB: Initializing buffer pool, size = 10.0G 2015-10-29 8:21:44 140708422530880 [Note] InnoDB: Completed initialization of buffer pool 2015-10-29 8:21:44 140708422530880 [Note] InnoDB: Highest supported file format is Barracuda. 2015-10-29 8:21:44 140708422530880 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164960 in the ib_logfiles! 2015-10-29 8:21:44 140708422530880 [Note] InnoDB: Database was not shutdown normally! 2015-10-29 8:21:44 140708422530880 [Note] InnoDB: Starting crash recovery. 2015-10-29 8:21:44 140708422530880 [Note] InnoDB: Reading tablespace information from the .ibd files... 2015-10-29 8:22:03 140708422530880 [Note] InnoDB: Restoring possible half-written data pages 2015-10-29 8:22:03 140708422530880 [Note] InnoDB: from the doublewrite buffer... InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 1716444, file name mysql-bin.020607 2015-10-29 8:22:04 140708422530880 [Note] InnoDB: 128 rollback segment(s) are active. 2015-10-29 8:22:04 140708422530880 [Note] InnoDB: Waiting for purge to start 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Error: trying to access tablespace 909587505 page no. 959523894, InnoDB: but the tablespace does not exist or is just being dropped. InnoDB: Error: Unable to read tablespace 909587505 page no 959523894 into the buffer pool after 100 attempts InnoDB: The most probable cause of this error may be that the table has been corrupted. InnoDB: You can try to fix this problem by using innodb_force_recovery. InnoDB: Please see reference manual for more details. InnoDB: Aborting... 2015-10-29 08:22:04 7ff6048b0700 InnoDB: Assertion failure in thread 140694614902528 in file buf0buf.cc line 2962 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 151029 8:22:04 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.8-MariaDB-log key_buffer_size=2147483648 read_buffer_size=4194304 max_used_connections=0 max_threads=502 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x40000 2015-10-29 8:22:04 140708422530880 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164960 2015-10-29 8:22:04 140694589724416 [Note] InnoDB: Dumping buffer pool(s) not yet started 2015-10-29 8:22:04 140708422530880 [Note] Plugin 'FEEDBACK' is disabled. 2015-10-29 8:22:04 140708422530880 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin 2015-10-29 8:22:04 140708422530880 [Note] Starting crash recovery... 2015-10-29 8:22:04 140708422530880 [Note] Crash recovery finished. 2015-10-29 8:22:04 140694542206720 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2015-10-29 8:22:04 140708422530880 [Note] Server socket created on IP: '::'. 2015-10-29 8:22:04 140708422530880 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it 2015-10-29 8:22:04 140708422530880 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A 2015-10-29 8:22:04 140708422530880 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. 2015-10-29 8:22:04 140694541674240 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2015-10-29 8:22:04 140708422530880 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem. /usr/local/mariadb10.1/bin/mysqld(my_print_stacktrace+0x2e)[0x55e31936cdbe] /usr/local/mariadb10.1/bin/mysqld(handle_fatal_signal+0x34d)[0x55e318f2ccfd] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7ff93b48f8d0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7ff93a4af107] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7ff93a4b04e8] /usr/local/mariadb10.1/bin/mysqld(+0x7f9ef7)[0x55e3191acef7] /usr/local/mariadb10.1/bin/mysqld(+0x7e1777)[0x55e319194777] /usr/local/mariadb10.1/bin/mysqld(+0x77b5c3)[0x55e31912e5c3] 2015-10-29 8:22:04 140708422530880 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections. Version: '10.1.8-MariaDB-log' socket: '/var/run/mysqld/mysqld44-3.sock' port: 3740 Source distribution /usr/local/mariadb10.1/bin/mysqld(+0x77c1d4)[0x55e31912f1d4] /usr/local/mariadb10.1/bin/mysqld(+0x7489b8)[0x55e3190fb9b8] /usr/local/mariadb10.1/bin/mysqld(+0x7a61b6)[0x55e3191591b6] /usr/local/mariadb10.1/bin/mysqld(+0x7968f9)[0x55e3191498f9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7ff93b4880a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff93a56004d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file Looks like a different error. Due to the errors being so different from the initial binary i've used, i also tried compiling MariaDB from the sources available at downloads/mariadb-10.1.8.tar.gz - but i get the same "InnoDB: Failing assertion: rb_ctx != RB_NONE" error out there.

            Hmm, I do not see the error message I added on latest commit to git, but I see errors from dropped tablespace. Very strange, did you see similar messages on 10.0 ?

            jplindst Jan Lindström (Inactive) added a comment - Hmm, I do not see the error message I added on latest commit to git, but I see errors from dropped tablespace. Very strange, did you see similar messages on 10.0 ?
            sysmonk aleksandr stankevic added a comment - - edited

            I also noticed that it's missing the debug info you added in your commit, thus i tested the compiled sources from download.mariadb.org to see if i get the same error.
            No, 10.0 does a crash recovery successfully and starts up fine.

            Here's a log from running 10.0.21 against the same datadir (didn't run mysql_upgrade on this one yet):

            151029 10:42:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
            151029 10:42:52 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
            151029 10:42:52 [Note] /usr/local/mariadb10.0/bin/mysqld (mysqld 10.0.21-MariaDB-log) starting as process 4202 ...
            151029 10:42:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages
            151029 10:42:53 [Note] InnoDB: The InnoDB memory heap is disabled
            151029 10:42:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            151029 10:42:53 [Note] InnoDB: Memory barrier is not used
            151029 10:42:53 [Note] InnoDB: Compressed tables use zlib 1.2.3
            151029 10:42:53 [Note] InnoDB: Using Linux native AIO
            151029 10:42:53 [Note] InnoDB: Using CPU crc32 instructions
            151029 10:42:53 [Note] InnoDB: Initializing buffer pool, size = 10.0G
            151029 10:42:56 [Note] InnoDB: Completed initialization of buffer pool
            151029 10:42:56 [Note] InnoDB: Highest supported file format is Barracuda.
            151029 10:42:56 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164980 in the ib_logfiles!
            151029 10:42:56 [Note] InnoDB: Database was not shutdown normally!
            151029 10:42:56 [Note] InnoDB: Starting crash recovery.
            151029 10:42:56 [Note] InnoDB: Reading tablespace information from the .ibd files...
            151029 10:43:17 [Note] InnoDB: Restoring possible half-written data pages
            151029 10:43:17 [Note] InnoDB: from the doublewrite buffer...InnoDB: In a MySQL replication slave the last master binlog file
            InnoDB: position 0 1716444, file name mysql-bin.020607
            151029 10:43:17 [Note] InnoDB: 128 rollback segment(s) are active.
            151029 10:43:17 [Note] InnoDB: Waiting for purge to start
            151029 10:43:17 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 17122164980
            151029 10:43:17 [Note] Plugin 'FEEDBACK' is disabled.
            151029 10:43:17 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
            151029 10:43:17 [Note] Starting crash recovery...
            151029 10:43:17 [Note] Crash recovery finished.
            151029 10:43:17 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            151029 10:43:17 [Note] Server socket created on IP: '::'.
            151029 10:43:17 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
            151029 10:43:17 [ERROR] Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
            151029 10:43:17 [ERROR] Column count of mysql.events_waits_history is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
            151029 10:43:17 [ERROR] Column count of mysql.events_waits_history_long is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_host_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.events_waits_summary_by_thread_by_event_name: expected column 'THREAD_ID' at position 0 to have type bigint(20), found type int(11).
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_user_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_account_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Column count of mysql.file_summary_by_event_name is wrong. Expected 23, found 5. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
            151029 10:43:17 [ERROR] Column count of mysql.file_summary_by_instance is wrong. Expected 25, found 6. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'host_cache' has the wrong structure
            151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.mutex_instances: expected column 'LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11).
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'objects_summary_global_by_type' has the wrong structure
            151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.rwlock_instances: expected column 'WRITE_LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11).
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'setup_actors' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'setup_objects' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_table' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_lock_waits_summary_by_table' has the wrong structure
            151029 10:43:17 [ERROR] Column count of mysql.threads is wrong. Expected 14, found 3. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_current' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_history' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_history_long' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_thread_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_account_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_user_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_host_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_global_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_current' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_history' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_history_long' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_thread_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_account_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_user_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_host_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_global_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_digest' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'users' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'accounts' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'hosts' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_instances' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_summary_by_instance' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_summary_by_event_name' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'session_connect_attrs' has the wrong structure
            151029 10:43:17 [ERROR] Native table 'performance_schema'.'session_account_connect_attrs' has the wrong structure
            51029 10:43:17 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
            151029 10:43:17 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
            151029 10:43:17 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
            151029 10:43:17 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
            151029 10:43:17 [Note] /usr/local/mariadb10.0/bin/mysqld: ready for connections.
            Version: '10.0.21-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  MariaDB Server

            sysmonk aleksandr stankevic added a comment - - edited I also noticed that it's missing the debug info you added in your commit, thus i tested the compiled sources from download.mariadb.org to see if i get the same error. No, 10.0 does a crash recovery successfully and starts up fine. Here's a log from running 10.0.21 against the same datadir (didn't run mysql_upgrade on this one yet): 151029 10:42:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3 151029 10:42:52 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 151029 10:42:52 [Note] /usr/local/mariadb10.0/bin/mysqld (mysqld 10.0.21-MariaDB-log) starting as process 4202 ... 151029 10:42:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages 151029 10:42:53 [Note] InnoDB: The InnoDB memory heap is disabled 151029 10:42:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 151029 10:42:53 [Note] InnoDB: Memory barrier is not used 151029 10:42:53 [Note] InnoDB: Compressed tables use zlib 1.2.3 151029 10:42:53 [Note] InnoDB: Using Linux native AIO 151029 10:42:53 [Note] InnoDB: Using CPU crc32 instructions 151029 10:42:53 [Note] InnoDB: Initializing buffer pool, size = 10.0G 151029 10:42:56 [Note] InnoDB: Completed initialization of buffer pool 151029 10:42:56 [Note] InnoDB: Highest supported file format is Barracuda. 151029 10:42:56 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164980 in the ib_logfiles! 151029 10:42:56 [Note] InnoDB: Database was not shutdown normally! 151029 10:42:56 [Note] InnoDB: Starting crash recovery. 151029 10:42:56 [Note] InnoDB: Reading tablespace information from the .ibd files... 151029 10:43:17 [Note] InnoDB: Restoring possible half-written data pages 151029 10:43:17 [Note] InnoDB: from the doublewrite buffer...InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 1716444, file name mysql-bin.020607 151029 10:43:17 [Note] InnoDB: 128 rollback segment(s) are active. 151029 10:43:17 [Note] InnoDB: Waiting for purge to start 151029 10:43:17 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 17122164980 151029 10:43:17 [Note] Plugin 'FEEDBACK' is disabled. 151029 10:43:17 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin 151029 10:43:17 [Note] Starting crash recovery... 151029 10:43:17 [Note] Crash recovery finished. 151029 10:43:17 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 151029 10:43:17 [Note] Server socket created on IP: '::'. 151029 10:43:17 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it 151029 10:43:17 [ERROR] Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error. 151029 10:43:17 [ERROR] Column count of mysql.events_waits_history is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error. 151029 10:43:17 [ERROR] Column count of mysql.events_waits_history_long is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error. 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_host_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.events_waits_summary_by_thread_by_event_name: expected column 'THREAD_ID' at position 0 to have type bigint(20), found type int(11). 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_user_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_account_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Column count of mysql.file_summary_by_event_name is wrong. Expected 23, found 5. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error. 151029 10:43:17 [ERROR] Column count of mysql.file_summary_by_instance is wrong. Expected 25, found 6. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error. 151029 10:43:17 [ERROR] Native table 'performance_schema'.'host_cache' has the wrong structure 151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.mutex_instances: expected column 'LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11). 151029 10:43:17 [ERROR] Native table 'performance_schema'.'objects_summary_global_by_type' has the wrong structure 151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.rwlock_instances: expected column 'WRITE_LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11). 151029 10:43:17 [ERROR] Native table 'performance_schema'.'setup_actors' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'setup_objects' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_table' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_lock_waits_summary_by_table' has the wrong structure 151029 10:43:17 [ERROR] Column count of mysql.threads is wrong. Expected 14, found 3. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error. 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_current' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_history' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_history_long' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_thread_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_account_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_user_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_host_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_global_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_current' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_history' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_history_long' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_thread_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_account_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_user_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_host_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_global_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_digest' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'users' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'accounts' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'hosts' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_instances' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_summary_by_instance' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_summary_by_event_name' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'session_connect_attrs' has the wrong structure 151029 10:43:17 [ERROR] Native table 'performance_schema'.'session_account_connect_attrs' has the wrong structure 51029 10:43:17 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A 151029 10:43:17 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. 151029 10:43:17 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 151029 10:43:17 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem. 151029 10:43:17 [Note] /usr/local/mariadb10.0/bin/mysqld: ready for connections. Version: '10.0.21-MariaDB-log' socket: '/var/run/mysqld/mysqld44-3.sock' port: 3740 MariaDB Server

            By the way, tried compiling from latest git with your patch (0fe5eb5cae7cf7da90c8f46997bb331ce2253ac6) reverted - i get the rb_ctx != RB_NONE assertion, so that's definitely your patch that changes the behaviour.

            sysmonk aleksandr stankevic added a comment - By the way, tried compiling from latest git with your patch (0fe5eb5cae7cf7da90c8f46997bb331ce2253ac6) reverted - i get the rb_ctx != RB_NONE assertion, so that's definitely your patch that changes the behaviour.
            CypherOz Kym Farnik added a comment - See also https://mariadb.atlassian.net/browse/MDEV-9040

            Hi, Can you try with latest source version from git using normal build (not debug) and provide me the error log, I do not get this, because that part of code has not changed from 10.0, so where this comes from ?

            jplindst Jan Lindström (Inactive) added a comment - Hi, Can you try with latest source version from git using normal build (not debug) and provide me the error log, I do not get this, because that part of code has not changed from 10.0, so where this comes from ?

            Better now, only pasting the related info, if you don't mind:

            2015-10-30  7:20:57 140667493930752 [ERROR] InnoDB: Unwritten off-page columns in rollback context 0. Table wpmu_23d/wp_66829272_postmeta index "PRIMARY" space_id 0 index space 10.
            2015-10-30 07:20:57 7fefb401d700  InnoDB: Assertion failure in thread 140667493930752 in file btr0cur.cc line 5356InnoDB: Failing assertion: rb_ctx != RB_NONE

            sysmonk aleksandr stankevic added a comment - Better now, only pasting the related info, if you don't mind: 2015-10-30 7:20:57 140667493930752 [ERROR] InnoDB: Unwritten off-page columns in rollback context 0. Table wpmu_23d/wp_66829272_postmeta index "PRIMARY" space_id 0 index space 10. 2015-10-30 07:20:57 7fefb401d700 InnoDB: Assertion failure in thread 140667493930752 in file btr0cur.cc line 5356InnoDB: Failing assertion: rb_ctx != RB_NONE

            Ok, and can you provide the show create table for that table, I would really need some db where this repeats as this code where it happens is not that familiar to me.

            jplindst Jan Lindström (Inactive) added a comment - Ok, and can you provide the show create table for that table, I would really need some db where this repeats as this code where it happens is not that familiar to me.

            Here's the show create table output:

            CREATE TABLE `wp_66829272_postmeta` (
              `meta_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
              `post_id` bigint(20) unsigned NOT NULL DEFAULT '0',
              `meta_key` varchar(255) DEFAULT NULL,
              `meta_value` longtext,
              PRIMARY KEY (`meta_id`),
              KEY `post_id` (`post_id`),
              KEY `meta_key` (`meta_key`)
            ) ENGINE=InnoDB AUTO_INCREMENT=11671766 DEFAULT CHARSET=latin1

            Unfortunately, we can't provide the data files as they have customer data

            sysmonk aleksandr stankevic added a comment - Here's the show create table output: CREATE TABLE `wp_66829272_postmeta` ( `meta_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `post_id` bigint(20) unsigned NOT NULL DEFAULT '0', `meta_key` varchar(255) DEFAULT NULL, `meta_value` longtext, PRIMARY KEY (`meta_id`), KEY `post_id` (`post_id`), KEY `meta_key` (`meta_key`) ) ENGINE=InnoDB AUTO_INCREMENT=11671766 DEFAULT CHARSET=latin1 Unfortunately, we can't provide the data files as they have customer data

            Anything i can do, except for providing data files, to help get this issue resolved?

            sysmonk aleksandr stankevic added a comment - Anything i can do, except for providing data files, to help get this issue resolved?

            Hmm, actually there is, try to use innodb_plugin instead of xtradb to start your 10.1 database and provide me the results (error log).

            jplindst Jan Lindström (Inactive) added a comment - Hmm, actually there is, try to use innodb_plugin instead of xtradb to start your 10.1 database and provide me the results (error log).

            I used the initial mariadb 10.1 binary (without patches / debug ):
            Without innodb plugin:
            2015-11-03 11:13:28 7f85e57fc700 InnoDB: Assertion failure in thread 140213057734400 in file btr0cur.cc line 5341InnoDB: Failing assertion: rb_ctx != RB_NONE

            With innodb plugin:

            2015-11-03 11:25:02 140716576610176 [Note] InnoDB: Waiting for purge to start
            2015-11-03 11:25:02 7ff7c3ffc700  InnoDB: Error: trying to access tablespace 909390902 page no. 959720502,
            InnoDB: but the tablespace does not exist or is just being dropped.
            ....
            2015-11-03 11:25:02 7ff7c3ffc700  InnoDB: Error: trying to access tablespace 909390902 page no. 959720502,
            InnoDB: but the tablespace does not exist or is just being dropped.
            2015-11-03 11:25:02 7ff7c3ffc700  InnoDB: Error: trying to access tablespace 909390902 page no. 959720502,
            InnoDB: but the tablespace does not exist or is just being dropped.
            InnoDB: Error: Unable to read tablespace 909390902 page no 959720502 into the buffer pool after 100 attempts
            InnoDB: The most probable cause of this error may be that the table has been corrupted.
            ....
            stack_bottom = 0x0 thread_stack 0x40000
            mysys/stacktrace.c:247(my_print_stacktrace)[0xbf55be]
            sql/signal_handler.cc:160(handle_fatal_signal)[0x75e6c4]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7ffb214e68d0]
            /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7ffb200da107]
            /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7ffb200db4e8]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x5e900)[0x7ffa94bb2900]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x3947b)[0x7ffa94b8d47b]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1ab833)[0x7ffa94cff833]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x17750e)[0x7ffa94ccb50e]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1db325)[0x7ffa94d2f325]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1c68cf)[0x7ffa94d1a8cf]
            /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1cb8aa)[0x7ffa94d1f8aa]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7ffb214df0a4]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffb2018b04d]

            sysmonk aleksandr stankevic added a comment - I used the initial mariadb 10.1 binary (without patches / debug ): Without innodb plugin: 2015-11-03 11:13:28 7f85e57fc700 InnoDB: Assertion failure in thread 140213057734400 in file btr0cur.cc line 5341InnoDB: Failing assertion: rb_ctx != RB_NONE With innodb plugin: 2015-11-03 11:25:02 140716576610176 [Note] InnoDB: Waiting for purge to start 2015-11-03 11:25:02 7ff7c3ffc700 InnoDB: Error: trying to access tablespace 909390902 page no. 959720502, InnoDB: but the tablespace does not exist or is just being dropped. .... 2015-11-03 11:25:02 7ff7c3ffc700 InnoDB: Error: trying to access tablespace 909390902 page no. 959720502, InnoDB: but the tablespace does not exist or is just being dropped. 2015-11-03 11:25:02 7ff7c3ffc700 InnoDB: Error: trying to access tablespace 909390902 page no. 959720502, InnoDB: but the tablespace does not exist or is just being dropped. InnoDB: Error: Unable to read tablespace 909390902 page no 959720502 into the buffer pool after 100 attempts InnoDB: The most probable cause of this error may be that the table has been corrupted. .... stack_bottom = 0x0 thread_stack 0x40000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbf55be] sql/signal_handler.cc:160(handle_fatal_signal)[0x75e6c4] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7ffb214e68d0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7ffb200da107] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7ffb200db4e8] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x5e900)[0x7ffa94bb2900] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x3947b)[0x7ffa94b8d47b] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1ab833)[0x7ffa94cff833] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x17750e)[0x7ffa94ccb50e] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1db325)[0x7ffa94d2f325] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1c68cf)[0x7ffa94d1a8cf] /usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1cb8aa)[0x7ffa94d1f8aa] /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7ffb214df0a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffb2018b04d]

            Hi, can you try with latest version on git with commit 25f8738112b05f33cfa45eabfebf6edfc80e6d8a to verify does that also help here (it should).

            jplindst Jan Lindström (Inactive) added a comment - Hi, can you try with latest version on git with commit 25f8738112b05f33cfa45eabfebf6edfc80e6d8a to verify does that also help here (it should).
            jplindst Jan Lindström (Inactive) added a comment - Duplicate to MDEV-9040 .

            Hi,

            Apologies in the delay of testing this - had to afk unexpectedly for a week. Tested the todays git checkout - seems to work properly, no more crashes.

            Thanks!

            sysmonk aleksandr stankevic added a comment - Hi, Apologies in the delay of testing this - had to afk unexpectedly for a week. Tested the todays git checkout - seems to work properly, no more crashes. Thanks!

            People

              jplindst Jan Lindström (Inactive)
              sysmonk aleksandr stankevic
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.