[MDEV-9032] MariaDB 10.1.8 crashes at startup Created: 2015-10-28  Updated: 2015-11-18  Resolved: 2015-11-05

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.1.8
Fix Version/s: 10.1.9

Type: Bug Priority: Critical
Reporter: aleksandr stankevic Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 1
Labels: innodb
Environment:

Linux 4.2.0-0.bpo.1-amd64 #1 SMP Debian 4.2.3-2~bpo8+1 (2015-10-20) x86_64 GNU/Linux
Debian Jessie 8.2


Issue Links:
Blocks
blocks MDEV-8672 Mariadb 10.1.6 crashed on upgrade fr... Closed
Sprint: 10.1.9-2

 Description   

We are migrating a bunch of databases from Percona 5.5.33 to MariaDB 10.1.8. Unfortunately, a lot of them crash during startup.

We do hot backups of the databases with LVM snapshots (flush tables with read lock, make snapshot, tar -cf the snapshot). We move the backup to a different server and extract it.

MariaDB 5.5.46 works fine with the backup
MariaDB 10.0.21 works fine with the backup
MariaDB 10.1.8 crashes

I've tried going through Percona 5.5 -> MariaDB 5.5.46 -> MariaDB 10.0.21 -> MariaDB 10.1.8 with full mysql_upgrade's and clean shutdowns, but it would still crash on startup with 10.1.8.

Looks similar to https://mariadb.atlassian.net/browse/MDEV-8672

Here are the logs from MariaDB:

2015-10-28 12:09:23 139878086616960 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 37700 ...
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: The InnoDB memory heap is disabled
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Memory barrier is not used
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Using Linux native AIO
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Using CPU crc32 instructions
2015-10-28 12:09:23 139878086616960 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Completed initialization of buffer pool
2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Highest supported file format is Barracuda.
2015-10-28 12:09:27 139878086616960 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164830 in the ib_logfiles!
2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Database was not shutdown normally!
2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Starting crash recovery.
2015-10-28 12:09:27 139878086616960 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-10-28 12:09:30 139878086616960 [Note] InnoDB: Restoring possible half-written data pages
2015-10-28 12:09:30 139878086616960 [Note] InnoDB: from the doublewrite buffer...
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 1716444, file name mysql-bin.020607
2015-10-28 12:09:30 139878086616960 [Note] InnoDB: 128 rollback segment(s) are active.
2015-10-28 12:09:30 139878086616960 [Note] InnoDB: Waiting for purge to start
2015-10-28 12:09:30 7f348a3fc700  InnoDB: Assertion failure in thread 139863634462464 in file btr0cur.cc line 5341
InnoDB: Failing assertion: rb_ctx != RB_NONE
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
151028 12:09:30 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.1.8-MariaDB-log
key_buffer_size=2147483648
read_buffer_size=4194304  
max_used_connections=0
max_threads=502
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
 
015-10-28 12:09:30 139878086616960 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164830
2015-10-28 12:09:30 139863609284352 [Note] InnoDB: Dumping buffer pool(s) not yet started
2015-10-28 12:09:30 139878086616960 [Note] Plugin 'FEEDBACK' is disabled.
2015-10-28 12:09:30 139878086616960 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
2015-10-28 12:09:30 139878086616960 [Note] Starting crash recovery...
2015-10-28 12:09:30 139878086616960 [Note] Crash recovery finished.
2015-10-28 12:09:30 139878086117120 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-28 12:09:30 139878086616960 [Note] Server socket created on IP: '::'.
2015-10-28 12:09:30 139878086616960 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
2015-10-28 12:09:30 139878086616960 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
2015-10-28 12:09:30 139878086616960 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
2015-10-28 12:09:30 139878085584640 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-28 12:09:30 139878086616960 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
2015-10-28 12:09:30 139878086616960 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
Version: '10.1.8-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  MariaDB Server
/usr/local/mariadb10.1/bin/mysqld(my_print_stacktrace+0x2e)[0xbf55be]
/usr/local/mariadb10.1/bin/mysqld(handle_fatal_signal+0x464)[0x75e6c4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f37e76868d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f37e627a107]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f37e627b4e8]
/usr/local/mariadb10.1/bin/mysqld[0xa0e9d3]
/usr/local/mariadb10.1/bin/mysqld[0x9ae60c]
/usr/local/mariadb10.1/bin/mysqld[0x979bd4]
/usr/local/mariadb10.1/bin/mysqld[0x9dba74]
/usr/local/mariadb10.1/bin/mysqld[0x9c4507]
/usr/local/mariadb10.1/bin/mysqld[0x9ca5ca]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f37e767f0a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f37e632b04d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
151028 12:09:31 mysqld_safe Number of processes running now: 0
151028 12:09:31 mysqld_safe mysqld restarted



 Comments   
Comment by Jan Lindström (Inactive) [ 2015-10-28 ]

Do you know what is the value of rb_ctx in this case ?

Comment by aleksandr stankevic [ 2015-10-28 ]

Hi,

No, i don't. But i can compile mysqld with debuging to see if i can get one.

Comment by aleksandr stankevic [ 2015-10-28 ]

MariaDB 10.1.8 compiled with debug gives more info:

151028 13:53:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
2015-10-28 13:53:55 140640495765312 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
2015-10-28 13:53:55 140640495765312 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-debug-log) starting as process 115715 ...
2015-10-28 13:53:56 140640495765312 [ERROR] mysqld: file-key-management-filename is not set
2015-10-28 13:53:56 140640495765312 [ERROR] Plugin 'file_key_management' init function returned error.
2015-10-28 13:53:56 140640495765312 [ERROR] Plugin 'file_key_management' registration as a ENCRYPTION failed.
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2015-10-28 13:53:56 140640495765312 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2015-10-28 13:53:56 140640495765312 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: The InnoDB memory heap is disabled
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Memory barrier is not used
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Compressed tables use zlib 1.2.8
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Using CPU crc32 instructions
2015-10-28 13:53:56 140640495765312 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Completed initialization of buffer pool
2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Highest supported file format is Barracuda.
2015-10-28 13:54:00 140640495765312 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164850 in the ib_logfiles!
2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Database was not shutdown normally!
2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Starting crash recovery.
2015-10-28 13:54:00 140640495765312 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-10-28 13:54:20 140640495765312 [Note] InnoDB: Restoring possible half-written data pages 
2015-10-28 13:54:20 140640495765312 [Note] InnoDB: from the doublewrite buffer...
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 1716444, file name mysql-bin.020607
2015-10-28 13:54:22 140640495765312 [Note] InnoDB: 128 rollback segment(s) are active.
2015-10-28 13:54:22 140640495765312 [Note] InnoDB: Waiting for purge to start
2015-10-28 13:54:22 140640495765312 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164850
2015-10-28 13:54:22 140626372826880 [Note] InnoDB: Dumping buffer pool(s) not yet started
2015-10-28 13:54:22 140640495765312 [Note] CONNECT: Version 1.03.0007 July 05, 2015
2015-10-28 13:54:22 7fe6227fc700  InnoDB: Assertion failure in thread 140626398004992 in file btr0cur.cc line 5332
InnoDB: Failing assertion: mtr_memo_contains_page(local_mtr, field_ref, 2)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
151028 13:54:22 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.8-MariaDB-debug-log
key_buffer_size=2147483648
read_buffer_size=4194304
max_used_connections=0
max_threads=502
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6220045 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
2015-10-28 13:54:22 140640495765312 [Note] Plugin 'FEEDBACK' is disabled.
2015-10-28 13:54:22 140640495765312 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
2015-10-28 13:54:22 140640495765312 [Note] Starting crash recovery...
2015-10-28 13:54:22 140640495765312 [Note] Crash recovery finished.
2015-10-28 13:54:22 140640252671744 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-28 13:54:22 140640495765312 [Note] Server socket created on IP: '::'.
2015-10-28 13:54:22 140640495765312 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
2015-10-28 13:54:23 140640495765312 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
2015-10-28 13:54:23 140640495765312 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
2015-10-28 13:54:23 140640252139264 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-28 13:54:23 140640495765312 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
2015-10-28 13:54:23 140640495765312 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
Version: '10.1.8-MariaDB-debug-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  Source distribution
mysys/stacktrace.c:246(my_print_stacktrace)[0xf7711f]
sql/signal_handler.cc:162(handle_fatal_signal)[0x88339b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fe96a88c8d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fe969691107]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9696924e8]
btr/btr0cur.cc:5333(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0xcbd1fe]
row/row0purge.cc:711(row_purge_upd_exist_or_extern_func)[0xc18134]
row/row0purge.cc:869(row_purge_record_func)[0xc1859d]
row/row0purge.cc:912(row_purge)[0xc186e8]
row/row0purge.cc:994(row_purge_step(que_thr_t*))[0xc18990]
que/que0que.cc:1089(que_thr_step)[0xbb93e4]
que/que0que.cc:1151(que_run_threads_low)[0xbb95e8]
que/que0que.cc:1194(que_run_threads(que_thr_t*))[0xbb9774]
trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0xc67535]
srv/srv0srv.cc:3432(srv_do_purge)[0xc4ad31]
srv/srv0srv.cc:3618(srv_purge_coordinator_thread)[0xc4b445]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fe96a8850a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe96974204d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
151028 13:54:23 mysqld_safe Number of processes running now: 0
151028 13:54:23 mysqld_safe mysqld restarted

Comment by Jan Lindström (Inactive) [ 2015-10-28 ]

This is not the same assertion, do you have db that you could share ?

Comment by aleksandr stankevic [ 2015-10-28 ]

The DB is around 150gb and has private customer data so probably can't share it.
If you give me any steps to help debug this i will do my best to help debug this issue.

Comment by Jan Lindström (Inactive) [ 2015-10-29 ]

Tried migrate from Percona 5.5 -> MariaDB 5.5.46 -> MariaDB 10.0.21 -> MariaDB 10.1.8 and everything went smoothly, so I need more information. Can you try version from git repository so that you have commit 0fe5eb5cae7cf7da90c8f46997bb331ce2253ac6 compile using normal setup (no debug) and provide me a full unedited error log (do not delete the old one).

Comment by aleksandr stankevic [ 2015-10-29 ]

The error isn't reproducible on every installl. Out of ~10 db's that we tried to migrate, 4 have this error. Others don't.

If it makes any difference - most of these databases have been started as 4.1 and have been migrated up to 5.5 in years.

I've already had the error log truncated, so i compiled the git version and started with the mariadb 10.1.8 from package, then started with mariadb10.1.8 from git. Here are the results:

151029 08:18:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
2015-10-29  8:18:54 140291706009472 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
2015-10-29  8:18:54 140291706009472 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 92493 ...
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: The InnoDB memory heap is disabled
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Memory barrier is not used
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Using Linux native AIO
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Using CPU crc32 instructions
2015-10-29  8:18:54 140291706009472 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Completed initialization of buffer pool
2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Highest supported file format is Barracuda.
2015-10-29  8:18:57 140291706009472 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164950 in the ib_logfiles!
2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Database was not shutdown normally!
2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Starting crash recovery.
2015-10-29  8:18:57 140291706009472 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-10-29  8:19:17 140291706009472 [Note] InnoDB: Restoring possible half-written data pages 
2015-10-29  8:19:17 140291706009472 [Note] InnoDB: from the doublewrite buffer...
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 1716444, file name mysql-bin.020607
2015-10-29  8:19:18 140291706009472 [Note] InnoDB: 128 rollback segment(s) are active.
2015-10-29  8:19:18 140291706009472 [Note] InnoDB: Waiting for purge to start
2015-10-29 08:19:18 7f94d7ffc700  InnoDB: Assertion failure in thread 140277255751424 in file btr0cur.cc line 5341
InnoDB: Failing assertion: rb_ctx != RB_NONE
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
151029  8:19:18 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.8-MariaDB-log
key_buffer_size=2147483648
read_buffer_size=4194304
max_used_connections=0
max_threads=502
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
2015-10-29  8:19:18 140291706009472 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164950
2015-10-29  8:19:18 140277230573312 [Note] InnoDB: Dumping buffer pool(s) not yet started
2015-10-29  8:19:18 140291706009472 [Note] Plugin 'FEEDBACK' is disabled.
2015-10-29  8:19:18 140291706009472 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
2015-10-29  8:19:18 140291706009472 [Note] Starting crash recovery...
2015-10-29  8:19:18 140291706009472 [Note] Crash recovery finished.
2015-10-29  8:19:18 140291705509632 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-29  8:19:18 140291706009472 [Note] Server socket created on IP: '::'.
2015-10-29  8:19:18 140291706009472 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
2015-10-29  8:19:18 140291706009472 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
2015-10-29  8:19:18 140291706009472 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
2015-10-29  8:19:18 140291704977152 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-29  8:19:18 140291706009472 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
2015-10-29  8:19:18 140291706009472 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
Version: '10.1.8-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  MariaDB Server
mysys/stacktrace.c:247(my_print_stacktrace)[0xbf55be]
sql/signal_handler.cc:160(handle_fatal_signal)[0x75e6c4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f98350b88d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f9833cac107]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f9833cad4e8]
btr/btr0cur.cc:5470(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0xa0e9d3]
row/row0purge.cc:712(row_purge_upd_exist_or_extern_func)[0x9ae60c]
que/que0que.cc:1089(que_thr_step)[0x979bd4]
trx/trx0purge.cc:1253(trx_purge(unsigned long, unsigned long, bool))[0x9dba74]
srv/srv0srv.cc:3434(srv_do_purge)[0x9c4507]
srv/srv0srv.cc:3618(srv_purge_coordinator_thread)[0x9ca5ca]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f98350b10a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f9833d5d04d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
151029 08:21:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
2015-10-29  8:21:40 140708422530880 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
2015-10-29  8:21:40 140708422530880 [Note] /usr/local/mariadb10.1/bin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 94154 ...
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: The InnoDB memory heap is disabled
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Memory barrier is not used
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Using CPU crc32 instructions
2015-10-29  8:21:40 140708422530880 [Note] InnoDB: Initializing buffer pool, size = 10.0G
2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Completed initialization of buffer pool
2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Highest supported file format is Barracuda.
2015-10-29  8:21:44 140708422530880 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164960 in the ib_logfiles!
2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Database was not shutdown normally!
2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Starting crash recovery.
2015-10-29  8:21:44 140708422530880 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-10-29  8:22:03 140708422530880 [Note] InnoDB: Restoring possible half-written data pages 
2015-10-29  8:22:03 140708422530880 [Note] InnoDB: from the doublewrite buffer...
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 1716444, file name mysql-bin.020607
2015-10-29  8:22:04 140708422530880 [Note] InnoDB: 128 rollback segment(s) are active.
2015-10-29  8:22:04 140708422530880 [Note] InnoDB: Waiting for purge to start
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Error: trying to access tablespace 909587505 page no. 959523894,
InnoDB: but the tablespace does not exist or is just being dropped.
InnoDB: Error: Unable to read tablespace 909587505 page no 959523894 into the buffer pool after 100 attempts
InnoDB: The most probable cause of this error may be that the table has been corrupted.
InnoDB: You can try to fix this problem by using innodb_force_recovery.
InnoDB: Please see reference manual for more details.
InnoDB: Aborting...
2015-10-29 08:22:04 7ff6048b0700  InnoDB: Assertion failure in thread 140694614902528 in file buf0buf.cc line 2962
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
151029  8:22:04 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.8-MariaDB-log
key_buffer_size=2147483648
read_buffer_size=4194304
max_used_connections=0
max_threads=502
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6219767 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
2015-10-29  8:22:04 140708422530880 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 17122164960
2015-10-29  8:22:04 140694589724416 [Note] InnoDB: Dumping buffer pool(s) not yet started
2015-10-29  8:22:04 140708422530880 [Note] Plugin 'FEEDBACK' is disabled.
2015-10-29  8:22:04 140708422530880 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
2015-10-29  8:22:04 140708422530880 [Note] Starting crash recovery...
2015-10-29  8:22:04 140708422530880 [Note] Crash recovery finished.
2015-10-29  8:22:04 140694542206720 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-29  8:22:04 140708422530880 [Note] Server socket created on IP: '::'.
2015-10-29  8:22:04 140708422530880 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
2015-10-29  8:22:04 140708422530880 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
2015-10-29  8:22:04 140708422530880 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
2015-10-29  8:22:04 140694541674240 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
2015-10-29  8:22:04 140708422530880 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
/usr/local/mariadb10.1/bin/mysqld(my_print_stacktrace+0x2e)[0x55e31936cdbe]
/usr/local/mariadb10.1/bin/mysqld(handle_fatal_signal+0x34d)[0x55e318f2ccfd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7ff93b48f8d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7ff93a4af107]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7ff93a4b04e8]
/usr/local/mariadb10.1/bin/mysqld(+0x7f9ef7)[0x55e3191acef7]
/usr/local/mariadb10.1/bin/mysqld(+0x7e1777)[0x55e319194777]
/usr/local/mariadb10.1/bin/mysqld(+0x77b5c3)[0x55e31912e5c3]
2015-10-29  8:22:04 140708422530880 [Note] /usr/local/mariadb10.1/bin/mysqld: ready for connections.
Version: '10.1.8-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  Source distribution
/usr/local/mariadb10.1/bin/mysqld(+0x77c1d4)[0x55e31912f1d4]
/usr/local/mariadb10.1/bin/mysqld(+0x7489b8)[0x55e3190fb9b8]
/usr/local/mariadb10.1/bin/mysqld(+0x7a61b6)[0x55e3191591b6]
/usr/local/mariadb10.1/bin/mysqld(+0x7968f9)[0x55e3191498f9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7ff93b4880a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff93a56004d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

Looks like a different error. Due to the errors being so different from the initial binary i've used, i also tried compiling MariaDB from the sources available at downloads/mariadb-10.1.8.tar.gz - but i get the same "InnoDB: Failing assertion: rb_ctx != RB_NONE" error out there.

Comment by Jan Lindström (Inactive) [ 2015-10-29 ]

Hmm, I do not see the error message I added on latest commit to git, but I see errors from dropped tablespace. Very strange, did you see similar messages on 10.0 ?

Comment by aleksandr stankevic [ 2015-10-29 ]

I also noticed that it's missing the debug info you added in your commit, thus i tested the compiled sources from download.mariadb.org to see if i get the same error.
No, 10.0 does a crash recovery successfully and starts up fine.

Here's a log from running 10.0.21 against the same datadir (didn't run mysql_upgrade on this one yet):

151029 10:42:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql44-3
151029 10:42:52 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
151029 10:42:52 [Note] /usr/local/mariadb10.0/bin/mysqld (mysqld 10.0.21-MariaDB-log) starting as process 4202 ...
151029 10:42:53 [Note] InnoDB: Using mutexes to ref count buffer pool pages
151029 10:42:53 [Note] InnoDB: The InnoDB memory heap is disabled
151029 10:42:53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
151029 10:42:53 [Note] InnoDB: Memory barrier is not used
151029 10:42:53 [Note] InnoDB: Compressed tables use zlib 1.2.3
151029 10:42:53 [Note] InnoDB: Using Linux native AIO
151029 10:42:53 [Note] InnoDB: Using CPU crc32 instructions
151029 10:42:53 [Note] InnoDB: Initializing buffer pool, size = 10.0G
151029 10:42:56 [Note] InnoDB: Completed initialization of buffer pool
151029 10:42:56 [Note] InnoDB: Highest supported file format is Barracuda.
151029 10:42:56 [Note] InnoDB: The log sequence numbers 17122163212 and 17122163212 in ibdata files do not match the log sequence number 17122164980 in the ib_logfiles!
151029 10:42:56 [Note] InnoDB: Database was not shutdown normally!
151029 10:42:56 [Note] InnoDB: Starting crash recovery.
151029 10:42:56 [Note] InnoDB: Reading tablespace information from the .ibd files...
151029 10:43:17 [Note] InnoDB: Restoring possible half-written data pages
151029 10:43:17 [Note] InnoDB: from the doublewrite buffer...InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 1716444, file name mysql-bin.020607
151029 10:43:17 [Note] InnoDB: 128 rollback segment(s) are active.
151029 10:43:17 [Note] InnoDB: Waiting for purge to start
151029 10:43:17 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 17122164980
151029 10:43:17 [Note] Plugin 'FEEDBACK' is disabled.
151029 10:43:17 [Note] Recovering after a crash using /var/log/mysql44-3/mysql-bin
151029 10:43:17 [Note] Starting crash recovery...
151029 10:43:17 [Note] Crash recovery finished.
151029 10:43:17 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
151029 10:43:17 [Note] Server socket created on IP: '::'.
151029 10:43:17 [ERROR] Missing system table mysql.roles_mapping; please run mysql_upgrade to create it
151029 10:43:17 [ERROR] Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
151029 10:43:17 [ERROR] Column count of mysql.events_waits_history is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
151029 10:43:17 [ERROR] Column count of mysql.events_waits_history_long is wrong. Expected 19, found 16. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_host_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.events_waits_summary_by_thread_by_event_name: expected column 'THREAD_ID' at position 0 to have type bigint(20), found type int(11).
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_user_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_account_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Column count of mysql.file_summary_by_event_name is wrong. Expected 23, found 5. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
151029 10:43:17 [ERROR] Column count of mysql.file_summary_by_instance is wrong. Expected 25, found 6. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
151029 10:43:17 [ERROR] Native table 'performance_schema'.'host_cache' has the wrong structure
151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.mutex_instances: expected column 'LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11).
151029 10:43:17 [ERROR] Native table 'performance_schema'.'objects_summary_global_by_type' has the wrong structure
151029 10:43:17 [ERROR] Incorrect definition of table performance_schema.rwlock_instances: expected column 'WRITE_LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11).
151029 10:43:17 [ERROR] Native table 'performance_schema'.'setup_actors' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'setup_objects' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_table' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'table_lock_waits_summary_by_table' has the wrong structure
151029 10:43:17 [ERROR] Column count of mysql.threads is wrong. Expected 14, found 3. Created with MariaDB 50528, now running 100021. Please use mysql_upgrade to fix this error.
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_current' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_history' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_history_long' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_thread_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_account_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_user_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_host_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_stages_summary_global_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_current' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_history' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_history_long' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_thread_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_account_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_user_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_host_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_global_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_digest' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'users' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'accounts' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'hosts' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_instances' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_summary_by_instance' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'socket_summary_by_event_name' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'session_connect_attrs' has the wrong structure
151029 10:43:17 [ERROR] Native table 'performance_schema'.'session_account_connect_attrs' has the wrong structure
51029 10:43:17 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A
151029 10:43:17 [ERROR] Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
151029 10:43:17 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist
151029 10:43:17 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld44-3-relay-bin' to avoid this problem.
151029 10:43:17 [Note] /usr/local/mariadb10.0/bin/mysqld: ready for connections.
Version: '10.0.21-MariaDB-log'  socket: '/var/run/mysqld/mysqld44-3.sock'  port: 3740  MariaDB Server

Comment by aleksandr stankevic [ 2015-10-29 ]

By the way, tried compiling from latest git with your patch (0fe5eb5cae7cf7da90c8f46997bb331ce2253ac6) reverted - i get the rb_ctx != RB_NONE assertion, so that's definitely your patch that changes the behaviour.

Comment by Kym Farnik [ 2015-10-30 ]

See also https://mariadb.atlassian.net/browse/MDEV-9040

Comment by Jan Lindström (Inactive) [ 2015-10-30 ]

Hi, Can you try with latest source version from git using normal build (not debug) and provide me the error log, I do not get this, because that part of code has not changed from 10.0, so where this comes from ?

Comment by aleksandr stankevic [ 2015-10-30 ]

Better now, only pasting the related info, if you don't mind:

2015-10-30  7:20:57 140667493930752 [ERROR] InnoDB: Unwritten off-page columns in rollback context 0. Table wpmu_23d/wp_66829272_postmeta index "PRIMARY" space_id 0 index space 10.
2015-10-30 07:20:57 7fefb401d700  InnoDB: Assertion failure in thread 140667493930752 in file btr0cur.cc line 5356InnoDB: Failing assertion: rb_ctx != RB_NONE

Comment by Jan Lindström (Inactive) [ 2015-10-30 ]

Ok, and can you provide the show create table for that table, I would really need some db where this repeats as this code where it happens is not that familiar to me.

Comment by aleksandr stankevic [ 2015-10-30 ]

Here's the show create table output:

CREATE TABLE `wp_66829272_postmeta` (
  `meta_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `post_id` bigint(20) unsigned NOT NULL DEFAULT '0',
  `meta_key` varchar(255) DEFAULT NULL,
  `meta_value` longtext,
  PRIMARY KEY (`meta_id`),
  KEY `post_id` (`post_id`),
  KEY `meta_key` (`meta_key`)
) ENGINE=InnoDB AUTO_INCREMENT=11671766 DEFAULT CHARSET=latin1

Unfortunately, we can't provide the data files as they have customer data

Comment by aleksandr stankevic [ 2015-11-03 ]

Anything i can do, except for providing data files, to help get this issue resolved?

Comment by Jan Lindström (Inactive) [ 2015-11-03 ]

Hmm, actually there is, try to use innodb_plugin instead of xtradb to start your 10.1 database and provide me the results (error log).

Comment by aleksandr stankevic [ 2015-11-03 ]

I used the initial mariadb 10.1 binary (without patches / debug ):
Without innodb plugin:
2015-11-03 11:13:28 7f85e57fc700 InnoDB: Assertion failure in thread 140213057734400 in file btr0cur.cc line 5341InnoDB: Failing assertion: rb_ctx != RB_NONE

With innodb plugin:

2015-11-03 11:25:02 140716576610176 [Note] InnoDB: Waiting for purge to start
2015-11-03 11:25:02 7ff7c3ffc700  InnoDB: Error: trying to access tablespace 909390902 page no. 959720502,
InnoDB: but the tablespace does not exist or is just being dropped.
....
2015-11-03 11:25:02 7ff7c3ffc700  InnoDB: Error: trying to access tablespace 909390902 page no. 959720502,
InnoDB: but the tablespace does not exist or is just being dropped.
2015-11-03 11:25:02 7ff7c3ffc700  InnoDB: Error: trying to access tablespace 909390902 page no. 959720502,
InnoDB: but the tablespace does not exist or is just being dropped.
InnoDB: Error: Unable to read tablespace 909390902 page no 959720502 into the buffer pool after 100 attempts
InnoDB: The most probable cause of this error may be that the table has been corrupted.
....
stack_bottom = 0x0 thread_stack 0x40000
mysys/stacktrace.c:247(my_print_stacktrace)[0xbf55be]
sql/signal_handler.cc:160(handle_fatal_signal)[0x75e6c4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7ffb214e68d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7ffb200da107]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7ffb200db4e8]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x5e900)[0x7ffa94bb2900]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x3947b)[0x7ffa94b8d47b]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1ab833)[0x7ffa94cff833]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x17750e)[0x7ffa94ccb50e]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1db325)[0x7ffa94d2f325]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1c68cf)[0x7ffa94d1a8cf]
/usr/local/mariadb10.1/lib/plugin/ha_innodb.so(+0x1cb8aa)[0x7ffa94d1f8aa]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7ffb214df0a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffb2018b04d]

Comment by Jan Lindström (Inactive) [ 2015-11-05 ]

Hi, can you try with latest version on git with commit 25f8738112b05f33cfa45eabfebf6edfc80e6d8a to verify does that also help here (it should).

Comment by Jan Lindström (Inactive) [ 2015-11-05 ]

Duplicate to MDEV-9040.

Comment by aleksandr stankevic [ 2015-11-12 ]

Hi,

Apologies in the delay of testing this - had to afk unexpectedly for a week. Tested the todays git checkout - seems to work properly, no more crashes.

Thanks!

Generated at Thu Feb 08 07:31:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.