Windows 2008 R2 , Debian sid

Moshe L,

Is it a test instance or production, and how big is the data set? Would you be able to upload the datadir to our ftp? (ftp.askmonty.org/private, only MariaDB developers will have access to it).
Please also attach your cnf file(s) and mention which, if any, changes you've made upon upgrade.

Thanks.

My.ini attached

dataset - 21GB compressed one database, all ~ 30GB, with innodb_file_per_table if it relevant (I can upload some of the tables, but not all of them, maybe will help you).

there is no changes opun upgrade (I was tried to use INNODB Recover to make it work, without help).

Thanks, yes, please upload whatever you can. The partial data is likely not to work, but it's worth trying anyway. Besides, it's possible the problem is in system tables which the server complains about in your error log (mysql.innodb_table_stats and such), which you will hopefully be able to provide.

1. please send me an email (m@inn.co.il). I will upload an non-sensitive information so you can check.

2. tables `x has the wrong structure` - there is a tables in 10.0 format (the old one), so I just need to run mysql_upgrade to make it works.
I can upload these table, as these not store sensitive data.

please send me an email (m@inn.co.il). I will upload an non-sensitive information so you can check.

What kind of email? If you mean credentials for FTP (ftp.askmonty.org/private), anyone can upload there, but only MariaDB developers can read from there.

tables `x has the wrong structure` - there is a tables in 10.0 format (the old one), so I just need to run mysql_upgrade to make it works.

"has wrong structure" error messages are of course indeed just that.
The problem is, you also had "There are 1 foreign key(s) pointing to "mysql"."innodb_table_stats", but there must be 0." messages which are far less normal – and you had them even before 10.0=>10.1 upgrade. Same for messages about performance_schema tables.

1. yes, FTP cred.
Uploaded some frm/ibd files - tables you requested and some random tables without sensitive information. `performance_schema` folder is empty.

2. I was tried to upgrade via wizard and mysqld was crashed in the wizard. maybe some tables was updated before the crash.

[ https://mariadb.atlassian.net/browse/MDEV-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=75022#comment-75022 ]

Elena Stepanova commented &#111n MDEV-8672:
---------------------------------------

please send me an email (m@inn.co.il). I will upload an non-sensitive information so you can check.

What kind of email? If you mean credentials for FTP (ftp.askmonty.org/private), anyone can upload there, but &#111nly MariaDB developers can read from there.

tables `x has the wrong structure` - there is a tables in 10.0 format (the old &#111ne), so I just need to run mysql_upgrade to make it works.

"has wrong structure" error messages are of course indeed just that.
The problem is, you also had "There are 1 foreign key(s) pointing to "mysql"."innodb_table_stats", but there must be 0." messages which are far less normal – and you had them even before 10.0=&gt10.1 upgrade. Same for messages about performance_schema tables.

–
This message was sent by Atlassian JIRA
(v7.0.0-OD-02-247#70102)

Thanks. One more thing, in the 'Environment' field you put both Windows 2008 R2 and Debian sid. Does it mean you actually have two instances which have the same problem? Can you provide the error log from the Debian instance as well?

I was tried a month ago to do the same thing on a clone of our production server, and upgrading to 10.1.5 (as I remember), and see the same problem (won't load, things like this).

I do not have the logs from the Debian machine, as it was deleted...

I've just tried upgrading from 10.0.21 to 10.1.8 and got this same error. Couldn't make it work at all, so I had to revert to 10.0.21

In my case it's Windows 2012 R2.

Same problem. Tried updating from 10.0.21 to 10.1.8
Ubuntu 14.04

Here are some of the entries on syslog:

{{2015-10-18 02:07:06 7f63aa7f9700 InnoDB: Error: trying to access tablespace 805539775 page no. 1342472192,
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: but the tablespace does not exist or is just being dropped.
Oct 18 02:07:06 LION mysqld_safe[11757]: 2015-10-18 02:07:06 7f63aa7f9700 InnoDB: Error: trying to access tablespace 805539775 page no. 1342472192,
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: but the tablespace does not exist or is just being dropped.
Oct 18 02:07:06 LION mysqld_safe[11757]: 2015-10-18 02:07:06 7f63aa7f9700 InnoDB: Error: trying to access tablespace 805539775 page no. 1342472192,
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: but the tablespace does not exist or is just being dropped.
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: Error: Unable to read tablespace 805539775 page no 1342472192 into the buffer pool after 100 attempts
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: The most probable cause of this error may be that the table has been corrupted.
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: You can try to fix this problem by using innodb_force_recovery.
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: Please see reference manual for more details.
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: Aborting...
Oct 18 02:07:06 LION mysqld_safe[11757]: 2015-10-18 02:07:06 7f63aa7f9700 InnoDB: Assertion failure in thread 140066038978304 in file buf0buf.cc line 2962
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: We intentionally generate a memory trap.
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: If you get repeated assertion failures or crashes, even
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: immediately after the mysqld startup, there may be
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
Oct 18 02:07:06 LION mysqld_safe[11757]: InnoDB: about forcing recovery.
Oct 18 02:07:06 LION mysqld_safe[11757]: 151018 2:07:06 [ERROR] mysqld got signal 6 ;
Oct 18 02:07:06 LION mysqld_safe[11757]: This could be because you hit a bug. It is also possible that this binary
Oct 18 02:07:06 LION mysqld_safe[11757]: or one of the libraries it was linked against is corrupt, improperly built,
Oct 18 02:07:06 LION mysqld_safe[11757]: or misconfigured. This error can also be caused by malfunctioning hardware.
Oct 18 02:07:06 LION mysqld_safe[11757]:
Oct 18 02:07:06 LION mysqld_safe[11757]: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Oct 18 02:07:06 LION mysqld_safe[11757]:
Oct 18 02:07:06 LION mysqld_safe[11757]: We will try our best to scrape up some info that will hopefully help
Oct 18 02:07:06 LION mysqld_safe[11757]: diagnose the problem, but since we have already crashed,
Oct 18 02:07:06 LION mysqld_safe[11757]: something is definitely wrong and this may fail.
Oct 18 02:07:06 LION mysqld_safe[11757]:
Oct 18 02:07:06 LION mysqld_safe[11757]: Server version: 10.1.8-MariaDB-1~trusty
Oct 18 02:07:06 LION mysqld_safe[11757]: key_buffer_size=134217728
Oct 18 02:07:06 LION mysqld_safe[11757]: read_buffer_size=4194304
Oct 18 02:07:06 LION mysqld_safe[11757]: max_used_connections=0
Oct 18 02:07:06 LION mysqld_safe[11757]: max_threads=102
Oct 18 02:07:06 LION mysqld_safe[11757]: thread_count=0
Oct 18 02:07:06 LION mysqld_safe[11757]: It is possible that mysqld could use up to
Oct 18 02:07:06 LION mysqld_safe[11757]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1386510 K bytes of memory
Oct 18 02:07:06 LION mysqld_safe[11757]: Hope that's ok; if not, decrease some variables in the equation.
Oct 18 02:07:06 LION mysqld_safe[11757]:
Oct 18 02:07:06 LION mysqld_safe[11757]: Thread pointer: 0x0x0
Oct 18 02:07:06 LION mysqld_safe[11757]: Attempting backtrace. You can use the following information to find out
Oct 18 02:07:06 LION mysqld_safe[11757]: where mysqld died. If you see no messages after this, something went
Oct 18 02:07:06 LION mysqld_safe[11757]: terribly wrong...
Oct 18 02:07:06 LION mysqld_safe[11757]: stack_bottom = 0x0 thread_stack 0x48000
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f665d6bd29e]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7f665d1ed8ad]
Oct 18 02:07:07 LION mysqld_safe[11757]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f665b74e340]
Oct 18 02:07:07 LION mysqld_safe[11757]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f665ada5cc9]
Oct 18 02:07:07 LION mysqld_safe[11757]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f665ada90d8]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x939b00)[0x7f665d580b00]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x91faca)[0x7f665d566aca]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x8aaa49)[0x7f665d4f1a49]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x8ab6c8)[0x7f665d4f26c8]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x872767)[0x7f665d4b9767]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x8d90ee)[0x7f665d5200ee]
Oct 18 02:07:07 LION mysqld_safe[11757]: /usr/sbin/mysqld(+0x8c703f)[0x7f665d50e03f]
Oct 18 02:07:07 LION mysqld_safe[11757]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f665b746182]
Oct 18 02:07:07 LION mysqld_safe[11757]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f665ae6947d]}}

upgraded to 10.1.8 , working .

I created database using 10.0.22, created some tables with data and shutdown. Then started MariaDB 10.1.8 and mysql_upgrade. No crashes, but following errors:

2015-10-28 09:35:48 7f3953502700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).

2015-10-28 09:35:48 7f3953502700 InnoDB: Error: Fetch of persistent statistics requested for table "mysql"."gtid_slave_pos" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

mysql_upgrade should remove that foreign key from innodb_table_stats, similarly it should fix the last_update field, but something is not right.

Actually, I see it correctly after shutdown, so you should shutdown after mysql_upgrade.

 innodb_table_stats | CREATE TABLE `innodb_table_stats` (

  `database_name` varchar(64) COLLATE utf8_bin NOT NULL,

  `table_name` varchar(64) COLLATE utf8_bin NOT NULL,

  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

  `n_rows` bigint(20) unsigned NOT NULL,

  `clustered_index_size` bigint(20) unsigned NOT NULL,

  `sum_of_other_index_sizes` bigint(20) unsigned NOT NULL,

  PRIMARY KEY (`database_name`,`table_name`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0 |

See also MDEV-9032

Can someone provide full unedited error log from this upgrade, and if you have a database where this repeats, providing that will also help to get bottom of this. I tested migrate reported on MDEV-9032 from Percona 5.5 -> MariaDB 5.5.46 -> MariaDB 10.0.21 -> MariaDB 10.1.8 and everything went smoothly. Also Windows migrate done by Elena did not repeat the issue, so we are missing something.

See https://mariadb.atlassian.net/browse/MDEV-9040

Same on CentOS 7.1

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Using mutexes to ref count buffer pool pages

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: The InnoDB memory heap is disabled

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Memory barrier is not used

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Compressed tables use zlib 1.2.7

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Using Linux native AIO

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Using CPU crc32 instructions

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Initializing buffer pool, size = 128.0M

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Completed initialization of buffer pool

2015-10-19 15:59:58 139690003601536 [Note] InnoDB: Highest supported file format is Barracuda.

2015-10-19 16:00:00 139690003601536 [Note] InnoDB: 128 rollback segment(s) are active.

2015-10-19 16:00:00 139690003601536 [Note] InnoDB: Waiting for purge to start

2015-10-19 16:00:00 139690003601536 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 686560023

2015-10-19 16:00:00 139690003601536 [Note] Plugin 'FEEDBACK' is disabled.

2015-10-19 16:00:00 139689298200320 [Note] InnoDB: Dumping buffer pool(s) not yet started

2015-10-19 16:00:00 139690003601536 [Note] Server socket created on IP: '::1'.

2015-10-19 16:00:01 7f0bf3ffa700  InnoDB: Assertion failure in thread 139689314985728 in file btr0cur.cc line 5341

InnoDB: Failing assertion: rb_ctx != RB_NONE

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

151019 16:00:01 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.1.8-MariaDB

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=0

max_threads=153

thread_count=0

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467104 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x0 thread_stack 0x48000

2015-10-19 16:00:01 139690003601536 [Note] Event Scheduler: Loaded 0 events

2015-10-19 16:00:01 7f0c1d04b700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).

2015-10-19 16:00:01 7f0c1d04b700 InnoDB: Error: Fetch of persistent statistics requested for table "mysql"."gtid_slave_pos" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.

2015-10-19 16:00:01 139690003601536 [Note] /usr/sbin/mysqld: bereit für Verbindungen.

Version: '10.1.8-MariaDB'  Socket: '/var/lib/mysql/mysql.sock'  Port: 3306  MariaDB Server

mysys/stacktrace.c:247(my_print_stacktrace)[0x7f0c1db438be]

sql/signal_handler.cc:160(handle_fatal_signal)[0x7f0c1d673d0d]

/lib64/libpthread.so.0(+0xf130)[0x7f0c1cc9c130]

/lib64/libc.so.6(gsignal+0x37)[0x7f0c1adf45d7]

/lib64/libc.so.6(abort+0x148)[0x7f0c1adf5cc8]

btr/btr0cur.cc:5341(btr_free_externally_stored_field(dict_index_t*, unsigned char*, unsigned char const*, unsigned long const*, page_zip_des_t*, unsigned long, trx_rb_ctx, mtr_t*))[0x7f0c1d923a7e]

row/row0purge.cc:712(row_purge_upd_exist_or_extern_func)[0x7f0c1d8ae879]

row/row0purge.cc:869(row_purge_record_func)[0x7f0c1d8af4f8]

que/que0que.cc:1089(que_thr_step)[0x7f0c1d876707]

trx/trx0purge.cc:1254(trx_purge(unsigned long, unsigned long, bool))[0x7f0c1d8dcd4e]

srv/srv0srv.cc:3432(srv_do_purge)[0x7f0c1d8cac9f]

/lib64/libpthread.so.0(+0x7df5)[0x7f0c1cc94df5]

/lib64/libc.so.6(clone+0x6d)[0x7f0c1aeb51ad]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Hi, can you try with latest version on git with commit 25f8738112b05f33cfa45eabfebf6edfc80e6d8a to verify does that also help here (it should).

Duplicate to MDEV-9040.

Upgraded to 10.1.9, fixed.