[MDEV-21863] InnoDB: Assertion failure rem0rec.cc 1878 - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 10.3.11
Fix Version/s: N/A
Component/s: Storage Engine - InnoDB
Labels:
None

Description

2020-03-03 17:11:31 0x7f03bcb92700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.11/storage/innobase/rem/rem0rec.cc line 1878

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/

InnoDB: about forcing recovery.

200303 17:11:31 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.3.11-MariaDB-log

key_buffer_size=134217728

read_buffer_size=2097152

max_used_connections=13

max_threads=2002

thread_count=23

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8375880 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f03480009a8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f03bcb91d00 thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b0b491061e]

mysys/stacktrace.c:270(my_print_stacktrace)[0x55b0b43b13a7]

sigaction.c:0(__restore_rt)[0x7f38ad7a75d0]

:0(__GI_raise)[0x7f38aba7b207]

:0(__GI_abort)[0x7f38aba7c8f8]

/usr/sbin/mysqld(+0x4c62b6)[0x55b0b40fb2b6]

ut/ut0rbt.cc:221(rbt_tree_add_child(ib_rbt_t const*, ib_rbt_bound_t*, ib_rbt_node_t*) [clone .isra.4] [clone .part.5])[0x55b0b46222c3]

rem/rem0rec.cc:1878(rec_copy_prefix_to_buf(unsigned char const*, dict_index_t const*, unsigned long, unsigned char**, unsigned long*))[0x55b0b46fc4d8]

btr/btr0pcur.cc:174(btr_pcur_store_position(btr_pcur_t*, mtr_t*))[0x55b0b4668828]

row/row0sel.cc:5597(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x55b0b4590a99]

handler/ha_innodb.cc:9530(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int))[0x55b0b43b6d29]

sql/handler.cc:2894(handler::ha_index_prev(unsigned char*))[0x55b0b421e33c]

sql/sql_select.cc:20444(join_read_prev(READ_RECORD*))[0x55b0b421360f]

sql/sql_select.cc:19417(sub_select(JOIN*, st_join_table*, bool))[0x55b0b4234cfb]

sql/sql_select.cc:18937(do_select)[0x55b0b4234f13]

sql/sql_select.cc:3835(JOIN::exec())[0x55b0b423506a]

sql/sql_select.cc:4241(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55b0b4235b7c]

sql/sql_select.cc:385(handle_select(THD*, LEX*, select_result*, unsigned long))[0x55b0b40eb6b7]

sql/sql_parse.cc:6547(execute_sqlcom_select(THD*, TABLE_LIST*))[0x55b0b41e1af3]

sql/sql_parse.cc:3767(mysql_execute_command(THD*))[0x55b0b41e42db]

sql/sql_parse.cc:8091(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55b0b41e6e35]

sql/sql_parse.cc:1850(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55b0b41e7c9e]

sql/sql_parse.cc:1397(do_command(THD*))[0x55b0b42b79da]

sql/sql_connect.cc:1402(do_handle_one_connection(CONNECT*))[0x55b0b42b7afd]

sql/sql_connect.cc:1310(handle_one_connection)[0x55b0b455bd3d]

pthread_create.c:0(start_thread)[0x7f38ad79fdd5]

/lib64/libc.so.6(clone+0x6d)[0x7f38abb42ead]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f0348011de0): SELECT `ssladmin_certificatemodel`.`id`, `ssladmin_certificatemodel`.`provider`, `ssladmin_certificatemodel`.`orderid`, `ssladmin_certificatemodel`.`common_name`, `ssladmin_certificatemodel`.`key_alg`, `ssladmin_certificatemodel`.`private_key_pem`, `ssladmin_certificatemodel`.`private_key_fernet`, `ssladmin_certificatemodel`.`certificate_request_pem`, `ssladmin_certificatemodel`.`certificate_pem`, `ssladmin_certificatemodel`.`status`, `ssladmin_certificatemodel`.`created`, `ssladmin_certificatemodel`.`modified` FROM `ssladmin_certificatemodel` WHERE (`ssladmin_certificatemodel`.`id` LIKE '%aazfby%' OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%') ORDER BY `ssladmin_certificatemodel`.`id` DESC

Connection ID (thread ID): 75887

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

The query who caused this crash :

CREATE TABLE `ssladmin_certificatemodel` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  `provider` varchar(30) NOT NULL,

  `orderid` varchar(30) NOT NULL,

  `common_name` varchar(100) NOT NULL,

  `key_alg` varchar(10) NOT NULL,

  `private_key_pem` longtext NOT NULL,

  `certificate_request_pem` longtext NOT NULL,

  `certificate_pem` longtext NOT NULL,

  `status` longtext NOT NULL,

  `created` datetime(6) NOT NULL,

  `modified` datetime(6) NOT NULL,

  `private_key_fernet` longtext NOT NULL,

  PRIMARY KEY (`id`),

  UNIQUE KEY `ssladmin_certificatemode_provider_orderid_common__bc90565b_uniq` (`provider`,`orderid`,`common_name`,`key_alg`)

) ENGINE=InnoDB AUTO_INCREMENT=24264 DEFAULT CHARSET=utf8;

SELECT

  `ssladmin_certificatemodel`.`id`,

  `ssladmin_certificatemodel`.`provider`,

  `ssladmin_certificatemodel`.`orderid`,

  `ssladmin_certificatemodel`.`common_name`,

  `ssladmin_certificatemodel`.`key_alg`,

  `ssladmin_certificatemodel`.`private_key_pem`,

  `ssladmin_certificatemodel`.`private_key_fernet`,

  `ssladmin_certificatemodel`.`certificate_request_pem`,

  `ssladmin_certificatemodel`.`certificate_pem`,

  `ssladmin_certificatemodel`.`status`,

  `ssladmin_certificatemodel`.`created`,

  `ssladmin_certificatemodel`.`modified`

FROM

  `ssladmin_certificatemodel`

WHERE

    `ssladmin_certificatemodel`.`id` LIKE '%aazfby%'

    OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%'

ORDER BY

  `ssladmin_certificatemodel`.`id` DESC

Unfortunatly I cannot provide data (private keys)

Workaround to fix :

mysql_upgrade --force

--nothing special

MariaDB [netportal_db]> SELECT    `ssladmin_certificatemodel`.`id`,    `ssladmin_certificatemodel`.`provider`,    `ssladmin_certificatemodel`.`orderid`,    `ssladmin_certificatemodel`.`common_name`,    `ssladmin_certificatemodel`.`key_alg`,    `ssladmin_certificatemodel`.`private_key_pem`,    `ssladmin_certificatemodel`.`private_key_fernet`,    `ssladmin_certificatemodel`.`certificate_request_pem`,    `ssladmin_certificatemodel`.`certificate_pem`,    `ssladmin_certificatemodel`.`status`,    `ssladmin_certificatemodel`.`created`,    `ssladmin_certificatemodel`.`modified`  FROM    `ssladmin_certificatemodel`  WHERE    (     `ssladmin_certificatemodel`.`id` LIKE '%aazfby%'      OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%'   )  ORDER BY    `ssladmin_certificatemodel`.`id` DESC;

--crash mariadb got signal 6

MariaDB [netportal_db]> ALTER TABLE ssladmin_certificatemodel ENGINE=INNODB;

Query OK, 0 rows affected (21.818 sec)     -- <<= 26 000 lignes and not big at all !

Records: 0  Duplicates: 0  Warnings: 0

MariaDB [netportal_db]> SELECT    `ssladmin_certificatemodel`.`id`,    `ssladmin_certificatemodel`.`provider`,    `ssladmin_certificatemodel`.`orderid`,    `ssladmin_certificatemodel`.`common_name`,    `ssladmin_certificatemodel`.`key_alg`,    `ssladmin_certificatemodel`.`private_key_pem`,    `ssladmin_certificatemodel`.`private_key_fernet`,    `ssladmin_certificatemodel`.`certificate_request_pem`,    `ssladmin_certificatemodel`.`certificate_pem`,    `ssladmin_certificatemodel`.`status`,    `ssladmin_certificatemodel`.`created`,    `ssladmin_certificatemodel`.`modified`  FROM    `ssladmin_certificatemodel`  WHERE    (     `ssladmin_certificatemodel`.`id` LIKE '%aazfby%'      OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%'   )  ORDER BY    `ssladmin_certificatemodel`.`id` DESC;

-- secret results

2 rows in set (0.061 sec)

Attachments

Issue Links

is blocked by

MDEV-21890 cannot connect to phpAdmin

Closed

relates to

MDEV-13542 Crashing on a corrupted page is unhelpful

Closed

Activity

Ascending order - Click to sort in descending order

View 13 older comments

Aurélien LEQUOY added a comment - 2020-03-10 09:27 - edited

About IMPORT table space i do this to not impact production, I still have one server with this issue.
Marko Mäkelä : export / import was the solution to work quietly, i am sure the problem not come from there. (maybe we not the good things with this server)

It's happen on master and I got exactly the same trouble and both slaves behind. I can isolate this slave and make a debug too.

Aurélien LEQUOY added a comment - 2020-03-10 09:27 - edited About IMPORT table space i do this to not impact production, I still have one server with this issue. Marko Mäkelä : export / import was the solution to work quietly, i am sure the problem not come from there. (maybe we not the good things with this server) It's happen on master and I got exactly the same trouble and both slaves behind. I can isolate this slave and make a debug too.

Aurélien LEQUOY added a comment - 2020-03-10 09:50 - edited

I think it's more complicated than that, I clearly identified the problem with PRIMARY INDEX.

MariaDB [test]> show create table ssladmin_certificatemodel\G

*************************** 1. row ***************************

       Table: ssladmin_certificatemodel

Create Table: CREATE TABLE `ssladmin_certificatemodel` (

  `id` int(11) NOT NULL,

  `provider` varchar(30) NOT NULL,

  `orderid` varchar(30) NOT NULL,

  `common_name` varchar(100) NOT NULL,

  `key_alg` varchar(10) NOT NULL,

  `private_key_pem` longtext NOT NULL,

  `certificate_request_pem` longtext NOT NULL,

  `certificate_pem` longtext NOT NULL,

  `status` longtext NOT NULL,

  `created` datetime(6) NOT NULL,

  `modified` datetime(6) NOT NULL,

  `private_key_fernet` longtext NOT NULL,

  PRIMARY KEY (`id`),

  UNIQUE KEY `ssladmin_certificatemode_provider_orderid_common__bc90565b_uniq` (`provider`,`orderid`,`common_name`,`key_alg`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8

1 row in set (0.000 sec)

MariaDB [test]> ALTER TABLE ssladmin_certificatemodel DROP INDEX ssladmin_certificatemode_provider_orderid_common__bc90565b_uniq;

Query OK, 0 rows affected (0.019 sec)

Records: 0  Duplicates: 0  Warnings: 0

MariaDB [test]> SELECT

  `ssladmin_certificatemodel`.`id`,

  `ssladmin_certificatemodel`.`provider`,

  `ssladmin_certificatemodel`.`orderid`,

  `ssladmin_certificatemodel`.`common_name`,

  `ssladmin_certificatemodel`.`key_alg`,

  `ssladmin_certificatemodel`.`private_key_pem`,

  `ssladmin_certificatemodel`.`private_key_fernet`,

  `ssladmin_certificatemodel`.`certificate_request_pem`,

  `ssladmin_certificatemodel`.`certificate_pem`,

  `ssladmin_certificatemodel`.`status`,

  `ssladmin_certificatemodel`.`created`,

  `ssladmin_certificatemodel`.`modified`

FROM

  `ssladmin_certificatemodel`

WHERE

    `ssladmin_certificatemodel`.`id` LIKE '%aazfby%'

    OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%'

ORDER BY

  `ssladmin_certificatemodel`.`id` DESC

ERROR 2013 (HY000): Lost connection to MySQL server during query

On original server I couldn't DROP PK, on copy i can :

MariaDB [test]> ALTER TABLE ssladmin_certificatemodel DROP PRIMARY KEY;

Query OK, 23585 rows affected (2.523 sec)

Records: 23585  Duplicates: 0  Warnings: 0

-- and now this query works good :

MariaDB [test]> SELECT

  `ssladmin_certificatemodel`.`id`,

  `ssladmin_certificatemodel`.`provider`,

  `ssladmin_certificatemodel`.`orderid`,

  `ssladmin_certificatemodel`.`common_name`,

  `ssladmin_certificatemodel`.`key_alg`,

  `ssladmin_certificatemodel`.`private_key_pem`,

  `ssladmin_certificatemodel`.`private_key_fernet`,

  `ssladmin_certificatemodel`.`certificate_request_pem`,

  `ssladmin_certificatemodel`.`certificate_pem`,

  `ssladmin_certificatemodel`.`status`,

  `ssladmin_certificatemodel`.`created`,

  `ssladmin_certificatemodel`.`modified`

FROM

  `ssladmin_certificatemodel`

WHERE

    `ssladmin_certificatemodel`.`id` LIKE '%aazfby%'

    OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%'

ORDER BY

  `ssladmin_certificatemodel`.`id` DESC

...

2 rows in set (0.027 sec)

Aurélien LEQUOY added a comment - 2020-03-10 09:50 - edited I think it's more complicated than that, I clearly identified the problem with PRIMARY INDEX. MariaDB [test]> show create table ssladmin_certificatemodel\G *************************** 1. row *************************** Table : ssladmin_certificatemodel Create Table : CREATE TABLE `ssladmin_certificatemodel` ( `id` int (11) NOT NULL , `provider` varchar (30) NOT NULL , `orderid` varchar (30) NOT NULL , `common_name` varchar (100) NOT NULL , `key_alg` varchar (10) NOT NULL , `private_key_pem` longtext NOT NULL , `certificate_request_pem` longtext NOT NULL , `certificate_pem` longtext NOT NULL , `status` longtext NOT NULL , `created` datetime(6) NOT NULL , `modified` datetime(6) NOT NULL , `private_key_fernet` longtext NOT NULL , PRIMARY KEY (`id`), UNIQUE KEY `ssladmin_certificatemode_provider_orderid_common__bc90565b_uniq` (`provider`,`orderid`,`common_name`,`key_alg`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 1 row in set (0.000 sec) MariaDB [test]> ALTER TABLE ssladmin_certificatemodel DROP INDEX ssladmin_certificatemode_provider_orderid_common__bc90565b_uniq; Query OK, 0 rows affected (0.019 sec) Records: 0 Duplicates: 0 Warnings: 0 MariaDB [test]> SELECT `ssladmin_certificatemodel`.`id`, `ssladmin_certificatemodel`.`provider`, `ssladmin_certificatemodel`.`orderid`, `ssladmin_certificatemodel`.`common_name`, `ssladmin_certificatemodel`.`key_alg`, `ssladmin_certificatemodel`.`private_key_pem`, `ssladmin_certificatemodel`.`private_key_fernet`, `ssladmin_certificatemodel`.`certificate_request_pem`, `ssladmin_certificatemodel`.`certificate_pem`, `ssladmin_certificatemodel`.`status`, `ssladmin_certificatemodel`.`created`, `ssladmin_certificatemodel`.`modified` FROM `ssladmin_certificatemodel` WHERE ( `ssladmin_certificatemodel`.`id` LIKE '%aazfby%' OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%' ) ORDER BY `ssladmin_certificatemodel`.`id` DESC ERROR 2013 (HY000): Lost connection to MySQL server during query On original server I couldn't DROP PK, on copy i can : MariaDB [test]> ALTER TABLE ssladmin_certificatemodel DROP PRIMARY KEY ; Query OK, 23585 rows affected (2.523 sec) Records: 23585 Duplicates: 0 Warnings: 0 -- and now this query works good : MariaDB [test]> SELECT `ssladmin_certificatemodel`.`id`, `ssladmin_certificatemodel`.`provider`, `ssladmin_certificatemodel`.`orderid`, `ssladmin_certificatemodel`.`common_name`, `ssladmin_certificatemodel`.`key_alg`, `ssladmin_certificatemodel`.`private_key_pem`, `ssladmin_certificatemodel`.`private_key_fernet`, `ssladmin_certificatemodel`.`certificate_request_pem`, `ssladmin_certificatemodel`.`certificate_pem`, `ssladmin_certificatemodel`.`status`, `ssladmin_certificatemodel`.`created`, `ssladmin_certificatemodel`.`modified` FROM `ssladmin_certificatemodel` WHERE ( `ssladmin_certificatemodel`.`id` LIKE '%aazfby%' OR `ssladmin_certificatemodel`.`common_name` LIKE '%aazfby%' ) ORDER BY `ssladmin_certificatemodel`.`id` DESC ... 2 rows in set (0.027 sec)

Aurélien LEQUOY added a comment - 2020-03-10 09:58

I already faced to some kind of corruption with InnoDB. My problem there it's the corruption happen on all the servers included in cluster Master/Slave !

Aurélien LEQUOY added a comment - 2020-03-10 09:58 I already faced to some kind of corruption with InnoDB. My problem there it's the corruption happen on all the servers included in cluster Master/Slave !

Marko Mäkelä added a comment - 2020-03-10 10:51

If some columns had been added to the table in MariaDB 10.3 before version 10.3.17, then the corruption of the primary key could be explained by ~~MDEV-19916~~.

If you cannot provide exact steps for repeating the bug, then I do not think that we can do much more unless you are a paying customer of MariaDB Corporation.

Marko Mäkelä added a comment - 2020-03-10 10:51 If some columns had been added to the table in MariaDB 10.3 before version 10.3.17, then the corruption of the primary key could be explained by MDEV-19916 . If you cannot provide exact steps for repeating the bug, then I do not think that we can do much more unless you are a paying customer of MariaDB Corporation.

Aurélien LEQUOY added a comment - 2020-03-26 17:16

I upgraded to lastest of 10.3 with "corrupted" file and it's working fine !

You can close ticket

Aurélien LEQUOY added a comment - 2020-03-26 17:16 I upgraded to lastest of 10.3 with "corrupted" file and it's working fine ! You can close ticket

MariaDB Server

InnoDB: Assertion failure rem0rec.cc 1878

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration