Marko Mäkelä added a comment - 2017-08-07 11:23 Andrei, can you debug this further? I tried with the following patch to both storage/innobase and storage/xtradb, and it will fail in binlog: diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 526836646d2..031ab54d4c5 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -14446,6 +14446,10 @@ innobase_commit_by_xid( DBUG_ASSERT(hton == innodb_hton_ptr); + if (high_level_read_only) { + return(XAER_RMFAIL); + } + trx = trx_get_trx_by_xid(xid); if (trx) { @@ -14473,6 +14477,10 @@ innobase_rollback_by_xid( DBUG_ASSERT(hton == innodb_hton_ptr); + if (high_level_read_only) { + return(XAER_RMFAIL); + } + trx = trx_get_trx_by_xid(xid); if (trx) { The failure was like this: 170807 14:19:29 [Note] Heuristic crash recovery mode 2017-08-07 14:19:29 7fc95dcc0540 InnoDB: Starting recovery for XA transactions... 2017-08-07 14:19:29 7fc95dcc0540 InnoDB: Transaction 1283 in prepared state after recovery 2017-08-07 14:19:29 7fc95dcc0540 InnoDB: Transaction contains changes to 1 rows 2017-08-07 14:19:29 7fc95dcc0540 InnoDB: 1 transactions in prepared state after recovery 170807 14:19:29 [Note] Found 1 prepared transaction(s) in InnoDB 170807 14:19:29 [Note] rollback xid 'MySQLXid\1\0\0\0\0\0\0\0\26\0\0\0\0\0\0\0' 170807 14:19:29 [Note] Please restart mysqld without --tc-heuristic-recover mysqld: /mariadb/10.0/sql/log.cc:3258: bool MYSQL_BIN_LOG::open(const char *, enum_log_type, const char *, enum cache_type, ulong, bool, bool): Assertion `(&(&LOCK_log)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&LOCK_log)->m_mutex)->thread)' failed. 170807 14:19:29 [ERROR] mysqld got signal 6 ; Stack trace: #7 0x00007fc95c446f12 in __GI___assert_fail (assertion=0x10c994a "(&(&LOCK_log)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&LOCK_log)->m_mutex)->thread)", file=0x10c8c0e "/mariadb/10.0/sql/log.cc", line=3258, function=0x10c99af "bool MYSQL_BIN_LOG::open(const char *, enum_log_type, const char *, enum cache_type, ulong, bool, bool)") at assert.c:101 #8 0x0000000000934a7c in MYSQL_BIN_LOG::open (this=0x18b7820 <mysql_bin_log>, log_name=0x3c13d98 "master-bin", log_type_arg=LOG_BIN, new_name=0x0, io_cache_type_arg=WRITE_CACHE, max_size_arg=1073741824, null_created_arg=false, need_mutex=true) at /mariadb/10.0/sql/log.cc:3258 #9 0x0000000000943bde in MYSQL_BIN_LOG::open (this=0x18b7820 <mysql_bin_log>, opt_name=0x3c13d98 "master-bin") at /mariadb/10.0/sql/log.cc:8964 #10 0x000000000057e8f2 in init_server_components () at /mariadb/10.0/sql/mysqld.cc:4905 #11 0x000000000057b5c4 in mysqld_main (argc=137, argv=0x3af0388) at /mariadb/10.0/sql/mysqld.cc:5371 #12 0x00000000005782c2 in main (argc=22, argv=0x7ffc17d97628) at /mariadb/10.0/sql/main.cc:25 It appears that a mutex is not being released in an error handling code path.

Marko Mäkelä added a comment - 2017-08-15 11:51 Also, it looks like we should revert part of this InnoDB change in MySQL 5.0.3 (making innodb_force_recovery=1 and innodb_force_recovery=2 initiate rollback of a transaction that reached the XA PREPARE state before previous crash or shutdown). It seems that the option tc_heuristic_recover=ROLLBACK was introduced for exactly this purpose; there is no need to convolute the logic of an InnoDB parameter.

Andrei Elkin added a comment - 2017-08-21 10:03 Elena, hello. There are small changes to mtr that's why you. I am asking Marko to review the rest of this work. The latest patch is pushed to bb-10.0-mdev_13437-andrei. Thanks! Andrei

Elena Stepanova added a comment - 2017-08-21 13:32 The only change I see in MTR is exporting of an extra variable. It's not dangerous, so if it works for you, I have no objections. In general, we should invent a better way to fail-start the server, but it doesn't have to be done now. Re-assigning to Marko for real review.

Marko Mäkelä added a comment - 2017-08-21 13:41 Elkin , the new logic looks OK to me. I only requested some minor clean-up to both the code and the test case.

Marko Mäkelä added a comment - 2017-08-24 11:27 Looks OK now; the newest revision to the test even works with Valgrind: main.tc_heuristic_recover 'innodb_plugin' [ pass ] 54201 main.tc_heuristic_recover 'xtradb' [ pass ] 73606 valgrind_report [ pass ]