Kenneth Dyer (Inactive) added a comment - 2018-12-10 03:38 - edited Hi valerii , I updated the --log-tc listing on the mysqld options page to show tc.log as the default value. But, I ran into a couple issues while researching for the rest of the ticket: wsrep is not an XA-capable storage engine. The Galera Replication plugin itself doesn't play nice with XA operations, causing conflicts and rollbacks. Codership documents this on the Limitations page in their Docs. We have it copied to our Galera limitations page as well as called out specifically on the XA Transactions docs. So, unless something has changed upstream, these operations still aren't supported. wsrep is also not a storage engine. Not sure what's up with the SHOW PLUGINS output, but Galera replication requires InnoDB tables. So, presumably even in a hypothetical case where Codership rolled out XA support, it wouldn't cause issues like what's happening on MDEV-9214 , since there'd still be no entry for it on SHOW ENGINES . Does that cover or mitigate the issue?

Valerii Kravchuk added a comment - 2018-12-10 07:21 What is NOT covered is that in MariaDB 10.1+ we'll have tc.log created in the datadir and maintained by default, with just InnoDB or XtraDB as the onlt XA-capable storage engine, even if wsrep_on is set to OFF and InnoDB is the only XA-capable storage engine listed in SHOW ENGINES. That's because of WSREP support we have included. It does not matter if Galera itself supports XA transactions completely, correctly or not, do they plan to support it or not. tc.log is created and maintained, and it leads to overhead discussed in other MDEVs (like https://jira.mariadb.org/browse/MDEV-16509 ). This must be documented (or fixed). Fix for MDEV-9214 (you can check commit) just changed error message and few tests to reflect this, but had not removed the creation of the file.

Valerii Kravchuk added a comment - 2019-03-05 15:22 I do not see any link to any KB article that documents Galera as XA-capable storage engine or explains that tc.log is created and maintained by default in 10.1+ (correct me if I missed it, but --log-tc option description is NOT about this at all). So if you ask me, the problem is not yet fixed and this MDEV can NOT be closed.

Monty Taylor added a comment - 2019-03-05 16:25 I agree with @Valerii, the underlying issue has not be addressed. This just hit me last night - we restarted some mariadb containers as part of a docker-compose action, and were left with mariadb unable to start because it was unhappy with the tc.log file. We're only using innodb, so there is nothing useful in the file. Deleting it caused mariadb to be able to start. What would be better is if tc.log weren't created if it isn't needed - or if startup could be successful after a restart without the user needing to delete the tc.log file. I mean, I can make things always delete tc.log as part of restarting ... but that seems a bit unfortunate.

Geoff Montee (Inactive) added a comment - 2019-03-05 22:44 - edited valerii , How's the note here?: Starting with MariaDB 10.1, the MySQL-wsrep patch that powers MariaDB Galera Cluster is enabled by default on most MariaDB builds on Linux. Internally, the wsrep plugin is considered an XA-capable storage engine, because it supports MariaDB's internal two-phase commit API. Therefore, these builds have multiple XA-capable storage engines by default, even if the only "real" storage engine that supports external XA transactions enabled on these builds by default is InnoDB. See MDEV-9214 and MDEV-16442 for more information. https://mariadb.com/kb/en/library/transaction-coordinator-log-overview/ https://mariadb.com/kb/en/library/xa-transactions/#internal-xa-vs-external-xa

Valerii Kravchuk added a comment - 2019-03-06 05:19 GeoffMontee , Yes, these texts cover the topic of this bug report properly. Sorry that I missed them. Based on other comments I think we still need to explain users better (maybe in some notes) why they may unexpectedly see messages about tc.log upon startup in 10.1, when they should just delete it and what are possible performance implications (see MDEV-16509 etc).

Geoff Montee (Inactive) added a comment - 2019-03-06 05:27 No worries, valerii . You didn't miss anything. I just added that text today. I just created the entire Transaction Coordinator Log category today. Yeah, those other points are good to ask. We might need input from Codership for those.

Geoff Montee (Inactive) added a comment - 2019-03-06 05:58 Interestingly, in MariaDB 10.4, the wsrep plugin type has changed from MYSQL_STORAGE_ENGINE_PLUGIN to MYSQL_REPLICATION_PLUGIN. Here's the definition in the 10.3 branch: maria_declare_plugin(wsrep) { MYSQL_STORAGE_ENGINE_PLUGIN, &wsrep_storage_engine, "wsrep", "Codership Oy", "A pseudo storage engine to represent transactions in multi-master " "synchornous replication", PLUGIN_LICENSE_GPL, wsrep_hton_init, /* Plugin Init */ NULL, /* Plugin Deinit */ 0x0100 /* 1.0 */, NULL, /* status variables */ NULL, /* system variables */ "1.0", /* string version */ MariaDB_PLUGIN_MATURITY_STABLE /* maturity */ } maria_declare_plugin_end; https://github.com/MariaDB/server/blob/57e0da50bbef8164635317785b67dd468a908327/sql/wsrep_hton.cc#L641 And here's the definition in the 10.4 branch: maria_declare_plugin(wsrep) { MYSQL_REPLICATION_PLUGIN, &wsrep_plugin, "wsrep", "Codership Oy", "Wsrep replication plugin", PLUGIN_LICENSE_GPL, wsrep_plugin_init, wsrep_plugin_deinit, 0x0100, NULL, /* Status variables */ NULL, /* System variables */ "1.0", /* Version (string) */ MariaDB_PLUGIN_MATURITY_STABLE /* Maturity */ } maria_declare_plugin_end; https://github.com/MariaDB/server/blob/41a9a677ebfa0133df9abb48adc33a3555ecb490/sql/wsrep_plugin.cc#L37 Since it's not even a storage engine in MariaDB 10.4, I doubt that it can be a XA-capable storage engine in that version. valerii , do you have a MariaDB 10.4 instance on-hand to see if tc.log is still created when InnoDB is the only XA-capable storage engine?

Valerii Kravchuk added a comment - 2019-03-06 08:26 From what I see with MariaDB 10.4.4 built from GitHub today WSREP is indeed REPLICATION plugin now and tc.log file is NOT created: ... [openxs@fc23 maria10.4]$ ls -l data total 110636 -rw-rw----. 1 openxs openxs 24576 Mar 6 10:18 aria_log.00000001 -rw-rw----. 1 openxs openxs 52 Mar 6 10:18 aria_log_control -rw-rw----. 1 openxs openxs 972 Mar 6 10:18 ib_buffer_pool -rw-rw----. 1 openxs openxs 12582912 Mar 6 10:18 ibdata1 -rw-rw----. 1 openxs openxs 50331648 Mar 6 10:18 ib_logfile0 -rw-rw----. 1 openxs openxs 50331648 Mar 6 10:18 ib_logfile1 drwx------. 2 openxs openxs 4096 Mar 6 10:18 mysql drwx------. 2 openxs openxs 4096 Mar 6 10:18 performance_schema drwxrwxr-x. 2 openxs openxs 4096 Mar 6 10:09 test [openxs@fc23 maria10.4]$ bin/mysqld_safe --no-defaults --port=3308 --socket=/tmp/mariadb.sock & [1] 31165 [openxs@fc23 maria10.4]$ 190306 10:19:04 mysqld_safe Logging to '/home/openxs/dbs/maria10.4/data/fc23.err'. 190306 10:19:04 mysqld_safe Starting mysqld daemon with databases from /home/openxs/dbs/maria10.4/data   [openxs@fc23 maria10.4]$ bin/mysql -uroot --socket=/tmp/mariadb.sock ERROR 1698 (28000): Access denied for user 'root'@'localhost' [openxs@fc23 maria10.4]$ bin/mysql --socket=/tmp/mariadb.sock Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 10.4.4-MariaDB Source distribution   Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.   Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.   MariaDB [(none)]> select current_user(), @@datadir; +------------------+----------------------------------+ | current_user() | @@datadir | +------------------+----------------------------------+ | openxs@localhost | /home/openxs/dbs/maria10.4/data/ | +------------------+----------------------------------+ 1 row in set (0.000 sec)   MariaDB [(none)]> show engines; +--------------------+---------+-------------------------------------------------------------------------------------------------+--------------+------+------------+ | Engine | Support | Comment | Transactions | XA | Savepoints | +--------------------+---------+-------------------------------------------------------------------------------------------------+--------------+------+------------+ | CSV | YES | Stores tables as CSV files | NO | NO | NO | | MRG_MyISAM | YES | Collection of identical MyISAM tables | NO | NO | NO | | MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO | | Aria | YES | Crash-safe tables with MyISAM heritage. Used for internal temporary tables and privilege tables | NO | NO | NO | | MyISAM | YES | Non-transactional engine with good performance and small data footprint | NO | NO | NO | | SEQUENCE | YES | Generated tables filled with sequential values | YES | NO | YES | | InnoDB | DEFAULT | Supports transactions, row-level locking, foreign keys and encryption for tables | YES | YES | YES | | PERFORMANCE_SCHEMA | YES | Performance Schema | NO | NO | NO | +--------------------+---------+-------------------------------------------------------------------------------------------------+--------------+------+------------+ 8 rows in set (0.000 sec)   MariaDB [(none)]> show plugins; +-------------------------------+----------+--------------------+---------+---------+ | Name | Status | Type | Library | License | +-------------------------------+----------+--------------------+---------+---------+ | binlog | ACTIVE | STORAGE ENGINE | NULL | GPL | | mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL | | mysql_old_password | ACTIVE | AUTHENTICATION | NULL | GPL | | wsrep | ACTIVE | REPLICATION | NULL | GPL | | CSV | ACTIVE | STORAGE ENGINE | NULL | GPL | | MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL | | Aria | ACTIVE | STORAGE ENGINE | NULL | GPL | | MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL | | MRG_MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL | | CLIENT_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INDEX_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | TABLE_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | USER_STATISTICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | SQL_SEQUENCE | ACTIVE | STORAGE ENGINE | NULL | GPL | | InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL | | INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_LOCKS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_LOCK_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_PER_INDEX | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_CMP_PER_INDEX_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_METRICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_DEFAULT_STOPWORD | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_BEING_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_CONFIG | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_INDEX_CACHE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_FT_INDEX_TABLE | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLESTATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_COLUMNS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FIELDS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FOREIGN | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_FOREIGN_COLS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_DATAFILES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_VIRTUAL | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_MUTEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_SYS_SEMAPHORE_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | INNODB_TABLESPACES_ENCRYPTION | ACTIVE | INFORMATION SCHEMA | NULL | BSD | | INNODB_TABLESPACES_SCRUBBING | ACTIVE | INFORMATION SCHEMA | NULL | BSD | | PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL | | SEQUENCE | ACTIVE | STORAGE ENGINE | NULL | GPL | | unix_socket | ACTIVE | AUTHENTICATION | NULL | GPL | | FEEDBACK | DISABLED | INFORMATION SCHEMA | NULL | GPL | | user_variables | ACTIVE | INFORMATION SCHEMA | NULL | GPL | | partition | ACTIVE | STORAGE ENGINE | NULL | GPL | +-------------------------------+----------+--------------------+---------+---------+ 54 rows in set (0.001 sec)   MariaDB [(none)]> \! ls -l data/*.log ls: cannot access 'data/*.log': No such file or directory MariaDB [(none)]> \! ls -l data/ total 122932 -rw-rw----. 1 openxs openxs 24576 Mar 6 10:18 aria_log.00000001 -rw-rw----. 1 openxs openxs 52 Mar 6 10:18 aria_log_control -rw-rw----. 1 openxs openxs 2092 Mar 6 10:19 fc23.err -rw-rw----. 1 openxs openxs 6 Mar 6 10:19 fc23.pid -rw-rw----. 1 openxs openxs 972 Mar 6 10:18 ib_buffer_pool -rw-rw----. 1 openxs openxs 12582912 Mar 6 10:18 ibdata1 -rw-rw----. 1 openxs openxs 50331648 Mar 6 10:19 ib_logfile0 -rw-rw----. 1 openxs openxs 50331648 Mar 6 10:18 ib_logfile1 -rw-rw----. 1 openxs openxs 12582912 Mar 6 10:19 ibtmp1 -rw-rw----. 1 openxs openxs 0 Mar 6 10:19 multi-master.info drwx------. 2 openxs openxs 4096 Mar 6 10:18 mysql drwx------. 2 openxs openxs 4096 Mar 6 10:18 performance_schema drwxrwxr-x. 2 openxs openxs 4096 Mar 6 10:09 test MariaDB [(none)]> So, this change of 10.4+ is also something to document properly.

Geoff Montee (Inactive) added a comment - 2019-03-06 18:46 Thanks, valerii . How's this?: MariaDB Galera Cluster builds include a built-in plugin called wsrep. Prior to MariaDB 10.4.3, this plugin was internally considered an XA-capable storage engine. Consequently, these MariaDB Galera Cluster builds have multiple XA-capable storage engines by default, even if the only "real" storage engine that supports external XA transactions enabled on these builds by default is InnoDB. Therefore, when using one these builds MariaDB would be forced to use a transaction coordinator log by default, which could have performance implications. For example, MDEV-16509 describes performance problems where MariaDB Galera Cluster actually performs better when the binary log is enabled. It is possible that this is caused by the fact that MariaDB is forced to use the memory-mapped file-based transaction coordinator log in this case, which may not perform as well. This became a bigger issue in MariaDB 10.1 when the MySQL-wsrep patch that powers MariaDB Galera Cluster was enabled on most MariaDB builds on Linux by default. Consequently, this built-in wsrep plugin would exist on those MariaDB builds on Linux by default. Therefore, MariaDB users might pay a performance penalty, even if they never actually intended to use the MariaDB Galera Cluster features included in MariaDB 10.1. In MariaDB 10.4.3 and later, the built-in wsrep plugin has been changed to a replication plugin. Therefore, it is no longer considered an XA-capable storage engine, so it no longer forces MariaDB to use a transaction coordinator log by default. See MDEV-9214 and MDEV-16442 for more information. https://mariadb.com/kb/en/library/transaction-coordinator-log-overview/#mariadb-galera-cluster

Valerii Kravchuk added a comment - 2019-03-06 19:03 The above is good for my original request, but does not cover concern from the comment by Monty Taylor - when one may have to and can safely remove tc.log upon startup.

Geoff Montee (Inactive) added a comment - 2019-03-06 23:44 Hi valerii , If he saw errors like this: 2018-11-30 23:08:49 140046048638848 [ERROR] Recovery failed! You must enable exactly 3 storage engines that support two-phase commit protocol 2018-11-30 23:08:49 140046048638848 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback} 2018-11-30 23:08:49 140046048638848 [ERROR] Can't init tc log 2018-11-30 23:08:49 140046048638848 [ERROR] Aborting Then that's already documented here: https://mariadb.com/kb/en/library/transaction-coordinator-log-overview/#you-must-enable-exactly-n-storage-engines mordred , Can you please share what errors you saw?

Monty Taylor added a comment - 2019-03-07 00:00 I got a different error, basically the one reported here: https://github.com/docker-library/mariadb/issues/201 The "Bad Magic Number" being the key. We've made changes similar to the recommendations in that issue in our deployment, and deleting the tc.log worked as expected since we were only using innodb.

Geoff Montee (Inactive) added a comment - 2019-03-07 00:20 Deleting tc.log is a temporary fix. It sounds like this problem can happen every time a docker instance is rebooted. Which of the specific changes mentioned there fixed the problem with the docker instance permanently for you? i.e.: Running mysql_upgrade? Explicitly specifying a major release (like "mariadb: 10.3") in the compose file? Setting a "stop timeout" for the container? Something else?

Geoff Montee (Inactive) added a comment - 2019-03-07 03:53 mordred , I went ahead and created this section: If you are using the memory-mapped file-based transaction coordinator log, then it is possible to see errors like the following: 2018-09-19 4:29:31 0 [Note] Recovering after a crash using tc.log 2018-09-19 4:29:31 0 [ERROR] Bad magic header in tc log 2018-09-19 4:29:31 0 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover= Unknown macro: {commit|rollback} 2018-09-19 4:29:31 0 [ERROR] Can't init tc log 2018-09-19 4:29:31 0 [ERROR] Aborting This means that the header of the memory-mapped file-based transaction coordinator log is corrupt. To recover from this error, delete the file defined by the --log-tc server option, and then restart the server with the --tc-heuristic-recover option set. This issue is known to occur when using docker. In that case, the problem may be caused by using a MariaDB container version with a data directory from a different MariaDB or MySQL version. Therefore, some potential fixes are: Pinning the docker instance to a specific MariaDB version in the docker compose file, so that it consistently uses the same version. Running mysql_upgrade to ensure that the data directory is upgraded to match the server version. See this docker issue for more information. https://mariadb.com/kb/en/library/transaction-coordinator-log-overview/#bad-magic-header-in-tc-log