Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12179

Per-engine mysql.gtid_slave_pos tables

Details

    • 10.3.1-1

    Description

      Implement that server reads/updates the GTID position using multiple tables
      mysql.gtid_slave_pos_XXX, each using a different storage engine. Replicated
      transactions update the version of the table in the same engine, if
      available. This avoids the overhead of cross-engine transactions on servers
      where multiple storage engines are in use at the same time (but not in the
      same transactions).

      See mailing list thread: https://lists.launchpad.net/maria-developers/msg10453.html

      Attachments

        Issue Links

          Activity

            MDEV-11657 suggests an alternative approach (with a much wider scope): storing the transaction state in a global table. This would require substantial changes to transactional storage engines, but it would avoid the need of using binlog’s XA 2PC mechanism for multi-engine transactions.
            With MDEV-11657, persistent transactions would be registered in only one place, with GTID, XID and everything in a single place.

            marko Marko Mäkelä added a comment - MDEV-11657 suggests an alternative approach (with a much wider scope): storing the transaction state in a global table. This would require substantial changes to transactional storage engines, but it would avoid the need of using binlog’s XA 2PC mechanism for multi-engine transactions. With MDEV-11657 , persistent transactions would be registered in only one place, with GTID, XID and everything in a single place.

            Reading in the associated MDEV-11655:

            "There will be a common redo log for and all crash-safe storage engines."

            The lack of a common redo log indeed is the source of endless amounts of
            complexity and performance penalties, in replication and probably elsewhere
            also. Resolving this would be very interesting, (and of much wider scope, as
            you said).

            The main point of per-engine mysql.gtid_slave_pos table is to allow
            replication transactional commits with just one transactional log (if
            --log-slave-updates=0). So if my understanding is correct, common redo log
            would be a pre-requisite for MDEV-11657 to be a full alternative to this
            feature?

            It was interesting to see the MDEV-11657 and related designs, I have not
            seen any mention of them before.

            knielsen Kristian Nielsen added a comment - Reading in the associated MDEV-11655 : "There will be a common redo log for and all crash-safe storage engines." The lack of a common redo log indeed is the source of endless amounts of complexity and performance penalties, in replication and probably elsewhere also. Resolving this would be very interesting, (and of much wider scope, as you said). The main point of per-engine mysql.gtid_slave_pos table is to allow replication transactional commits with just one transactional log (if --log-slave-updates=0). So if my understanding is correct, common redo log would be a pre-requisite for MDEV-11657 to be a full alternative to this feature? It was interesting to see the MDEV-11657 and related designs, I have not seen any mention of them before.

            An interesting property of the current patch:

            If one does

             alter table mysql.gtid_slave_pos engine=innobase;
            

            (suppose the above changes the storage engine of gtid_slave_pos from MyISAM to InnoDB)
            then Transactions_gtid_foreign_engine counter will continue to be incremented as if ALTER TABLE didn't happen. STOP SLAVE ; START SLAVE fixes the issue.

            psergei Sergei Petrunia added a comment - An interesting property of the current patch: If one does alter table mysql.gtid_slave_pos engine=innobase; (suppose the above changes the storage engine of gtid_slave_pos from MyISAM to InnoDB) then Transactions_gtid_foreign_engine counter will continue to be incremented as if ALTER TABLE didn't happen. STOP SLAVE ; START SLAVE fixes the issue.

            Another thing to investigate: I have set

            gtid_pos_auto_engines='Aria,HEAP,TokuDB';
            

            and I can observe gtid_slave_pos_TokuDB table to be created when a transaction involving a TokuDB table is replayed on the slave.

            However, I dont observe a gtid_slave_pos_Aria table to be created when a transaction with Aria table is replicated. The Aria table does have TRANSACTIONAL=1 flag.

            psergei Sergei Petrunia added a comment - Another thing to investigate: I have set gtid_pos_auto_engines='Aria,HEAP,TokuDB'; and I can observe gtid_slave_pos_TokuDB table to be created when a transaction involving a TokuDB table is replayed on the slave. However, I dont observe a gtid_slave_pos_Aria table to be created when a transaction with Aria table is replicated. The Aria table does have TRANSACTIONAL=1 flag.

            Aria (or HEAP, or MyISAM) is not transactional, in the sense that it does
            not keep a transaction open.

            So after an event has been executed by the slave, and the COMMIT event runs,
            there is no open transaction involving the Aria storage engine. Therefore,
            record_gtid() cannot choose an Aria table. Nor would there be any benefit,
            since whatever is done, record_gtid() needs to commit a new transaction.

            One could make the default mysql.gtid_slave_pos table use Aria, if desired.

            knielsen Kristian Nielsen added a comment - Aria (or HEAP, or MyISAM) is not transactional, in the sense that it does not keep a transaction open. So after an event has been executed by the slave, and the COMMIT event runs, there is no open transaction involving the Aria storage engine. Therefore, record_gtid() cannot choose an Aria table. Nor would there be any benefit, since whatever is done, record_gtid() needs to commit a new transaction. One could make the default mysql.gtid_slave_pos table use Aria, if desired.

            knielsen, thanks for clarification

            psergei Sergei Petrunia added a comment - knielsen , thanks for clarification

            Ok I have no issues with the patch (Review Done), except the question of how to package this, which I have described here: https://lists.launchpad.net/maria-developers/msg10779.html .

            psergei Sergei Petrunia added a comment - Ok I have no issues with the patch (Review Done), except the question of how to package this, which I have described here: https://lists.launchpad.net/maria-developers/msg10779.html .

            Merged the code to MariaDB 10.3 (not automatic but nothing complex): https://github.com/MariaDB/server/tree/bb-10.3-mdev12179

            psergei Sergei Petrunia added a comment - Merged the code to MariaDB 10.3 (not automatic but nothing complex): https://github.com/MariaDB/server/tree/bb-10.3-mdev12179

            Pushed to 10.3.1.

            For now, the default for --gtid-pos-auto-engines is empty. This can be easily changed if desired, just update the value in sql/mysqld.cc.

            knielsen Kristian Nielsen added a comment - Pushed to 10.3.1. For now, the default for --gtid-pos-auto-engines is empty. This can be easily changed if desired, just update the value in sql/mysqld.cc.
            jeanfrancois.gagne Jean-François Gagné added a comment - Leaving some links about this work here: http://kristiannielsen.livejournal.com/19223.html https://jfg-mysql.blogspot.com/2017/05/better-replication-with-many-storage-engine.html

            People

              knielsen Kristian Nielsen
              knielsen Kristian Nielsen
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.