Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12179

Per-engine mysql.gtid_slave_pos tables

Details

    • 10.3.1-1

    Description

      Implement that server reads/updates the GTID position using multiple tables
      mysql.gtid_slave_pos_XXX, each using a different storage engine. Replicated
      transactions update the version of the table in the same engine, if
      available. This avoids the overhead of cross-engine transactions on servers
      where multiple storage engines are in use at the same time (but not in the
      same transactions).

      See mailing list thread: https://lists.launchpad.net/maria-developers/msg10453.html

      Attachments

        Issue Links

          Activity

            knielsen Kristian Nielsen created issue -
            knielsen Kristian Nielsen made changes -
            Field Original Value New Value
            Description Implement that server reads/updates the GTID position using multiple tables
            mysql.gtid_slave_pos_XXX, each using a different storage engine. Replicated
            transactions update the version of the table in the same engine, if
            available. This avoids the overhead of cross-engine transactions on servers
            where multiple storage engines are in use at the same time (but not in the
            same transactions).
            Implement that server reads/updates the GTID position using multiple tables
            mysql.gtid_slave_pos_XXX, each using a different storage engine. Replicated
            transactions update the version of the table in the same engine, if
            available. This avoids the overhead of cross-engine transactions on servers
            where multiple storage engines are in use at the same time (but not in the
            same transactions).

            See mailing list thread: https://lists.launchpad.net/maria-developers/msg10453.html

            MDEV-11657 suggests an alternative approach (with a much wider scope): storing the transaction state in a global table. This would require substantial changes to transactional storage engines, but it would avoid the need of using binlog’s XA 2PC mechanism for multi-engine transactions.
            With MDEV-11657, persistent transactions would be registered in only one place, with GTID, XID and everything in a single place.

            marko Marko Mäkelä added a comment - MDEV-11657 suggests an alternative approach (with a much wider scope): storing the transaction state in a global table. This would require substantial changes to transactional storage engines, but it would avoid the need of using binlog’s XA 2PC mechanism for multi-engine transactions. With MDEV-11657 , persistent transactions would be registered in only one place, with GTID, XID and everything in a single place.
            marko Marko Mäkelä made changes -

            Reading in the associated MDEV-11655:

            "There will be a common redo log for and all crash-safe storage engines."

            The lack of a common redo log indeed is the source of endless amounts of
            complexity and performance penalties, in replication and probably elsewhere
            also. Resolving this would be very interesting, (and of much wider scope, as
            you said).

            The main point of per-engine mysql.gtid_slave_pos table is to allow
            replication transactional commits with just one transactional log (if
            --log-slave-updates=0). So if my understanding is correct, common redo log
            would be a pre-requisite for MDEV-11657 to be a full alternative to this
            feature?

            It was interesting to see the MDEV-11657 and related designs, I have not
            seen any mention of them before.

            knielsen Kristian Nielsen added a comment - Reading in the associated MDEV-11655 : "There will be a common redo log for and all crash-safe storage engines." The lack of a common redo log indeed is the source of endless amounts of complexity and performance penalties, in replication and probably elsewhere also. Resolving this would be very interesting, (and of much wider scope, as you said). The main point of per-engine mysql.gtid_slave_pos table is to allow replication transactional commits with just one transactional log (if --log-slave-updates=0). So if my understanding is correct, common redo log would be a pre-requisite for MDEV-11657 to be a full alternative to this feature? It was interesting to see the MDEV-11657 and related designs, I have not seen any mention of them before.
            ratzpo Rasmus Johansson (Inactive) made changes -
            Sprint 10.3.1-1 [ 164 ]
            serg Sergei Golubchik made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.3 [ 22126 ]
            psergei Sergei Petrunia made changes -
            Assignee Kristian Nielsen [ knielsen ] Sergei Petrunia [ psergey ]
            serg Sergei Golubchik made changes -
            Priority Critical [ 2 ] Major [ 3 ]

            An interesting property of the current patch:

            If one does

             alter table mysql.gtid_slave_pos engine=innobase;
            

            (suppose the above changes the storage engine of gtid_slave_pos from MyISAM to InnoDB)
            then Transactions_gtid_foreign_engine counter will continue to be incremented as if ALTER TABLE didn't happen. STOP SLAVE ; START SLAVE fixes the issue.

            psergei Sergei Petrunia added a comment - An interesting property of the current patch: If one does alter table mysql.gtid_slave_pos engine=innobase; (suppose the above changes the storage engine of gtid_slave_pos from MyISAM to InnoDB) then Transactions_gtid_foreign_engine counter will continue to be incremented as if ALTER TABLE didn't happen. STOP SLAVE ; START SLAVE fixes the issue.

            Another thing to investigate: I have set

            gtid_pos_auto_engines='Aria,HEAP,TokuDB';
            

            and I can observe gtid_slave_pos_TokuDB table to be created when a transaction involving a TokuDB table is replayed on the slave.

            However, I dont observe a gtid_slave_pos_Aria table to be created when a transaction with Aria table is replicated. The Aria table does have TRANSACTIONAL=1 flag.

            psergei Sergei Petrunia added a comment - Another thing to investigate: I have set gtid_pos_auto_engines='Aria,HEAP,TokuDB'; and I can observe gtid_slave_pos_TokuDB table to be created when a transaction involving a TokuDB table is replayed on the slave. However, I dont observe a gtid_slave_pos_Aria table to be created when a transaction with Aria table is replicated. The Aria table does have TRANSACTIONAL=1 flag.

            Aria (or HEAP, or MyISAM) is not transactional, in the sense that it does
            not keep a transaction open.

            So after an event has been executed by the slave, and the COMMIT event runs,
            there is no open transaction involving the Aria storage engine. Therefore,
            record_gtid() cannot choose an Aria table. Nor would there be any benefit,
            since whatever is done, record_gtid() needs to commit a new transaction.

            One could make the default mysql.gtid_slave_pos table use Aria, if desired.

            knielsen Kristian Nielsen added a comment - Aria (or HEAP, or MyISAM) is not transactional, in the sense that it does not keep a transaction open. So after an event has been executed by the slave, and the COMMIT event runs, there is no open transaction involving the Aria storage engine. Therefore, record_gtid() cannot choose an Aria table. Nor would there be any benefit, since whatever is done, record_gtid() needs to commit a new transaction. One could make the default mysql.gtid_slave_pos table use Aria, if desired.

            knielsen, thanks for clarification

            psergei Sergei Petrunia added a comment - knielsen , thanks for clarification

            Ok I have no issues with the patch (Review Done), except the question of how to package this, which I have described here: https://lists.launchpad.net/maria-developers/msg10779.html .

            psergei Sergei Petrunia added a comment - Ok I have no issues with the patch (Review Done), except the question of how to package this, which I have described here: https://lists.launchpad.net/maria-developers/msg10779.html .

            Merged the code to MariaDB 10.3 (not automatic but nothing complex): https://github.com/MariaDB/server/tree/bb-10.3-mdev12179

            psergei Sergei Petrunia added a comment - Merged the code to MariaDB 10.3 (not automatic but nothing complex): https://github.com/MariaDB/server/tree/bb-10.3-mdev12179
            serg Sergei Golubchik made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            knielsen Kristian Nielsen made changes -
            Assignee Sergei Petrunia [ psergey ] Kristian Nielsen [ knielsen ]

            Pushed to 10.3.1.

            For now, the default for --gtid-pos-auto-engines is empty. This can be easily changed if desired, just update the value in sql/mysqld.cc.

            knielsen Kristian Nielsen added a comment - Pushed to 10.3.1. For now, the default for --gtid-pos-auto-engines is empty. This can be easily changed if desired, just update the value in sql/mysqld.cc.
            knielsen Kristian Nielsen made changes -
            Fix Version/s 10.3.1 [ 22532 ]
            Fix Version/s 10.3 [ 22126 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            elenst Elena Stepanova made changes -
            jeanfrancois.gagne Jean-François Gagné added a comment - Leaving some links about this work here: http://kristiannielsen.livejournal.com/19223.html https://jfg-mysql.blogspot.com/2017/05/better-replication-with-many-storage-engine.html
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 79843 ] MariaDB v4 [ 133145 ]
            knielsen Kristian Nielsen made changes -

            People

              knielsen Kristian Nielsen
              knielsen Kristian Nielsen
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.