Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22327

Feature request: replicate_do_engine/replicate_ignore_engine

Details

    • Task
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • Replication
    • None

    Description

      When using remote table engines, like FEDERATEDX or CONNECT, usually one wouldn't want to have master side changes to be replicated to slaves, as these would just lead to duplicate actions on the remote tables by default.

      In edge cases though, where e.g. CONNECTION parameters for CONNECT tables on the slave side have been altered to point to different servers than on the master, replication for such tables may actually be wanted though.

      Right now this could only be solved by using replicate_do/ignore_table rules, and having a consistent table naming scheme that would allow for CONNECT tables to be easily identified, or by adding per-table rules one by one.

      So I'd request to be able to outright ignore CONNECT or FEDERATEDX tables by default, by having a new filter like

      replicate_ignore_engine=CONNECT,FEDERATEDX

      This would obviously be a ROW-only filter.

      Attachments

        Issue Links

          Activity

            hholzgra Hartmut Holzgraefe created issue -
            serg Sergei Golubchik made changes -
            Field Original Value New Value
            maxmether Max Mether made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Ralf Gebhardt [ ralf.gebhardt@mariadb.com ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Description When using remote table engines, like FEDERATEDX or CONNECT, usually one wouldn't want to have master side changes to be replicated to slaves, as these would just lead to duplicate actions on the remote tables by default.

            In edge cases though, where e.g. CONNECTION parameters for CONNECT tables ont he slave side have been altered to point to different servers than on the master, replication for such tables may actually be wanted though.

            Right now this could only be solved by using replicate_do/ignore_table rules, and having a consistent table naming scheme that would allow for CONNECT tables to be easily identified, or by adding per-table rules one by one.

            So I'd request to be able to outright ignore CONNECT or FEDERATEDX tables by default, by having a new filter like

            {{replicate_ignore_engine=CONNECT,FEDERATEDX}}

            This would obviously be a ROW-only filter.
            When using remote table engines, like FEDERATEDX or CONNECT, usually one wouldn't want to have master side changes to be replicated to slaves, as these would just lead to duplicate actions on the remote tables by default.

            In edge cases though, where e.g. CONNECTION parameters for CONNECT tables on the slave side have been altered to point to different servers than on the master, replication for such tables may actually be wanted though.

            Right now this could only be solved by using replicate_do/ignore_table rules, and having a consistent table naming scheme that would allow for CONNECT tables to be easily identified, or by adding per-table rules one by one.

            So I'd request to be able to outright ignore CONNECT or FEDERATEDX tables by default, by having a new filter like

            {{replicate_ignore_engine=CONNECT,FEDERATEDX}}

            This would obviously be a ROW-only filter.
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 107528 ] MariaDB v4 [ 131268 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Status Open [ 1 ] Needs Feedback [ 10501 ]
            hholzgra Hartmut Holzgraefe made changes -
            Comment [ A comment with security level 'Developers' was removed. ]

            DDL for CONNECT, FEDERATED and the likes should still be replicated, only DML needs to be filtered indeed as that is already replicated otherwise.

            So my request indeed looks similar to s3_slave_ignore_updates, just in a more general form that handles this on the server level for all potentially affected engines.

            hholzgra Hartmut Holzgraefe added a comment - DDL for CONNECT, FEDERATED and the likes should still be replicated, only DML needs to be filtered indeed as that is already replicated otherwise. So my request indeed looks similar to s3_slave_ignore_updates, just in a more general form that handles this on the server level for all potentially affected engines.
            hholzgra Hartmut Holzgraefe made changes -
            Status Needs Feedback [ 10501 ] Open [ 1 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Summary Feature request: replicate_do/ignore_engine Feature request: replicate_do_engine/replicate_ignore_engine
            AirFocus AirFocus made changes -
            Description When using remote table engines, like FEDERATEDX or CONNECT, usually one wouldn't want to have master side changes to be replicated to slaves, as these would just lead to duplicate actions on the remote tables by default.

            In edge cases though, where e.g. CONNECTION parameters for CONNECT tables on the slave side have been altered to point to different servers than on the master, replication for such tables may actually be wanted though.

            Right now this could only be solved by using replicate_do/ignore_table rules, and having a consistent table naming scheme that would allow for CONNECT tables to be easily identified, or by adding per-table rules one by one.

            So I'd request to be able to outright ignore CONNECT or FEDERATEDX tables by default, by having a new filter like

            {{replicate_ignore_engine=CONNECT,FEDERATEDX}}

            This would obviously be a ROW-only filter.
            When using remote table engines, like FEDERATEDX or CONNECT, usually one wouldn't want to have master side changes to be replicated to slaves, as these would just lead to duplicate actions on the remote tables by default.

            In edge cases though, where e.g. CONNECTION parameters for CONNECT tables on the slave side have been altered to point to different servers than on the master, replication for such tables may actually be wanted though.

            Right now this could only be solved by using replicate_do/ignore_table rules, and having a consistent table naming scheme that would allow for CONNECT tables to be easily identified, or by adding per\-table rules one by one.

            So I'd request to be able to outright ignore CONNECT or FEDERATEDX tables by default, by having a new filter like

            {{replicate_ignore_engine=CONNECT,FEDERATEDX}}

            This would obviously be a ROW\-only filter.
            Elkin Andrei Elkin added a comment - - edited

            ralf.gebhardt, hholzgra: while the proposed one is a generalization of an existing replicate_do_... let me first comment on how the desired effect can be achieve with a bit different user interface. That is a new option to Change-Master can trigger the desired filtering and already on the master side.
            The new Change-Master's option would be passed to the actual master at handshake time so in the following
            it would watch over specifically marked events.

            As a crude idea, could not master actually always compute *on its own* what Connect/Federated table events to filter out?
            After all the master is aware of both: the tables' hosts and slave hosts. Of course a slave may placehold a certain set of tables while being a user
            of another set which makes the slave interested in a selective automated

            {if possible}

            filtering.

            ralf.gebhardt, to your questions, sure the task is doable and there seems to be more than 1 approach how to.
            To any existing 'hard coded engine' filters - we don't have. But we have a "soft" s3_slave_ignore_updates one which also may serve as a pattern to follow (or be refactored, hopefully replaced by the automatic policy mentioned above.

            Elkin Andrei Elkin added a comment - - edited ralf.gebhardt , hholzgra : while the proposed one is a generalization of an existing replicate_do_... let me first comment on how the desired effect can be achieve with a bit different user interface. That is a new option to Change-Master can trigger the desired filtering and already on the master side. The new Change-Master's option would be passed to the actual master at handshake time so in the following it would watch over specifically marked events. As a crude idea, could not master actually always compute * on its own * what Connect/Federated table events to filter out? After all the master is aware of both: the tables' hosts and slave hosts. Of course a slave may placehold a certain set of tables while being a user of another set which makes the slave interested in a selective automated {if possible} filtering. ralf.gebhardt , to your questions, sure the task is doable and there seems to be more than 1 approach how to. To any existing 'hard coded engine' filters - we don't have. But we have a "soft" s3_slave_ignore_updates one which also may serve as a pattern to follow (or be refactored, hopefully replaced by the automatic policy mentioned above.
            ralf.gebhardt Ralf Gebhardt made changes -
            Status Open [ 1 ] Needs Feedback [ 10501 ]
            hholzgra Hartmut Holzgraefe made changes -
            Status Needs Feedback [ 10501 ] Open [ 1 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Ralf Gebhardt [ ralf.gebhardt@mariadb.com ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 147062 146234
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 146234 147062 146234 147062 203525
            Zendesk active tickets 203525
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk active tickets 203525

            People

              Unassigned Unassigned
              hholzgra Hartmut Holzgraefe
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.