Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19371

Implement binlog_expire_logs_seconds for purging of binary logs

Details

    Description

      Oracle MySQL implemented binary log expiration period in seconds via binlog_expire_logs_seconds variable and expire_logs_days is deprecated in MySQL 8.
      It would be nice to have similar option for MariaDB Server. Expiring binary logs within seconds is extremely useful in high volume writes on master and where disk space is limited option.

      Attachments

        Issue Links

          Activity

            muhammad.irfan Muhammad Irfan created issue -
            ralf.gebhardt Ralf Gebhardt made changes -
            Field Original Value New Value
            Assignee Andrei Elkin [ elkin ]
            Elkin Andrei Elkin added a comment -

            ralf.gebhardt@mariadb.com This one looks pretty easy cherry-picking of WL#9237: Add a new variable binlog_expire_logs_seconds

            commit a7e1ef858ee82493dd8ad9a76bc9c22fe3b8c05b
            Author: Neha Kumari <neha.n.kumari@oracle.com>
            Date:   Thu Mar 2 12:30:08 2017 +0530
            
            

            Elkin Andrei Elkin added a comment - ralf.gebhardt@mariadb.com This one looks pretty easy cherry-picking of WL#9237: Add a new variable binlog_expire_logs_seconds commit a7e1ef858ee82493dd8ad9a76bc9c22fe3b8c05b Author: Neha Kumari <neha.n.kumari@oracle.com> Date: Thu Mar 2 12:30:08 2017 +0530
            Elkin Andrei Elkin made changes -
            Labels replication replication upstream-fixed
            julien.fritsch Julien Fritsch made changes -
            Assignee Andrei Elkin [ elkin ] Ralf Gebhardt [ ralf.gebhardt@mariadb.com ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Ralf Gebhardt [ ralf.gebhardt@mariadb.com ] Andrei Elkin [ elkin ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Component/s Replication [ 10100 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.6 [ 24028 ]
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Sujatha Sivakumar [ sujatha.sivakumar ]
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            Following commits need to be backported.

            1)

            commit a7e1ef858ee82493dd8ad9a76bc9c22fe3b8c05b
            Author: Neha Kumari <neha.n.kumari@oracle.com>
            Date:   Thu Mar 2 12:30:08 2017 +0530
             
                WL#9237: Add a new variable binlog_expire_logs_seconds
            

            2)

            commit 17bd6e0aee8963275f2b5153c90aad5e3a94b74f
            Author: Sven Sandberg <sven.sandberg@oracle.com>
            Date:   Tue Apr 10 14:50:04 2018 +0200
             
                BUG#27699608: WARNING FOR CONFLICTING EXPIRE-LOGS-[DAYS|SECONDS] SUPPRESSED WITH SKIP-LOG-BIN
            

            3)

            commit 9d591e29af1654b99a670d975ad8b9c95b011747
            Author: Neha Kumari <neha.n.kumari@oracle.com>
            Date:   Fri Feb 2 09:12:08 2018 +0530
             
                Bug#27252658 :CHANGE DEFAULT VALUE OF BINLOG_EXPIRE_LOGS_SECONDS
            

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Following commits need to be backported. 1) commit a7e1ef858ee82493dd8ad9a76bc9c22fe3b8c05b Author: Neha Kumari <neha.n.kumari@oracle.com> Date: Thu Mar 2 12:30:08 2017 +0530   WL#9237: Add a new variable binlog_expire_logs_seconds 2) commit 17bd6e0aee8963275f2b5153c90aad5e3a94b74f Author: Sven Sandberg <sven.sandberg@oracle.com> Date: Tue Apr 10 14:50:04 2018 +0200   BUG#27699608: WARNING FOR CONFLICTING EXPIRE-LOGS-[DAYS|SECONDS] SUPPRESSED WITH SKIP-LOG-BIN 3) commit 9d591e29af1654b99a670d975ad8b9c95b011747 Author: Neha Kumari <neha.n.kumari@oracle.com> Date: Fri Feb 2 09:12:08 2018 +0530   Bug#27252658 :CHANGE DEFAULT VALUE OF BINLOG_EXPIRE_LOGS_SECONDS

            Hello Andrei,

            Can you please review the changes for MDEV-19371.

            There is only one minor change between upstream patch and its corresponding MariaDB
            version, i.e the deprecation warnings are not written to server log, warnings get
            printed only when user tries to set the value using mysql client.

            Patch: https://github.com/MariaDB/server/commit/7ede75e1d2279e0cce3a27a93cc337d29af0fad7

            BuildBot: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-sujatha

            Thank you

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - - edited Hello Andrei, Can you please review the changes for MDEV-19371 . There is only one minor change between upstream patch and its corresponding MariaDB version, i.e the deprecation warnings are not written to server log, warnings get printed only when user tries to set the value using mysql client. Patch: https://github.com/MariaDB/server/commit/7ede75e1d2279e0cce3a27a93cc337d29af0fad7 BuildBot: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-sujatha Thank you
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Assignee Sujatha Sivakumar [ sujatha.sivakumar ] Andrei Elkin [ elkin ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Sujatha Sivakumar [ sujatha.sivakumar ]
            Elkin Andrei Elkin made changes -
            Assignee Sujatha Sivakumar [ sujatha.sivakumar ] Andrei Elkin [ elkin ]
            Elkin Andrei Elkin added a comment -

            Suggested to extend the work content for the current user to stay with
            their whole-days values if they are happy with that.
            The existing option is proposed to become of a decimal type, so the fractional part to represent a part of the whole 24 hr and those who need second or so precision could specify that.
            At the same time the oracle purge-in-seconds option will be accepted. The two are to show
            the same value just in different units.

            Elkin Andrei Elkin added a comment - Suggested to extend the work content for the current user to stay with their whole-days values if they are happy with that. The existing option is proposed to become of a decimal type, so the fractional part to represent a part of the whole 24 hr and those who need second or so precision could specify that. At the same time the oracle purge-in-seconds option will be accepted. The two are to show the same value just in different units.
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Sujatha Sivakumar [ sujatha.sivakumar ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]

            Hello Andrei,

            I have addressed your review comments. Please review the latest changes.

            Patch: https://github.com/MariaDB/server/commit/14ba1d3c29880314ca61ad2feafe53a1d4c8a3da

            BuildBot Testing: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-sujatha

            Thank you.

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Hello Andrei, I have addressed your review comments. Please review the latest changes. Patch: https://github.com/MariaDB/server/commit/14ba1d3c29880314ca61ad2feafe53a1d4c8a3da BuildBot Testing: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-sujatha Thank you.
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Assignee Sujatha Sivakumar [ sujatha.sivakumar ] Andrei Elkin [ elkin ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            Elkin Andrei Elkin added a comment -

            The patch looks close to be fine. I left few notes on 14ba1d3c298.

            Elkin Andrei Elkin added a comment - The patch looks close to be fine. I left few notes on 14ba1d3c298 .
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Sujatha Sivakumar [ sujatha.sivakumar ]
            Status In Review [ 10002 ] Stalled [ 10000 ]

            Hello Andrei,

            Thank you for the review comments.
            I have addressed them in the latest patch.

            Please review them.

            https://github.com/MariaDB/server/commit/888af81f942e3d537b99f073d9504b4c8a7323b8
            buildbot: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-sujatha

            Thank you.

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Hello Andrei, Thank you for the review comments. I have addressed them in the latest patch. Please review them. https://github.com/MariaDB/server/commit/888af81f942e3d537b99f073d9504b4c8a7323b8 buildbot: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-sujatha Thank you.
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Assignee Sujatha Sivakumar [ sujatha.sivakumar ] Andrei Elkin [ elkin ]
            Status Stalled [ 10000 ] In Review [ 10002 ]
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Assignee Andrei Elkin [ elkin ] Sujatha Sivakumar [ sujatha.sivakumar ]

            Patch was approved by Elkin and Brandon.

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Patch was approved by Elkin and Brandon.
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sujatha.sivakumar Sujatha Sivakumar (Inactive) made changes -
            Fix Version/s 10.6.1 [ 24437 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            Elkin Andrei Elkin added a comment -

            For the documentation, the patch 49ff2cbff4
            1. changes the type of expire_log_days to double to allow factional part of the whole day.
            Backward compatible setting to an integer value is preserved (no need any post-upgrade actions)
            2. adopts from the upstream binlog_expire_logs_seconds as an alias to expire_log_days so
            the two variables show the same value just in different units and formats.

            The changes to --help for expire_log_days can be seen in the patch:

            "If non-zero, binary logs will be purged after expire_logs_days "

            • "days; possible purges happen at startup and at binary log rotation",
              + "days; It and binlog_expire_logs_seconds are aliases, such that "
              + "changes in one are converted into the other, presentable as a "
              + "decimal value with 1/1000000 of the day precision; possible "
              + "purges happen at startup and at binary log rotation",

            The upstream new binlog_expire_logs_seconds is described as

            + "If non-zero, binary logs will be purged after "
            + "binlog_expire_logs_seconds seconds; It and expire_logs_days are "
            + "aliases, such that changes in one are converted into the other. "
            + "Possible purges happen at startup and at binary log rotation.",

            the type is double,
            the valid range is from 0 to 99*86400 seconds.
            Values from 1 to 11 are rounded to zero 0.000000 of expire_log_days.

            Elkin Andrei Elkin added a comment - For the documentation, the patch 49ff2cbff4 1. changes the type of expire_log_days to double to allow factional part of the whole day. Backward compatible setting to an integer value is preserved (no need any post-upgrade actions) 2. adopts from the upstream binlog_expire_logs_seconds as an alias to expire_log_days so the two variables show the same value just in different units and formats. The changes to --help for expire_log_days can be seen in the patch: "If non-zero, binary logs will be purged after expire_logs_days " "days; possible purges happen at startup and at binary log rotation", + "days; It and binlog_expire_logs_seconds are aliases, such that " + "changes in one are converted into the other, presentable as a " + "decimal value with 1/1000000 of the day precision; possible " + "purges happen at startup and at binary log rotation", The upstream new binlog_expire_logs_seconds is described as + "If non-zero, binary logs will be purged after " + "binlog_expire_logs_seconds seconds; It and expire_logs_days are " + "aliases, such that changes in one are converted into the other. " + "Possible purges happen at startup and at binary log rotation.", the type is double, the valid range is from 0 to 99*86400 seconds. Values from 1 to 11 are rounded to zero 0.000000 of expire_log_days .
            greenman Ian Gilfillan added a comment -

            I suggest not using the word "alias" in the descriptions, as this implies that they are the same, and if a user doesn't look beyond that, they could end up specifying the wrong units. You could replace it with the word "linked", or "forms of aliases".

            greenman Ian Gilfillan added a comment - I suggest not using the word "alias" in the descriptions, as this implies that they are the same, and if a user doesn't look beyond that, they could end up specifying the wrong units. You could replace it with the word "linked", or "forms of aliases".
            Elkin Andrei Elkin added a comment -

            Thanks, Ian! Agreed here. sujatha.sivakumar, could you please pick it up to a patch that fixes the valid ranges?

            Elkin Andrei Elkin added a comment - Thanks, Ian! Agreed here. sujatha.sivakumar , could you please pick it up to a patch that fixes the valid ranges?

            Thank greenman and Elkin. I will do the changes.

            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Thank greenman and Elkin . I will do the changes.
            sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - - edited Hello Elkin Please find the following post push fixes. https://github.com/MariaDB/server/commit/2ea32e65a34ba065b3dd63d4c80071404dad1ca8 BuildBot: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-MDEV-19371 Please review them. Thank you.
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 96511 ] MariaDB v4 [ 133944 ]
            nayuta-yanagisawa Nayuta Yanagisawa (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 177191 170479

            People

              sujatha.sivakumar Sujatha Sivakumar (Inactive)
              muhammad.irfan Muhammad Irfan
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.