[MDEV-4989] Support for GTID in mysqlbinlog Created: 2013-09-03  Updated: 2023-08-07  Resolved: 2022-01-31

Status: Closed
Project: MariaDB Server
Component/s: Replication, Scripts & Clients
Fix Version/s: 10.8.1

Type: Task Priority: Critical
Reporter: Kristian Nielsen Assignee: Brandon Nesterenko
Resolution: Fixed Votes: 23
Labels: Preview_10.8, gsoc14, gsoc15, gsoc16, gsoc17, gsoc18, gsoc19, gsoc20, gsoc21, gtid, mysqlbinlog, replication

Issue Links:
PartOf
is part of MDEV-27373 Q1 2022 release merge Closed
Problem/Incident
causes MDEV-29803 Unexpected ERROR: Found out of order ... Open
Relates
relates to MDEV-20119 Implement the --do-domain-ids, --igno... Closed
relates to MXS-2589 Adapt binlog router to Closed
relates to MDEV-25348 Using Point-in-time-Recovery with mys... Open
relates to MXS-2581 Support DO_DOMAIN_IDS and IGNORE_DOMA... Closed

 Description   

The mysqlbinlog client program needs to be updated to support GTID.

Here is a suggested list of things to be done:

  • The --start-position and --stop-position options should be able to take
    GTID positions; or maybe there should be new --start-gtid and --stop-gtid
    options. Like --start-gtid=0-1-100,1-2-200,2-1-1000.
  • A GTID position means the point just after that GTID. So starting from
    GTID 0-1-100 and stopping at GTID 0-1-200, the first GTID output will
    probably be 0-1-101 and the last one 0-1-200. Note that if some domain is
    not specified in the position, it means to start from the begining,
    respectively stop immediately in that domain.
  • Starting and stopping GTID should work both with local files, and with
    --read-from-remote-server. For the latter, there are a couple of extra
    things that need doing in the master-slave protocol, see
    get_master_version_and_clock() in sql/slave.cc.
  • At the end of the dump, put these statements, to reduce the risk of those session variables incorrectly spilling into subsequent statements run in the same session:

       SET session.server_id = @@global.server_id,
           session.gtid_domain_id=@@global.gtid_domain_id;

    Probably some more things will come up during the work, but this looks like a
    reasonable start.



 Comments   
Comment by Kristian Nielsen [ 2014-06-30 ]

This is actually a really good starter project for someone to get into working with the GTID code.

The tasks requires getting some moderate understanding on how GTID works in general, but the actual code that needs to be worked on is fairly isolated and doesn't need to touch a majority of the GTID and replication stuff.

Comment by Kristian Nielsen [ 2015-03-16 ]

Some suggestions for getting started with this:

I would recommend to start by reading the GTID documentation, and
experimenting with all the relevant features described there, to familiarise
yourself with it. The main thing is to understand the concept of GTID position
and binlog order, and the relation to replication domains:

https://mariadb.com/kb/en/mariadb/global-transaction-id/

There are also some blog posts:

http://kristiannielsen.livejournal.com/16826.html
http://kristiannielsen.livejournal.com/17008.html
http://kristiannielsen.livejournal.com/17238.html

And similarly, familiarise yourself with mysqlbinlog and how it would make
sense for it to be extended to support GTID.

Then make a detailed proposal for how the user-visible interface of
mysqlbinlog should be extended, and send it to the list for comments.

The next step will be to understand the concept of binlog state
(@@gtid_binlog_state), and how it is used to deal correctly with out-of-order
GTID sequence numbers in binlogs. This is mostly an internal implementation
detail, but necessary to be able to correctly identify starting and stopping
GTID positions in all cases.

Comment by Frederik Brinck Jensen [ 2016-03-19 ]

Hi Kristian,

You have logged this issue/idea to four weeks only. Do you recommend GSoC applicants to propose to take on other ideas as well, or elaborate on this one, considering that they might finish this implementation within the estimated time.

Thanks in advance,
Frederik

Comment by Amol Holani [ 2017-03-12 ]

Hi Kristian,
I'm a senior year undergraduate student majoring in Computer Science. I
intend to be a part of the MariaDB Developer Community and would love to
start my journey through the Google Summer of Code this summer.
Do you expect any task that I should complete before submitting proposal for this particular project so that I can get a positive response.
Why this project was not allocated last year, even though it is being considered for Gsoc from last 3 years.

Comment by Mike Reiss [ 2017-03-12 ]

Hi,
I am Mike Reiss a MS Computer Science student and I would love to work on this project during GSoC 2017.I have studied the mysqlbinlog.cc file and experimented with the various options available in the my_options array of the tool. The proposed idea is straight forward to implement: We add the --start-gtid and --stop-gtid options to the array of options then parse the GTID passed to these options to get the GTID sequence number for the binlog. One quick question. Is there any relationship between a GTID sequence number and the argument passed to the --start-position option? that way we need to set the start position to the GTID sequence number. I am still doing more research into the source files and I will love to discuss this more with the mentor. I am having difficulties debugging mysqlbinlog on OS X sierra because it does not support GDB any ideas please?

Comment by Daniel Black [ 2017-03-13 ]

mikereiss - a number of gdb options for OS X Sierra are mentioned here http://stackoverflow.com/questions/39702871/gdb-kind-of-doesnt-work-on-macos-sierra . Its essential that you can work an debugger in your development environment and utilising a mentors time to achieve this isn't a productive use of their time especially if they use a different OS. There are plenty of other people/resources that should be able to help you with this.

Also related: https://lists.launchpad.net/maria-developers/msg09332.html and
https://lists.launchpad.net/maria-developers/msg08186.html

Comment by Mike Reiss [ 2017-03-13 ]

@Daniel Black - I'm sorry I should have crafted my question as what is the recommended debugger used by MariaDB developers on OS X ? Thank you for the tip on gdb. Vicențiu on Maria-Developers recommended that I use lldb and it works great for me. Hope its fine with you I continue using lldb.

Comment by Geoff Montee (Inactive) [ 2019-07-22 ]

MDEV-20119 is somewhat related.

Comment by Andrei Elkin [ 2019-08-15 ]

{--start.stop}-position may be reused to accept the new type of GTID format string.

Comment by Brandon Nesterenko [ 2021-08-24 ]

This is ready for review.

Commit: f7e582ff9c5ace871c9b98ea8465956c1e94fb96

Buildbot: bb-10.7-MDEV-4989

    • Updated commit Aug 27 with some documentation, code formatting, and slight code changes to help with MDEV-20119
Comment by Brandon Nesterenko [ 2021-09-10 ]

I have pushed a new commit which addresses an initial round of reviews from Andrei:

Commit: 02f98a22d2caef7c2953fe02b7f7516cd08c9544
Buildbot: bb-10.7-MDEV-4989

Comment by Alice Sherepa [ 2022-01-25 ]

it is ok to push it into 10.8

Generated at Thu Feb 08 07:00:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.