Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11154

Write_on_release_cache(log_event.cc) function will not write "COMMIT", if use "mysqlbinlog ... | mysql ..."

Details

    Description

      Executing command, "mysqlbinlog --read-from-remote-server --host='xx.xx.xx.xx' --port=3306 --user=xxx --password=xxx --database=mysql --to-last-log mysql-bin.000001 --start-position=1098699 --stop-never |mysql -uxxx -pxxx", we found that last data read from remote couldn't commit. Write_on_release_cache(log_event.cc) function may have some problems.

      In Write_on_release_cache(log_event.cc) function have following code:

      public:
        enum flag
        {
          FLUSH_F
        };
      ...
        Write_on_release_cache(IO_CACHE *cache, FILE *file, flag_set flags = 0)
          : m_cache(cache), m_file(file), m_flags(flags)
        {
          reinit_io_cache(m_cache, WRITE_CACHE, 0L, FALSE, TRUE);
        }
       
        ~Write_on_release_cache()
        {
          copy_event_cache_to_file_and_reinit(m_cache, m_file);
          if (m_flags & FLUSH_F)
            fflush(m_file);
        }
      ...
      

      The right code maybe:

      public:
        enum flag
        {
          FLUSH_F = 1
        };
      ...
      

      Attachments

        1. mysqlbinlog1.PNG
          mysqlbinlog1.PNG
          8 kB
        2. mysqlbinlog2.PNG
          mysqlbinlog2.PNG
          12 kB
        3. mysqlbinlog3.PNG
          mysqlbinlog3.PNG
          4 kB
        4. mysqlbinlog4.PNG
          mysqlbinlog4.PNG
          8 kB
        5. mysqlbinlog5.PNG
          mysqlbinlog5.PNG
          4 kB
        6. mysqlbinlog6.PNG
          mysqlbinlog6.PNG
          30 kB

        Activity

          Fix has been implemented in 10.1.41.

          There are some additional changes required for 10.2+ versions.

          Please refer
          https://github.com/MariaDB/server/commit/096722b61ebe66ffe510156d358d4ddbeb53241a

          The patch has been tested on all higher versions.

          sujatha.sivakumar Sujatha Sivakumar (Inactive) added a comment - Fix has been implemented in 10.1.41. There are some additional changes required for 10.2+ versions. Please refer https://github.com/MariaDB/server/commit/096722b61ebe66ffe510156d358d4ddbeb53241a The patch has been tested on all higher versions.
          yury.chaikou Yury Chaikou added a comment -

          This patch should also be ported to 10.5 and all later versions, including the latest 11.1

          yury.chaikou Yury Chaikou added a comment - This patch should also be ported to 10.5 and all later versions, including the latest 11.1
          Elkin Andrei Elkin added a comment -

          yury.chaikou, do you mean the bug is present in higher versions (than those in Fixed box)? Just in case the higher version are not listed 'cos they did not exist at time
          of 10.2.26.

          Elkin Andrei Elkin added a comment - yury.chaikou , do you mean the bug is present in higher versions (than those in Fixed box)? Just in case the higher version are not listed 'cos they did not exist at time of 10.2.26.
          yury.chaikou Yury Chaikou added a comment - - edited

          Andrei, correct, the bug is present in any version higher than those in Fixed box.
          For example, check the latest https://github.com/MariaDB/server/blob/11.1/sql/log_event.cc:

            enum flag
            {
              FLUSH_F
            };
          

          yury.chaikou Yury Chaikou added a comment - - edited Andrei, correct, the bug is present in any version higher than those in Fixed box. For example, check the latest https://github.com/MariaDB/server/blob/11.1/sql/log_event.cc: enum flag { FLUSH_F };
          knielsen Kristian Nielsen added a comment - - edited

          As Yury Chaikou noticed, the fix for this MDEV-11154 was merged incorrectly to 10.5:

          commit 624dd71b9419555eca8baadc695e3376de72286f
          Merge: d4d865fcc80 e9c1701e11e
          Author: Marko Mäkelä <marko.makela@mariadb.com>
          Date:   Tue Aug 13 18:57:00 2019 +0300
           
              Merge 10.4 into 10.5
          

          The fix got null-merged to 10.5, which probably is not what was intended, as now all the logic using FLUSH_F is dead code:

          $ git diff 624dd71b9419555eca8baadc695e3376de72286f^1 624dd71b9419555eca8baadc695e3376de72286f -- sql/log_event.cc
          # Empty output
          

          However, the original bug, which is missing fflush() by mysqlbinlog on the output, seems nevertheless to not occur. The reason for this is another merge:

          commit 69628551858825bebc3f1653882920e4f6555cbb
          Merge: 26b594e4110 10ebdb7f1d7
          Author: Marko Mäkelä <marko.makela@mariadb.com>
          Date:   Thu Jul 18 13:10:09 2019 +0300
           
              Merge 10.1 into 10.2
          

          $ git diff 696285518588^1 696285518588 -- client/mysqlbinlog.cc
            diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc
            index 1516e08284b..acccf0a3411 100644
            --- a/client/mysqlbinlog.cc
            +++ b/client/mysqlbinlog.cc
            @@ -1534,6 +1534,7 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev,
          	 {
          	   my_fwrite(result_file, (const uchar *) tmp_str.str, tmp_str.length,
          		     MYF(MY_NABP));
            +        fflush(result_file);
          	   my_free(tmp_str.str);
          	 }
                 }
          

          I think this occured because the mysqlbinlog code is somewhat different between 10.1 and 10.2, so the MDEV-11154 flush bug is fixed differently with the new code with this fflush().
          As a result, the whole FLUSH_F code in log_event_client.cc may now be mostly dead code, and should maybe be removed; or if still in use, the FLUSH_F enum should be fixed.

          Interestingly, the test case for this bug (binlog_mysqlbinlog_stop_never.test) seems also to be wrong now. It is trying to do something like:

              mysqlbinlog ... | mysql ...
          

          But the problem is that the events will be applied by the `mysql` command with binlogging enabled, which will loop around and endlessly stream events through mysqlbinlog (or rather, until a duplicate GTID or row error stops it); in any case it doesn't reliably test the missing fflush() reliably. So this should also be fixed. I think can be fixed with something like this:

              ( echo 'set @@SESSION.SQL_LOG_BIN = 0;' ; mysqlbinlog ... ) | mysql ...
          

          Since the bug itself seems to not be present, reopening with a low priority.

          Hope this helps,

          - Kristian.

          knielsen Kristian Nielsen added a comment - - edited As Yury Chaikou noticed, the fix for this MDEV-11154 was merged incorrectly to 10.5: commit 624dd71b9419555eca8baadc695e3376de72286f Merge: d4d865fcc80 e9c1701e11e Author: Marko Mäkelä <marko.makela@mariadb.com> Date: Tue Aug 13 18:57:00 2019 +0300   Merge 10.4 into 10.5 The fix got null-merged to 10.5, which probably is not what was intended, as now all the logic using FLUSH_F is dead code: $ git diff 624dd71b9419555eca8baadc695e3376de72286f^1 624dd71b9419555eca8baadc695e3376de72286f -- sql/log_event.cc # Empty output However, the original bug, which is missing fflush() by mysqlbinlog on the output, seems nevertheless to not occur. The reason for this is another merge: commit 69628551858825bebc3f1653882920e4f6555cbb Merge: 26b594e4110 10ebdb7f1d7 Author: Marko Mäkelä <marko.makela@mariadb.com> Date: Thu Jul 18 13:10:09 2019 +0300   Merge 10.1 into 10.2 $ git diff 696285518588^1 696285518588 -- client/mysqlbinlog.cc diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc index 1516e08284b..acccf0a3411 100644 --- a/client/mysqlbinlog.cc +++ b/client/mysqlbinlog.cc @@ -1534,6 +1534,7 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, { my_fwrite(result_file, (const uchar *) tmp_str.str, tmp_str.length, MYF(MY_NABP)); + fflush(result_file); my_free(tmp_str.str); } } I think this occured because the mysqlbinlog code is somewhat different between 10.1 and 10.2, so the MDEV-11154 flush bug is fixed differently with the new code with this fflush(). As a result, the whole FLUSH_F code in log_event_client.cc may now be mostly dead code, and should maybe be removed; or if still in use, the FLUSH_F enum should be fixed. Interestingly, the test case for this bug (binlog_mysqlbinlog_stop_never.test) seems also to be wrong now. It is trying to do something like: mysqlbinlog ... | mysql ... But the problem is that the events will be applied by the `mysql` command with binlogging enabled, which will loop around and endlessly stream events through mysqlbinlog (or rather, until a duplicate GTID or row error stops it); in any case it doesn't reliably test the missing fflush() reliably. So this should also be fixed. I think can be fixed with something like this: ( echo 'set @@SESSION.SQL_LOG_BIN = 0;' ; mysqlbinlog ... ) | mysql ... Since the bug itself seems to not be present, reopening with a low priority. Hope this helps, - Kristian.

          People

            knielsen Kristian Nielsen
            vincen vincen
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.