Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17133

Binlog truncated in the middle of event; consider out of disk space on master

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.1.36, 10.2.17
    • 10.2.19, 10.1.38
    • Replication

    Description

      Experts,
      After applying latest MariaDB patches on Replication master and Replication Slave, we are facing a strange issue.
      Once per nearly 3 hours (random time) replication process at slave stopped with following errors

      2018-09-04 6:33:28 140064771901184 [ERROR] Error reading packet from server: binlog truncated in the middle of event; consider out of disk space on master; the first event 'XXXXX-bin.000727' at 431628288, the last event read from 'XXXXX-bin.000728' at 399111340, the last byte read from 'XXXXX-bin.000728' at 399111359. (server_errno=1236)

      2018-09-04 6:33:28 140064771901184 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'binlog truncated in the middle of event; consider out of disk space on master; the first event 'XXXXX-bin.000727' at 431628288, the last event read from 'XXXXX-bin.000728' at 399111340, the last byte read from 'XXXXX-bin.000728' at 399111359.', Internal MariaDB error code: 1236

      This issue resolves when i submit command "stop slave; start slave;" .
      This is something weird and strange because let's assume if problem persists on master replication server how it gone away by "stop slave; start slave;" simple command.

      That make me uncomfortable and unconvinced.
      thanks
      Devang

      Attachments

        1. 712.sql
          2 kB
        2. MASTER14.log
          4.99 MB
        3. MASTER14-bin.000712.1033405612.19_bytes.dump
          0.0 kB
        4. MASTER14-bin.000761.zip.001
          10.00 MB
        5. MASTER14-bin.000761.zip.001
          10.00 MB
        6. MASTER14-bin.000761.zip.002
          10.00 MB
        7. MASTER14-bin.000761.zip.002
          10.00 MB
        8. MASTER14-bin.000761.zip.003
          10.00 MB
        9. MASTER14-bin.000761.zip.003
          10.00 MB
        10. MASTER14-bin.000761.zip.004
          10.00 MB
        11. MASTER14-bin.000761.zip.004
          10.00 MB
        12. MASTER14-bin.000761.zip.005
          10.00 MB
        13. MASTER14-bin.000761.zip.005
          10.00 MB
        14. MASTER14-bin.000761.zip.006
          10.00 MB
        15. MASTER14-bin.000761.zip.006
          10.00 MB
        16. MASTER14-bin.000761.zip.007
          10.00 MB
        17. MASTER14-bin.000761.zip.007
          10.00 MB
        18. MASTER14-bin.000761.zip.008
          10.00 MB
        19. MASTER14-bin.000761.zip.008
          10.00 MB
        20. MASTER14-bin.000761.zip.009
          10.00 MB
        21. MASTER14-bin.000761.zip.009
          10.00 MB
        22. MASTER14-bin.000761.zip.010
          10.00 MB
        23. MASTER14-bin.000761.zip.010
          10.00 MB
        24. MASTER14-bin.000761.zip.011
          10.00 MB
        25. MASTER14-bin.000761.zip.011
          10.00 MB
        26. MASTER14-bin.000761.zip.012
          10.00 MB
        27. MASTER14-bin.000761.zip.012
          10.00 MB
        28. MASTER14-bin.000761.zip.013
          1.66 MB
        29. MASTER14-bin.000761.zip.013
          1.66 MB
        30. MASTER14-Log.txt
          39 kB
        31. MASTER14-Processes.txt
          11 kB
        32. MASTER14-Status.txt
          28 kB
        33. MASTER14-Variables.txt
          466 kB
        34. MDEV-17133.png
          MDEV-17133.png
          21 kB
        35. mysqld-master14.log
          95 kB
        36. mysqld-slave10.log
          71 kB
        37. SLAVE10.log
          4.99 MB
        38. SLAVE10-Status.txt
          28 kB
        39. SLAVE10-Variables.txt
          312 kB

        Issue Links

          Activity

            devang_modi Devang added a comment -

            Dear Andrei,
            I was following you and all the communication or action carried out under this ticket.
            Thanks a lot for your initiatives and actions taken to resolve it.
            Can you suggest next action because i m now confused with version for which i have reported issue and version showing in FIXED both are quite unmatching i.e. reported ver was 10.2.17 and fixed ver 10.1.37 ?
            thanks
            Devang

            devang_modi Devang added a comment - Dear Andrei, I was following you and all the communication or action carried out under this ticket. Thanks a lot for your initiatives and actions taken to resolve it. Can you suggest next action because i m now confused with version for which i have reported issue and version showing in FIXED both are quite unmatching i.e. reported ver was 10.2.17 and fixed ver 10.1.37 ? thanks Devang
            Elkin Andrei Elkin added a comment -

            Devang, thank you for a great assistance! I will let you know when the current patch branch 10.1 is going
            to be merged to 10.2.

            Cheers, Andrei

            Elkin Andrei Elkin added a comment - Devang, thank you for a great assistance! I will let you know when the current patch branch 10.1 is going to be merged to 10.2. Cheers, Andrei
            Elkin Andrei Elkin added a comment -

            https://jira.mariadb.org/secure/Dashboard.jspa promises the next release on Nov 2.

            Elkin Andrei Elkin added a comment - https://jira.mariadb.org/secure/Dashboard.jspa promises the next release on Nov 2.
            devang_modi Devang added a comment -

            Andrei
            It is 10-Nov but i did not found updates of Maria DB when i tried YUM DOWNLOAD command on CentOS 6.10.
            please confirm
            thanks
            Devang

            devang_modi Devang added a comment - Andrei It is 10-Nov but i did not found updates of Maria DB when i tried YUM DOWNLOAD command on CentOS 6.10. please confirm thanks Devang
            Elkin Andrei Elkin added a comment -

            Devang,

            I see the release has been delayed for 3 days by now:
            https://jira.mariadb.org/projects/MDEV/versions/23207
            because some of builders have not completed yet.

            I may look into the latest http://hasky.askmonty.org/archive/10.2/build-22746/ which includes the current bug fixes
            in packages that are available.

            Cheers,
            Andrei.

            Elkin Andrei Elkin added a comment - Devang, I see the release has been delayed for 3 days by now: https://jira.mariadb.org/projects/MDEV/versions/23207 because some of builders have not completed yet. I may look into the latest http://hasky.askmonty.org/archive/10.2/build-22746/ which includes the current bug fixes in packages that are available. Cheers, Andrei.

            People

              Elkin Andrei Elkin
              devang_modi Devang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.