Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17133

Binlog truncated in the middle of event; consider out of disk space on master

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.1.36, 10.2.17
    • 10.2.19, 10.1.38
    • Replication

    Description

      Experts,
      After applying latest MariaDB patches on Replication master and Replication Slave, we are facing a strange issue.
      Once per nearly 3 hours (random time) replication process at slave stopped with following errors

      2018-09-04 6:33:28 140064771901184 [ERROR] Error reading packet from server: binlog truncated in the middle of event; consider out of disk space on master; the first event 'XXXXX-bin.000727' at 431628288, the last event read from 'XXXXX-bin.000728' at 399111340, the last byte read from 'XXXXX-bin.000728' at 399111359. (server_errno=1236)

      2018-09-04 6:33:28 140064771901184 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'binlog truncated in the middle of event; consider out of disk space on master; the first event 'XXXXX-bin.000727' at 431628288, the last event read from 'XXXXX-bin.000728' at 399111340, the last byte read from 'XXXXX-bin.000728' at 399111359.', Internal MariaDB error code: 1236

      This issue resolves when i submit command "stop slave; start slave;" .
      This is something weird and strange because let's assume if problem persists on master replication server how it gone away by "stop slave; start slave;" simple command.

      That make me uncomfortable and unconvinced.
      thanks
      Devang

      Attachments

        1. 712.sql
          2 kB
        2. MASTER14.log
          4.99 MB
        3. MASTER14-bin.000712.1033405612.19_bytes.dump
          0.0 kB
        4. MASTER14-bin.000761.zip.001
          10.00 MB
        5. MASTER14-bin.000761.zip.001
          10.00 MB
        6. MASTER14-bin.000761.zip.002
          10.00 MB
        7. MASTER14-bin.000761.zip.002
          10.00 MB
        8. MASTER14-bin.000761.zip.003
          10.00 MB
        9. MASTER14-bin.000761.zip.003
          10.00 MB
        10. MASTER14-bin.000761.zip.004
          10.00 MB
        11. MASTER14-bin.000761.zip.004
          10.00 MB
        12. MASTER14-bin.000761.zip.005
          10.00 MB
        13. MASTER14-bin.000761.zip.005
          10.00 MB
        14. MASTER14-bin.000761.zip.006
          10.00 MB
        15. MASTER14-bin.000761.zip.006
          10.00 MB
        16. MASTER14-bin.000761.zip.007
          10.00 MB
        17. MASTER14-bin.000761.zip.007
          10.00 MB
        18. MASTER14-bin.000761.zip.008
          10.00 MB
        19. MASTER14-bin.000761.zip.008
          10.00 MB
        20. MASTER14-bin.000761.zip.009
          10.00 MB
        21. MASTER14-bin.000761.zip.009
          10.00 MB
        22. MASTER14-bin.000761.zip.010
          10.00 MB
        23. MASTER14-bin.000761.zip.010
          10.00 MB
        24. MASTER14-bin.000761.zip.011
          10.00 MB
        25. MASTER14-bin.000761.zip.011
          10.00 MB
        26. MASTER14-bin.000761.zip.012
          10.00 MB
        27. MASTER14-bin.000761.zip.012
          10.00 MB
        28. MASTER14-bin.000761.zip.013
          1.66 MB
        29. MASTER14-bin.000761.zip.013
          1.66 MB
        30. MASTER14-Log.txt
          39 kB
        31. MASTER14-Processes.txt
          11 kB
        32. MASTER14-Status.txt
          28 kB
        33. MASTER14-Variables.txt
          466 kB
        34. MDEV-17133.png
          MDEV-17133.png
          21 kB
        35. mysqld-master14.log
          95 kB
        36. mysqld-slave10.log
          71 kB
        37. SLAVE10.log
          4.99 MB
        38. SLAVE10-Status.txt
          28 kB
        39. SLAVE10-Variables.txt
          312 kB

        Issue Links

          Activity

            devang_modi Devang created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Epic/Theme service
            elenst Elena Stepanova made changes -
            Fix Version/s 10.2 [ 14601 ]
            Assignee Andrei Elkin [ elkin ]
            Labels replication
            devang_modi Devang made changes -
            Attachment mysqld-master14.log [ 46408 ]
            devang_modi Devang made changes -
            Attachment mysqld-slave10.log [ 46409 ]
            devang_modi Devang made changes -
            Attachment MDEV-17133.png [ 46410 ]
            devang_modi Devang made changes -
            Attachment 712.sql [ 46414 ]
            devang_modi Devang made changes -
            Attachment SLAVE10-Variables.txt [ 46416 ]
            Attachment SLAVE10-Status.txt [ 46417 ]
            Attachment MASTER14-Status.txt [ 46418 ]
            Attachment MASTER14-Variables.txt [ 46419 ]
            Attachment MASTER14-Log.txt [ 46420 ]
            Attachment MASTER14-Processes.txt [ 46421 ]
            devang_modi Devang made changes -
            devang_modi Devang made changes -
            Attachment MASTER14.log [ 46431 ]
            Attachment SLAVE10.log [ 46432 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.001 [ 46439 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.013 [ 46440 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.012 [ 46441 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.011 [ 46442 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.010 [ 46443 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.009 [ 46444 ]
            Attachment MASTER14-bin.000761.zip.008 [ 46445 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.007 [ 46446 ]
            Attachment MASTER14-bin.000761.zip.006 [ 46447 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.005 [ 46448 ]
            Attachment MASTER14-bin.000761.zip.004 [ 46449 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.003 [ 46450 ]
            Attachment MASTER14-bin.000761.zip.002 [ 46451 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.001 [ 46458 ]
            Attachment MASTER14-bin.000761.zip.002 [ 46459 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.003 [ 46460 ]
            Attachment MASTER14-bin.000761.zip.004 [ 46461 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.005 [ 46462 ]
            Attachment MASTER14-bin.000761.zip.006 [ 46463 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.007 [ 46464 ]
            Attachment MASTER14-bin.000761.zip.008 [ 46465 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.009 [ 46466 ]
            Attachment MASTER14-bin.000761.zip.010 [ 46467 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.011 [ 46468 ]
            Attachment MASTER14-bin.000761.zip.012 [ 46469 ]
            devang_modi Devang made changes -
            Attachment MASTER14-bin.000761.zip.013 [ 46470 ]
            Elkin Andrei Elkin made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Sergei Golubchik [ serg ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            Elkin Andrei Elkin made changes -
            Elkin Andrei Elkin made changes -
            Elkin Andrei Elkin made changes -
            Elkin Andrei Elkin made changes -
            Fix Version/s 10.1 [ 16100 ]
            Fix Version/s 10.2 [ 14601 ]
            GeoffMontee Geoff Montee (Inactive) made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Andrei Elkin [ elkin ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            Elkin Andrei Elkin made changes -
            issue.field.resolutiondate 2018-10-21 14:53:47.0 2018-10-21 14:53:47.821
            Elkin Andrei Elkin made changes -
            Fix Version/s 10.1.37 [ 23204 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            Elkin Andrei Elkin made changes -
            Affects Version/s 10.1.36 [ 23117 ]
            marko Marko Mäkelä made changes -
            Fix Version/s 10.1.38 [ 23209 ]
            Fix Version/s 10.1.37 [ 23204 ]
            Elkin Andrei Elkin made changes -
            Fix Version/s 10.2.19 [ 23207 ]
            Elkin Andrei Elkin made changes -
            Labels IO_CACHE
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 89221 ] MariaDB v4 [ 154889 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 184868 189185

            People

              Elkin Andrei Elkin
              devang_modi Devang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.