Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16091

Seconds_Behind_Master spikes to millions of seconds

Details

    Description

      Hey everyone

      Rarely, we have a weird issue happening. We have a master <-> master setup. The writes are going to only one of them. Every now and then, the replication lag on the master where we send writes jumps to millions of seconds (ex: 2879736). Then, after 1s, is back to 0. After all, there's no lag, as the writes and the binlog from the other master should be ignored, as this is the server where we write to.
      Both servers have the same timezone and ntp running.
      Replication is set with GTIDs, and parallel threads.

      Sample config:
      slave_net_timeout=2
      gtid-strict-mode=1
      gtid-domain-id=1
      server_id = 101
      sync_binlog = 1
      binlog_format = ROW
      log-slave-updates
      replicate-same-server-id = 0
      log_bin=/var/log/mysql/binlog/mysql-bin.log
      log_bin_index=/var/log/mysql/binlog/mysql-bin.log.index
      relay_log=/var/log/mysql/binlog/mysql-relay-bin
      relay_log_index=/var/log/mysql/binlog/mysql-relay-bin.index
      expire_logs_days=5
      max_binlog_size=1G
      slave_parallel_threads=10

      We have a bunch of monitoring scripts looking at that value and triggering actions and alerts and we want to know what can cause this and if there's anything we can do to avoid this in the future.

      Thank you

      Attachments

        Issue Links

          Activity

            Transition Time In Source Status Execution Times
            Julien Fritsch made transition -
            In Progress Closed
            307d 2h 22m 1
            Andrei Elkin made transition -
            Closed Stalled
            4d 1h 9m 1
            Brandon Nesterenko made transition -
            Stalled In Progress
            7d 19h 1
            Julien Fritsch made transition -
            In Progress Needs Feedback
            7d 9h 41m 1
            Brandon Nesterenko made transition -
            Needs Feedback Open
            23d 8h 20m 1
            Brandon Nesterenko made transition -
            Open In Progress
            986d 13h 19m 2
            Brandon Nesterenko made transition -
            In Progress In Review
            2m 34s 1
            Andrei Elkin made transition -
            In Review Stalled
            4d 19h 26m 1
            Brandon Nesterenko made transition -
            Stalled Closed
            20h 41m 1

            People

              bnestere Brandon Nesterenko
              genesisdigital raul (Inactive)
              Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.