Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41231

mariadb-backup --safe-slave-backup stops only the default replication channel while another channel keeps applying

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 12.3.3
    • None
    • mariabackup
    • None

    Description

      On a replica with a default replication channel and a named `analytics`
      channel, both IO and SQL threads are initially running. During a real
      `mariadb-backup --safe-slave-backup`, the default channel changes to `Yes/No`
      but `analytics` remains `Yes/Yes`. The backup exits 0 and prints `completed
      OK!`.

      The option therefore does not provide a replica-wide safe state in a
      multi-source setup.

      Steps to reproduce

      #!/usr/bin/env bash
      set -euo pipefail
      IMAGE='mariadb@sha256:dd9b303aed4f4890ed09f766d8ca9ddfd176c0c6f6267feff53b3192ec65a979'
      OWNER="mdev-mcf204-$$"
      S1="mdev-mcf204-source1-$$"; S2="mdev-mcf204-source2-$$"; R="mdev-mcf204-replica-$$"
      NAMES=("$S1" "$S2" "$R")
      cleanup() {
        for C in "${NAMES[@]}"; do
          if docker inspect --format '{{ index .Config.Labels "io.encryptiondbfuzz.owner" }}' \
               "$C" 2>/dev/null | grep -Fqx "$OWNER"; then
            docker rm -f "$C" >/dev/null 2>&1 || true
          fi
        done
      }
      trap cleanup EXIT
       
      run_server() {
        NAME=$1; NETWORK=$2; PORT=$3; ID=$4; shift 4
        docker run -d --name "$NAME" --label "io.encryptiondbfuzz.owner=$OWNER" \
          --network "$NETWORK" --read-only --cap-drop ALL \
          --security-opt no-new-privileges=true --pids-limit 180 --memory 1100m \
          --cpus 1 --user 999:999 \
          --tmpfs /var/lib/mysql:rw,nosuid,nodev,size=450m,uid=999,gid=999 \
          --tmpfs /run/mysqld:rw,nosuid,nodev,size=16m,uid=999,gid=999 \
          --tmpfs /tmp:rw,nosuid,nodev,size=500m,uid=999,gid=999 \
          --env MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 "$IMAGE" \
          --port="$PORT" --bind-address=127.0.0.1 --skip-name-resolve \
          --server-id="$ID" "$@" >/dev/null
      }
      sql() { docker exec --user 999:999 "$1" mariadb --protocol=socket --socket=/run/mysqld/mysqld.sock -uroot --batch --skip-column-names -e "$2"; }
      ready() {
        for I in $(seq 1 90); do
          V=$(sql "$1" "SELECT CONCAT(VERSION(),'|',@@port)" 2>/dev/null || true)
          [ "$V" = "12.3.3-MariaDB-ubu2404-log|$2" ] || [ "$V" = "12.3.3-MariaDB-ubu2404|$2" ] || { sleep 1; continue; }
          return 0
        done
        return 1
      }
       
      run_server "$S1" none 3307 31 --log-bin=mariadb-bin
      ready "$S1" 3307
      run_server "$S2" "container:$S1" 3308 32 --log-bin=mariadb-bin
      ready "$S2" 3308
      run_server "$R" "container:$S1" 3306 33 --relay-log=relay-bin
      ready "$R" 3306
       
      for S in "$S1" "$S2"; do
        sql "$S" "CREATE USER repl@'127.0.0.1' IDENTIFIED BY 'replpass'; GRANT REPLICATION SLAVE ON *.* TO repl@'127.0.0.1'"
      done
      read F1 P1 <<EOF
      $(sql "$S1" 'SHOW MASTER STATUS' | awk '{print $1, $2}')
      EOF
      read F2 P2 <<EOF
      $(sql "$S2" 'SHOW MASTER STATUS' | awk '{print $1, $2}')
      EOF
      sql "$R" "CHANGE MASTER TO MASTER_HOST='127.0.0.1',MASTER_PORT=3307,MASTER_USER='repl',MASTER_PASSWORD='replpass',MASTER_LOG_FILE='$F1',MASTER_LOG_POS=$P1,MASTER_CONNECT_RETRY=1; CHANGE MASTER 'analytics' TO MASTER_HOST='127.0.0.1',MASTER_PORT=3308,MASTER_USER='repl',MASTER_PASSWORD='replpass',MASTER_LOG_FILE='$F2',MASTER_LOG_POS=$P2,MASTER_CONNECT_RETRY=1; START ALL SLAVES"
       
      for I in $(seq 1 60); do
        STATE=$(sql "$R" 'SHOW ALL SLAVES STATUS' | awk -F '\t' '{print $1 ":" $13 ":" $14}' | paste -sd'|' -)
        [ "$STATE" = ':Yes:Yes|analytics:Yes:Yes' ] && break
        sleep 1
      done
      echo "before=$STATE"
       
      docker exec --detach --user 999:999 "$R" sh -c \
        "i=0; : >/tmp/channel-poll; while test \$i -lt 500; do mariadb --protocol=socket --socket=/run/mysqld/mysqld.sock -uroot --batch --skip-column-names -e 'SHOW ALL SLAVES STATUS' 2>/dev/null | awk -F '\\t' '{print \$1 \":\" \$13 \":\" \$14}' | paste -sd'|' - >>/tmp/channel-poll; i=\$((i+1)); sleep 0.02; done" >/dev/null
      sleep 0.1
      docker exec --user 999:999 "$R" timeout 150 mariadb-backup --backup \
        --safe-slave-backup --safe-slave-backup-timeout=9 --protocol=socket \
        --socket=/run/mysqld/mysqld.sock --user=root --target-dir=/tmp/backup \
        2>&1 | tee /tmp/mcf204-backup.log
      sleep 1
      docker exec --user 999:999 "$R" sort -u /tmp/channel-poll | grep -E '^:Yes:(Yes|No)\|analytics:Yes:Yes$'
      STATE=$(sql "$R" 'SHOW ALL SLAVES STATUS' | awk -F '\t' '{print $1 ":" $13 ":" $14}' | paste -sd'|' -)
      echo "after=$STATE"
      grep 'completed OK!' /tmp/mcf204-backup.log
      

      Actual result

      The significant output from the reproduced run was:

      before=:Yes:Yes|analytics:Yes:Yes
      :Yes:No|analytics:Yes:Yes
      :Yes:Yes|analytics:Yes:Yes
      after=:Yes:Yes|analytics:Yes:Yes
      [00] ... completed OK!
      

      Forty-seven complete polling snapshots were captured. At least one had the
      default SQL thread stopped while the named channel continued. All three
      servers remained healthy; no crash was observed.

      Expected result

      `--safe-slave-backup` should either stop and later restart the SQL thread for
      every active replication channel, or reject multi-source replicas explicitly.
      It should not claim a safe backup while one channel is still applying events.

      Attachments

        Issue Links

          Activity

            People

              Deepthi ES Deepthi Eranti Sreenivas
              csfuzz csfuzz
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.