Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9785

Galera node cant connect to cluster

Details

    Description

      Hello am using Galera with 10.1.12-MariaDB and SST method is xtrabackup-v2

      please dont recommend SST=rsync it's not work for me

      I have healthy cluster 8 nodes, sometimes one or few nodes goes down.
      im just `service mysql` start on it and they successfully connecting to cluster and all is OK.

      BUT sometimes, when disconnected nodes down few days i cant connect they to cluster.

      after few tries im
      rm -fr /var/lib/mysql/* & rm -fr /var/log/mysql/*
      and nothing too, they have this message in syslog:

      mysqld: [ERROR] Binlog file '/var/log/mysql/mariadb-bin.003079' not found in binlog index, needed for recovery. Aborting.

      i know how work with this, i can recover cluster when i have nodes which can't connect to cluster with message above, so i do this:

      1. shutdown all nodes, and leave only one node
      2. shutdown last node and rm -fr /var/log/mysql/*
      3. bootstrap this last node with deleted binlog
      4. connect other nodes to cluster service mysql start
      5. profit - all is OK

      But problem is:

      I cant down all production nodes, and down last node too, because i have 8 nodes to serve big site traffic and one running node immediately down when all traffic goes to it (of course because overload)

      QUESTION IS:

      Please help me. How connect nodes to cluster when they won't connect and have error
      mysqld: [ERROR] Binlog file '/var/log/mysql/mariadb-bin.003079' not found in binlog index, needed for recovery. Aborting.

      Attachments

        Issue Links

          Activity

            nirbhay_c Nirbhay Choubey (Inactive) added a comment - - edited

            at@rustex.ru Could share some more details?

            • The binlog index file on joiner node after this failure.
            • The binlog index file on donor node, and
            • last couple of binlog files from donor node.

            Also, why isn't rsync SST working for you?

            nirbhay_c Nirbhay Choubey (Inactive) added a comment - - edited at@rustex.ru Could share some more details? The binlog index file on joiner node after this failure. The binlog index file on donor node, and last couple of binlog files from donor node. Also, why isn't rsync SST working for you?

            it working but very terrible. DB size is 6.5 Gb.
            When im use SST=rsync i have this bad things:

            • joining node always drop donor node (after transfer complete - donor goes up)
            • joining is a long time operation, xtrabackup-v2 make it more faster
            • joining fails more often
            • maybe, nodes sometimes drops from cluster self, xtrabackup-v2 feel more stable (not shure)

            in my case xtrabackup-v2 feels more stable and easy.

            Nirbay, please tell me - do you have some ideas about my trouble?
            why mysql cant find bin log when joining to cluster?
            When node is joining it download bin log from donor? (or not? im using sst xtrabackup-v2)

            at@rustex.ru Alexey Teterin added a comment - it working but very terrible. DB size is 6.5 Gb. When im use SST=rsync i have this bad things: joining node always drop donor node (after transfer complete - donor goes up) joining is a long time operation, xtrabackup-v2 make it more faster joining fails more often maybe, nodes sometimes drops from cluster self, xtrabackup-v2 feel more stable (not shure) in my case xtrabackup-v2 feels more stable and easy. Nirbay, please tell me - do you have some ideas about my trouble? why mysql cant find bin log when joining to cluster? When node is joining it download bin log from donor? (or not? im using sst xtrabackup-v2)

            People

              nirbhay_c Nirbhay Choubey (Inactive)
              at@rustex.ru Alexey Teterin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.