[MDEV-9785] Galera node cant connect to cluster Created: 2016-03-24  Updated: 2016-03-30  Resolved: 2016-03-30

Status: Closed
Project: MariaDB Server
Component/s: Galera, Galera SST
Affects Version/s: 10.1.12
Fix Version/s: 10.1.13

Type: Bug Priority: Major
Reporter: Alexey Teterin Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 0
Labels: galera, sst, xtrabackup
Environment:

ubuntu 14.04 x64


Issue Links:
Relates
relates to MDEV-9498 MariaDB server with Galera replicatio... Closed

 Description   

Hello am using Galera with 10.1.12-MariaDB and SST method is xtrabackup-v2

please dont recommend SST=rsync it's not work for me

I have healthy cluster 8 nodes, sometimes one or few nodes goes down.
im just `service mysql` start on it and they successfully connecting to cluster and all is OK.

BUT sometimes, when disconnected nodes down few days i cant connect they to cluster.

after few tries im
rm -fr /var/lib/mysql/* & rm -fr /var/log/mysql/*
and nothing too, they have this message in syslog:

mysqld: [ERROR] Binlog file '/var/log/mysql/mariadb-bin.003079' not found in binlog index, needed for recovery. Aborting.

i know how work with this, i can recover cluster when i have nodes which can't connect to cluster with message above, so i do this:

1. shutdown all nodes, and leave only one node
2. shutdown last node and rm -fr /var/log/mysql/*
3. bootstrap this last node with deleted binlog
4. connect other nodes to cluster service mysql start
5. profit - all is OK

But problem is:

I cant down all production nodes, and down last node too, because i have 8 nodes to serve big site traffic and one running node immediately down when all traffic goes to it (of course because overload)

QUESTION IS:

Please help me. How connect nodes to cluster when they won't connect and have error
mysqld: [ERROR] Binlog file '/var/log/mysql/mariadb-bin.003079' not found in binlog index, needed for recovery. Aborting.



 Comments   
Comment by Nirbhay Choubey (Inactive) [ 2016-03-24 ]

at@rustex.ru Could share some more details?

  • The binlog index file on joiner node after this failure.
  • The binlog index file on donor node, and
  • last couple of binlog files from donor node.

Also, why isn't rsync SST working for you?

Comment by Alexey Teterin [ 2016-03-25 ]

it working but very terrible. DB size is 6.5 Gb.
When im use SST=rsync i have this bad things:

  • joining node always drop donor node (after transfer complete - donor goes up)
  • joining is a long time operation, xtrabackup-v2 make it more faster
  • joining fails more often
  • maybe, nodes sometimes drops from cluster self, xtrabackup-v2 feel more stable (not shure)

in my case xtrabackup-v2 feels more stable and easy.

Nirbay, please tell me - do you have some ideas about my trouble?
why mysql cant find bin log when joining to cluster?
When node is joining it download bin log from donor? (or not? im using sst xtrabackup-v2)

Generated at Thu Feb 08 07:37:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.