[MCOL-938] columnstore replication failover from master updates server-id Created: 2017-09-26  Updated: 2023-10-26  Resolved: 2017-10-31

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.0.11
Fix Version/s: 1.1.1

Type: Bug Priority: Major
Reporter: David Thompson (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 2
Labels: None

Sprint: 2017-20, 2017-21, 2017-22

 Description   

If you have a multi node system with replication then if UM1 is to go down then the system will automate failover to another UM, however say UM2 has it' server-id value changed to 1 from 2. This is bad and we should simply make UM2 the new master only. This will avoid problems with InnoDB replication (no impact to ColumnStore tables) Presumably this was done for a reason so should make sure that we cover this.



 Comments   
Comment by David Hill (Inactive) [ 2017-10-23 ]

fixed

1.0.12

commit 58a77b708cb0306892cbfd7a1be3b72246a0dbf6
Author: david hill <david.hill@mariadb.com>
Date: Mon Oct 23 14:50:43 2017 -0500

MCOL-938

procmon/processmonitor.cpp | 13 ++++++-------

1.1.1

commit 8706f0a9ef39329331761980df483806f942b0b8
Author: david hill <david.hill@mariadb.com>
Date: Mon Oct 23 15:04:56 2017 -0500

MCOL-938

procmon/processmonitor.cpp | 6 +++---

1.2

commit 4e7e8487f7c711405dd1fccde17c84b181c3dfd3
Author: david hill <david.hill@mariadb.com>
Date: Mon Oct 23 15:03:40 2017 -0500

MCOL-938

procmon/processmonitor.cpp | 6 +++---

------------------------------------------------------------------------------
Change - previous code would always set the MasterRep serer-id to '1' in my.cnf and all slaves to X as the id from ExeMgrX from Columnstore.xml.
New code will set both master/slave server-id to ExeMgrX from Columnstore.xml. So server-id in the my.cnf doesn't change now during failover

To test:

install system with replication set, do a failover on um1 to cause um1 to go active. Bring back um1. check both nodes to make sure the ID server-id doesnt change

Comment by Daniel Lee (Inactive) [ 2017-10-26 ]

Build tested: Github source for 1.0.12

/root/columnstore/mariadb-columnstore-server
commit a42eb6d1e74e44c9e8fd9bb8290e6ce7dbf909f5
Merge: 2965fc8 6a14ced
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Oct 3 10:12:33 2017 -0500

Merge pull request #69 from mariadb-corporation/MCOL-940

MCOL-940

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit a8414b9a8f586917e20c2688f053d30d36b725d4
Merge: 22f5c04 16990c8
Author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Date: Wed Oct 25 14:28:31 2017 +0300

Merge pull request #297 from mariadb-corporation/MCOL-985-1.0

MCOL-985 Add return code test after call to buildReturnedColumn

Build tested: Github source for 1.1.1

/root/columnstore/mariadb-columnstore-server
commit f6cd94ea167789970db7b5b501569a6549495d10
Merge: 3d846d3 91b2553
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Oct 24 09:15:58 2017 -0500

Merge pull request #72 from mariadb-corporation/MCOL-982

MCOL-982 Merge MariaDB 10.2.9

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit a1c82cb2b7cfc501b64843f1cc31336303e79051
Merge: 1360de6 1601d6f
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Wed Oct 25 15:59:42 2017 -0500

Merge pull request #299 from mariadb-corporation/mcol-667

Mcol 667

For both 1.0 and 1.1, I did a 2pm combo system. After the system came up, both PMs have "server-id = 1" in my.cnf. server-id should be unique among servers

For 1.1,

PM1:

MariaDB [(none)]> show master status;
-------------------------------------------------------+

File Position Binlog_Do_DB Binlog_Ignore_DB

-------------------------------------------------------+

mysql-bin.000003 150502085    

-------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> show slave status;
Empty set (0.00 sec)

PM2:

MariaDB [(none)]> show master status;
------------------------------------------------------+

File Position Binlog_Do_DB Binlog_Ignore_DB

------------------------------------------------------+

mysql-bin.000002 342    

------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [(none)]> show slave status;
Empty set (0.00 sec)

Comment by David Hill (Inactive) [ 2017-10-26 ]

fixed....

commit af35345273b85b55f755303abc8cb0d1bea5fb89
Author: david hill <david.hill@mariadb.com>
Date: Thu Oct 26 17:38:25 2017 -0500

mcol-938 - fixed pm server ids

procmon/processmonitor.cpp | 29 ++---------------------------

[root@ip-172-30-0-161 mysql]# grep server-id my.cnf
server-id = 2
[root@ip-172-30-0-161 mysql]#
[root@ip-172-30-0-161 mysql]#
[root@ip-172-30-0-161 mysql]# ma getsystemn
getsystemnetworkconfig Thu Oct 26 22:36:49 2017

System Network Configuration

Module Name Module Description NIC ID Host Name IP Address
----------- ------------------------- ------ ------------------------------------------ ---------------
um1 User Module #1 1 um1 172.30.0.59
pm1 Performance Module #1 1 ip-172-30-0-161.us-west-2.compute.internal 172.30.0.161
pm2 Performance Module #2 1 pm2 172.30.0.152

[root@ip-172-30-0-161 mysql]# ssh 172.30.0.152
Last failed login: Thu Oct 26 21:34:07 UTC 2017 from 195.144.238.68 on ssh:notty
There were 41 failed login attempts since the last successful login.
Last login: Thu Oct 26 15:46:11 2017 from ip-172-30-0-161.us-west-2.compute.internal

------------------------------------
MariaDB ColumnStore Amazon Linux AMI
------------------------------------

Take Control of Your Data with MariaDB
Columnstore - The High Performance, Scalable
Data Platform that enables Deep Analytics

www.mariadb.com

---------------------------------------------

Please refer to the README file for
information on using this AMI

[root@ip-172-30-0-152 ~]# home
[root@ip-172-30-0-152 columnstore]# cd mysql/
[root@ip-172-30-0-152 mysql]# grep server-id my.cnf
server-id = 3
[root@ip-172-30-0-152 mysql]#

Comment by Daniel Lee (Inactive) [ 2017-10-27 ]

Build tested: Github source for 1.1.1

/root/columnstore/mariadb-columnstore-server
commit f6cd94ea167789970db7b5b501569a6549495d10
Merge: 3d846d3 91b2553
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Oct 24 09:15:58 2017 -0500

Merge pull request #72 from mariadb-corporation/MCOL-982

MCOL-982 Merge MariaDB 10.2.9

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit af35345273b85b55f755303abc8cb0d1bea5fb89
Author: david hill <david.hill@mariadb.com>
Date: Thu Oct 26 17:38:25 2017 -0500

mcol-938 - fixed pm server ids

I did a combo installation and still have the same issues. Both PMs have server id of 1.

Comment by Daniel Lee (Inactive) [ 2017-10-27 ]

Build tested: Github source for 1.0.12

/root/columnstore/mariadb-columnstore-server
commit a42eb6d1e74e44c9e8fd9bb8290e6ce7dbf909f5
Merge: 2965fc8 6a14ced
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Oct 3 10:12:33 2017 -0500

Merge pull request #69 from mariadb-corporation/MCOL-940

MCOL-940

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 28d26c89018faa3ec02fd49559b2fb53e6847e97
Merge: a8414b9 5ab7538
Author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Date: Thu Oct 26 20:22:27 2017 +0300

Merge pull request #304 from mariadb-corporation/MCOL-979-1.0

MCOL-979 getNullValueByType() should return string for all char types

Comment by David Hill (Inactive) [ 2017-10-30 ]

test fix, working for combo 2pm now... need to retest other install types

from pm2

[root@ip-172-30-0-152 ~]# home
[root@ip-172-30-0-152 columnstore]# cd mysql/
[root@ip-172-30-0-152 mysql]# grep server-id my.cnf
server-id = 2
[root@ip-172-30-0-152 mysql]# mcsmysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 16
Server version: 10.2.9-MariaDB-log Columnstore 1.1.1-1

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show slave status\G;

                                                      • 1. row ***************************
                                                        Slave_IO_State: Waiting for master to send event
Comment by David Hill (Inactive) [ 2017-10-30 ]

fixed

develop-1.1

commit 6277371669953e000ae91ea92466d0f4907722f4
Author: david hill <david.hill@mariadb.com>
Date: Mon Oct 30 13:42:31 2017 -0500

MCOL-938 - fixed server-id setup on combo systems

oam/install_scripts/rsync.sh | 40 +++++++++++++++++++++-------------------
oamapps/postConfigure/helpers.cpp | 26 +++++++++++++++++++-------
procmgr/processmanager.cpp | 9 ++++-----
procmon/processmonitor.cpp | 15 ++++++++++-----
4 files changed, 54 insertions, 36 deletions

Comment by Daniel Lee (Inactive) [ 2017-10-31 ]

Build tested: 1.1.1-1 Centos 7 rpm package released to the shared today.

I performed an installation of 2pm combo, the server-id are correct (pm1=1, pm2=2), and the MySQLRep flag was setup to y. But databases on PM1 are not being replicated to PM2.

Comment by David Hill (Inactive) [ 2017-10-31 ]

tried to reproduce Daniels issue with both binary and rpm packages and couldn't. I was able to create database and tables and they were getting replicated on pm2..

ma getso
getsoftwareinfo Tue Oct 31 19:36:29 2017

Name : mariadb-columnstore-platform
Version : 1.1.1
Release : 1
Architecture: x86_64
Install Date: Tue 31 Oct 2017 07:28:12 PM UTC
Group : Applications/Databases
Size : 113946160
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.
Signature : (none)
Source RPM : mariadb-columnstore-platform-1.1.1-1.src.rpm
Build Date : Tue 31 Oct 2017 05:19:30 PM UTC

Comment by Daniel Lee (Inactive) [ 2017-10-31 ]

Build verified: 1.1.1-1 rpm package released to QA today

Verified, Fixed

Build tested: 1.0.12-1 GitHub source

Still having an issue

/root/columnstore/mariadb-columnstore-server
commit a42eb6d1e74e44c9e8fd9bb8290e6ce7dbf909f5
Merge: 2965fc8 6a14ced
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Oct 3 10:12:33 2017 -0500

Merge pull request #69 from mariadb-corporation/MCOL-940

MCOL-940

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 28d26c89018faa3ec02fd49559b2fb53e6847e97
Merge: a8414b9 5ab7538
Author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Date: Thu Oct 26 20:22:27 2017 +0300

Merge pull request #304 from mariadb-corporation/MCOL-979-1.0

MCOL-979 getNullValueByType() should return string for all char types

Tested on a 2um2pm stack. Because of the issues in MCOL-998, I had to use the following steps to get a working replication stack:

1. Installed a 2um2pm stack with mysqlrep off
2. Enable mysqlrep
3. Verified mysqlrep is working

Testing steps

4. suspended um1 VM (in vagrant)
um failed over to um2 (noticed the issue documented in MCOL-983)
UM2's server-id in my.cnf did not change (that's what the fix is for)
5. resumed UM1, UM1 recovered
Now UM1's server-id is also 2.

At the end, both UM1 and UM2 have the same server-id, 2.

Comment by Daniel Lee (Inactive) [ 2017-10-31 ]

This ticket is now for 1.1.1-1 only. MCOL-999 has been opened to track the issue found in 1.0.12-1. Closing it.

Generated at Thu Feb 08 02:24:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.