Testing on Amazon EC2 with latest Centos 7 build, same as Daniel was using.

Issue number #1 - Stack configurations tested: 2pm combo, 2umpm
Install the replication stack, verified server-id are correct, but databases are not replicated.

• • • • I Couldn't reproduce

1. performed clean install on a 2 um/pm with local storage and mysqlrep enabled
pm1

MariaDB [(none)]> show databases;
---------------------

---------------------

---------------------
8 rows in set (0.00 sec)

pm2

MariaDB [(none)]> show databases;
---------------------

---------------------

---------------------
8 rows in set (0.00 sec)

MariaDB [(none)]>

2. create database pm1 and its replicated on pm2

pm1

MariaDB [(none)]> create database david;
Query OK, 1 row affected (0.00 sec)

pm2

MariaDB [(none)]> show databases;
---------------------

---------------------

---------------------
9 rows in set (0.00 sec)

3. create table on pm1 and its replicated on pm2

pm1

MariaDB [(none)]> use david
Database changed
MariaDB [david]> create table tmp (c1 int ) engine=columnstore;
Query OK, 0 rows affected (0.22 sec)

pm2

MariaDB [(none)]> use david
Database changed
MariaDB [david]> show tables;
-----------------

-----------------

-----------------
1 row in set (0.00 sec)

Issue 2 - Shutdownsystem and startsystem, system seemed to be normal. But mcsmysql on um2 failed due to missing mysql.sock file. I verified the mysqld process is running and supposed to be using the mysql.sock.

• • • • I couldn't reproduce

1. did a shutdown on pm1, mysqld not running on pm1 or pm2 afterwards. running commands from pm1

ma shutd y
shutdownsystem Wed Nov 1 15:13:37 2017

This command stops the processing of applications on all Modules within the MariaDB ColumnStore System

Checking for active transactions

Stopping System...
Successful stop of System

Shutting Down System...
Successful shutdown of System

[root@ip-172-30-0-161 ~]# ps -ef | grep mysqld
root 26595 17590 0 15:14 pts/0 00:00:00 grep --color=auto mysqld

2. did a startsystem

pm1

ma startsystem Calpont1
startsystem Wed Nov 1 15:15:38 2017
startSystem command, 'columnstore' service is down, sending command to
start the 'columnstore' service on all modules

System being started, please wait.......................
Successful start of System

[root@ip-172-30-0-161 ~]# mcsmysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.2.9-MariaDB-log Columnstore 1.1.1-1

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

pm2

mcsmysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 13
Server version: 10.2.9-MariaDB-log Columnstore 1.1.1-1

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

Issue 3 - Thinking that some processes may not have been started, I tried the enableMySQLRep commands while MySQLRep also has been enabled and got api errors:

• • • I couldn't reproduce

1. enableMySqlRep from pm1 console - worked

mcsadmin> enableMySQLReplication
enablemysqlreplication Wed Nov 1 15:19:13 2017

MariaDB ColumnStore Replication Feature is already enabled
Do you want to proceed: (y or n) [n]: y

Enter the 'User' Password or 'ssh' if configured with ssh-keys
Please enter: Calpont1

Successful Enabling of MariaDB ColumnStore Replication

MariaDB ColumnStore Replication Master Node is pm1

2. disableMySQLReplication - worked

disableMySQLReplication
disablemysqlreplication Wed Nov 1 15:20:23 2017

Successful Disable of MariaDB ColumnStore Replication

So I couldn't reproduce the issue Daniels had. He is running on VMS, I'm running on Amazon EC2. My local workstation VMs don't work to the point I can do multi-node testing. So cant try to reproduce in VM environment. I tried with both binary and rpm package installs, root install

Maybe someone else can try with VMs. Centos 7.

I have a feeling that it is a timing issue in 1.1.1-1. I did the exact same test using 1.1.0-1 beta, it worked fine.

Fixed in develop-1.1

commit c84ea561e1b6006b526b370bc5f8ed5c4b2285fb
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 12:10:01 2017 -0600

MCOL-998 - added in binlog_format

Added in binlog_fotmat=row into the released my.cnf .

Build verified: 1.1.2 Git source

/root/columnstore/mariadb-columnstore-server
commit ed21e674cfd70db421957d0212ff7fc1835d06d5
Author: david hill <david.hill@mariadb.com>
Date: Mon Nov 6 08:41:53 2017 -0600

Update README.md

updated version

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 1603ce925f4cbffef8754663ae0a3058bafc4c65
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 13:06:45 2017 -0600

rechecking back in version of calpontsystemcatalog, accidental commit old versions

Tested with 2pm.combo and 1um2pm.localquery

2pm.combo worked

1um2pm.localquery is not replicating, including database and tables. Restarting the stack did not help. "show slave status\G" on PMs returned empty set. I verified the bingo_format=ROW statement in the my.cnf file in both UM and PMs.

When I tried enablemysqlreplication, replication is working. "Show slave status\G" on PMs now returned normal status output. Database and table are now replicated to PMs.

mcsadmin> enablemysql
enablemysqlreplication Wed Nov 8 22:34:26 2017

MariaDB ColumnStore Replication Feature is already enabled
Do you want to proceed: (y or n) [n]: y

Enter the 'User' Password or 'ssh' if configured with ssh-keys
Please enter: vagrant

Successful Enabling of MariaDB ColumnStore Replication

MariaDB ColumnStore Replication Master Node is um1

postConfigure reported the following error:

Found this error in PM1's debug.log

Nov 8 22:57:46 localhost writeengine[9240]: 46.787258 |0|0|0| C 19 CAL0060: dbbuilder system catalog error: Creating TableName column OID: 1001 The File already exists. [BRM error status: UNKNOWN (255)]
Nov 8 22:59:47 localhost IDBFile[8551]: 47.480105 |0|0|0| D 35 CAL0002: Failed to open file: /usr/local/mariadb/columnstore/data1/versionbuffer.cdf, exception: unable to open Buffered file

But create database and tables, DDL, DML, and cpimport all worked on UM1, just not replicating

This is a new installation and no system catalog should have existed.

On my 1um / 2pm with local query, salves were setup on pm1 and pm2 and the replication was working correctly... I couldn't reproduce the issue Daniel reported.

on local query 1um / 2pm install, infinidb_local_query=0 and it should be 1 on um1

The dbbuilder system catalog error does not always happen. It happened 1 out of 3 installation this morning. When it does happen, postConfigure skips replication setup and cause the test for this ticket to fail. I have created MCOL-1024 to track the issue.

Build verified: 1.1.2 GitHub source

/root/columnstore/mariadb-columnstore-server
commit ed21e674cfd70db421957d0212ff7fc1835d06d5
Author: david hill <david.hill@mariadb.com>
Date: Mon Nov 6 08:41:53 2017 -0600

Update README.md

updated version

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit e0febdcd1a88bc3fab55519bd679f93719687ccc
Merge: 1603ce9 f1dd92a
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 15:32:50 2017 -0600

Merge pull request #313 from mariadb-corporation/MCOL-989

MCOL-989