[MCOL-4291] columnstore-docker-cluster: single node cluster can't tolerate container restart - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Not a Bug
Affects Version/s: None
Fix Version/s: N/A
Component/s: Docker
Labels:
- skysql
Environment:
mariadb/skysql-enterprise-columnstore-handoff sky-1.5.4-1 95e34753d40f

Description

The columnstore-docker-cluster image (c.f. environment) doesn't handle container restarts in case the ColumnStore cluster only comprises one ColumnStore node. After the ColumnStore node is restarted queries to the ColumnStore engine aren't possible anymore.

How to reproduce:
Unzip attached columnstore-docker-cluster-sn.tar.gz and follow the steps as shown below.

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker-compose up -d

Creating network "columnstore-docker-cluster-sn_default" with the default driver

Creating volume "columnstore-docker-cluster-sn_data1" with default driver

Creating volume "columnstore-docker-cluster-sn_storagemanager" with default driver

Creating mcs1 ... done

Creating mx1  ... done

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker logs mcs1

Starting Cluster Manager API...

CMAPI PID = 552

Starting MariaDB.200903 12:14:17 mysqld_safe Logging to '/var/log/mariadb/columnstore/mariadb-error.log'.

200903 12:14:17 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql

 SUCCESS!

 New Monit id: 9da5b60feaf161f9f6ec354154faef3a

 Stored in '/root/.monit.id'

Starting Monit 5.26.0 daemon with http interface at [localhost]:2812

Monit will delay for 60s on first start after reboot ...

'mcs1' Monit 5.26.0 started

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker logs mx1

Starting...

MaxScale PID = 16

 New Monit id: 919a65a61e5fa2dff4c326bbc79d09e8

 Stored in '/root/.monit.id'

Starting Monit 5.26.0 daemon with http interface at [localhost]:2812

'mx1' Monit 5.26.0 started

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker cp demo mcs1:/usr/bin/demo

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker exec -it mcs1 cat /usr/bin/demo

#!/bin/bash

CMAPI_KEY="${CMAPI_KEY:-somekey123}"

IFLAG='/etc/columnstore/container-initialized'

MAX_USER="${MAX_USER:-maxscale}"

MAX_PASS="${MAX_PASS:-C0lumnStore!}"

REP_USER="${REP_USER:-idbrep}"

REP_PASS="${REP_PASS:-C0lumnStore!}"

RED='\033[0;31m'

GREEN='\033[0;32m'

YELLOW='\033[0;33m'

NC='\033[0m'

CMAPI="curl --write-out '%{http_code}' --silent --output /dev/null -X PUT https://${PM1}:8640/cmapi/0.4.0/cluster"

MAXAPI="curl --write-out '%{http_code}' --silent --output /dev/null -X POST -u admin:mariadb ${MX1}:8989/v1"

HEADER1="'Content-Type:application/json'"

HEADER2="'x-api-key:$CMAPI_KEY'"

PAYLOAD1='{"timeout":60, "node": "'$PM1'"}'

PAYLOAD2='{"timeout":60, "node": "'$PM2'"}'

PAYLOAD3='{"timeout":60, "node": "'$PM3'"}'

PAYLOAD4='{"data":{"id":"pm3","type":"servers","attributes":{"parameters":{"address":"'$PM3'","protocol":"MariaDBBackend"}}}}'

PAYLOAD5='{"data":{"id":"pm2","type":"servers","attributes":{"parameters":{"address":"'$PM2'","protocol":"MariaDBBackend"}}}}'

PAYLOAD6='{"data":{"id":"pm1","type":"servers","attributes":{"parameters":{"address":"'$PM1'","protocol":"MariaDBBackend"}}}}'

#PAYLOAD7='{"data":{"id":"mcs_service","type":"services","attributes":{"router":"readwritesplit","parameters":{"user":"'$MAX_USER'","password":"'$MAX_PASS'"}},"relationships":{"servers":{"data":[{"id":"pm1","type":"servers"},{"id":"pm2","type":"servers"},{"id":"pm3","type":"servers"}]}}}}'

PAYLOAD7='{"data":{"id":"mcs_service","type":"services","attributes":{"router":"readwritesplit","parameters":{"user":"'$MAX_USER'","password":"'$MAX_PASS'"}},"relationships":{"servers":{"data":[{"id":"pm1","type":"servers"}]}}}}'

PAYLOAD8='{"data":{"id":"mcs_listener","type":"listeners","attributes":{"parameters":{"protocol":"MariaDBClient","port":3306}},"relationships":{"services":{"data":[{"id":"mcs_service","type":"services"}]}}}}'

#PAYLOAD9='{"data":{"id":"mcs_monitor","type":"monitors","attributes":{"module":"mariadbmon","parameters":{"monitor_interval":1000,"user":"'$MAX_USER'","password":"'$MAX_PASS'","replication_user":"'$REP_USER'","replication_password":"'$REP_PASS'","detect_replication_lag":true,"enforce_read_only_slaves":true,"auto_failover":true,"auto_rejoin":true}},"relationships":{"servers":{"data":[{"id":"pm1","type":"servers"},{"id":"pm2","type":"servers"},{"id":"pm3","type":"servers"}]}}}}'

PAYLOAD9='{"data":{"id":"mcs_monitor","type":"monitors","attributes":{"module":"mariadbmon","parameters":{"monitor_interval":1000,"user":"'$MAX_USER'","password":"'$MAX_PASS'","replication_user":"'$REP_USER'","replication_password":"'$REP_PASS'","detect_replication_lag":true,"enforce_read_only_slaves":true,"auto_failover":true,"auto_rejoin":true}},"relationships":{"servers":{"data":[{"id":"pm1","type":"servers"}]}}}}'

RESOURCE1='add-node'

RESOURCE2='servers'

RESOURCE3='services'

RESOURCE4='listeners'

RESOURCE5='monitors'

#MODULES=("$CMAPI/$RESOURCE1 --header $HEADER1 --header $HEADER2 --data '$PAYLOAD1' -k" "$CMAPI/$RESOURCE1 --header $HEADER1 --header $HEADER2 --data '$PAYLOAD2' -k" "$CMAPI/$RESOURCE1 --header $HEADER1 --header $HEADER2 --data '$PAYLOAD3' -k")

MODULES=("$CMAPI/$RESOURCE1 --header $HEADER1 --header $HEADER2 --data '$PAYLOAD1' -k")

#SERVERS=("$MAXAPI/$RESOURCE2 -d '$PAYLOAD4'" "$MAXAPI/$RESOURCE2 -d '$PAYLOAD5'" "$MAXAPI/$RESOURCE2 -d '$PAYLOAD6'")

SERVERS=("$MAXAPI/$RESOURCE2 -d '$PAYLOAD6'")

SERVICE="$MAXAPI/$RESOURCE3 -d '$PAYLOAD7'"

LISTENER="$MAXAPI/$RESOURCE4 -d '$PAYLOAD8'"

MONITOR="$MAXAPI/$RESOURCE5 -d '$PAYLOAD9'"

# Wait For Primary Node To Start

if [ ! -f $IFLAG ]; then

    echo -n 'Waiting for PM1 to be initialized '

    while [ ! -f $IFLAG ]

do

        printf "."

        sleep 2

    done

    printf " ${GREEN}done${NC}\n"

fi

sleep 120

# Add Nodes To Cluster

count=1

for i in "${MODULES[@]}"

do

    RESULT1=$(bash -c "$i")

    if [[ "$RESULT1" == 200 ]]; then

        printf "Adding PM$count to cluster ... ${GREEN}done${NC}\n"

    else

        printf "Adding PM$count to cluster ... ${RED}fail${NC}\n"

        exit 1

fi

    (( count++ ))

done

sleep 5

# Validate

QUERY=$(mariadb -sN -e "DROP DATABASE IF EXISTS validate; CREATE DATABASE validate; CREATE TABLE validate.sample (id INT) ENGINE=columnstore; INSERT INTO validate.sample VALUES (99); SELECT id FROM validate.sample;" 2>&1)

if [ $? = 0 ] && [ $QUERY = 99 ]; then

    printf "Validating ... ${GREEN}done${NC}\n"

    mariadb -e "DROP DATABASE IF EXISTS validate;" 2>&1

else

    printf "Validating ... ${RED}fail${NC}\n"

    exit 1

fi

# Add Servers To MaxScale

count=1

for i in "${SERVERS[@]}"

do

    RESULT1=$(bash -c "$i")

    if [[ "$RESULT1" == 403 ]]; then

        printf "Adding PM$count to MaxScale ... ${YELLOW}already exists${NC}\n"

    else

        if [[ "$RESULT1" == 204 ]]; then

            printf "Adding PM$count to MaxScale ... ${GREEN}done${NC}\n"

        else

            printf "Adding PM$count to MaxScale ... ${RED}fail${NC}\n"

            exit 1

fi

fi

    (( count++ ))

done

# Add Router To MaxScale

RESULT2=$(bash -c "$SERVICE")

if [[ "$RESULT2" == 403 ]]; then

    printf "Adding service to MaxScale ... ${YELLOW}already exists${NC}\n"

else

    if [[ "$RESULT2" == 204 ]]; then

        printf "Adding service ... ${GREEN}done${NC}\n"

    else

        printf "Adding service ... ${RED}fail${NC}\n"

        exit 1

fi

fi

# Add Listener To MaxScale

RESULT3=$(bash -c "$LISTENER")

if [[ "$RESULT3" == 403 ]]; then

    printf "Adding listener to MaxScale ... ${YELLOW}already exists${NC}\n"

else

    if [[ "$RESULT3" == 204 ]]; then

        printf "Adding listener ... ${GREEN}done${NC}\n"

    else

        printf "Adding listener ... ${RED}fail${NC}\n"

        exit 1

fi

fi

# Add Monitor To MaxScale

RESULT4=$(bash -c "$MONITOR")

if [[ "$RESULT4" == 403 ]]; then

    printf "Adding monitor to MaxScale ... ${YELLOW}already exists${NC}\n"

else

    if [[ "$RESULT4" == 204 ]]; then

        printf "Adding monitor ... ${GREEN}done${NC}\n"

    else

        printf "Adding monitor ... ${RED}fail${NC}\n"

        exit 1

fi

fi

unset REP_USER

unset REP_PASS

unset MAX_USER

unset MAX_PASS

#curl -X PATCH -u admin:mariadb ${MX1}:8989/v1/monitors/mcs_monitor/relationships/servers -d '{"data":[]}'

#curl -X PATCH -u admin:mariadb ${MX1}:8989/v1/services/mcs_service/relationships/servers -d '{"data":[]}'

#curl -X DELETE -u admin:mariadb ${MX1}:8989/v1/listeners/mcs_listener

#curl -X DELETE -u admin:mariadb ${MX1}:8989/v1/services/mcs_service

#curl -X DELETE -u admin:mariadb ${MX1}:8989/v1/monitors/mcs_monitor

#curl -X DELETE -u admin:mariadb ${MX1}:8989/v1/servers/pm1

#curl -X DELETE -u admin:mariadb ${MX1}:8989/v1/servers/pm2

#curl -X DELETE -u admin:mariadb ${MX1}:8989/v1/servers/pm3

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker exec -it mcs1 demo

Adding PM1 to cluster ... done

Validating ... done

Adding PM1 to MaxScale ... done

Adding service ... done

Adding listener ... done

Adding monitor ... done

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker exec -it mcs1 mariadb

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 10

Server version: 10.5.6-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select mcssystemready();

+------------------+

| mcssystemready() |

+------------------+

|                1 |

+------------------+

1 row in set (0.001 sec)

MariaDB [(none)]> create database test;

Query OK, 1 row affected (0.000 sec)

MariaDB [(none)]> use test;

Database changed

MariaDB [test]> create table tmp1 (i int) engine=columnstore;

Query OK, 0 rows affected (1.251 sec)

MariaDB [test]> insert into tmp1 values (42);

Query OK, 1 row affected (0.211 sec)

MariaDB [test]> select * FROM tmp1;

+------+

| i    |

+------+

|   42 |

+------+

1 row in set (0.068 sec)

MariaDB [test]> Bye

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker restart mcs1

mcs1

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker logs mcs1

Starting Cluster Manager API...

CMAPI PID = 552

Starting MariaDB.200903 12:14:17 mysqld_safe Logging to '/var/log/mariadb/columnstore/mariadb-error.log'.

200903 12:14:17 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql

 SUCCESS!

 New Monit id: 9da5b60feaf161f9f6ec354154faef3a

 Stored in '/root/.monit.id'

Starting Monit 5.26.0 daemon with http interface at [localhost]:2812

Monit will delay for 60s on first start after reboot ...

'mcs1' Monit 5.26.0 started

'mcs1' unmonitor on user request

Monit daemon with PID 14 awakened

Awakened by User defined signal 1

'mcs1' unmonitor action done

Stopping Cluster Manager API...

Shutting down MariaDB. SUCCESS!

Monit daemon with pid [14] killed

Starting Cluster Manager API...

CMAPI PID = 27

Starting MariaDB.200903 12:20:42 mysqld_safe Logging to '/var/log/mariadb/columnstore/mariadb-error.log'.

200903 12:20:42 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql

 SUCCESS!

Starting Monit 5.26.0 daemon with http interface at [localhost]:2812

Monit will delay for 60s on first start after reboot ...

'mcs1' Monit 5.26.0 started

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker exec -it mcs1 mariadb

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 6

Server version: 10.5.6-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select mcssystemready();

+------------------+

| mcssystemready() |

+------------------+

|                1 |

+------------------+

1 row in set (0.003 sec)

MariaDB [(none)]> select * FROM test.tmp1;

ERROR 1815 (HY000): Internal error: DBRM is not responding. Cannot accept queries

MariaDB [(none)]> select sleep(120);

+------------+

| sleep(120) |

+------------+

|          0 |

+------------+

1 row in set (2 min 0.001 sec)

MariaDB [(none)]> select * FROM test.tmp1;

ERROR 1815 (HY000): Internal error: DBRM is not responding. Cannot accept queries

MariaDB [(none)]> create database test2;

Query OK, 1 row affected (0.000 sec)

MariaDB [(none)]> create table test2.tmp2 (i int) engine=columnstore;

ERROR 1815 (HY000): Internal error: Cannot execute the statement. DBRM is read only!

MariaDB [(none)]> Bye

jens@ubuntu18:~/columnstore-docker-cluster-sn$ docker exec -it mcs1 bash

[root@mcs1 /]# columnstoreDBWrite -c resume

DBRM::send_recv caught: InetStreamSocket::connect: connect() error: Connection refused to: InetStreamSocket: sd: 9 inet: 172.19.0.2 port: 8616

DBRM::send_recv caught: InetStreamSocket::connect: connect() error: Connection refused to: InetStreamSocket: sd: 9 inet: 172.19.0.2 port: 8616

Resume Columnstore Database Writes Request successfully completed

[root@mcs1 /]# mariadb

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 7

Server version: 10.5.6-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select * FROM test.tmp1;

ERROR 1815 (HY000): Internal error: DBRM is not responding. Cannot accept queries

MariaDB [(none)]> create table test2.tmp2 (i int) engine=columnstore;

ERROR 1815 (HY000): Internal error: Cannot execute the statement. DBRM is read only!

MariaDB [(none)]> Bye

[root@mcs1 /]# exit

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

columnstore-docker-cluster-sn.tar.gz
2020-09-03 12:40
2 kB
Jens Röwekamp

Activity

People

Assignee:: Todd Stoffel (Inactive)

Reporter:: Jens Röwekamp (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2020-09-03 12:41

Updated:: 2020-09-25 15:09

Resolved:: 2020-09-25 15:07

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.