[#MXS-2459] Replication error: 1236 after startup Master Slave with Maxscale monitor

Hello,

I am trying to make a Master Slave cluster that is resilient to failover and rejoining.
Whenever I start the cluster the replication starts and ends up on the wrong GTID position and fails with a 1236 error code.

my.cnf for the MariaDB servers

[mariadb]

log-bin

max_allowed_packet=128M

gtid_strict_mode=1

skip-name-resolve

Here is the bash script to start the MariaDB servers

[[ $(hostname) =~ -([0-9]+)$ ]] || exit 1

export server_id=${BASH_REMATCH[1]}

/usr/local/bin/docker-entrypoint.sh mysqld --log-bin=mariadb-bin --binlog-format=ROW --server-id=$((3000 + $server_id)) --log-slave-updates=1 --gtid-strict-mode=1 --innodb-flush-method=fsync --extra-port=3307 --extra_max_connections=1

maxscale.cnf - the configuration file for the maxscale instance

[maxscale]

threads=auto

admin_enabled=false

[server0]

type=server

address=mariadb-0.mariadb.mariadb.svc.cluster.local

port=3306

protocol=MariaDBBackend

persistpoolmax=10

persistmaxtime=999999999999

[server1]

type=server

address=mariadb-1.mariadb.mariadb.svc.cluster.local

port=3306

protocol=MariaDBBackend

persistpoolmax=10

persistmaxtime=999999999999

[server2]

type=server

address=mariadb-2.mariadb.mariadb.svc.cluster.local

port=3306

protocol=MariaDBBackend

persistpoolmax=10

persistmaxtime=999999999999

[Replication-Monitor]

type=monitor

module=mariadbmon

servers=server0,server1,server2

user=*****

password=*****

replication_user=*****

replication_password=*****

monitor_interval=2000

auto_failover=true

auto_rejoin=true

[Splitter-Service]

type=service

router=readwritesplit

servers=server0,server1,server2

user=*****

password=*****

[Splitter-Listener]

type=listener

service=Splitter-Service

protocol=MariaDBClient

port=3306

# MaxAdmin administrative interface

[CLI]

type=service

router=cli

[CLI-Listener]

type=listener

service=CLI

protocol=maxscaled

socket=default

I checked the permissions for the user on the Maxscale monitor and I think the user has all the permissions it needs. Just in case, I tried this with the root user and an admin user with all of the possible permissions and I still get the same error.

Here are the logs for the Master MariaDB server (serverid = 3000):

Initializing database

2019-04-23 13:01:03 6 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !

To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'

'/usr/bin/mysqladmin' -u root -h  password 'new-password'

Alternatively you can run:

'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test

databases and anonymous user created by default.  This is

strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the

MySQL manual for more instructions.

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.

You can find additional information about the MySQL part at:

http://dev.mysql.com

Consider joining MariaDB's strong and vibrant community:

https://mariadb.org/get-involved/

Database initialized

MySQL init process in progress...

2019-04-23 13:01:05 0 [Note] mysqld (mysqld 10.3.13-MariaDB-1:10.3.13+maria~bionic-log) starting as process 108 ...

2019-04-23 13:01:05 0 [Note] InnoDB: Using Linux native AIO

2019-04-23 13:01:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2019-04-23 13:01:05 0 [Note] InnoDB: Uses event mutexes

2019-04-23 13:01:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2019-04-23 13:01:05 0 [Note] InnoDB: Number of pools: 1

2019-04-23 13:01:05 0 [Note] InnoDB: Using SSE2 crc32 instructions

2019-04-23 13:01:05 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M

2019-04-23 13:01:05 0 [Note] InnoDB: Completed initialization of buffer pool

2019-04-23 13:01:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2019-04-23 13:01:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2019-04-23 13:01:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2019-04-23 13:01:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2019-04-23 13:01:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2019-04-23 13:01:05 0 [Note] InnoDB: Waiting for purge to start

2019-04-23 13:01:06 0 [Note] InnoDB: 10.3.13 started; log sequence number 1630815; transaction id 21

2019-04-23 13:01:06 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2019-04-23 13:01:06 0 [Note] Plugin 'FEEDBACK' is disabled.

2019-04-23 13:01:06 0 [Note] InnoDB: Buffer pool(s) load completed at 190423 13:01:06

2019-04-23 13:01:06 0 [Warning] 'user' entry 'root@mariadb-0' ignored in --skip-name-resolve mode.

2019-04-23 13:01:06 0 [Warning] 'user' entry '@mariadb-0' ignored in --skip-name-resolve mode.

2019-04-23 13:01:06 0 [Warning] 'proxies_priv' entry '@% root@mariadb-0' ignored in --skip-name-resolve mode.

2019-04-23 13:01:06 0 [Note] Reading of all Master_info entries succeded

2019-04-23 13:01:06 0 [Note] Added new Master_info '' to hash table

2019-04-23 13:01:06 0 [Note] mysqld: ready for connections.

Version: '10.3.13-MariaDB-1:10.3.13+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution

Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.

2019-04-23 13:01:09 11 [Warning] 'proxies_priv' entry '@% root@mariadb-0' ignored in --skip-name-resolve mode.

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql

2019-04-23 13:01:09 0 [Note] mysqld (initiated by: unknown): Normal shutdown

2019-04-23 13:01:09 0 [Note] Event Scheduler: Purging the queue. 0 events

2019-04-23 13:01:09 0 [Note] InnoDB: FTS optimize thread exiting.

2019-04-23 13:01:09 0 [Note] InnoDB: Starting shutdown...

2019-04-23 13:01:09 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool

2019-04-23 13:01:09 0 [Note] InnoDB: Buffer pool(s) dump completed at 190423 13:01:09

2019-04-23 13:01:10 0 [Note] InnoDB: Shutdown completed; log sequence number 1630824; transaction id 24

2019-04-23 13:01:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2019-04-23 13:01:10 0 [Note] mysqld: Shutdown complete

MySQL init process done. Ready for start up.

2019-04-23 13:01:10 0 [Note] mysqld (mysqld 10.3.13-MariaDB-1:10.3.13+maria~bionic-log) starting as process 7 ...

2019-04-23 13:01:11 0 [Note] InnoDB: Using Linux native AIO

2019-04-23 13:01:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2019-04-23 13:01:11 0 [Note] InnoDB: Uses event mutexes

2019-04-23 13:01:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2019-04-23 13:01:11 0 [Note] InnoDB: Number of pools: 1

2019-04-23 13:01:11 0 [Note] InnoDB: Using SSE2 crc32 instructions

2019-04-23 13:01:11 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M

2019-04-23 13:01:11 0 [Note] InnoDB: Completed initialization of buffer pool

2019-04-23 13:01:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2019-04-23 13:01:11 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2019-04-23 13:01:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2019-04-23 13:01:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2019-04-23 13:01:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2019-04-23 13:01:11 0 [Note] InnoDB: Waiting for purge to start

2019-04-23 13:01:11 0 [Note] InnoDB: 10.3.13 started; log sequence number 1630824; transaction id 21

2019-04-23 13:01:11 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2019-04-23 13:01:11 0 [Note] Plugin 'FEEDBACK' is disabled.

2019-04-23 13:01:11 0 [Note] InnoDB: Buffer pool(s) load completed at 190423 13:01:11

2019-04-23 13:01:11 0 [Note] Server socket created on IP: '::'.

2019-04-23 13:01:11 0 [Note] Server socket created on IP: '::'.

2019-04-23 13:01:11 0 [Warning] 'proxies_priv' entry '@% root@mariadb-0' ignored in --skip-name-resolve mode.

2019-04-23 13:01:11 0 [Note] Reading of all Master_info entries succeded

2019-04-23 13:01:11 0 [Note] Added new Master_info '' to hash table

2019-04-23 13:01:11 0 [Note] mysqld: ready for connections.

Version: '10.3.13-MariaDB-1:10.3.13+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

2019-04-23 13:01:13 17 [Note] Start binlog_dump to slave_server(3001), pos(, 4)

2019-04-23 13:01:19 19 [Note] Start binlog_dump to slave_server(3002), pos(, 4)

Here are the logs for one of the slave MariaDB servers (serverid = 3001):

Initializing database

2019-04-23 13:01:06 6 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !

To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'

'/usr/bin/mysqladmin' -u root -h  password 'new-password'

Alternatively you can run:

'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test

databases and anonymous user created by default.  This is

strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the

MySQL manual for more instructions.

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.

You can find additional information about the MySQL part at:

http://dev.mysql.com

Consider joining MariaDB's strong and vibrant community:

https://mariadb.org/get-involved/

Database initialized

MySQL init process in progress...

2019-04-23 13:01:08 0 [Note] mysqld (mysqld 10.3.13-MariaDB-1:10.3.13+maria~bionic-log) starting as process 111 ...

2019-04-23 13:01:08 0 [Note] InnoDB: Using Linux native AIO

2019-04-23 13:01:08 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2019-04-23 13:01:08 0 [Note] InnoDB: Uses event mutexes

2019-04-23 13:01:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2019-04-23 13:01:08 0 [Note] InnoDB: Number of pools: 1

2019-04-23 13:01:08 0 [Note] InnoDB: Using SSE2 crc32 instructions

2019-04-23 13:01:08 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M

2019-04-23 13:01:08 0 [Note] InnoDB: Completed initialization of buffer pool

2019-04-23 13:01:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2019-04-23 13:01:08 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2019-04-23 13:01:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2019-04-23 13:01:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2019-04-23 13:01:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2019-04-23 13:01:08 0 [Note] InnoDB: Waiting for purge to start

2019-04-23 13:01:08 0 [Note] InnoDB: 10.3.13 started; log sequence number 1630815; transaction id 21

2019-04-23 13:01:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2019-04-23 13:01:08 0 [Note] Plugin 'FEEDBACK' is disabled.

2019-04-23 13:01:08 0 [Note] InnoDB: Buffer pool(s) load completed at 190423 13:01:08

2019-04-23 13:01:08 0 [Warning] 'user' entry 'root@mariadb-1' ignored in --skip-name-resolve mode.

2019-04-23 13:01:08 0 [Warning] 'user' entry '@mariadb-1' ignored in --skip-name-resolve mode.

2019-04-23 13:01:08 0 [Warning] 'proxies_priv' entry '@% root@mariadb-1' ignored in --skip-name-resolve mode.

2019-04-23 13:01:08 0 [Note] Reading of all Master_info entries succeded

2019-04-23 13:01:08 0 [Note] Added new Master_info '' to hash table

2019-04-23 13:01:08 0 [Note] mysqld: ready for connections.

Version: '10.3.13-MariaDB-1:10.3.13+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution

Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.

2019-04-23 13:01:11 11 [Warning] 'proxies_priv' entry '@% root@mariadb-1' ignored in --skip-name-resolve mode.

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql

2019-04-23 13:01:11 0 [Note] mysqld (initiated by: unknown): Normal shutdown

2019-04-23 13:01:11 0 [Note] Event Scheduler: Purging the queue. 0 events

2019-04-23 13:01:11 0 [Note] InnoDB: FTS optimize thread exiting.

2019-04-23 13:01:11 0 [Note] InnoDB: Starting shutdown...

2019-04-23 13:01:11 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool

2019-04-23 13:01:11 0 [Note] InnoDB: Buffer pool(s) dump completed at 190423 13:01:11

2019-04-23 13:01:12 0 [Note] InnoDB: Shutdown completed; log sequence number 1630824; transaction id 24

2019-04-23 13:01:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2019-04-23 13:01:12 0 [Note] mysqld: Shutdown complete

MySQL init process done. Ready for start up.

2019-04-23 13:01:12 0 [Note] mysqld (mysqld 10.3.13-MariaDB-1:10.3.13+maria~bionic-log) starting as process 8 ...

2019-04-23 13:01:12 0 [Note] InnoDB: Using Linux native AIO

2019-04-23 13:01:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2019-04-23 13:01:12 0 [Note] InnoDB: Uses event mutexes

2019-04-23 13:01:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2019-04-23 13:01:12 0 [Note] InnoDB: Number of pools: 1

2019-04-23 13:01:12 0 [Note] InnoDB: Using SSE2 crc32 instructions

2019-04-23 13:01:12 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M

2019-04-23 13:01:12 0 [Note] InnoDB: Completed initialization of buffer pool

2019-04-23 13:01:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2019-04-23 13:01:12 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2019-04-23 13:01:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2019-04-23 13:01:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2019-04-23 13:01:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2019-04-23 13:01:12 0 [Note] InnoDB: Waiting for purge to start

2019-04-23 13:01:12 0 [Note] InnoDB: 10.3.13 started; log sequence number 1630824; transaction id 21

2019-04-23 13:01:12 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2019-04-23 13:01:12 0 [Note] Plugin 'FEEDBACK' is disabled.

2019-04-23 13:01:12 0 [Note] InnoDB: Buffer pool(s) load completed at 190423 13:01:12

2019-04-23 13:01:12 0 [Note] Server socket created on IP: '::'.

2019-04-23 13:01:12 0 [Note] Server socket created on IP: '::'.

2019-04-23 13:01:12 0 [Warning] 'proxies_priv' entry '@% root@mariadb-1' ignored in --skip-name-resolve mode.

2019-04-23 13:01:12 0 [Note] Reading of all Master_info entries succeded

2019-04-23 13:01:12 0 [Note] Added new Master_info '' to hash table

2019-04-23 13:01:12 0 [Note] mysqld: ready for connections.

Version: '10.3.13-MariaDB-1:10.3.13+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

2019-04-23 13:01:13 10 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'

2019-04-23 13:01:13 10 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.

2019-04-23 13:01:13 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='mariadb-0.mariadb.mariadb.svc.cluster.local', master_port='3306', master_log_file='', master_log_pos='4'.

2019-04-23 13:01:13 10 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos

2019-04-23 13:01:13 11 [Note] Slave I/O thread: Start asynchronous replication to master 'root@mariadb-0.mariadb.mariadb.svc.cluster.local:3306' in log '' at position 4

2019-04-23 13:01:13 12 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-3001-7235'

2019-04-23 13:01:13 11 [Note] Slave I/O thread: connected to master 'root@mariadb-0.mariadb.mariadb.svc.cluster.local:3306',replication starts at GTID position '0-3001-7235'

2019-04-23 13:01:13 11 [ERROR] Error reading packet from server: Error: connecting slave requested to start from GTID 0-3001-7235, which is not in the master's binlog. Since the master's binlog contains GTIDs with higher sequence numbers, it probably means that the slave has diverged due to executing extra erroneous transactions (server_errno=1236)

2019-04-23 13:01:13 11 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 0-3001-7235, which is not in the master's binlog. Since the master's binlog contains GTIDs with higher sequence numbers, it probably means that the slave has diverged due to executing extra erroneous transactions', Internal MariaDB error code: 1236

2019-04-23 13:01:13 11 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4; GTID position 0-3001-7235

Here are the logs from the Maxscale instance

Info : MaxScale will be run in the terminal process.

2019-04-23 13:01:13   notice : syslog logging is enabled.

2019-04-23 13:01:13   notice : maxlog logging is enabled.

2019-04-23 13:01:13   notice : Using up to 3.90GiB of memory for query classifier cache

2019-04-23 13:01:13   notice : The collection of SQLite memory allocation statistics turned off.

2019-04-23 13:01:13   notice : Threading mode of SQLite set to Multi-thread.

2019-04-23 13:01:13   notice : MariaDB MaxScale 2.3.4 started (Commit: aea64aede280558ca6b55500dfa7eb049ec9c377)

2019-04-23 13:01:13   notice : MaxScale is running in process 1

Configuration file : /etc/maxscale.cnf

Log directory      : /var/log/maxscale

Data directory     : /var/lib/maxscale

Module directory   : /usr/lib/x86_64-linux-gnu/maxscale

Service cache      : /var/cache/maxscale

2019-04-23 13:01:13   notice : Configuration file: /etc/maxscale.cnf

2019-04-23 13:01:13   notice : Log directory: /var/log/maxscale

2019-04-23 13:01:13   notice : Data directory: /var/lib/maxscale

2019-04-23 13:01:13   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale

2019-04-23 13:01:13   notice : Service cache: /var/cache/maxscale

2019-04-23 13:01:13   notice : No query classifier specified, using default 'qc_sqlite'.

2019-04-23 13:01:13   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so

2019-04-23 13:01:13   notice : Query classification results are cached and reused. Memory used per thread: 665.14MiB

2019-04-23 13:01:13   notice : Loading /etc/maxscale.cnf.

2019-04-23 13:01:13   notice : /etc/maxscale.cnf.d does not exist, not reading.

2019-04-23 13:01:13   notice : Loaded module maxscaled: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmaxscaled.so

2019-04-23 13:01:13   warning: THE 'cli' MODULE AND 'maxadmin' ARE DEPRECATED: Use 'maxctrl' instead

2019-04-23 13:01:13   notice : Loaded module cli: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libcli.so

2019-04-23 13:01:13   notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so

2019-04-23 13:01:13   notice : Initializing statement-based read/write split router module.

2019-04-23 13:01:13   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so

2019-04-23 13:01:13   notice : Initialise the MariaDB Monitor module.

2019-04-23 13:01:13   notice : Loaded module mariadbmon: V1.5.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbmon.so

2019-04-23 13:01:13   notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so

2019-04-23 13:01:13   notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmysqlbackendauth.so

2019-04-23 13:01:13   notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmaxadminauth.so

2019-04-23 13:01:13   notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmysqlauth.so

2019-04-23 13:01:13   notice : Housekeeper thread started.

2019-04-23 13:01:13   warning: Removing stale journal file for monitor 'Replication-Monitor'.

2019-04-23 13:01:13   notice : Encrypted password file /var/lib/maxscale/.secrets can't be accessed (No such file or directory). Password encryption is not used.

2019-04-23 13:01:13   notice : Starting a total of 2 services...

2019-04-23 13:01:13   notice : Listening for connections at [/var/run/maxscale/maxadmin.sock]:0 with protocol MaxScale Admin

2019-04-23 13:01:13   notice : Service 'CLI' started (1/2)

2019-04-23 13:01:13   error  : [Splitter-Service] Failed to connect to server 'server2' ([mariadb-2.mariadb.mariadb.svc.cluster.local]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on 'mariadb-2.mariadb.mariadb.svc.cluster.local' (115)

2019-04-23 13:01:13   notice : [Splitter-Service] Loaded 4 MySQL users for listener Splitter-Listener.

2019-04-23 13:01:13   notice : Listening for connections at [::]:3306 with protocol MySQL

2019-04-23 13:01:13   notice : Service 'Splitter-Service' started (2/2)

2019-04-23 13:01:13   error  : Monitor was unable to connect to server server2[mariadb-2.mariadb.mariadb.svc.cluster.local:3306] : 'Can't connect to MySQL server on 'mariadb-2.mariadb.mariadb.svc.cluster.local' (115)'

2019-04-23 13:01:13   notice : Selecting new master server.

2019-04-23 13:01:13   warning: 'server2' is not a valid master candidate because it is down.

2019-04-23 13:01:13   notice : Setting 'server0' as master.

2019-04-23 13:01:13   notice : Server changed state: server0[mariadb-0.mariadb.mariadb.svc.cluster.local:3306]: new_master. [Running] -> [Master, Running]

2019-04-23 13:01:13   notice : Server changed state: server2[mariadb-2.mariadb.mariadb.svc.cluster.local:3306]: server_down. [Running] -> [Down]

2019-04-23 13:01:13   notice : Directing standalone server 'server1' to replicate from 'server0'.

2019-04-23 13:01:13   notice : Slave connection from server1 to [mariadb-0.mariadb.mariadb.svc.cluster.local]:3306 created and started.

2019-04-23 13:01:13   notice : 1 server(s) redirected or rejoined the cluster.

2019-04-23 13:01:13   notice : MaxScale started with 6 worker threads, each with a stack size of 8388608 bytes.

2019-04-23 13:01:19   notice : Server changed state: server2[mariadb-2.mariadb.mariadb.svc.cluster.local:3306]: server_up. [Down] -> [Running]

2019-04-23 13:01:19   notice : Directing standalone server 'server2' to replicate from 'server0'.

2019-04-23 13:01:19   notice : Slave connection from server2 to [mariadb-0.mariadb.mariadb.svc.cluster.local]:3306 created and started.

2019-04-23 13:01:19   notice : 1 server(s) redirected or rejoined the cluster.

[MXS-2459] Replication error: 1236 after startup Master Slave with Maxscale monitor Created: 2019-04-29 Updated: 2019-10-28 Resolved: 2019-10-28
Status:	Closed
Project:	MariaDB MaxScale
Component/s:	failover, mariadbmon, Monitor
Affects Version/s:	2.3.4
Fix Version/s:	N/A

[MXS-2459] Replication error: 1236 after startup Master Slave with Maxscale monitor Created: 2019-04-29 Updated: 2019-10-28 Resolved: 2019-10-28