[MXS-2110] swithover fails with error Unknown error'. Query: 'FLUSH LOGS Created: 2018-10-24  Updated: 2019-01-23  Resolved: 2019-01-23

Status: Closed
Project: MariaDB MaxScale
Component/s: failover
Affects Version/s: 2.2.4
Fix Version/s: 2.2.4

Type: Bug Priority: Major
Reporter: Aftab Khan Assignee: Esa Korhonen
Resolution: Not a Bug Votes: 0
Labels: Maxscale
Environment:

test


Attachments: File maxscale.conf     Text File maxscale.log    
Sprint: MXS-SPRINT-71, MXS-SPRINT-72, MXS-SPRINT-73, MXS-SPRINT-74

 Description   

We are testing MaxScale with a simple Master-Slave servers running MariaDB 10.1. Automatic failover works fine.

We have two servers:

server1 = INFDMYS001.example.gov.uk
server2= npedbmys001.example.gov.uk

We have successfully tested following event series:

  • Server1 goes down -> server2 is promoted as new Master
  • Server1 comes back -> server1 rejoins cluster as 'new_slave'
  • Server2 goes down -> server1 is promoted as new Master
  • Server2 comes back -> server2 rejoins cluster as 'new_slave'

[root@INFDMYS001 ~]# maxadmin list servers
Servers.
-------------------+-----------------+-------+-------------+--------------------
Server             | Address         | Port  | Connections | Status
-------------------+-----------------+-------+-------------+--------------------
server1            | INFDMYS001.example.gov.uk |  3306 |           0 | Master, Running
server2            | npedbmys001.example.gov.uk |  3306 |           0 | Slave, Running
-------------------+-----------------+-------+-------------+--------------------

But switchover does not work:

# maxadmin call command mariadbmon switchover ReplMonitor server2 server1
2018-10-24 08:56:14   notice : (15) [mariadbmon] Stopped the monitor ReplMonitor for the duration of switchover.
2018-10-24 08:56:14   notice : (15) [mariadbmon] Demoting server 'server1'.
2018-10-24 08:56:14   error  : (15) [mariadbmon] Demotion failed due to a query error: 'Unknown error'. Query: 'FLUSH LOGS;'.
2018-10-24 08:56:14   error  : (15) [mariadbmon] Switchover server1 -> server2 failed, failover has been disabled..
{
    "errors": [
        {
            "detail": "Demotion failed due to a query error: 'Unknown error'. Query: 'FLUSH LOGS;'."
        },
        {
            "detail": "Switchover server1 -> server2 failed, failover has been disabled.."
        }
    ]
}

Monitor user has all required privileges:

MariaDB [(none)]> show grants for maxscale@'%';
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for maxscale@%                                                                                                                                                  |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT RELOAD, SHOW DATABASES, SUPER, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'maxscale'@'%' IDENTIFIED BY PASSWORD '*53E5D0C7885BD540911663B04133F20C94AD4306' |
| GRANT SELECT ON `mysql`.* TO 'maxscale'@'%'                                                                                                                            |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

I have attached maxscale log and config files.



 Comments   
Comment by Esa Korhonen [ 2018-11-28 ]

Tried reproducing this but no luck. Does this happen with more recent MaxScale/Server versions as well?

The error message means that the monitor tried to "FLUSH LOGS;" on the current master when demoting it, however the query failed. Then, the monitor queried the connector driver for an error description and got the "Unknown error". This is a valid server error code but doesn't explain anything. Is this a repeating occurrence, happening every time? What if you log into the server using a client program and the monitor user and try to "FLUSH LOGS;"?

Comment by Aftab Khan [ 2018-12-11 ]

>What if you log into the server using a client program and the monitor user and try to "FLUSH LOGS;"?
Yes it works when I tried it with monitor user. Not sure if its relevant, in our setup one MariaDB instance was setup on Windows and second one on Linux.

Comment by Esa Korhonen [ 2019-01-08 ]

I suggest trying with a more recent version of MaxScale. Alternatively, you can add the monitor settings "backend_read_timeout" and "backend_write_timeout" and set them to e.g. 3. In 2.2.4 the default values for these settings are somewhat strict and may cause connection errors for the monitor.

Comment by Esa Korhonen [ 2019-01-23 ]

Closing for now. Reopen if issue is unsolved.

Generated at Thu Feb 08 04:11:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.