[MXS-4845] Maxscale: SSL certificates don't show up in SLAVE STATUS when Master rejoins after being Down Created: 2023-11-01  Updated: 2023-12-14  Resolved: 2023-11-30

Status: Closed
Project: MariaDB MaxScale
Component/s: mariadbmon
Affects Version/s: 6.4.11, 22.08.9
Fix Version/s: 23.02.5, 23.08.2

Type: Bug Priority: Major
Reporter: Susmeet Khaire Assignee: Esa Korhonen
Resolution: Duplicate Votes: 0
Labels: None

Attachments: File master_client.cnf     File master_server.cnf     File maxscale.cnf     File slave_client.cnf     File slave_server.cnf    
Issue Links:
Relates
relates to MXS-4718 Add replication_custom_options to ena... Closed
Sprint: MXS-SPRINT-195

 Description   

SSL certificates don't show up in SLAVE STATUS when Master rejoins after being Down.

MariaDB ver(Master & Slave): 10.6.7-3
Maxscale ver: 6.4.11

MariaDB Master config: master_server.cnf master_client.cnf
MariaDB Slave config: slave_server.cnf slave_client.cnf
Maxscale config: maxscale.cnf

Steps to reproduce:
1. Enable SSL replication and have Maxscale monitor the servers with SSL
I followed this blog to create SSL certificates and enable SSL replication.

[root@localhost ~]# maxctrl list servers
┌─────────┬───────────────┬──────┬─────────────┬─────────────────┬──────────┐
│ Server  │ Address       │ Port │ Connections │ State           │ GTID     │
├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────┤
│ server1 │ 192.168.20.2233060           │ Master, Running │ 0-100-54
├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────┤
│ server2 │ 192.168.20.2833060           │ Slave, Running  │ 0-100-54
└─────────┴───────────────┴──────┴─────────────┴─────────────────┴──────────┘

2. SHOW SLAVE STATUS\G output from the slave server2

MariaDB [(none)]> SHOW ALL SLAVES STATUS\G
*************************** 1. row ***************************
               Connection_name:
               Slave_SQL_State: Slave has read all relay log; waiting for more updates
                Slave_IO_State: Waiting for master to send event
                   Master_Host: 192.168.20.22
                   Master_User: repl_ssl
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: mariadb-server2-bin.000016
           Read_Master_Log_Pos: 405
                Relay_Log_File: mariadb_slave3-relay-bin.000021
                 Relay_Log_Pos: 714
         Relay_Master_Log_File: mariadb-server2-bin.000016
              Slave_IO_Running: Yes
             Slave_SQL_Running: Yes
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 0
                    Last_Error:
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 405
               Relay_Log_Space: 1089
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: Yes
            Master_SSL_CA_File: /etc/my.cnf.d/certs/ca-cert.pem
            Master_SSL_CA_Path:
               Master_SSL_Cert: /etc/my.cnf.d/certs/client-cert.pem
             Master_SSL_Cipher:
                Master_SSL_Key: /etc/my.cnf.d/certs/client-key.pem
         Seconds_Behind_Master: 0
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 0
                 Last_IO_Error:
                Last_SQL_Errno: 0
                Last_SQL_Error:
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 100
                Master_SSL_Crl: /etc/my.cnf.d/certs/ca-cert.pem
            Master_SSL_Crlpath:
                    Using_Gtid: Slave_Pos
                   Gtid_IO_Pos: 0-100-54
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: optimistic
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
              Slave_DDL_Groups: 8
Slave_Non_Transactional_Groups: 1
    Slave_Transactional_Groups: 0
          Retried_transactions: 0
            Max_relay_log_size: 1073741824
          Executed_log_entries: 128
     Slave_received_heartbeats: 0
        Slave_heartbeat_period: 30.000
                Gtid_Slave_Pos: 0-100-54
1 row in set (0.000 sec)

3. stop mariadb service on Master and let the slave become the new Master

systemctl stop mariadb
 
maxctrl list servers
┌─────────┬───────────────┬──────┬─────────────┬────────────────┬──────────┐
│ Server  │ Address       │ Port │ Connections │ State          │ GTID     │
├─────────┼───────────────┼──────┼─────────────┼────────────────┼──────────┤
│ server1 │ 192.168.20.2233060           │ Down           │ 0-100-54
├─────────┼───────────────┼──────┼─────────────┼────────────────┼──────────┤
│ server2 │ 192.168.20.2833060           │ Slave, Running │ 0-100-54
└─────────┴───────────────┴──────┴─────────────┴────────────────┴──────────┘
 
maxctrl list servers
┌─────────┬───────────────┬──────┬─────────────┬─────────────────┬──────────┐
│ Server  │ Address       │ Port │ Connections │ State           │ GTID     │
├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────┤
│ server1 │ 192.168.20.2233060           │ Down            │ 0-100-54
├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────┤
│ server2 │ 192.168.20.2833060           │ Master, Running │ 0-100-54
└─────────┴───────────────┴──────┴─────────────┴─────────────────┴──────────┘

4. Now start the mariadb service

systemctl start mariadb
 
maxctrl list servers
┌─────────┬───────────────┬──────┬─────────────┬─────────────────┬──────────┐
│ Server  │ Address       │ Port │ Connections │ State           │ GTID     │
├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────┤
│ server1 │ 192.168.20.2233060           │ Slave, Running  │ 0-100-54
├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────┤
│ server2 │ 192.168.20.2833060           │ Master, Running │ 0-100-54
└─────────┴───────────────┴──────┴─────────────┴─────────────────┴──────────┘

5. SLAVE STATUS output from server1 new Slave (previous Master)

MariaDB [(none)]> SHOW ALL SLAVES STATUS\G
*************************** 1. row ***************************
               Connection_name:
               Slave_SQL_State: Slave has read all relay log; waiting for more updates
                Slave_IO_State: Waiting for master to send event
                   Master_Host: 192.168.20.28
                   Master_User: repl_ssl
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: mariadb_slave3-bin.000002
           Read_Master_Log_Pos: 2091
                Relay_Log_File: mariadb-server2-relay-bin.000002
                 Relay_Log_Pos: 702
         Relay_Master_Log_File: mariadb_slave3-bin.000002
              Slave_IO_Running: Yes
             Slave_SQL_Running: Yes
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 0
                    Last_Error:
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 2091
               Relay_Log_Space: 1021
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: Yes
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master: 0
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 0
                 Last_IO_Error:
                Last_SQL_Errno: 0
                Last_SQL_Error:
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 200
                Master_SSL_Crl:
            Master_SSL_Crlpath:
                    Using_Gtid: Current_Pos
                   Gtid_IO_Pos: 0-100-54
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: optimistic
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
              Slave_DDL_Groups: 0
Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 0
          Retried_transactions: 0
            Max_relay_log_size: 1073741824
          Executed_log_entries: 8
     Slave_received_heartbeats: 0
        Slave_heartbeat_period: 30.000
                Gtid_Slave_Pos: 0-100-54
1 row in set (0.000 sec)

Shows Master_SSL_Allowed: Yes but certificates are not displayed.

Error log

2023-11-01 11:34:06 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.7-3-MariaDB-enterprise-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Enterprise Server
2023-11-01 11:34:07 4 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
2023-11-01 11:34:07 4 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log-basename=#' or '--relay-log=mariadb-server2-relay-bin' to avoid this problem.
2023-11-01 11:34:07 4 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='192.168.20.28', master_port='3306', master_log_file='', master_log_pos='4'.
2023-11-01 11:34:07 4 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos
2023-11-01 11:34:07 5 [Note] Slave I/O thread: Start asynchronous replication to master 'repl_ssl@192.168.20.28:3306' in log '' at position 4
2023-11-01 11:34:07 6 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './mariadb-server2-relay-bin.000001' position: 4; GTID position '0-100-54'
2023-11-01 11:34:07 5 [Note] Slave I/O thread: connected to master 'repl_ssl@192.168.20.28:3306',replication starts at GTID position '0-100-54'



 Comments   
Comment by Susmeet Khaire [ 2023-11-30 ]

Thanks esa.korhonen
With replication_custom_options, it works properly now.

Generated at Thu Feb 08 04:31:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.