Details
- 
    Bug 
- 
    Status: Closed (View Workflow)
- 
    Major 
- 
    Resolution: Fixed
- 
    None
- 
    None
- 
    MaxScale server karma172:
 OS: CentOS 7
 Version: MaxScale 2.3.5 Commit: ab96606269d6526bb20c1b8585edb0a242408320
 Clustrix nodes:
 OS: RHEL 7 / CentOS 7
 Version: Fred
- 
        MXS-SPRINT-81, MXS-SPRINT-85
Description
I’m seeing a Fatal on MaxScale:
| 2019-04-20 03:31:31   notice : (40000201) [MySQLAuth] [GluttonRCR-vqc007c] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. | 
| 2019-04-20 03:31:31   error  : [clustrixmon] Clustrix: Could not execute 'SELECT ni.nodeid, ni.iface_ip, ni.mysql_port, ni.healthmon_port, sn.nodeid FROM system.nodeinfo AS ni LEFT JOIN system.softfailed_nodes AS sn ON ni.nodeid = sn.nodeid' on 10.2.15.13: [11280] Permission denied: User 'maxscale'@'%' is missing SELECT on `system`.`softfailed_nodes`; is missing SELECT on `system`.`nodeinfo`. | 
| 2019-04-20 03:31:31   alert  : Fatal: MaxScale 2.3.5 received fatal signal 11. Attempting backtrace. | 
| 2019-04-20 03:31:31   alert  : Commit ID: ab96606269d6526bb20c1b8585edb0a242408320 System name: Linux Release string: CentOS Linux release 7.6.1810 (Core) | 
| 2019-04-20 03:31:31   notice : (40000202) [MySQLAuth] [GluttonRCR-vqc007c] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded.
 | 
I had a working setup with MaxScale and Clustrix and then I reimaged the OS (from RHEL 7 to CentOS 7) on all clustrix nodes to take it to 9.1.4 from Fred release, reinstalled Clustrix and created the clustrix group again. maxscale user is not yet created in Clustrix.
I think, that’s when this Fatal occurred. Maxscale service stopped. The third setup monitored by the same maxscale server was running fine; but monitoring it stopped as maxscale service crashed.
This may not be a very frequent scenario, but I did not expect a Fatal.
Config file:
| [root@karma172 log]# cat /etc/clustrix_allsetups_ssl.cnf | 
| [maxscale] | 
| #log_info=1 | 
| logdir=/data/clustrix/log | 
| threads=auto | 
|  | 
| ######## Glutton karma067 ####### | 
|  | 
| [Bootstrap1] | 
| type=server | 
| address=10.2.14.123 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
| #karma067 | 
|  | 
| # [Bootstrap2] | 
| # type=server | 
| # address=10.2.224.102 | 
| # port=3306 | 
| # protocol=mariadbbackend | 
|  | 
| # [Bootstrap3] | 
| # type=server | 
| # address=10.2.224.103 | 
| # port=3306 | 
| # protocol=mariadbbackend | 
|  | 
| [Clustrix] | 
| type=monitor | 
| module=clustrixmon | 
| servers=Bootstrap1 | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster_monitor_interval=10000 | 
|  | 
| [RCR] | 
| type=service | 
| router=readconnroute | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster=Clustrix | 
|  | 
| [RCR-Listener] | 
| type=listener | 
| service=RCR | 
| protocol=MariaDBClient | 
| #address=127.0.0.1 | 
| port=4008 | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
|  | 
| ######## Dlong ####### | 
|  | 
| [Dlong1] | 
| type=server | 
| address=10.2.15.83 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
| #karma031 | 
| # can't enable SSL until 33424 fixed. | 
|  | 
| [Dlong2] | 
| type=server | 
| address=10.2.15.121 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
| #karma044 | 
|  | 
| [Dlong3] | 
| type=server | 
| address=10.2.15.114 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
| # karma006 | 
|  | 
| [Clustrix-Dlong] | 
| type=monitor | 
| module=clustrixmon | 
| servers=Dlong1,Dlong2,Dlong3 | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster_monitor_interval=10000 | 
|  | 
| [RCR-Dlong] | 
| type=service | 
| router=readconnroute | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster=Clustrix-Dlong | 
|  | 
| [RCR-Listener-Dlong] | 
| type=listener | 
| service=RCR-Dlong | 
| protocol=MariaDBClient | 
| #address=127.0.0.1 | 
| port=4009 | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
|  | 
| ######## Glutton karma050 ####### | 
|  | 
| [vqc007c-karma050] | 
| type=server | 
| address=10.2.15.127 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
| #karma050 | 
|  | 
| [vqc007c-karma058] | 
| type=server | 
| address=10.2.15.147 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
|  | 
| [vqc007c-karma073] | 
| type=server | 
| address=10.2.12.164 | 
| port=3306 | 
| protocol=mariadbbackend | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
|  | 
| [Gluttonmon-vqc007c] | 
| type=monitor | 
| module=clustrixmon | 
| servers=vqc007c-karma050,vqc007c-karma058,vqc007c-karma073 | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster_monitor_interval=10000 | 
|  | 
| [GluttonRCR-vqc007c] | 
| type=service | 
| router=readconnroute | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster=Gluttonmon-vqc007c | 
|  | 
| [GluttonRCR-Listener-vqc007c] | 
| type=listener | 
| service=GluttonRCR-vqc007c | 
| protocol=MariaDBClient | 
| #address=127.0.0.1 | 
| port=4010 | 
| ssl=required | 
| #ssl_version=TLSv10 | 
| ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
|  | 
| ######## Glutton karma118 vqc005c ####### | 
| [vqc005c-karma118] | 
| type=server | 
| address=10.2.15.67 | 
| port=3306 | 
| protocol=mariadbbackend | 
| #ssl=required | 
| #ssl_version=TLSv10 | 
| #ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| #ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| #ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
| #karma118 | 
|  | 
| [vqc005c-karma148] | 
| type=server | 
| address=10.2.15.193 | 
| port=3306 | 
| protocol=mariadbbackend | 
|  | 
| [vqc005c-karma110] | 
| type=server | 
| address=10.2.15.51 | 
| port=3306 | 
| protocol=mariadbbackend | 
|  | 
| [Gluttonmon-vqc005c] | 
| type=monitor | 
| module=clustrixmon | 
| servers=vqc005c-karma118,vqc005c-karma148,vqc005c-karma110 | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster_monitor_interval=10000 | 
|  | 
| [GluttonRCR-vqc005c] | 
| type=service | 
| router=readconnroute | 
| user=maxscale | 
| password=maxscale_pw | 
| cluster=Gluttonmon-vqc005c | 
|  | 
| [GluttonRCR-Listener-vqc005c] | 
| type=listener | 
| service=GluttonRCR-vqc005c | 
| protocol=MariaDBClient | 
| #address=127.0.0.1 | 
| port=4011 | 
| #ssl=required | 
| #ssl_version=TLSv10 | 
| #ssl_cert=/etc/my.cnf.d/certs/client-cert.pem | 
| #ssl_key=/etc/my.cnf.d/certs/client-key.pem | 
| #ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem | 
|  | 
|  | 
| ###################### | 
| [MaxAdmin-Service] | 
| type=service | 
| router=cli | 
|  | 
| [MaxAdmin-Unix-Listener] | 
| type=listener | 
| service=MaxAdmin-Service | 
| protocol=maxscaled | 
| socket=default
 |