Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2446

Fatal on Maxscale server on reimaging clustrix setup being monitored.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.1
    • Component/s: clustrixmon
    • Labels:
      None
    • Environment:
      MaxScale server karma172:
      OS: CentOS 7
      Version: MaxScale 2.3.5 Commit: ab96606269d6526bb20c1b8585edb0a242408320
      Clustrix nodes:
      OS: RHEL 7 / CentOS 7
      Version: Fred
    • Sprint:
      MXS-SPRINT-81, MXS-SPRINT-85

      Description

      I’m seeing a Fatal on MaxScale:

      2019-04-20 03:31:31   notice : (40000201) [MySQLAuth] [GluttonRCR-vqc007c] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded.
      2019-04-20 03:31:31   error  : [clustrixmon] Clustrix: Could not execute 'SELECT ni.nodeid, ni.iface_ip, ni.mysql_port, ni.healthmon_port, sn.nodeid FROM system.nodeinfo AS ni LEFT JOIN system.softfailed_nodes AS sn ON ni.nodeid = sn.nodeid' on 10.2.15.13: [11280] Permission denied: User 'maxscale'@'%' is missing SELECT on `system`.`softfailed_nodes`; is missing SELECT on `system`.`nodeinfo`.
      2019-04-20 03:31:31   alert  : Fatal: MaxScale 2.3.5 received fatal signal 11. Attempting backtrace.
      2019-04-20 03:31:31   alert  : Commit ID: ab96606269d6526bb20c1b8585edb0a242408320 System name: Linux Release string: CentOS Linux release 7.6.1810 (Core)
      2019-04-20 03:31:31   notice : (40000202) [MySQLAuth] [GluttonRCR-vqc007c] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded.
      

      I had a working setup with MaxScale and Clustrix and then I reimaged the OS (from RHEL 7 to CentOS 7) on all clustrix nodes to take it to 9.1.4 from Fred release, reinstalled Clustrix and created the clustrix group again. maxscale user is not yet created in Clustrix.
      I think, that’s when this Fatal occurred. Maxscale service stopped. The third setup monitored by the same maxscale server was running fine; but monitoring it stopped as maxscale service crashed.
      This may not be a very frequent scenario, but I did not expect a Fatal.

      Config file:

      [root@karma172 log]# cat /etc/clustrix_allsetups_ssl.cnf
      [maxscale]
      #log_info=1
      logdir=/data/clustrix/log
      threads=auto
       
      ######## Glutton karma067 #######
       
      [Bootstrap1]
      type=server
      address=10.2.14.123
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
      #karma067
       
      # [Bootstrap2]
      # type=server
      # address=10.2.224.102
      # port=3306
      # protocol=mariadbbackend
       
      # [Bootstrap3]
      # type=server
      # address=10.2.224.103
      # port=3306
      # protocol=mariadbbackend
       
      [Clustrix]
      type=monitor
      module=clustrixmon
      servers=Bootstrap1
      user=maxscale
      password=maxscale_pw
      cluster_monitor_interval=10000
       
      [RCR]
      type=service
      router=readconnroute
      user=maxscale
      password=maxscale_pw
      cluster=Clustrix
       
      [RCR-Listener]
      type=listener
      service=RCR
      protocol=MariaDBClient
      #address=127.0.0.1
      port=4008
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      ######## Dlong #######
       
      [Dlong1]
      type=server
      address=10.2.15.83
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
      #karma031
      # can't enable SSL until 33424 fixed.
       
      [Dlong2]
      type=server
      address=10.2.15.121
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
      #karma044
       
      [Dlong3]
      type=server
      address=10.2.15.114
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
      # karma006
       
      [Clustrix-Dlong]
      type=monitor
      module=clustrixmon
      servers=Dlong1,Dlong2,Dlong3
      user=maxscale
      password=maxscale_pw
      cluster_monitor_interval=10000
       
      [RCR-Dlong]
      type=service
      router=readconnroute
      user=maxscale
      password=maxscale_pw
      cluster=Clustrix-Dlong
       
      [RCR-Listener-Dlong]
      type=listener
      service=RCR-Dlong
      protocol=MariaDBClient
      #address=127.0.0.1
      port=4009
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      ######## Glutton karma050 #######
       
      [vqc007c-karma050]
      type=server
      address=10.2.15.127
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
      #karma050
       
      [vqc007c-karma058]
      type=server
      address=10.2.15.147
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      [vqc007c-karma073]
      type=server
      address=10.2.12.164
      port=3306
      protocol=mariadbbackend
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      [Gluttonmon-vqc007c]
      type=monitor
      module=clustrixmon
      servers=vqc007c-karma050,vqc007c-karma058,vqc007c-karma073
      user=maxscale
      password=maxscale_pw
      cluster_monitor_interval=10000
       
      [GluttonRCR-vqc007c]
      type=service
      router=readconnroute
      user=maxscale
      password=maxscale_pw
      cluster=Gluttonmon-vqc007c
       
      [GluttonRCR-Listener-vqc007c]
      type=listener
      service=GluttonRCR-vqc007c
      protocol=MariaDBClient
      #address=127.0.0.1
      port=4010
      ssl=required
      #ssl_version=TLSv10
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      ######## Glutton karma118 vqc005c #######
      [vqc005c-karma118]
      type=server
      address=10.2.15.67
      port=3306
      protocol=mariadbbackend
      #ssl=required
      #ssl_version=TLSv10
      #ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      #ssl_key=/etc/my.cnf.d/certs/client-key.pem
      #ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
      #karma118
       
      [vqc005c-karma148]
      type=server
      address=10.2.15.193
      port=3306
      protocol=mariadbbackend
       
      [vqc005c-karma110]
      type=server
      address=10.2.15.51
      port=3306
      protocol=mariadbbackend
       
      [Gluttonmon-vqc005c]
      type=monitor
      module=clustrixmon
      servers=vqc005c-karma118,vqc005c-karma148,vqc005c-karma110
      user=maxscale
      password=maxscale_pw
      cluster_monitor_interval=10000
       
      [GluttonRCR-vqc005c]
      type=service
      router=readconnroute
      user=maxscale
      password=maxscale_pw
      cluster=Gluttonmon-vqc005c
       
      [GluttonRCR-Listener-vqc005c]
      type=listener
      service=GluttonRCR-vqc005c
      protocol=MariaDBClient
      #address=127.0.0.1
      port=4011
      #ssl=required
      #ssl_version=TLSv10
      #ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      #ssl_key=/etc/my.cnf.d/certs/client-key.pem
      #ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
       
      ######################
      [MaxAdmin-Service]
      type=service
      router=cli
       
      [MaxAdmin-Unix-Listener]
      type=listener
      service=MaxAdmin-Service
      protocol=maxscaled
      socket=default
      

        Attachments

          Activity

            People

            Assignee:
            johan.wikman Johan Wikman
            Reporter:
            rahul.joshi@mariadb.com Rahul Joshi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: