XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 2.2.21
    • N/A
    • failover
    • Ubuntu 18.04 VM on ESXi Vmware Host.

    Description

      Hallo together,

      im not sure if this is a bug. We found out, that we got some high peaks in mtr on the MASTER maxscale VM, the load of the VM is fine. And everything works great, the problem is, sometimes we get ping peaks from 800 - 3000ms:

      64 bytes from 192.168.x.x: icmp_seq=99 ttl=64 time=93.3 ms
      64 bytes from 192.168.x.x: icmp_seq=100 ttl=64 time=205 ms
      64 bytes from 192.168.x.x: icmp_seq=101 ttl=64 time=289 ms
      64 bytes from 192.168.x.x: icmp_seq=102 ttl=64 time=57.6 ms
      64 bytes from 192.168.x.x: icmp_seq=103 ttl=64 time=0.094 ms

      64 bytes from 192.168.x.x: icmp_seq=645 ttl=64 time=0.072 ms
      64 bytes from 192.168.x.x: icmp_seq=646 ttl=64 time=0.092 ms
      64 bytes from 192.168.x.x: icmp_seq=647 ttl=64 time=0.856 ms
      64 bytes from 192.168.x.x: icmp_seq=648 ttl=64 time=0.083 ms
      64 bytes from 192.168.x.x: icmp_seq=649 ttl=64 time=2996 ms
      64 bytes from 192.168.x.x: icmp_seq=650 ttl=64 time=1975 ms
      64 bytes from 192.168.x.x: icmp_seq=651 ttl=64 time=952 ms
      64 bytes from 192.168.x.x: icmp_seq=652 ttl=64 time=1.29 ms
      64 bytes from 192.168.x.x: icmp_seq=653 ttl=64 time=1402 ms
      64 bytes from 192.168.x.x: icmp_seq=654 ttl=64 time=394 ms
      64 bytes from 192.168.x.x: icmp_seq=655 ttl=64 time=0.188 ms
      64 bytes from 192.168.x.x: icmp_seq=656 ttl=64 time=0.435 ms
      64 bytes from 192.168.x.x: icmp_seq=657 ttl=64 time=0.116 ms
      64 bytes from 192.168.x.x: icmp_seq=658 ttl=64 time=0.237 ms

      that brings us to the problem, @the peak keeplaived vrrp packets wont reach the BACKUP node and it takes over for on second, than it all goes back.

      Sep 13 12:00:17 maxscale01 systemd[1]: Started Session 10 of user root.
      Sep 13 12:17:01 maxscale01 CRON[28261]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Sep 13 12:19:16 maxscale01 systemd[1]: Started Session 12 of user root.
      Sep 13 12:21:00 maxscale01 systemd[1]: Started Session 13 of user root.
      Sep 13 12:37:49 maxscale01 Keepalived_vrrp[1208]: /home/scripts/is_maxscale_running.sh exited due to signal 15
      Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Script(chk_myscript) timed out
      Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: /home/scripts/is_maxscale_running.sh exited due to signal 15
      Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering FAULT STATE
      Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
      Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Now in FAULT state
      Sep 13 12:37:51 maxscale01 maxscale[1391]: Updated 'passive' from 'false' to 'true'
      Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: VRRP_Script(chk_myscript) succeeded
      Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering BACKUP STATE
      Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
      Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: SMTP connection ERROR to [127.0.0.1]:25.
      Sep 13 12:37:54 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) forcing a new MASTER election
      Sep 13 12:37:55 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Transition to MASTER STATE
      Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering MASTER STATE
      Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
      Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: SMTP connection ERROR to [127.0.0.1]:25.
      Sep 13 12:38:00 maxscale01 maxscale[1391]: Updated 'passive' from 'true' to 'false'
      Sep 13 12:41:28 maxscale01 systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 2044 (sysctl)
      Sep 13 12:41:28 maxscale01 systemd[1]: Mounting Arbitrary Executable File Formats File System...
      Sep 13 12:41:28 maxscale01 systemd[1]: Mounted Arbitrary Executable File Formats File System.

      Because of this we get some sql errors in our application and the maxscale process/load goes up.

      I tested a lot of VM's with mtr none of them got this problem highest ping is around 10ms

      Even the backup maxscale which is in passive mode has a stable ping.

      Do you guys have any idea?

      Attachments

        1. high_ping_fallout.png
          188 kB
          Sebastian Halfar
        2. maxscalespecs
          2 kB
          Sebastian Halfar

        Activity

          People

            Unassigned Unassigned
            Seb0 Sebastian Halfar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.