Details
-
Bug
-
Status: Closed (View Workflow)
-
Minor
-
Resolution: Fixed
-
2.2.21
-
Ubuntu 18.04 VM on ESXi Vmware Host.
Description
Hallo together,
im not sure if this is a bug. We found out, that we got some high peaks in mtr on the MASTER maxscale VM, the load of the VM is fine. And everything works great, the problem is, sometimes we get ping peaks from 800 - 3000ms:
64 bytes from 192.168.x.x: icmp_seq=99 ttl=64 time=93.3 ms
64 bytes from 192.168.x.x: icmp_seq=100 ttl=64 time=205 ms
64 bytes from 192.168.x.x: icmp_seq=101 ttl=64 time=289 ms
64 bytes from 192.168.x.x: icmp_seq=102 ttl=64 time=57.6 ms
64 bytes from 192.168.x.x: icmp_seq=103 ttl=64 time=0.094 ms
64 bytes from 192.168.x.x: icmp_seq=645 ttl=64 time=0.072 ms
64 bytes from 192.168.x.x: icmp_seq=646 ttl=64 time=0.092 ms
64 bytes from 192.168.x.x: icmp_seq=647 ttl=64 time=0.856 ms
64 bytes from 192.168.x.x: icmp_seq=648 ttl=64 time=0.083 ms
64 bytes from 192.168.x.x: icmp_seq=649 ttl=64 time=2996 ms
64 bytes from 192.168.x.x: icmp_seq=650 ttl=64 time=1975 ms
64 bytes from 192.168.x.x: icmp_seq=651 ttl=64 time=952 ms
64 bytes from 192.168.x.x: icmp_seq=652 ttl=64 time=1.29 ms
64 bytes from 192.168.x.x: icmp_seq=653 ttl=64 time=1402 ms
64 bytes from 192.168.x.x: icmp_seq=654 ttl=64 time=394 ms
64 bytes from 192.168.x.x: icmp_seq=655 ttl=64 time=0.188 ms
64 bytes from 192.168.x.x: icmp_seq=656 ttl=64 time=0.435 ms
64 bytes from 192.168.x.x: icmp_seq=657 ttl=64 time=0.116 ms
64 bytes from 192.168.x.x: icmp_seq=658 ttl=64 time=0.237 ms
that brings us to the problem, @the peak keeplaived vrrp packets wont reach the BACKUP node and it takes over for on second, than it all goes back.
Sep 13 12:00:17 maxscale01 systemd[1]: Started Session 10 of user root.
Sep 13 12:17:01 maxscale01 CRON[28261]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 13 12:19:16 maxscale01 systemd[1]: Started Session 12 of user root.
Sep 13 12:21:00 maxscale01 systemd[1]: Started Session 13 of user root.
Sep 13 12:37:49 maxscale01 Keepalived_vrrp[1208]: /home/scripts/is_maxscale_running.sh exited due to signal 15
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Script(chk_myscript) timed out
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: /home/scripts/is_maxscale_running.sh exited due to signal 15
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering FAULT STATE
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Now in FAULT state
Sep 13 12:37:51 maxscale01 maxscale[1391]: Updated 'passive' from 'false' to 'true'
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: VRRP_Script(chk_myscript) succeeded
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: SMTP connection ERROR to [127.0.0.1]:25.
Sep 13 12:37:54 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) forcing a new MASTER election
Sep 13 12:37:55 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: SMTP connection ERROR to [127.0.0.1]:25.
Sep 13 12:38:00 maxscale01 maxscale[1391]: Updated 'passive' from 'true' to 'false'
Sep 13 12:41:28 maxscale01 systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 2044 (sysctl)
Sep 13 12:41:28 maxscale01 systemd[1]: Mounting Arbitrary Executable File Formats File System...
Sep 13 12:41:28 maxscale01 systemd[1]: Mounted Arbitrary Executable File Formats File System.
Because of this we get some sql errors in our application and the maxscale process/load goes up.
I tested a lot of VM's with mtr none of them got this problem highest ping is around 10ms
Even the backup maxscale which is in passive mode has a stable ping.
Do you guys have any idea?