[MXS-2676] High Ping peaks Created: 2019-09-13  Updated: 2020-03-02  Resolved: 2020-03-02

Status: Closed
Project: MariaDB MaxScale
Component/s: failover
Affects Version/s: 2.2.21
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Sebastian Halfar Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: performance
Environment:

Ubuntu 18.04 VM on ESXi Vmware Host.


Attachments: PNG File high_ping_fallout.png     HTML File maxscalespecs    

 Description   

Hallo together,

im not sure if this is a bug. We found out, that we got some high peaks in mtr on the MASTER maxscale VM, the load of the VM is fine. And everything works great, the problem is, sometimes we get ping peaks from 800 - 3000ms:

64 bytes from 192.168.x.x: icmp_seq=99 ttl=64 time=93.3 ms
64 bytes from 192.168.x.x: icmp_seq=100 ttl=64 time=205 ms
64 bytes from 192.168.x.x: icmp_seq=101 ttl=64 time=289 ms
64 bytes from 192.168.x.x: icmp_seq=102 ttl=64 time=57.6 ms
64 bytes from 192.168.x.x: icmp_seq=103 ttl=64 time=0.094 ms

64 bytes from 192.168.x.x: icmp_seq=645 ttl=64 time=0.072 ms
64 bytes from 192.168.x.x: icmp_seq=646 ttl=64 time=0.092 ms
64 bytes from 192.168.x.x: icmp_seq=647 ttl=64 time=0.856 ms
64 bytes from 192.168.x.x: icmp_seq=648 ttl=64 time=0.083 ms
64 bytes from 192.168.x.x: icmp_seq=649 ttl=64 time=2996 ms
64 bytes from 192.168.x.x: icmp_seq=650 ttl=64 time=1975 ms
64 bytes from 192.168.x.x: icmp_seq=651 ttl=64 time=952 ms
64 bytes from 192.168.x.x: icmp_seq=652 ttl=64 time=1.29 ms
64 bytes from 192.168.x.x: icmp_seq=653 ttl=64 time=1402 ms
64 bytes from 192.168.x.x: icmp_seq=654 ttl=64 time=394 ms
64 bytes from 192.168.x.x: icmp_seq=655 ttl=64 time=0.188 ms
64 bytes from 192.168.x.x: icmp_seq=656 ttl=64 time=0.435 ms
64 bytes from 192.168.x.x: icmp_seq=657 ttl=64 time=0.116 ms
64 bytes from 192.168.x.x: icmp_seq=658 ttl=64 time=0.237 ms

that brings us to the problem, @the peak keeplaived vrrp packets wont reach the BACKUP node and it takes over for on second, than it all goes back.

Sep 13 12:00:17 maxscale01 systemd[1]: Started Session 10 of user root.
Sep 13 12:17:01 maxscale01 CRON[28261]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 13 12:19:16 maxscale01 systemd[1]: Started Session 12 of user root.
Sep 13 12:21:00 maxscale01 systemd[1]: Started Session 13 of user root.
Sep 13 12:37:49 maxscale01 Keepalived_vrrp[1208]: /home/scripts/is_maxscale_running.sh exited due to signal 15
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Script(chk_myscript) timed out
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: /home/scripts/is_maxscale_running.sh exited due to signal 15
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering FAULT STATE
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
Sep 13 12:37:51 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Now in FAULT state
Sep 13 12:37:51 maxscale01 maxscale[1391]: Updated 'passive' from 'false' to 'true'
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: VRRP_Script(chk_myscript) succeeded
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
Sep 13 12:37:53 maxscale01 Keepalived_vrrp[1208]: SMTP connection ERROR to [127.0.0.1]:25.
Sep 13 12:37:54 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) forcing a new MASTER election
Sep 13 12:37:55 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: Opening script file /home/scripts/notify_script.sh
Sep 13 12:37:56 maxscale01 Keepalived_vrrp[1208]: SMTP connection ERROR to [127.0.0.1]:25.
Sep 13 12:38:00 maxscale01 maxscale[1391]: Updated 'passive' from 'true' to 'false'
Sep 13 12:41:28 maxscale01 systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 2044 (sysctl)
Sep 13 12:41:28 maxscale01 systemd[1]: Mounting Arbitrary Executable File Formats File System...
Sep 13 12:41:28 maxscale01 systemd[1]: Mounted Arbitrary Executable File Formats File System.

Because of this we get some sql errors in our application and the maxscale process/load goes up.

I tested a lot of VM's with mtr none of them got this problem highest ping is around 10ms

Even the backup maxscale which is in passive mode has a stable ping.

Do you guys have any idea?



 Comments   
Comment by markus makela [ 2019-10-07 ]

If I'm reading this correctly, the ping only jumps up on the active MaxScale. The only explanation would be that the increase in system load is what causes this and if it only happens on the active MaxScale, then it's likely due to that.

Comment by markus makela [ 2020-03-02 ]

2.2 is EOL

Generated at Thu Feb 08 04:15:52 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.