[MXS-3650] Maxscale crashes while logging to GUI Created: 2021-06-30  Updated: 2021-08-04  Resolved: 2021-07-09

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 2.5.13
Fix Version/s: 2.5.14

Type: Bug Priority: Major
Reporter: suresh ramagiri Assignee: Johan Wikman
Resolution: Fixed Votes: 0
Labels: None
Environment:

VM Virtual Box -6.1 , RHEL 7.7


Sprint: MXS-SPRINT-135

 Description   

One of our customers reported that at VM instance using OS RHEL 7, changing the CPU's from 2 to 4 and trying to login the GUI, causes the maxscale to crash.

I can locally able to reproduce the issue. Tried this at our latest release maxscale - 2.5.13
Here is the procedure which needs to be followed:

1. Run MaxScale on a 2 CPU VM.
1) Stop MaxScale
2) Set threads=auto in /etc/maxscale.cnf file.
3) Remove the /var/lib/maxscale/maxscale.cnf.d/maxscale.cnf file if it exists.
4) Run MaxScale

2. Check the MaxScale parameters and values.
$ maxctrl show maxscale
=> "threads" = 2

3. Change the value of the MaxScale parameter with the alter maxscale command.
e.g $ maxctrl alter maxscale passive false
At this time, the file /var/lib/maxscale/maxscale.cnf.d/maxscale.cnf is created.
This file records threads=2.

4. Stop the VM, change the VM's CPU to 4, then start the VM.

5. Check the MaxScale parameters and values.
$ grep threads /etc/maxscale.cnf
=> threads=auto
$ grep threads /var/lib/maxscale/maxscale.cnf.d/maxscale.cnf
=> threads=2
$ maxctrl show maxscale | grep threads
=> "threads" = 2
※ /var/lib/maxscale/maxscale.cnf.d/maxscale.cnf settings take precedence over /etc/maxscale.cnf

6. Log in to the MaxScale Web GUI admin page.
e.g. http://<IP>:8989
Upon login, the MaxScale process terminated with error messages and process core dump.
If MaxScale is run as a Linux service, no process core dump is generated and the process restarts.

7. Temporary workaround
1) $ vi /var/lib/maxscale/maxscale.cnf.d/maxscale.cnf
Change threads=2 to threads=auto or threads=4.
2) Restart MaxScale
3) $ maxctrl show maxscale | grep threads
=> "threads" = 4
4) If you log in to Web GUI admin page(e.g. http://<IP>:8989), it works normally.

Logs -

      • Error in `/usr/bin/maxscale': free(): invalid next size (fast): 0x00000000016c5470 ***
        ======= Backtrace: =========
        /lib64/libc.so.6(+0x816b9)[0x7fc9b4c7f6b9]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_Z24mxs_rworker_list_to_jsonPKc+0x9d)[0x7fc9b791b96d]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x136378)[0x7fc9b790c378]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x14)[0x7fc9b790ec84]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x13a6b2)[0x7fc9b79106b2]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x13c210)[0x7fc9b7912210]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x159)[0x7fc9b795fa39]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138)[0x7fc9b7962d48]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be)[0x7fc9b7960a4e]
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53)[0x7fc9b7960c63]
        /usr/bin/maxscale(main+0x1f7a)[0x40939a]
        /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc9b4c20545]
        /usr/bin/maxscale[0x409ca2]
        ======= Memory map: ========
        00400000-00415000 r-xp 00000000 fd:00 12950925 /usr/bin/maxscale
        00614000-00615000 r--p 00014000 fd:00 12950925 /usr/bin/maxscale
        00615000-00616000 rw-p 00015000 fd:00 12950925 /usr/bin/maxscale
        00616000-00618000 rw-p 00000000 00:00 0
        01683000-016c5000 rw-p 00000000 00:00 0 [heap]
        016c5000-036e9000 rw-p 00000000 00:00 0 [heap]


 Comments   
Comment by Johan Wikman [ 2021-07-05 ]

Could not repeat this by duplicating the scenario outside a VM setup. However, it is clear that if the original configuration file has threads=auto, then that is also what should be stored in the generated config file under maxscale.cnf.d.

Comment by Johan Wikman [ 2021-07-09 ]

If the thread setting is auto it will also be stored as such and not as a specific value.

Generated at Thu Feb 08 04:22:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.