[MXS-2486] MaxScale 2.3.6 received fatal signal 11 Created: 2019-05-14  Updated: 2020-08-25  Resolved: 2019-07-24

Status: Closed
Project: MariaDB MaxScale
Component/s: readwritesplit, schemarouter
Affects Version/s: 2.3.6, 2.3.9
Fix Version/s: 2.3.10

Type: Bug Priority: Critical
Reporter: Nicolas De Amicis Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

SLES 12 SP4 x86_64, also happened on CentOS 7.4

Attachments: File maxscale.cnf    
Issue Links:
Blocks
is blocked by MXS-2599 Suse Entreprise Server 15 isn't suppo... Closed
Sprint: MXS-SPRINT-85, MXS-SPRINT-86

 Description   

This configuration runs in MaxScale 2.3.5, but crashes in 2.3.6:
2019-05-14T09:48:06.242113+02:00 siesrv154 maxscale[1234]: (31) Fatal: MaxScale 2.3.6 received fatal signal 11. Attempting backtrace.
2019-05-14T09:48:06.242666+02:00 siesrv154 maxscale[1234]: (31) Commit ID: 03dc969cf203decb4d0bc65280fe4f901da87830 System name: Linux Release string: NAME="SLES"
2019-05-14T09:48:06.255591+02:00 siesrv154 maxscale[1234]: (31) /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x40d5cb]: /home/ec2-user/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
2019-05-14T09:48:06.267646+02:00 siesrv154 maxscale[1234]: (31) /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x4e) [0x40d92e]: /usr/include/c++/4.8/functional:2029
2019-05-14T09:48:06.278464+02:00 siesrv154 maxscale[1234]: (31) /usr/bin/maxscale() [0x409fd9]: ??:0
2019-05-14T09:48:06.283333+02:00 siesrv154 maxscale[1234]: (31) /lib64/libpthread.so.0(+0x10c70) [0x7f950ad4ac70]: ??:?
2019-05-14T09:48:06.297423+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(_ZN8maxscale9RWBackend15process_packetsEP5gwbuf+0x52) [0x7f9505874782]: /home/ec2-user/MaxScale/server/modules/protocol/MySQL/rwbackend.cc:291
2019-05-14T09:48:06.311906+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(_ZN8maxscale9RWBackend13process_replyEP5gwbuf+0x26) [0x7f9505874ac6]: /home/ec2-user/MaxScale/server/modules/protocol/MySQL/rwbackend.cc:387
2019-05-14T09:48:06.330359+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libschemarouter.so(_ZN12schemarouter19SchemaRouterSession11clientReplyEP5gwbufP3dcb+0x3e) [0x7f950544b72e]: /home/ec2-user/MaxScale/server/modules/routing/schemarouter/schemaroutersession.cc:554
2019-05-14T09:48:06.346140+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libschemarouter.so(_ZN8maxscale6RouterIN12schemarouter12SchemaRouterENS1_19SchemaRouterSessionEE11clientReplyEP10mxs_routerP18mxs_router_sessionP5gwbufP3dcb+0x17) [0x7f9505447a17]: /home/ec2-user/MaxScale/include/maxscale/router.hh:206
2019-05-14T09:48:06.352839+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmariadbbackend.so(+0x4667) [0x7f9504ff5667]: /home/ec2-user/MaxScale/server/modules/protocol/MySQL/mariadbbackend/mysql_backend.cc:1041
2019-05-14T09:48:06.365486+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x81bf7) [0x7f950b4cebf7]: /home/ec2-user/MaxScale/server/core/dcb.cc:3136
2019-05-14T09:48:06.378524+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x81d91) [0x7f950b4ced91]: /home/ec2-user/MaxScale/server/core/dcb.cc:3228
2019-05-14T09:48:06.407601+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1b5) [0x7f950b52ab85]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:844
2019-05-14T09:48:06.434956+02:00 siesrv154 maxscale[1234]: (31) /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x51) [0x7f950b52ad81]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:545
2019-05-14T09:48:06.449957+02:00 siesrv154 maxscale[1234]: (31) /usr/bin/maxscale(main+0x2019) [0x409219]: /home/ec2-user/MaxScale/server/core/gateway.cc:2259
2019-05-14T09:48:06.469933+02:00 siesrv154 maxscale[1234]: (31) /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9508b78765]: ??:?
2019-05-14T09:48:06.480446+02:00 siesrv154 maxscale[1234]: (31) /usr/bin/maxscale() [0x409a42]: ??:0
2019-05-14T09:48:06.644916+02:00 siesrv154 systemd[1]: maxscale.service: Main process exited, code=dumped, status=11/SEGV

The



 Comments   
Comment by markus makela [ 2019-06-24 ]

Can you try this with the latest version?

Comment by markus makela [ 2019-06-25 ]

The place where it seems to crash is a bit odd.

void RWBackend::process_packets(GWBUF* result)
 
{
 
    mxs::Buffer buffer(result);
 
    auto it = buffer.begin();
 
 
 
    while (it != buffer.end())
 
    {
 
        // Extract packet length and command byte
 
        uint32_t len = *it++; // Crashed here?

Comment by markus makela [ 2019-07-05 ]

deamn any updates?

Comment by Nicolas De Amicis [ 2019-07-12 ]

I would like to test, but I configured a new server instance (on SLES15sp1) because the current server is in production, but they is no maxscale package for sles15.
I will open an issue for that.

Comment by Timofey Turenko [ 2019-07-12 ]

packages are here https://downloads.mariadb.com/MaxScale/latest/opensuse/15/x86_64/

Comment by Nicolas De Amicis [ 2019-07-15 ]

I need packages for SLES 15 SP1 (SuSE Linux Entreprise Server 15 SP1) not for opensuse 15. I know that opensuse 15 as the same base that SLES, but it's not a good idea to mix packages across distros (it's my point of view).

Comment by markus makela [ 2019-07-23 ]

Managed to hit a debug assertion that might be related to this:

2019-07-23 09:26:28   error  : (127) debug assert at /home/markusjm/MaxScale/server/modules/protocol/MySQL/rwbackend.cc:306 failed: used_len + len <= total_len
 
debug assert at /home/markusjm/MaxScale/server/modules/protocol/MySQL/rwbackend.cc:306 failed: used_len + len <= total_len
 
Aborted (core dumped)

Comment by markus makela [ 2019-07-23 ]

Managed to reproduce it, looks like the schemarouter capabilities did not meet the RWBackend requirements which caused partial result packets to be returned to the router.

Generated at Thu Feb 08 04:14:29 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.