[MXS-3332] MaxScale 2.5.5. Signal 11 on database select Created: 2020-12-09  Updated: 2021-03-10  Resolved: 2021-03-10

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 2.5.5
Fix Version/s: 2.5.9

Type: Bug Priority: Major
Reporter: Bryan Alsdorf Assignee: markus makela
Resolution: Fixed Votes: 1
Labels: None
Environment:

Ubuntu 18.04.4 LTS connecting to single SkySQL Server.

Attachments: Text File global.txt     File maxscale.cnf     File maxscale.log    
Sprint: MXS-SPRINT-122, MXS-SPRINT-123, MXS-SPRINT-124, MXS-SPRINT-125, MXS-SPRINT-126

 Description   

When selecting a database MaxScale is crashing with a signal 11.

$ mysql -h127.0.0.1 -uXUSERX -p --port=3306
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 1
Server version: 10.5.4-2-MariaDB-enterprise-log MariaDB Enterprise Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> \u XDATABASEX
ERROR 2013 (HY000): Lost connection to MySQL server during query

Specifying a database on initial connect works fine

mysql -h127.0.0.1 -uXUSERX -p --port=3306 XDATABASEX

Stacktrace:

2020-12-09 13:18:35   alert  : (1) MaxScale 2.5.5 received fatal signal 11. Commit ID: 91c3b76195d0057ddbe572bbb8d17f6ac6b09d5e System name: Linux Release string: Ubuntu 18.04.4 LTS
 
2020-12-09 13:18:35   alert  : (1) Statement currently being classified: none/unknown
 
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
 
2020-12-09 13:18:35   alert  : (1) 
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so(+0x26dba): server/modules/protocol/MariaDB/mariadb_backend.cc:71
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so(_ZN24MariaDBBackendConnection17process_ok_packetEN8maxscale6Buffer8iteratorES2_+0x139): server/modules/protocol/MariaDB/mariadb_backend.cc:114
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so(_ZN24MariaDBBackendConnection20process_result_startEN8maxscale6Buffer8iteratorES2_+0xd2): server/modules/protocol/MariaDB/mariadb_backend.cc:2076
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so(_ZN24MariaDBBackendConnection15process_packetsEPP5GWBUF+0x152): server/modules/protocol/MariaDB/mariadb_backend.cc:1811
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so(_ZN24MariaDBBackendConnection14track_responseEPP5GWBUF+0xf): server/modules/protocol/MariaDB/mariadb_backend.cc:1450
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so(_ZN24MariaDBBackendConnection11normal_readEv+0x628): server/modules/protocol/MariaDB/mariadb_backend.cc:574
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0x6c): server/core/dcb.cc:1291
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x21): server/core/dcb.cc:1350
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1ce): maxutils/maxbase/src/worker.cc:879
 
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
 
  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df): ??:?
 
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db): /build/glibc-S7xCS9/glibc-2.27/nptl/pthread_create.c:463
 
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f): ??:0



 Comments   
Comment by Bryan Alsdorf [ 2020-12-10 ]

I successfully downgraded to 2.4 as a workaround

Comment by markus makela [ 2021-01-11 ]

Based on the stacktrace, the most likely cause of this is the use of session_track_schema=ON in the server configuration. Obviously, even if it is on MaxScale shouldn't crash but, as a workaround, I believe it can be disabled.

Initial testing with the community 10.5.8 binaries in a docker container doesn't reveal anything wrong in the current code.

Comment by markus makela [ 2021-01-11 ]

Tested with 10.5.8 enterprise packages and MaxScale 2.5.5, wasn't able to reproduce it.

Same story with 10.5.5-3.

Comment by markus makela [ 2021-01-11 ]

Looking at the code, the most likely reason for the crash is a read past the end of the buffer. The code assumes that the server sends valid packets and I'm guessing that the server sent something that MaxScale wasn't able to decode which in turn caused it to think it needed to read more data than was available.

Comment by Bryan Alsdorf [ 2021-01-11 ]

I've attached show global variables for the server, since it is skysql I do not have access to the raw configuration files. You are correct that session_track_schema is on.

global.txt

Comment by markus makela [ 2021-03-08 ]

bryan do you remember of this ever happened with the readwritesplit router?

Comment by Bryan Alsdorf [ 2021-03-08 ]

I do not believe we had the readwritesplit setup (unless it is enabled by default). We only had one backend enabled though.

Comment by markus makela [ 2021-03-08 ]

If you have the chance to test this again, try changing the router from readconnroute to readwritesplit. This should give us more information about whether this is only a problem with readconnroute or if it affects readwritesplit as well. The number of the backends in this case doesn't matter.

I did find a bug where readconnroute was unnecessarily executing the code where the crash happened. With the fix in place I wouldn't expect this to happen again but I don't yet fully understand why the crash happened in the first place which is why reproducing this would still be valuable.

Comment by markus makela [ 2021-03-10 ]

I'll close this as fixed in 2.5.9 since the code that caused this isn't going to be used by readconnroute anymore. In addition, with the same exact versions of the products and the steps described in the issue, I wasn't able to reproduce the problem.

If you can, please let us know if you'll see this again now that 2.5.9 has been released.

Comment by Bryan Alsdorf [ 2021-03-10 ]

Thanks, I'll upgrade the server to 2.5.9 when I get a chance and report back.

Generated at Thu Feb 08 04:20:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.