[MXS-3200] Abort due to double free or corruption after "Write to Client DCB ... state DCB_STATE_POLLING failed" Created: 2020-09-21  Updated: 2021-04-19  Resolved: 2020-10-12

Status: Closed
Project: MariaDB MaxScale
Component/s: schemarouter
Affects Version/s: 2.4.12
Fix Version/s: 2.4.13, 2.5.5

Type: Bug Priority: Critical
Reporter: Hartmut Holzgraefe Assignee: Johan Wikman
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MXS-3070 regular aborts with DCB_STATE_POLLING... Closed

 Description   

Seeing a similar thing happening on customer case CS0122589, but with different stack trace, where

Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address

followed by

double free or corruption (out)

leads to an abort() call first, followed by a segfault during signal handling.

This is happening repeatedly on two maxcale instances, sometimes even right after maxscale process restart.

2020-09-18 01:24:54   error  : (974076) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address
2020-09-18 01:24:54   alert  : (974076) Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
2020-09-18 01:24:54   alert  : (974076) Statement currently being classified: none/unknown
double free or corruption (out)
Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
 
2020-09-18 01:24:54   alert  : (974076) 
  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fb5d37b7f47]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fb5d37b98b1]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(+0x89907) [0x7fb5d3802907]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(+0x9097a) [0x7fb5d380997a]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(cfree+0x525) [0x7fb5d3810ee5]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_Z10gwbuf_freeP5GWBUF+0x18) [0x7fb5d5784978]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x454a) [0x7fb5cf24954a]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x49f1) [0x7fb5cf2499f1]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c887) [0x7fb5d57af887]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c961) [0x7fb5d57af961]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1a6) [0x7fb5d582c646]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53) [0x7fb5d582c863]: binutils not installed
  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7fb5d45436df]: binutils not installed
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb5d4fea6db]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb5d389aa3f]: binutils not installed
2020-09-18 01:24:54   alert  : (974076) Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
2020-09-18 01:24:54   alert  : (974076) Statement currently being classified: none/unknown
 
 
Writing core dump
Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
 
 
 
Writing core dump
2020-09-18 01:24:54   alert  : (974076) 
  /lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7fb5d37b99a0]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(+0x89907) [0x7fb5d3802907]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(+0x9097a) [0x7fb5d380997a]: binutils not installed
  /lib/x86_64-linux-gnu/libc.so.6(cfree+0x525) [0x7fb5d3810ee5]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_Z10gwbuf_freeP5GWBUF+0x18) [0x7fb5d5784978]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x454a) [0x7fb5cf24954a]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x49f1) [0x7fb5cf2499f1]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c887) [0x7fb5d57af887]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c961) [0x7fb5d57af961]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1a6) [0x7fb5d582c646]: binutils not installed
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53) [0x7fb5d582c863]: binutils not installed
  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7fb5d45436df]: binutils not installed
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb5d4fea6db]: binutils not installed
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb5d389aa3f]: binutils not installed



 Comments   
Comment by Johan Wikman [ 2020-10-06 ]

Are those Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed always followed by a crash? And a crash is always preceded by that error?

Comment by Hartmut Holzgraefe [ 2020-10-06 ]

Not always, but almost. Also there are two different cases of DCB_STATE_POLLING log entries, only those with "14, Bad address" error message are directly related to crashes, while those showing "104, Connection reset by peer" are showing more often, and not directly related to crashes. On the first maxscale nodes log I see nine crashes, seven of which are directly following "DCB_STATE_POLLING Bad Address", one seems unrelated, and one has a ten second delay between Bad Address and crash:

#1 
{"log":"2020-09-18 01:24:54   error  : (974076) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-18T01:24:54.364117221Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:24:54.364191351Z"}
 
#2
{"log":"2020-09-18 01:26:09   error  : (1390) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-18T01:26:09.119713151Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:26:09.12025142Z"}
 
#3 - crash without DCB_STATE_POLLING failed: 14, Bad address
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:26:50.980079721Z"}
 
#4
{"log":"2020-09-19 01:37:20   error  : (3428495) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:37:20.954614354Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:37:20.954344435Z"}
 
#5
{"log":"2020-09-19 01:37:57   error  : (1778) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:37:57.428786922Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:37:57.428817716Z"}
 
#6
{"log":"2020-09-19 01:39:45   error  : (751) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:39:45.523828761Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:39:45.531472865Z"}
 
#7
{"log":"2020-09-19 01:41:07   error  : (3615) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:41:07.100807197Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:41:07.222915366Z"}
 
#8 10 seconds between message and crash
{"log":"2020-09-19 01:41:49   error  : (280) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:41:49.439615283Z"}
 
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:41:59.230705607Z"}
 
#9
{"log":"2020-09-19 01:43:11   error  : (2460) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:43:11.953636015Z"}
{"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:43:11.953653404Z"}

Comment by Johan Wikman [ 2020-10-12 ]

The behaviour could not be repeated, but a bug that would cause the reported behaviour was found and fixed. Closing the issue. Please reopen if the reported behaviour does not disappear.

Generated at Thu Feb 08 04:19:40 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.