Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-3200

Abort due to double free or corruption after "Write to Client DCB ... state DCB_STATE_POLLING failed"

Details

    Description

      Seeing a similar thing happening on customer case CS0122589, but with different stack trace, where

      Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address

      followed by

      double free or corruption (out)

      leads to an abort() call first, followed by a segfault during signal handling.

      This is happening repeatedly on two maxcale instances, sometimes even right after maxscale process restart.

      2020-09-18 01:24:54   error  : (974076) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address
      2020-09-18 01:24:54   alert  : (974076) Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
      2020-09-18 01:24:54   alert  : (974076) Statement currently being classified: none/unknown
      double free or corruption (out)
      Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
       
      2020-09-18 01:24:54   alert  : (974076) 
        /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fb5d37b7f47]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fb5d37b98b1]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(+0x89907) [0x7fb5d3802907]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(+0x9097a) [0x7fb5d380997a]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(cfree+0x525) [0x7fb5d3810ee5]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_Z10gwbuf_freeP5GWBUF+0x18) [0x7fb5d5784978]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x454a) [0x7fb5cf24954a]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x49f1) [0x7fb5cf2499f1]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c887) [0x7fb5d57af887]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c961) [0x7fb5d57af961]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1a6) [0x7fb5d582c646]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53) [0x7fb5d582c863]: binutils not installed
        /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7fb5d45436df]: binutils not installed
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb5d4fea6db]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb5d389aa3f]: binutils not installed
      2020-09-18 01:24:54   alert  : (974076) Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
      2020-09-18 01:24:54   alert  : (974076) Statement currently being classified: none/unknown
       
       
      Writing core dump
      Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS
       
       
       
      Writing core dump
      2020-09-18 01:24:54   alert  : (974076) 
        /lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7fb5d37b99a0]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(+0x89907) [0x7fb5d3802907]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(+0x9097a) [0x7fb5d380997a]: binutils not installed
        /lib/x86_64-linux-gnu/libc.so.6(cfree+0x525) [0x7fb5d3810ee5]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_Z10gwbuf_freeP5GWBUF+0x18) [0x7fb5d5784978]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x454a) [0x7fb5cf24954a]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x49f1) [0x7fb5cf2499f1]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c887) [0x7fb5d57af887]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x9c961) [0x7fb5d57af961]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1a6) [0x7fb5d582c646]: binutils not installed
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53) [0x7fb5d582c863]: binutils not installed
        /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7fb5d45436df]: binutils not installed
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fb5d4fea6db]: binutils not installed
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb5d389aa3f]: binutils not installed
      

      Attachments

        Issue Links

          Activity

            johan.wikman Johan Wikman added a comment -

            Are those Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed always followed by a crash? And a crash is always preceded by that error?

            johan.wikman Johan Wikman added a comment - Are those Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed always followed by a crash? And a crash is always preceded by that error?

            Not always, but almost. Also there are two different cases of DCB_STATE_POLLING log entries, only those with "14, Bad address" error message are directly related to crashes, while those showing "104, Connection reset by peer" are showing more often, and not directly related to crashes. On the first maxscale nodes log I see nine crashes, seven of which are directly following "DCB_STATE_POLLING Bad Address", one seems unrelated, and one has a ten second delay between Bad Address and crash:

            #1 
            {"log":"2020-09-18 01:24:54   error  : (974076) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-18T01:24:54.364117221Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:24:54.364191351Z"}
             
            #2
            {"log":"2020-09-18 01:26:09   error  : (1390) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-18T01:26:09.119713151Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:26:09.12025142Z"}
             
            #3 - crash without DCB_STATE_POLLING failed: 14, Bad address
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:26:50.980079721Z"}
             
            #4
            {"log":"2020-09-19 01:37:20   error  : (3428495) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:37:20.954614354Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:37:20.954344435Z"}
             
            #5
            {"log":"2020-09-19 01:37:57   error  : (1778) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:37:57.428786922Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:37:57.428817716Z"}
             
            #6
            {"log":"2020-09-19 01:39:45   error  : (751) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:39:45.523828761Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:39:45.531472865Z"}
             
            #7
            {"log":"2020-09-19 01:41:07   error  : (3615) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:41:07.100807197Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:41:07.222915366Z"}
             
            #8 10 seconds between message and crash
            {"log":"2020-09-19 01:41:49   error  : (280) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:41:49.439615283Z"}
             
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:41:59.230705607Z"}
             
            #9
            {"log":"2020-09-19 01:43:11   error  : (2460) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:43:11.953636015Z"}
            {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:43:11.953653404Z"}
            

            hholzgra Hartmut Holzgraefe added a comment - Not always, but almost. Also there are two different cases of DCB_STATE_POLLING log entries, only those with "14, Bad address" error message are directly related to crashes, while those showing "104, Connection reset by peer" are showing more often, and not directly related to crashes. On the first maxscale nodes log I see nine crashes, seven of which are directly following "DCB_STATE_POLLING Bad Address", one seems unrelated, and one has a ten second delay between Bad Address and crash: #1 {"log":"2020-09-18 01:24:54 error : (974076) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-18T01:24:54.364117221Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 6. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:24:54.364191351Z"}   #2 {"log":"2020-09-18 01:26:09 error : (1390) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-18T01:26:09.119713151Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:26:09.12025142Z"}   #3 - crash without DCB_STATE_POLLING failed: 14, Bad address {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-18T01:26:50.980079721Z"}   #4 {"log":"2020-09-19 01:37:20 error : (3428495) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:37:20.954614354Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:37:20.954344435Z"}   #5 {"log":"2020-09-19 01:37:57 error : (1778) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:37:57.428786922Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:37:57.428817716Z"}   #6 {"log":"2020-09-19 01:39:45 error : (751) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:39:45.523828761Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:39:45.531472865Z"}   #7 {"log":"2020-09-19 01:41:07 error : (3615) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:41:07.100807197Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:41:07.222915366Z"}   #8 10 seconds between message and crash {"log":"2020-09-19 01:41:49 error : (280) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:41:49.439615283Z"}   {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:41:59.230705607Z"}   #9 {"log":"2020-09-19 01:43:11 error : (2460) Write to Client DCB 10.10.130.24 in state DCB_STATE_POLLING failed: 14, Bad address\n","stream":"stdout","time":"2020-09-19T01:43:11.953636015Z"} {"log":"Fatal: MaxScale 2.4.12 received fatal signal 11. Commit ID: 7756d7c25708811b7a52b72156de752d6ac91a35 System name: Linux Release string: Ubuntu 18.04.5 LTS\n","stream":"stderr","time":"2020-09-19T01:43:11.953653404Z"}
            johan.wikman Johan Wikman added a comment -

            The behaviour could not be repeated, but a bug that would cause the reported behaviour was found and fixed. Closing the issue. Please reopen if the reported behaviour does not disappear.

            johan.wikman Johan Wikman added a comment - The behaviour could not be repeated, but a bug that would cause the reported behaviour was found and fixed. Closing the issue. Please reopen if the reported behaviour does not disappear.

            People

              johan.wikman Johan Wikman
              hholzgra Hartmut Holzgraefe
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.