Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15761

mysqld crashed after WSREP ERROR Trying to launch slave threads before creating connection at gcomm:// and mysqld got signal 6

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Duplicate
    • 10.3.5, 10.1(EOL), 10.2(EOL)
    • N/A
    • Galera, wsrep
    • None
    • 3 Nodes MariDB cluster;CentOS 7.4

    Description

      mysqld crashed after WSREP ERROR Trying to launch slave threads before creating connection at gcomm:// and mysqld got signal 6

      issue occurred after Galera cluster recovering
      and bootstrapping 1 Node , Node was bootstrapped successfully but on other Node mysql crashed
      after wsrep error and mysqld Assertion `0' failed

      2018-04-03 12:01:34 0 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196'
      mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.5/sql/wsrep_thd.cc:446: void wsrep_create_appliers(long int): Assertion `0' failed.
      180403 12:01:34 [ERROR] mysqld got signal 6 ;
      

      2018-04-03 12:01:34 0 [Note] WSREP: Signalling provider to continue.
      2018-04-03 12:01:34 0 [Note] WSREP: SST received: dff6e041-1005-11e8-85c9-965f304f37bc:131618
      2018-04-03 12:01:34 0 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196'
      mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.5/sql/wsrep_thd.cc:446: void wsrep_create_appliers(long int): Assertion `0' failed.
      180403 12:01:34 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.3.5-MariaDB
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=0
      max_threads=153
      thread_count=7
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467384 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      2018-04-03 12:01:34 1 [ERROR] WSREP: gcs/src/gcs.cpp:_join():892: Sending JOIN failed: -103 (Software caused connection abort).
      stack_bottom = 0x0 thread_stack 0x49000
      2018-04-03 12:01:34 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
      2018-04-03 12:01:34 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
      2018-04-03 12:01:34 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2018-04-03 12:01:34 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
      2018-04-03 12:01:34 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2018-04-03 12:01:34 1 [Note] WSREP: applier thread exiting (code:0)
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dd4ca1ccce]
      2018-04-03 12:01:34 0 [Note] InnoDB: Buffer pool(s) load completed at 180403 12:01:34
      /usr/sbin/mysqld(handle_fatal_signal+0x357)[0x55dd4c4aad17]
      2018-04-03 12:01:36 2 [Note] WSREP: rollbacker thread exiting
      /lib64/libpthread.so.0(+0xf5e0)[0x7f954a32e5e0]
      /lib64/libc.so.6(gsignal+0x37)[0x7f954883b1f7]
      :0(__GI_raise)[0x7f954883c8e8]
      :0(__GI_abort)[0x7f9548834266]
      :0(__assert_fail_base)[0x7f9548834312]
      /usr/sbin/mysqld(+0x6e0f0f)[0x55dd4c428f0f]
      /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xd50)[0x55dd4c22e360]
      /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9548827c05]
      /usr/sbin/mysqld(+0x4d93ed)[0x55dd4c2213ed]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      
      

      Attachments

        Issue Links

          Activity

            winstone Zdravelina Sokolovska (Inactive) created issue -
            winstone Zdravelina Sokolovska (Inactive) made changes -
            Field Original Value New Value
            Attachment full_log.txt [ 45428 ]
            elenst Elena Stepanova made changes -
            Fix Version/s 10.3 [ 22126 ]
            Assignee Sachin Setiya [ sachin.setiya.007 ]
            serg Sergei Golubchik made changes -
            Description mysqld crashed after WSREP ERROR Trying to launch slave threads before creating connection at gcomm:// and mysqld got signal 6



            issue occurred after Galera cluster recovering
            and bootstrapping 1 Node , Node was bootstrapped successfully but on other Node mysql crashed
            after wsrep error and mysqld Assertion `0' failed

            2018-04-03 12:01:34 0 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196'
            mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.5/sql/wsrep_thd.cc:446: void wsrep_create_appliers(long int): Assertion `0' failed.
            180403 12:01:34 [ERROR] mysqld got signal 6 ;





            {noformat}
            2018-04-03 12:01:34 0 [Note] WSREP: Signalling provider to continue.
            2018-04-03 12:01:34 0 [Note] WSREP: SST received: dff6e041-1005-11e8-85c9-965f304f37bc:131618
            2018-04-03 12:01:34 0 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196'
            mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.5/sql/wsrep_thd.cc:446: void wsrep_create_appliers(long int): Assertion `0' failed.
            180403 12:01:34 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.3.5-MariaDB
            key_buffer_size=134217728
            read_buffer_size=131072
            max_used_connections=0
            max_threads=153
            thread_count=7
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467384 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            2018-04-03 12:01:34 1 [ERROR] WSREP: gcs/src/gcs.cpp:_join():892: Sending JOIN failed: -103 (Software caused connection abort).
            stack_bottom = 0x0 thread_stack 0x49000
            2018-04-03 12:01:34 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
            2018-04-03 12:01:34 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
            2018-04-03 12:01:34 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            2018-04-03 12:01:34 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
            2018-04-03 12:01:34 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            2018-04-03 12:01:34 1 [Note] WSREP: applier thread exiting (code:0)
            /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dd4ca1ccce]
            2018-04-03 12:01:34 0 [Note] InnoDB: Buffer pool(s) load completed at 180403 12:01:34
            /usr/sbin/mysqld(handle_fatal_signal+0x357)[0x55dd4c4aad17]
            2018-04-03 12:01:36 2 [Note] WSREP: rollbacker thread exiting
            /lib64/libpthread.so.0(+0xf5e0)[0x7f954a32e5e0]
            /lib64/libc.so.6(gsignal+0x37)[0x7f954883b1f7]
            :0(__GI_raise)[0x7f954883c8e8]
            :0(__GI_abort)[0x7f9548834266]
            :0(__assert_fail_base)[0x7f9548834312]
            /usr/sbin/mysqld(+0x6e0f0f)[0x55dd4c428f0f]
            /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xd50)[0x55dd4c22e360]
            /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9548827c05]
            /usr/sbin/mysqld(+0x4d93ed)[0x55dd4c2213ed]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.

            {noformat}
            mysqld crashed after WSREP ERROR Trying to launch slave threads before creating connection at gcomm:// and mysqld got signal 6



            issue occurred after Galera cluster recovering
            and bootstrapping 1 Node , Node was bootstrapped successfully but on other Node mysql crashed
            after wsrep error and mysqld Assertion `0' failed
            {noformat}
            2018-04-03 12:01:34 0 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196'
            mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.5/sql/wsrep_thd.cc:446: void wsrep_create_appliers(long int): Assertion `0' failed.
            180403 12:01:34 [ERROR] mysqld got signal 6 ;
            {noformat}

            {noformat}
            2018-04-03 12:01:34 0 [Note] WSREP: Signalling provider to continue.
            2018-04-03 12:01:34 0 [Note] WSREP: SST received: dff6e041-1005-11e8-85c9-965f304f37bc:131618
            2018-04-03 12:01:34 0 [ERROR] WSREP: Trying to launch slave threads before creating connection at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196'
            mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.5/sql/wsrep_thd.cc:446: void wsrep_create_appliers(long int): Assertion `0' failed.
            180403 12:01:34 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.3.5-MariaDB
            key_buffer_size=134217728
            read_buffer_size=131072
            max_used_connections=0
            max_threads=153
            thread_count=7
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467384 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            2018-04-03 12:01:34 1 [ERROR] WSREP: gcs/src/gcs.cpp:_join():892: Sending JOIN failed: -103 (Software caused connection abort).
            stack_bottom = 0x0 thread_stack 0x49000
            2018-04-03 12:01:34 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
            2018-04-03 12:01:34 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
            2018-04-03 12:01:34 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            2018-04-03 12:01:34 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
            2018-04-03 12:01:34 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            2018-04-03 12:01:34 1 [Note] WSREP: applier thread exiting (code:0)
            /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dd4ca1ccce]
            2018-04-03 12:01:34 0 [Note] InnoDB: Buffer pool(s) load completed at 180403 12:01:34
            /usr/sbin/mysqld(handle_fatal_signal+0x357)[0x55dd4c4aad17]
            2018-04-03 12:01:36 2 [Note] WSREP: rollbacker thread exiting
            /lib64/libpthread.so.0(+0xf5e0)[0x7f954a32e5e0]
            /lib64/libc.so.6(gsignal+0x37)[0x7f954883b1f7]
            :0(__GI_raise)[0x7f954883c8e8]
            :0(__GI_abort)[0x7f9548834266]
            :0(__assert_fail_base)[0x7f9548834312]
            /usr/sbin/mysqld(+0x6e0f0f)[0x55dd4c428f0f]
            /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xd50)[0x55dd4c22e360]
            /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f9548827c05]
            /usr/sbin/mysqld(+0x4d93ed)[0x55dd4c2213ed]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.

            {noformat}
            serg Sergei Golubchik made changes -
            Fix Version/s 10.3 [ 22126 ]
            serg Sergei Golubchik made changes -
            Assignee Sachin Setiya [ sachin.setiya.007 ] Jan Lindström [ jplindst ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.3 [ 22126 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jplindst Jan Lindström (Inactive) added a comment - This looks very similar to https://jira.mariadb.org/browse/MDEV-15607 .
            jplindst Jan Lindström (Inactive) made changes -
            issue.field.resolutiondate 2018-04-25 05:20:45.0 2018-04-25 05:20:45.918
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.3 [ 22126 ]
            Resolution Duplicate [ 3 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            jplindst Jan Lindström (Inactive) made changes -
            jplindst Jan Lindström (Inactive) made changes -
            Affects Version/s 10.1 [ 16100 ]
            jplindst Jan Lindström (Inactive) made changes -
            Affects Version/s 10.2 [ 14601 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 86348 ] MariaDB v4 [ 154080 ]

            People

              jplindst Jan Lindström (Inactive)
              winstone Zdravelina Sokolovska (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.