[MDEV-21718] mysqld crash Created: 2020-02-12  Updated: 2020-07-24  Resolved: 2020-07-24

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.4.11, 10.4.12, 10.4.13
Fix Version/s: 10.4.14, 10.5.5

Type: Bug Priority: Major
Reporter: Kent Hoover Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: crash
Environment:
  1. mysql
    Welcome to the MariaDB monitor. Commands end with ; or \g.
    Your MariaDB connection id is 3007
    Server version: 10.4.11-MariaDB MariaDB Server
  1. rpm -qa | grep gal
    galera-4-26.4.3-1.rhel6.el6.x86_64
  1. uname -a
    Linux vertica01.cem.spirent.com 2.6.32-754.22.1.el6.x86_64 #1 SMP Tue Sep 17 16:24:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  2. cat /etc/centos-release
    CentOS release 6.10 (Final)
  3. cat /etc/lsb-release
    LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

Attachments: Microsoft Word my.cnf.host1.rtf    

 Description   

Hosts in this cluster are occasionally crashing, with this message each time.

Here's what got written to our mysql-error.log ...

mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.11/wsrep-lib/src/client_state.cpp:121: int wsrep::client_state::before_command(): Assertion `server_state_.rollback_mode() == wsrep::server_state::rm_async' failed.
200211 21:41:48 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.11-MariaDB
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=278
max_threads=65541
thread_count=238
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 142750971 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f113c0c73b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f12fb93ec98 thread_stack 0x49000

      • stack smashing detected ***: /usr/sbin/mysqld terminated
        ======= Backtrace: =========
        /lib64/libc.so.6(__fortify_fail+0x37)[0x7f1513812877]
        /lib64/libc.so.6(__fortify_fail+0x0)[0x7f1513812840]
        /usr/sbin/mysqld(+0xe3b93c)[0x562eb83c893c]
        /usr/sbin/mysqld(my_print_stacktrace+0x1c6)[0x562eb83b17a6]
        /usr/sbin/mysqld(handle_fatal_signal+0x4b7)[0x562eb7e80557]
        /lib64/libpthread.so.0(+0x3ff8c0f7e0)[0x7f15150e87e0]
        /lib64/libc.so.6(gsignal+0x35)[0x7f15137424f5]
        /lib64/libc.so.6(abort+0x175)[0x7f1513743cd5]
        /lib64/libc.so.6(+0x3ff882b66e)[0x7f151373b66e]
        /lib64/libc.so.6(__assert_perror_fail+0x0)[0x7f151373b730]
        /usr/sbin/mysqld(_ZN5wsrep12client_state14before_commandEv+0x320)[0x562eb8431150]
        /usr/sbin/mysqld(_Z10do_commandP3THD+0x1a4)[0x562eb7c9d1d4]
        /usr/sbin/mysqld(_Z11tp_callbackP13TP_connection+0x58)[0x562eb7e55568]
        /usr/sbin/mysqld(+0xa56dd8)[0x562eb7fe3dd8]
        /usr/sbin/mysqld(+0xa856fd)[0x562eb80126fd]
        /lib64/libpthread.so.0(+0x3ff8c07aa1)[0x7f15150e0aa1]
        /lib64/libc.so.6(clone+0x6d)[0x7f15137f8c4d]

 Comments   
Comment by Elena Stepanova [ 2020-02-12 ]

khoov, what did you mean by setting the label `buildbot`? Have you seen it happen in MariaDB buildbot? Or in your buildbot? Or in someone else's?

Comment by Kent Hoover [ 2020-02-12 ]

Hi, Elena:
I spotted "buildbot" in the pathname of the source file identified in the error log... so I just added it to the labels.

Thanks,
Kent

Comment by Kent Hoover [ 2020-06-25 ]

Hello, Elena:

Any update/progress regarding this error... We've upgraded this environment to 10.4.12, and are still encountering this crash.

Thanks,
Kent

Comment by Robert Klikics [ 2020-07-21 ]

Hi,

we're experiencing the same issue with 10.4.13. Any news on this?

Comment by Kent Hoover [ 2020-07-22 ]

I upgraded my site to 10.4.13/galera-4-26.4.4-1 , still experiencing these occasional crashes.

Typically, mysqld dies on 2 of my 3 servers, with the WSREP complaint about the failed assertion. The surviving host refuses updates at that point.
So, recovery is usually not straightforward.

Any updates?

Thanks,
Kent

Comment by Kent Hoover [ 2020-07-22 ]

This Percona bug [ https://jira.percona.com/browse/PXC-2935 ] (recently fixed) looks the same as this one .

Comment by Kent Hoover [ 2020-07-23 ]

Thanks, Jan and Teemu for digging in...

Question: If this does match the Percona case, is it reasonable that we could work around this by changing configuration from pool-of-threads to one-thread-per-connection (as long as our application can tolerate the effect of this change)?

Cheers,
Kent

Comment by Teemu Ollakka [ 2020-07-23 ]

Hi,

It looks that changing to one-thread-per-connection should make this crash to go away.

  • Teemu
Comment by Kent Hoover [ 2020-07-23 ]

Thanks for the quick reply, Teemu.
(We'll still be looking forward to the fix, of course � )

Kent

Generated at Thu Feb 08 09:09:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.