Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31320

Database crashing after upgrade to 10.11.3

    XMLWordPrintable

Details

    Description

      I have a 3 server Master Master cluster which was running on 10.11.2 without much issue. I upgraded a couple of the node to 10.11.3 and these have been crashing with a fair amount of regularity. One will stay up for 15 hours before the Mariadb process crashes and goes into loop of stopping and restarting. The other will almost immediately start crashing and looping as soon as I point some webservers at it. (This is a hosting environment with lots of webservers with multiple websites each of this is pointing at one or other of the DB servers. None of the websites are pointing at multiple DB servers to avoid any potential issues with that).

      I've turned on core files and the syslog output is showing this:

      May 21 00:41:01 wednesday mariadbd[4893]: 230521  0:41:01 [ERROR] mysqld got signal 11 ;
      May 21 00:41:01 wednesday mariadbd[4893]: This could be because you hit a bug. It is also possible that this binary
      May 21 00:41:01 wednesday mariadbd[4893]: or one of the libraries it was linked against is corrupt, improperly built,
      May 21 00:41:01 wednesday mariadbd[4893]: or misconfigured. This error can also be caused by malfunctioning hardware.
      May 21 00:41:01 wednesday mariadbd[4893]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      May 21 00:41:01 wednesday mariadbd[4893]: We will try our best to scrape up some info that will hopefully help
      May 21 00:41:01 wednesday mariadbd[4893]: diagnose the problem, but since we have already crashed,
      May 21 00:41:01 wednesday mariadbd[4893]: something is definitely wrong and this may fail.
      May 21 00:41:01 wednesday mariadbd[4893]: Server version: 10.11.3-MariaDB-1:10.11.3+maria~ubu2204-log source revision: 0bb31039f54bd6a0dc8f0fc7d40e6b58a51998b0
      May 21 00:41:01 wednesday mariadbd[4893]: key_buffer_size=134217728
      May 21 00:41:01 wednesday mariadbd[4893]: read_buffer_size=131072
      May 21 00:41:01 wednesday mariadbd[4893]: max_used_connections=31
      May 21 00:41:01 wednesday mariadbd[4893]: max_threads=50002
      May 21 00:41:01 wednesday mariadbd[4893]: thread_count=48
      May 21 00:41:01 wednesday mariadbd[4893]: It is possible that mysqld could use up to
      May 21 00:41:01 wednesday mariadbd[4893]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 110258497 K  bytes of memory
      May 21 00:41:01 wednesday mariadbd[4893]: Hope that's ok; if not, decrease some variables in the equation.
      May 21 00:41:01 wednesday mariadbd[4893]: Thread pointer: 0x7fe12400b588
      May 21 00:41:01 wednesday mariadbd[4893]: Attempting backtrace. You can use the following information to find out
      May 21 00:41:01 wednesday mariadbd[4893]: where mysqld died. If you see no messages after this, something went
      May 21 00:41:01 wednesday mariadbd[4893]: terribly wrong...
      May 21 00:41:01 wednesday mariadbd[4893]: stack_bottom = 0x7fe18cc71c78 thread_stack 0x30000
      May 21 00:41:01 wednesday mariadbd[4893]: /usr/sbin/mariadbd(my_print_stacktrace+0x32)[0x561b5786cb32]
      May 21 00:41:01 wednesday mariadbd[4893]: /usr/sbin/mariadbd(handle_fatal_signal+0x488)[0x561b57336158]
      May 21 00:41:01 wednesday mariadbd[4893]: /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fe4afd56520]
      May 21 00:41:01 wednesday kernel: [ 1795.932384] traps: mariadbd[5510] general protection fault ip:561b572b22b1 sp:7fe18cb43570 error:0 in mariadbd[561b56f04000+a2e000]
      May 21 00:41:04 wednesday systemd-coredump[5520]: Process 4893 (mariadbd) of user 107 dumped core.#012#012Found module linux-vdso.so.1 with build-id: 1102106fe94d69b7fb79c18cab997b4528ea2148#012Found module libresolv.so.2 with build-id: 7fd7253c61aa6fce2b7e13851c15afa14a5ab160#012Found module libkeyutils.so.1 with build-id: ff27227afa5eeddccab180dd29bd7fcff94aea7c#012Found module libkrb5support.so.0 with build-id: 85c1fccae74910b1afbe878af2202ec6139d8fc2#012Found module libcom_err.so.2 with build-id: ce0901f10854b3c9276066b98d9a72303206e0d5#012Found module libk5crypto.so.3 with build-id: 8bc1e44d4148b2b533d5a97335114565d94197f8#012Found module libkrb5.so.3 with build-id: 62434c49e8118c49a9d60a0795705c806524782d#012Found module libgssapi_krb5.so.2 with build-id: a05177e3a955af79b999bbc081b0f7bf9fb21c87#012Found module libtirpc.so.3 with build-id: 1a361e188043ff5abfdb655af6186b8a0f9b205f#012Found module libnsl.so.2 with build-id: 400d0e5ea8cb66596b2f49bfd0dfe0330ef9f51d#012Found module libnss_nis.so.2 with build-id: 19728f013851322d1aebd3682ca8bd4deef6a3f0#012Found module libnss_compat.so.2 with build-id: afb1eac8ffa86dac183c36720b9ca662a9b5d0f2#012Found module libgpg-error.so.0 with build-id: 3fbec71c67bee60d8aef00697ee187079b0fb307#012Found module libkmod.so.2 with build-id: c8ac4bc8d0fe03ceb8cad8d24484c5cbad9daf5a#012Found module libuuid.so.1 with build-id: 64c0d0cb22fa2bdeca075a0c0418ba5ff314b220#012Found module libudev.so.1 with build-id: 9183eed17e70543d81d9cddda160df12a445f94a#012Found module libgcc_s.so.1 with build-id: 09c4935b79388431a1248f6a98e00d7dc81b8513#012Found module libgcrypt.so.20 with build-id: 60a5e524de0ed8323edf33e9eb9127a9eee02359#012Found module libcap.so.2 with build-id: 9e11e3bca4b0a25d047cb36e933e1d727663cf8e#012Found module liblz4.so.1 with build-id: a85971851cd059f1af80d553c8e7170d42ec59a1#012Found module libzstd.so.1 with build-id: 5d9d0d946a3154a748e87e17af9d14764519237b#012Found module liblzma.so.5 with build-id: b85da6c48eb60a646615392559483b93617ef265#012Found module libdaxctl.so.1 with build-id: f7dfbca3d72bc7ba36d6b60a28119269f2504db2#012Found module libndctl.so.6 with build-id: 22fb97cc03c9bc2e81c12c5e1f82973cfea86338#012Found module ld-linux-x86-64.so.2 with build-id: 61ef896a699bb1c2e4e231642b2e1688b2f1a61e#012Found module libc.so.6 with build-id: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d#012Found module libm.so.6 with build-id: 27e82301dba6c3f644404d504e1bb1c97894b433#012Found module libstdc++.so.6 with build-id: f57e02bfadacc0c923c82457d5e18e1830b5faea#012Found module libcrypto.so.3 with build-id: 62ba5ee88d663a2396160fed1a1864f1f3b60103#012Found module libssl.so.3 with build-id: 4bc97c5bb581ccfe9ae803981f527ce321f16c7a#012Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f#012Found module libsystemd.so.0 with build-id: e45f7492c0f62251620378d7224ad0371a8d1f98#012Found module libpmem.so.1 with build-id: 24df7a25a4291c74c29f0e2bfac791c79fc5ad20#012Found module liburing.so.2 with build-id: 1b8e67a9c76b52628e14723e8890ec81a0d2e205#012Found module libcrypt.so.1 with build-id: 8f3f100ca1e8ff066713aa1e719ce71b46db0296#012Found module libpcre2-8.so.0 with build-id: 184a841c55fb7fe5e3873fcda8368c71016cd54c#012Found module mariadbd with build-id: a0c38761a675995644eed30fd388ffc221115bb6#012Stack trace of thread 5510:#012#0  0x0000561b572b22b1 _ZN13st_join_table21choose_best_splittingEjyPy (mariadbd + 0x9f82b1)#012#1  0x0000561b5711fd20 _Z16best_access_pathP4JOINP13st_join_tableyPK8POSITIONjbdPS3_S6_ (mariadbd + 0x865d20)#012#2  0x0000561b571235b3 n/a (mariadbd + 0x8695b3)#012#3  0x0000561b5712856a n/a (mariadbd + 0x86e56a)#012#4  0x0000561b57129aa8 _Z11choose_planP4JOINy (mariadbd + 0x86faa8)#012#5  0x0000561b5714fa96 n/a (mariadbd + 0x895a96)#012#6  0x0000561b57154882 _ZN4JOIN14optimize_innerEv (mariadbd + 0x89a882)#012#7  0x0000561b57154e9a _ZN4JOIN8optimizeEv (mariadbd + 0x89ae9a)#012#8  0x0000561b570b0095 _ZN13st_select_lex31optimize_unflattened_subqueriesEb (mariadbd + 0x7f6095)#012#9  0x0000561b571ba880 _Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_ (mariadbd + 0x900880)#012#10 0x0000561b570d7927 _Z21mysql_execute_commandP3THDb (mariadbd + 0x81d927)#012#11 0x0000561b570db6f7 _Z11mysql_parseP3THDPcjP12Parser_state (mariadbd + 0x8216f7)#012#12 0x0000561b570dde3d _Z16dispatch_command19enum_server_commandP3THDPcjb (mariadbd + 0x823e3d)#012#13 0x0000561b570dfa98 _Z10do_commandP3THDb (mariadbd + 0x825a98)#012#14 0x0000561b57203f7f _Z24do_handle_one_connectionP7CONNECTb (mariadbd + 0x949f7f)#012#15 0x0000561b572042cd handle_one_connection (mariadbd + 0x94a2cd)#012#16 0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#17 0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#18 0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4918:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7ac1 pthread_cond_wait (libc.so.6 + 0x93ac1)#012#2  0x0000561b57275e82 handle_rpl_parallel_thread (mariadbd + 0x9bbe82)#012#3  0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4899:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7ac1 pthread_cond_wait (libc.so.6 + 0x93ac1)#012#2  0x0000561b570c94f0 handle_manager (mariadbd + 0x80f4f0)#012#3  0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4896:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7ac1 pthread_cond_wait (libc.so.6 + 0x93ac1)#012#2  0x0000561b57773fe9 n/a (mariadbd + 0xeb9fe9)#012#3  0x00007fe4b00ff2b3 n/a (libstdc++.so.6 + 0xdc2b3)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4894:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7f1b pthread_cond_timedwait (libc.so.6 + 0x93f1b)#012#2  0x0000561b57870507 n/a (mariadbd + 0xfb6507)#012#3  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#4  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4909:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7ac1 pthread_cond_wait (libc.so.6 + 0x93ac1)#012#2  0x0000561b57275e82 handle_rpl_parallel_thread (mariadbd + 0x9bbe82)#012#3  0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4920:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7ac1 pthread_cond_wait (libc.so.6 + 0x93ac1)#012#2  0x0000561b57275e82 handle_rpl_parallel_thread (mariadbd + 0x9bbe82)#012#3  0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4911:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7ac1 pthread_cond_wait (libc.so.6 + 0x93ac1)#012#2  0x0000561b57275e82 handle_rpl_parallel_thread (mariadbd + 0x9bbe82)#012#3  0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4904:#012#0  0x00007fe4afd5721a __sigtimedwait (libc.so.6 + 0x4321a)#012#1  0x0000561b56fc17cb signal_hand (mariadbd + 0x7077cb)#012#2  0x0000561b5756fff6 n/a (mariadbd + 0xcb5ff6)#012#3  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#4  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4895:#012#0  0x00007fe4afda5197 n/a (libc.so.6 + 0x91197)#012#1  0x00007fe4afda7f1b pthread_cond_timedwait (libc.so.6 + 0x93f1b)#012#2  0x0000561b57504692 n/a (mariadbd + 0xc4a692)#012#3  0x0000561b574fca5e n/a (mariadbd + 0xc42a5e)#012#4  0x00007fe4afda8b43 n/a (libc.so.6 + 0x94b43)#012#5  0x00007fe4afe3aa00 n/a (libc.so.6 + 0x126a00)#012#012Stack trace of thread 4919:#012
      May 21 00:41:04 wednesday systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
      May 21 00:41:04 wednesday systemd[1]: mariadb.service: Failed with result 'signal'.
      May 21 00:41:04 wednesday systemd[1]: mariadb.service: Consumed 27.662s CPU time.
      May 21 00:41:09 wednesday systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
      May 21 00:41:09 wednesday systemd[1]: mariadb.service: Consumed 27.662s CPU time.
      

      I've also attached the core dump.

      I have run a mysqlcheck against both affected DB servers but this had no affect. When they are just working as slaves there are no issues. It's only when they are put under load that this happens.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              turf212 Derick Turner
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.