Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29190

10.6.8 (+uring compiled in but not used) hangs after sudden load spikes

    XMLWordPrintable

Details

    Description

      We are in the process of troubleshooting a 10.6.8 (compiled with uring) host that shows complete stalls after they get sudden load spikes.

      # ldd /opt/wmf-mariadb106/bin/mysqld | grep uring
      	liburing.so.1 => /lib/x86_64-linux-gnu/liburing.so.1 (0x00007f4d12123000)
      

      Along with Daniel and Vicențiu we got some stack traces
      See files attached for the full output.
      We also got some more specific ones requested by Marko and tracked at https://phabricator.wikimedia.org/P32031 but also pasting them here for the record

      root@db1132:/srv/tmp# gdb /opt/wmf-mariadb106/bin/mysqld core.3344701
      GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
      Copyright (C) 2021 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      Type "show copying" and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <https://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
          <http://www.gnu.org/software/gdb/documentation/>.
       
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from /opt/wmf-mariadb106/bin/mysqld...
      c[New LWP 3344701]
      [New LWP 3344702]
      [New LWP 3344703]
      [New LWP 3344704]
      [New LWP 3344707]
      [New LWP 3344708]
      [New LWP 3344709]
      [New LWP 3344710]
      [New LWP 3344711]
      [New LWP 3344712]
      [New LWP 3344713]
      [New LWP 1390476]
      [New LWP 1394265]
      [New LWP 1396255]
      [New LWP 1396643]
      [New LWP 1397334]
      [New LWP 1397339]
      [New LWP 1397368]
      [New LWP 1397405]
      [New LWP 1397494]
      [New LWP 1397698]
      [New LWP 1397721]
      [New LWP 1397753]
      [New LWP 1397815]
      [New LWP 1397919]
      [New LWP 1397927]
      [New LWP 1398086]
      [New LWP 1398119]
      [New LWP 1398179]
      [New LWP 1398180]
      [New LWP 1398239]
      [New LWP 1398242]
      [New LWP 1398278]
      [New LWP 1398528]
      [New LWP 1398717]
      [New LWP 1398775]
      [New LWP 1398821]
      [New LWP 1398855]
      [New LWP 1398860]
      [New LWP 1398892]
      [New LWP 1398913]
      [New LWP 1398935]
      [New LWP 1399058]
      [New LWP 1399138]
      [New LWP 1399140]
      [New LWP 1399146]
      [New LWP 1399179]
      [New LWP 1399181]
      [New LWP 1399226]
      [New LWP 1399339]
      [New LWP 1399480]
      [New LWP 1399491]
      [New LWP 1399517]
      [New LWP 1399586]
      [New LWP 1399629]
      [New LWP 1399639]
      [New LWP 1399770]
      [New LWP 1399867]
      [New LWP 1399877]
      [New LWP 1399878]
      [New LWP 1399960]
      [New LWP 1400005]
      [New LWP 1400015]
      [New LWP 1400037]
      [New LWP 1400043]
      [New LWP 1400080]
      [New LWP 1400081]
      [New LWP 1400082]
      [New LWP 1400084]
      [New LWP 1400086]
      [New LWP 1400196]
      [New LWP 1400212]
      [New LWP 1400266]
      [New LWP 1400267]
      [New LWP 1400288]
      [New LWP 1400292]
      [New LWP 1400358]
      [New LWP 1400364]
      [New LWP 1400393]
      [New LWP 1400394]
      [New LWP 1400396]
      [New LWP 1400402]
      [New LWP 1401080]
      [New LWP 1401164]
      [New LWP 1401169]
      [New LWP 1401173]
      [New LWP 1401181]
      [New LWP 1402234]
      [New LWP 1402235]
      [New LWP 1402370]
      [New LWP 1402379]
      [New LWP 1402403]
      [New LWP 1402405]
      [New LWP 1402407]
      [New LWP 1402419]
      [New LWP 1402437]
      [New LWP 1402458]
      [New LWP 1402477]
      [New LWP 1402493]
      [New LWP 1402514]
      [New LWP 1402521]
      [New LWP 1402522]
      [New LWP 1402536]
      [New LWP 1402537]
      [New LWP 1402539]
      [New LWP 1402591]
      [New LWP 1402606]
      [New LWP 1402661]
      [New LWP 1402721]
      [New LWP 1402737]
      [New LWP 1402748]
      [New LWP 1402776]
      [New LWP 1402777]
      [New LWP 1402778]
      [New LWP 1402780]
      [New LWP 1402781]
      [New LWP 1402783]
      [New LWP 1402784]
      [New LWP 1402792]
      [New LWP 1402793]
      [New LWP 1402797]
      [New LWP 1402807]
      [New LWP 1402830]
      [New LWP 1402832]
      [New LWP 1402902]
      [New LWP 1402903]
      [New LWP 1402917]
      [New LWP 1402935]
      [New LWP 1402949]
      [New LWP 1402953]
      [New LWP 1402954]
      [New LWP 1402963]
      [New LWP 1402965]
      [New LWP 1402982]
      [New LWP 1402983]
      [New LWP 1402985]
      [New LWP 1402994]
      [New LWP 1402998]
      [New LWP 1403004]
      [New LWP 1403018]
      [New LWP 1403019]
      [New LWP 1403035]
      [New LWP 1403036]
      [New LWP 1403037]
      [New LWP 1403057]
      [New LWP 1403071]
      [New LWP 1403072]
      [New LWP 1403073]
      [New LWP 1403074]
      [New LWP 1403078]
      [New LWP 1403097]
      [New LWP 1403098]
      [New LWP 1403099]
      [New LWP 1403118]
      [New LWP 1403119]
      [New LWP 1403123]
      [New LWP 1403124]
      [New LWP 1403125]
      [New LWP 1403167]
      [New LWP 1403168]
      [New LWP 1403169]
      [New LWP 1403170]
      [New LWP 1403171]
      [New LWP 1403172]
      [New LWP 1403182]
      [New LWP 1403183]
      [New LWP 1403185]
      [New LWP 1403241]
      [New LWP 1403242]
      [New LWP 1403243]
      [New LWP 1403245]
      [New LWP 1403247]
      [New LWP 1403248]
      [New LWP 1403249]
      [New LWP 1403258]
      [New LWP 1403259]
      [New LWP 1403260]
      [New LWP 1403261]
      [New LWP 1403266]
      [New LWP 1403267]
      [New LWP 1403268]
      [New LWP 1403269]
      [New LWP 1403270]
      [New LWP 1403271]
      [New LWP 1403272]
      [New LWP 1403273]
      [New LWP 1403274]
      [New LWP 1403275]
      [New LWP 1403277]
      [New LWP 1403279]
      [New LWP 1403280]
      [New LWP 1403282]
      [New LWP 1403283]
      [New LWP 1403284]
      [New LWP 1403285]
      [New LWP 1403288]
      [New LWP 1403290]
      [New LWP 1403291]
      [New LWP 1403297]
      [New LWP 1403298]
      [New LWP 1403299]
      [New LWP 1403301]
      [New LWP 1403303]
      [New LWP 1403304]
      [New LWP 1403305]
      [New LWP 1403307]
      [New LWP 1403311]
      [New LWP 1403313]
      [New LWP 1403314]
      [New LWP 1403316]
      [New LWP 1403318]
      [New LWP 1403320]
      [New LWP 1403324]
      [New LWP 1403325]
      [New LWP 1403326]
      [New LWP 1403327]
      [New LWP 1403328]
      [New LWP 1403329]
      [New LWP 1403330]
      [New LWP 1403333]
      [New LWP 1403334]
      [New LWP 1403336]
      [New LWP 1403339]
      [New LWP 1403341]
      [New LWP 1403345]
      [New LWP 1403346]
      [New LWP 1403347]
      [New LWP 1403349]
      [New LWP 1403350]
      [New LWP 1403352]
      [New LWP 1403355]
      [New LWP 1403356]
      [New LWP 1403357]
      [New LWP 1403359]
      [New LWP 1403362]
      [New LWP 1403363]
      [New LWP 1403365]
      [New LWP 1403366]
      [New LWP 1403367]
      [New LWP 1403368]
      [New LWP 1403369]
      [New LWP 1403373]
      [New LWP 1403375]
      [New LWP 1403380]
      [New LWP 1403381]
      [New LWP 1403383]
      [New LWP 1403384]
      [New LWP 1403385]
      [New LWP 1403387]
      [New LWP 1403388]
      [New LWP 1403389]
      [New LWP 1403398]
      [New LWP 1403400]
      [New LWP 1403422]
      [New LWP 1403435]
      [New LWP 1403515]
      [New LWP 1403554]
      [New LWP 1403592]
      [New LWP 1403721]
      [New LWP 1403730]
      [New LWP 1403750]
      [New LWP 1403785]
      [New LWP 1403840]
      [New LWP 1403843]
      [New LWP 1403869]
      [New LWP 1404226]
      [New LWP 1404615]
      [New LWP 1404754]
      [New LWP 1404763]
      [New LWP 1404819]
      [New LWP 1404821]
      [New LWP 1404829]
      [New LWP 1404852]
      [New LWP 1404913]
      [New LWP 1404918]
      [New LWP 1404921]
      [New LWP 1404946]
      [New LWP 1404963]
      [New LWP 1404965]
      [New LWP 1404966]
      [New LWP 1404976]
      [New LWP 1404984]
      [New LWP 1404985]
      [New LWP 1404987]
      [New LWP 1405028]
      [New LWP 1476733]
      [New LWP 1516656]
      [New LWP 1527625]
      [New LWP 1532592]
      [New LWP 1532599]
      [New LWP 1532680]
      [New LWP 1532682]
      [New LWP 1532683]
      [New LWP 1532684]
      [New LWP 1532685]
      [New LWP 1532686]
      [New LWP 1532687]
      [New LWP 1532692]
      [New LWP 1532736]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      --Type <RET> for more, q to quit, c to continue without paging--c
      Core was generated by `/opt/wmf-mariadb106/bin/mysqld'.
      #0  0x00007fac896ec3ff in poll () from /lib/x86_64-linux-gnu/libc.so.6
      [Current thread is 1 (Thread 0x7fac893631c0 (LWP 3344701))]
      (gdb) thread 11
      [Switching to thread 11 (Thread 0x7fac808e3700 (LWP 3344713))]
      #0  0x00007fac896f19b9 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
      (gdb) frame 6
      #6  btr_cur_search_to_nth_level_func (index=index@entry=0x7f4c5406d500, level=level@entry=0, tuple=tuple@entry=0x7f49a9189ab0, mode=mode@entry=PAGE_CUR_LE, latch_mode=<optimized out>, latch_mode@entry=33, cursor=cursor@entry=0x7fac808e05f0, ahi_latch=<optimized out>,
          mtr=<optimized out>, autoinc=<optimized out>) at /root/mariadb-10.6.8/storage/innobase/btr/btr0cur.cc:2229
      2229	/root/mariadb-10.6.8/storage/innobase/btr/btr0cur.cc: No such file or directory.
      (gdb) p *index
      $1 = {static MAX_N_FIELDS = 1023, id = 6376, heap = 0x7f4c5405a008, name = {m_name = 0x7f4c5406d690 "PRIMARY"}, table = 0x7f4c54059c90, page = 3, merge_threshold = 50, type = 3, trx_id_offset = 0, n_user_defined_cols = 2, nulls_equal = 0, n_uniq = 2, n_def = 5,
        n_fields = 5, n_nullable = 1, n_core_fields = 5, n_core_null_bytes = 1, static NO_CORE_NULL_BYTES = 255, static DICT_INDEXES_ID = 3, cached = 1, to_be_dropped = 0, online_status = 0, uncommitted = 0, fields = 0x7f4c5406d730, parser = 0x0, new_vcol_info = 0x0,
        indexes = {prev = 0x0, next = 0x7f4c5406dcc0}, search_info = 0x7f4c5406d828, online_log = 0x0, stat_n_diff_key_vals = 0x7f4c5406d7f8, stat_n_sample_sizes = 0x7f4c5406d808, stat_n_non_null_key_vals = 0x7f4c5406d818, stat_index_size = 1320173,
        stat_n_leaf_pages = 1151220, stats_error_printed = false, stat_defrag_modified_counter = 8582, stat_defrag_n_pages_freed = 0, stat_defrag_n_page_split = 8582, stat_defrag_data_size_sample = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, stat_defrag_sample_next_slot = 0, rtr_ssn = {
          m = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 0}, <No data fields>}}, rtr_track = 0x0, trx_id = 0, zip_pad = {mutex = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
                  __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, <No data fields>}, pad = {m = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 1408}, <No data fields>}},
          success = 125, failure = 0, n_rounds = 0}, lock = {lock = {pfs_psi = 0x7fac835f9b40, lock = {writer = {lock = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 2147483649}, <No data fields>}, static HOLDER = 2147483648},
              readers = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 213}, <No data fields>}, static WRITER = 2147483648}}, recursive = 65536, writer = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8,
              _M_i = 140378867906304}, <No data fields>}, static FOR_IO = <optimized out>, static RECURSIVE_X = 1, static RECURSIVE_U = 65536, static RECURSIVE_MAX = <optimized out>}}
      (gdb)
       
      (gdb) thread 11
      [Switching to thread 11 (Thread 0x7fac808e3700 (LWP 3344713))]
      #0  0x00007fac896f19b9 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
      (gdb) frame 6
      #6  btr_cur_search_to_nth_level_func (index=index@entry=0x7f4c5406d500, level=level@entry=0, tuple=tuple@entry=0x7f49a9189ab0, mode=mode@entry=PAGE_CUR_LE, latch_mode=<optimized out>, latch_mode@entry=33, cursor=cursor@entry=0x7fac808e05f0, ahi_latch=<optimized out>,
          mtr=<optimized out>, autoinc=<optimized out>) at /root/mariadb-10.6.8/storage/innobase/btr/btr0cur.cc:2229
      2229	/root/mariadb-10.6.8/storage/innobase/btr/btr0cur.cc: No such file or directory.
      (gdb) p index->table->space->chain.start->name
      $1 = 0x564d2408d618 "./enwiki/user_properties.ibd"
       
       
      (gdb) thread 11
      [Switching to thread 11 (Thread 0x7fac808e3700 (LWP 3344713))]
      #0  0x00007fac896f19b9 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
      (gdb) frame 6
      #6  btr_cur_search_to_nth_level_func (index=index@entry=0x7f4c5406d500, level=level@entry=0, tuple=tuple@entry=0x7f49a9189ab0, mode=mode@entry=PAGE_CUR_LE, latch_mode=<optimized out>, latch_mode@entry=33, cursor=cursor@entry=0x7fac808e05f0, ahi_latch=<optimized out>,
          mtr=<optimized out>, autoinc=<optimized out>) at /root/mariadb-10.6.8/storage/innobase/btr/btr0cur.cc:2229
      2229	/root/mariadb-10.6.8/storage/innobase/btr/btr0cur.cc: No such file or directory.
      (gdb) p index->table->space->chain.start->handle
      $1 = {m_file = 133, m_psi = 0x7fac8302afc0}
      
      

      Attachments

        1. mariadb_full_bt_all_threads.txt
          8.01 MB
        2. io_slots.txt
          124 kB
        3. gdb.txt
          126 kB
        4. gdb_db1132_24th_aug.txt
          5.32 MB

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              marostegui Manuel Arostegui
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.