Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8335

Thread stuck at `show create table` and using 1 core 100% CPU:

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.0.19
    • N/A
    • OTHER
    • Debian 7.2

    Description

      Thread stuck at `show create table` and using 1 core 100% CPU:

      MariaDB [(none)]> show processlist;
      +--------+------------+-----------+--------+---------+-------+----------------+---------------------------------+----------+
      | Id     | User       | Host      | db     | Command | Time  | State          | Info                            | Progress |
      +--------+------------+-----------+--------+---------+-------+----------------+---------------------------------+----------+
      | 1572   | monitoring | localhost | NULL   | Sleep   | 2     |                | NULL                            | 0.000    |
      | 64160  | monitoring | localhost | db_ugc | Killed  | 61061 | Opening tables | show create table ugc_state_que | 0.000    |
      | 117909 | monitoring | localhost | NULL   | Sleep   | 32    |                | NULL                            | 0.000    |
      | 119051 | monitoring | localhost | NULL   | Query   | 0     | init           | show processlist                | 0.000    |
      +--------+------------+-----------+--------+---------+-------+----------------+---------------------------------+----------+
      4 rows in set (0.00 sec)

      It's a rebuild of the packages with debugging symbols (so I can give you guys some info).

      MariaDB [performance_schema]> select version();
      +------------------------------+
      | version()                    |
      +------------------------------+
      | 10.0.19-MariaDB-2~wheezy-log |
      +------------------------------+
      1 row in set (0.00 sec)

      `mpstat -P ALL` shows 100% cpu on one thread

      09:06:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
      09:06:53     all    4.50    0.00    0.17    6.38    0.00    0.04    0.00    0.00   88.91
      09:06:53       0    2.02    0.00    1.01   95.96    0.00    1.01    0.00    0.00    0.00
      09:06:53       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53       2    0.00    0.00    0.00    4.04    0.00    0.00    0.00    0.00   95.96
      09:06:53       3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53       4    6.00    0.00    3.00   55.00    0.00    0.00    0.00    0.00   36.00
      09:06:53       5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53       6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53       7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53       8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53       9    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      10    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      13  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
      09:06:53      14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      15    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      17    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      19    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      20    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      21    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
      09:06:53      23    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

       50.22%  libpthread-2.13.so  [.] __pthread_mutex_unlock_usercnt
       37.76%  libpthread-2.13.so  [.] pthread_mutex_lock
        5.03%  ld-2.13.so          [.] 0x10c92
        1.50%  libpthread-2.13.so  [.] pthread_mutex_unlock
        1.06%  libz.so.1.2.7       [.] adler32
        0.44%  mysqld              [.] buf_calc_page_new_checksum(unsigned char const*)
        0.30%  [kernel]            [k] do_raw_spin_lock
        0.21%  mysqld              [.] os_aio_linux_handle(unsigned long, fil_node_t**, void**, unsigned long*, unsigned long*)
        0.18%  mysqld              [.] os_aio_func(unsigned long, unsigned long, char const*, int, void*, unsigned long, unsigned long, fil_node_t*, void*, unsigned long, trx_t*)
        0.15%  mysqld              [.] buf_do_flush_list_batch(buf_pool_t*, unsigned long, unsigned long)
        0.14%  mysqld              [.] buf_flush_try_neighbors(unsigned long, unsigned long, buf_flush_t, unsigned long, unsigned long)
        0.11%  libc-2.13.so        [.] 0x83fc2
        0.08%  [kernel]            [k] __blockdev_direct_IO
        0.08%  mysqld              [.] fil_page_get_type(unsigned char const*)
        0.08%  [kernel]            [k] native_write_msr_safe
        0.08%  [kernel]            [k] idle_cpu
        0.07%  [kernel]            [k] ____cache_alloc
        0.07%  mysqld              [.] buf_flush_remove(buf_page_t*)
        0.06%  [kernel]            [k] arch_local_save_flags
        0.05%  mysqld              [.] Protocol::send_result_set_row(List<Item>*)
        0.05%  mysqld              [.] sync_array_print_long_waits(unsigned long*, void const**)
        0.05%  [megaraid_sas]      [k] megasas_build_ldio_fusion
        0.05%  [megaraid_sas]      [k] megasas_build_and_issue_cmd_fusion
        0.05%  [kernel]            [k] gup_pte_range
        0.05%  [xfs]               [k] xfs_bmbt_get_startoff
        0.05%  [kernel]            [k] update_curr
        0.05%  mysqld              [.] _fil_io(unsigned long, bool, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, void*, void*, trx_t*)
        0.04%  [kernel]            [k] find_busiest_group
        0.03%  [kernel]            [k] do_io_submit
        0.03%  [kernel]            [k] kmem_cache_alloc
        0.03%  [kernel]            [k] arch_local_irq_restore

      `pt-pmp`

      root@dbrfat3:/home/kenny# pt-pmp 
      Thu Jun 18 09:08:44 UTC 2015
           78 pthread_cond_wait,toku_cond_wait(toku_pthread.h:309),kwait(toku_pthread.h:309),work_on_kibbutz(toku_pthread.h:309),start_thread(libpthread.so.0),clone(libc.so.6),??
           10 libaio::??(libaio.so.1),os_aio_linux_collect(os0file.cc:5114),os_aio_linux_handle(os0file.cc:5114),fil_aio_wait(fil0fil.cc:5686),io_handler_thread(srv0start.cc:524),start_thread(libpthread.so.0),clone(libc.so.6),??
            5 pthread_cond_wait,inline_mysql_cond_wait(mysql_thread.h:1165),inline_mysql_cond_wait(mysql_thread.h:1165),cache_thread(mysql_thread.h:1165),one_thread_per_connection_end(mysql_thread.h:1165),do_handle_one_connection(sql_connect.cc:1386),handle_one_connection(sql_connect.cc:1289),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
            2 pthread_cond_timedwait,toku_cond_timedwait(toku_pthread.h:325),minicron_do(void*)(toku_pthread.h:325),start_thread(libpthread.so.0),clone(libc.so.6),??
            2 poll(libc.so.6),vio_io_wait(viosocket.c:972),vio_socket_io_wait(viosocket.c:108),vio_read(viosocket.c:184),my_real_read(net_serv.cc:872),my_net_read_packet(net_serv.cc:1088),do_command(sql_parse.cc:931),do_handle_one_connection(sql_connect.cc:1375),handle_one_connection(sql_connect.cc:1289),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 select(libc.so.6),os_thread_sleep(os0thread.cc:285),srv_master_sleep(srv0srv.cc:2970),srv_master_thread(srv0srv.cc:2970),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 select(libc.so.6),os_thread_sleep(os0thread.cc:285),page_cleaner_sleep_if_needed(buf0flu.cc:2583),buf_flush_lru_manager_thread(buf0flu.cc:2583),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_mutex_lock(libpthread.so.0),ld-linux-x86-64::??(ld-linux-x86-64.so.2),__tls_get_addr(ld-linux-x86-64.so.2),get_or_alloc_thread_local_counter(partitioned_counter.cc:453),increment_partitioned_counter(partitioned_counter.cc:453),toku_txn_create_txn[clone.21261](txn.cc:376),toku_txn_begin_with_xid(txn.cc:229),toku_txn_begin[clone.part.2](ydb_txn.cc:617),toku_txn_begin(ydb_txn.cc:617),txn_begin(hatoku_defines.h:403),tokudb_discover3(hatoku_defines.h:403),tokudb_discover_table(hatoku_hton.cc:1036),discover_handlerton(handler.cc:4810),plugin_foreach_with_mask(sql_plugin.cc:2340),ha_discover_table(handler.cc:4849),open_table_def(table.cc:583),tdc_acquire_share(table_cache.cc:741),open_table(sql_base.cc:2371),open_and_process_table(sql_base.cc:3988),open_tables(sql_base.cc:3988),open_tables(sql_base.h:470),mysqld_show_create(sql_base.h:470),mysql_execute_command(sql_parse.cc:3271),mysql_parse(sql_parse.cc:6529),dispatch_command(sql_parse.cc:1780),do_handle_one_connection(sql_connect.cc:1375),handle_one_connection(sql_connect.cc:1289),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_wait,toku_cond_wait(toku_pthread.h:309),minicron_do(void*)(toku_pthread.h:309),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_wait,os_cond_wait(os0sync.cc:214),os_event_wait_low(os0sync.cc:214),srv_purge_coordinator_suspend(srv0srv.cc:3329),srv_purge_coordinator_thread(srv0srv.cc:3329),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_wait,os_cond_wait(os0sync.cc:214),os_event_wait_low(os0sync.cc:214),buf_dump_thread(buf0dump.cc:594),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_wait,inline_mysql_cond_wait(mysql_thread.h:1165),inline_mysql_cond_wait(mysql_thread.h:1165),binlog_background_thread(mysql_thread.h:1165),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,toku_cond_timedwait(toku_pthread.h:325),evictor::run_eviction_thread(toku_pthread.h:325),[clone(toku_pthread.h:325),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),srv_monitor_thread(srv0srv.cc:1978),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),srv_error_monitor_thread(srv0srv.cc:2242),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),lock_wait_timeout_thread(lock0wait.cc:499),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),ib_wqueue_timedwait(ut0wqueue.cc:154),fts_optimize_thread(fts0opt.cc:3001),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),dict_stats_thread(dict0stats_bg.cc:349),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 pthread_cond_timedwait,inline_mysql_cond_timedwait(mysql_thread.h:1202),my_service_thread_sleep(mysql_thread.h:1202),ma_checkpoint_background(ma_checkpoint.c:709),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 poll(libc.so.6),handle_connections_sockets(mysqld.cc:6130),mysqld_main(mysqld.cc:5562),__libc_start_main(libc.so.6),_start
            1 nanosleep(libc.so.6),usleep(libc.so.6),[clone(libc.so.6),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 io_submit(libaio.so.1),os_aio_linux_dispatch(os0file.cc:4679),os_aio_func(os0file.cc:4679),pfs_os_aio_func(os0file.ic:230),_fil_io(os0file.ic:230),buf_flush_write_block_low(buf0flu.cc:959),buf_flush_page(buf0flu.cc:959),buf_flush_try_neighbors(buf0flu.cc:1302),buf_flush_page_and_try_neighbors(buf0flu.cc:1390),buf_do_flush_list_batch(buf0flu.cc:1390),buf_flush_batch(buf0flu.cc:1796),buf_flush_list(buf0flu.cc:1796),page_cleaner_do_flush_batch(buf0flu.cc:2353),buf_flush_page_cleaner_thread(buf0flu.cc:2353),start_thread(libpthread.so.0),clone(libc.so.6),??
            1 do_sigwait(libpthread.so.0),sigwait(libpthread.so.0),signal_hand(mysqld.cc:3345),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
       

      Attachments

        1. gdb.bt
          100 kB
        2. gdb.bt.full
          151 kB
        3. show-global-variables.txt
          360 kB

        Activity

          People

            Unassigned Unassigned
            gryp Kenny
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.