[MDEV-8335] Thread stuck at `show create table` and using 1 core 100% CPU: Created: 2015-06-18  Updated: 2015-08-24  Resolved: 2015-08-17

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 10.0.19
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Kenny Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

Debian 7.2

Attachments: File gdb.bt     File gdb.bt.full     Text File show-global-variables.txt    

 Description   

Thread stuck at `show create table` and using 1 core 100% CPU:

MariaDB [(none)]> show processlist;
 
+--------+------------+-----------+--------+---------+-------+----------------+---------------------------------+----------+
 
| Id     | User       | Host      | db     | Command | Time  | State          | Info                            | Progress |
 
+--------+------------+-----------+--------+---------+-------+----------------+---------------------------------+----------+
 
| 1572   | monitoring | localhost | NULL   | Sleep   | 2     |                | NULL                            | 0.000    |
 
| 64160  | monitoring | localhost | db_ugc | Killed  | 61061 | Opening tables | show create table ugc_state_que | 0.000    |
 
| 117909 | monitoring | localhost | NULL   | Sleep   | 32    |                | NULL                            | 0.000    |
 
| 119051 | monitoring | localhost | NULL   | Query   | 0     | init           | show processlist                | 0.000    |
 
+--------+------------+-----------+--------+---------+-------+----------------+---------------------------------+----------+
 
4 rows in set (0.00 sec)

It's a rebuild of the packages with debugging symbols (so I can give you guys some info).

MariaDB [performance_schema]> select version();
 
+------------------------------+
 
| version()                    |
 
+------------------------------+
 
| 10.0.19-MariaDB-2~wheezy-log |
 
+------------------------------+
 
1 row in set (0.00 sec)

`mpstat -P ALL` shows 100% cpu on one thread

09:06:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
 
09:06:53     all    4.50    0.00    0.17    6.38    0.00    0.04    0.00    0.00   88.91
 
09:06:53       0    2.02    0.00    1.01   95.96    0.00    1.01    0.00    0.00    0.00
 
09:06:53       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53       2    0.00    0.00    0.00    4.04    0.00    0.00    0.00    0.00   95.96
 
09:06:53       3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53       4    6.00    0.00    3.00   55.00    0.00    0.00    0.00    0.00   36.00
 
09:06:53       5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53       6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53       7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53       8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53       9    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      10    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      13  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
 
09:06:53      14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      15    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      17    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      19    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      20    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      21    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 
09:06:53      23    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 

 50.22%  libpthread-2.13.so  [.] __pthread_mutex_unlock_usercnt
 
 37.76%  libpthread-2.13.so  [.] pthread_mutex_lock
 
  5.03%  ld-2.13.so          [.] 0x10c92
 
  1.50%  libpthread-2.13.so  [.] pthread_mutex_unlock
 
  1.06%  libz.so.1.2.7       [.] adler32
 
  0.44%  mysqld              [.] buf_calc_page_new_checksum(unsigned char const*)
 
  0.30%  [kernel]            [k] do_raw_spin_lock
 
  0.21%  mysqld              [.] os_aio_linux_handle(unsigned long, fil_node_t**, void**, unsigned long*, unsigned long*)
 
  0.18%  mysqld              [.] os_aio_func(unsigned long, unsigned long, char const*, int, void*, unsigned long, unsigned long, fil_node_t*, void*, unsigned long, trx_t*)
 
  0.15%  mysqld              [.] buf_do_flush_list_batch(buf_pool_t*, unsigned long, unsigned long)
 
  0.14%  mysqld              [.] buf_flush_try_neighbors(unsigned long, unsigned long, buf_flush_t, unsigned long, unsigned long)
 
  0.11%  libc-2.13.so        [.] 0x83fc2
 
  0.08%  [kernel]            [k] __blockdev_direct_IO
 
  0.08%  mysqld              [.] fil_page_get_type(unsigned char const*)
 
  0.08%  [kernel]            [k] native_write_msr_safe
 
  0.08%  [kernel]            [k] idle_cpu
 
  0.07%  [kernel]            [k] ____cache_alloc
 
  0.07%  mysqld              [.] buf_flush_remove(buf_page_t*)
 
  0.06%  [kernel]            [k] arch_local_save_flags
 
  0.05%  mysqld              [.] Protocol::send_result_set_row(List<Item>*)
 
  0.05%  mysqld              [.] sync_array_print_long_waits(unsigned long*, void const**)
 
  0.05%  [megaraid_sas]      [k] megasas_build_ldio_fusion
 
  0.05%  [megaraid_sas]      [k] megasas_build_and_issue_cmd_fusion
 
  0.05%  [kernel]            [k] gup_pte_range
 
  0.05%  [xfs]               [k] xfs_bmbt_get_startoff
 
  0.05%  [kernel]            [k] update_curr
 
  0.05%  mysqld              [.] _fil_io(unsigned long, bool, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, void*, void*, trx_t*)
 
  0.04%  [kernel]            [k] find_busiest_group
 
  0.03%  [kernel]            [k] do_io_submit
 
  0.03%  [kernel]            [k] kmem_cache_alloc
 
  0.03%  [kernel]            [k] arch_local_irq_restore

`pt-pmp`

root@dbrfat3:/home/kenny# pt-pmp 
Thu Jun 18 09:08:44 UTC 2015
 
     78 pthread_cond_wait,toku_cond_wait(toku_pthread.h:309),kwait(toku_pthread.h:309),work_on_kibbutz(toku_pthread.h:309),start_thread(libpthread.so.0),clone(libc.so.6),??
 
     10 libaio::??(libaio.so.1),os_aio_linux_collect(os0file.cc:5114),os_aio_linux_handle(os0file.cc:5114),fil_aio_wait(fil0fil.cc:5686),io_handler_thread(srv0start.cc:524),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      5 pthread_cond_wait,inline_mysql_cond_wait(mysql_thread.h:1165),inline_mysql_cond_wait(mysql_thread.h:1165),cache_thread(mysql_thread.h:1165),one_thread_per_connection_end(mysql_thread.h:1165),do_handle_one_connection(sql_connect.cc:1386),handle_one_connection(sql_connect.cc:1289),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      2 pthread_cond_timedwait,toku_cond_timedwait(toku_pthread.h:325),minicron_do(void*)(toku_pthread.h:325),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      2 poll(libc.so.6),vio_io_wait(viosocket.c:972),vio_socket_io_wait(viosocket.c:108),vio_read(viosocket.c:184),my_real_read(net_serv.cc:872),my_net_read_packet(net_serv.cc:1088),do_command(sql_parse.cc:931),do_handle_one_connection(sql_connect.cc:1375),handle_one_connection(sql_connect.cc:1289),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 select(libc.so.6),os_thread_sleep(os0thread.cc:285),srv_master_sleep(srv0srv.cc:2970),srv_master_thread(srv0srv.cc:2970),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 select(libc.so.6),os_thread_sleep(os0thread.cc:285),page_cleaner_sleep_if_needed(buf0flu.cc:2583),buf_flush_lru_manager_thread(buf0flu.cc:2583),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_mutex_lock(libpthread.so.0),ld-linux-x86-64::??(ld-linux-x86-64.so.2),__tls_get_addr(ld-linux-x86-64.so.2),get_or_alloc_thread_local_counter(partitioned_counter.cc:453),increment_partitioned_counter(partitioned_counter.cc:453),toku_txn_create_txn[clone.21261](txn.cc:376),toku_txn_begin_with_xid(txn.cc:229),toku_txn_begin[clone.part.2](ydb_txn.cc:617),toku_txn_begin(ydb_txn.cc:617),txn_begin(hatoku_defines.h:403),tokudb_discover3(hatoku_defines.h:403),tokudb_discover_table(hatoku_hton.cc:1036),discover_handlerton(handler.cc:4810),plugin_foreach_with_mask(sql_plugin.cc:2340),ha_discover_table(handler.cc:4849),open_table_def(table.cc:583),tdc_acquire_share(table_cache.cc:741),open_table(sql_base.cc:2371),open_and_process_table(sql_base.cc:3988),open_tables(sql_base.cc:3988),open_tables(sql_base.h:470),mysqld_show_create(sql_base.h:470),mysql_execute_command(sql_parse.cc:3271),mysql_parse(sql_parse.cc:6529),dispatch_command(sql_parse.cc:1780),do_handle_one_connection(sql_connect.cc:1375),handle_one_connection(sql_connect.cc:1289),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_wait,toku_cond_wait(toku_pthread.h:309),minicron_do(void*)(toku_pthread.h:309),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_wait,os_cond_wait(os0sync.cc:214),os_event_wait_low(os0sync.cc:214),srv_purge_coordinator_suspend(srv0srv.cc:3329),srv_purge_coordinator_thread(srv0srv.cc:3329),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_wait,os_cond_wait(os0sync.cc:214),os_event_wait_low(os0sync.cc:214),buf_dump_thread(buf0dump.cc:594),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_wait,inline_mysql_cond_wait(mysql_thread.h:1165),inline_mysql_cond_wait(mysql_thread.h:1165),binlog_background_thread(mysql_thread.h:1165),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,toku_cond_timedwait(toku_pthread.h:325),evictor::run_eviction_thread(toku_pthread.h:325),[clone(toku_pthread.h:325),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),srv_monitor_thread(srv0srv.cc:1978),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),srv_error_monitor_thread(srv0srv.cc:2242),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),lock_wait_timeout_thread(lock0wait.cc:499),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),ib_wqueue_timedwait(ut0wqueue.cc:154),fts_optimize_thread(fts0opt.cc:3001),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,os_cond_wait_timed(os0sync.cc:177),os_event_wait_time_low(os0sync.cc:177),dict_stats_thread(dict0stats_bg.cc:349),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 pthread_cond_timedwait,inline_mysql_cond_timedwait(mysql_thread.h:1202),my_service_thread_sleep(mysql_thread.h:1202),ma_checkpoint_background(ma_checkpoint.c:709),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 poll(libc.so.6),handle_connections_sockets(mysqld.cc:6130),mysqld_main(mysqld.cc:5562),__libc_start_main(libc.so.6),_start
 
      1 nanosleep(libc.so.6),usleep(libc.so.6),[clone(libc.so.6),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 io_submit(libaio.so.1),os_aio_linux_dispatch(os0file.cc:4679),os_aio_func(os0file.cc:4679),pfs_os_aio_func(os0file.ic:230),_fil_io(os0file.ic:230),buf_flush_write_block_low(buf0flu.cc:959),buf_flush_page(buf0flu.cc:959),buf_flush_try_neighbors(buf0flu.cc:1302),buf_flush_page_and_try_neighbors(buf0flu.cc:1390),buf_do_flush_list_batch(buf0flu.cc:1390),buf_flush_batch(buf0flu.cc:1796),buf_flush_list(buf0flu.cc:1796),page_cleaner_do_flush_batch(buf0flu.cc:2353),buf_flush_page_cleaner_thread(buf0flu.cc:2353),start_thread(libpthread.so.0),clone(libc.so.6),??
 
      1 do_sigwait(libpthread.so.0),sigwait(libpthread.so.0),signal_hand(mysqld.cc:3345),pfs_spawn_thread(pfs.cc:1860),start_thread(libpthread.so.0),clone(libc.so.6),??
 
 



 Comments   
Comment by Sergei Golubchik [ 2015-06-18 ]

IRC log:

<serg> gryp: is it mariadb specific? I'd guess you've tried that on tokudb+percona server too
<gryp> serg: nope, only something that happened random to me when working on Mariadb.
<gryp> serg: also, no tables are TokuDB, I am using read free replication though.
<serg> hmm, I'm confused
<serg> "nope" means you didn't try on percona server or it means it's not mariadb specific?
<gryp> Did not try on percona server.
<serg> "no tables are TokuDB" means exactly that - all tables are NOT tokudb?
<serg> tokudb seems to be pretty active in the traces :-[]
<gryp> everything is InnoDB <<
<gryp> 16 parallel slave threads, but I stopped replication.
<serg> "read free replication" - I thought we didn't merge it yet
<gryp> lol, okay.
<gryp> The settings are there.
<serg> and these settings are - read-only, binlog-format=row, and few tokudb-* variables?
<gryp> yes
<gryp> tokudb_rpl_lookup_rows=OFF
<gryp> tokudb_rpl_unique_checks=OFF
<gryp> tokudb_rpl_lookup_rows_delay=0
<gryp> tokudb_rpl_unique_checks_delay=0
<gryp> read_only=1
<serg> ok, thanks. if you don't mind I'll paste this log to MDEV-8335
<gryp> serg: sure, go ahead.

Comment by Torbjörn Norinder [ 2015-06-18 ]

Stack from the spinning thread:

#0  0x00007f39488f462e in __pthread_mutex_unlock_full (mutex=0x0, decr=0) at pthread_mutex_unlock.c:198
 
#1  0x0000000000000005 in ?? ()
 
#2  0x00007f3948b16c92 in ?? () from /lib64/ld-linux-x86-64.so.2
 
#3  0x00007f3948b17110 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2
 
#4  0x00007edf84ce5ef2 in get_or_alloc_thread_local_counter (pc=0x7edfa0413b40) at /home/vagrant/mariadb-10.0.19/storage/tokudb/ft-index/util/partitioned_counter.cc:453
 
#5  increment_partitioned_counter (pc=0x7edfa0413b40, amount=amount@entry=1) at /home/vagrant/mariadb-10.0.19/storage/tokudb/ft-index/util/partitioned_counter.cc:413
 
#6  0x00007edf84cfb5fc in toku_txn_create_txn(tokutxn**, tokutxn*, tokulogger*, __TXN_SNAPSHOT_TYPE, __toku_db_txn*, bool, bool) [clone .21261] (tokutxn=0x7f3948c5d620, parent_tokutxn=<optimized out>, logger=<optimized out>, 
    snapshot_type=<optimized out>, container_db_txn=<optimized out>, for_recovery=<optimized out>, read_only=false) at /home/vagrant/mariadb-10.0.19/storage/tokudb/ft-index/ft/txn/txn.cc:376
 
#7  0x00007edf84cfcfb1 in toku_txn_begin_with_xid (parent=0x0, txnp=0x7edf8fc10e28, logger=0x7ee264f18540, xid=..., snapshot_type=TXN_SNAPSHOT_NONE, container_db_txn=container_db_txn@entry=0x7edf8fc10da0, 
    for_recovery=for_recovery@entry=false, read_only=false) at /home/vagrant/mariadb-10.0.19/storage/tokudb/ft-index/ft/txn/txn.cc:229
 
#8  0x00007edf84cfd512 in toku_txn_begin(__toku_db_env*, __toku_db_txn*, __toku_db_txn**, unsigned int) [clone .part.2] (flags=<optimized out>, txn=0x7f3948c5d740, stxn=0x0, env=0x7edfa0474580)
 
    at /home/vagrant/mariadb-10.0.19/storage/tokudb/ft-index/src/ydb_txn.cc:617
 
#9  toku_txn_begin (env=0x7edfa0474580, stxn=0x0, txn=0x7f3948c5d740, flags=<optimized out>) at /home/vagrant/mariadb-10.0.19/storage/tokudb/ft-index/src/ydb_txn.cc:468
 
#10 0x00007edf84c9cc32 in txn_begin (thd=0x7edf8e698008, flags=0, txn=0x7f3948c5d740, parent=0x0, env=<optimized out>) at /home/vagrant/mariadb-10.0.19/storage/tokudb/hatoku_defines.h:403
 
#11 tokudb_discover3 (thd=thd@entry=0x7edf8e698008, db=<optimized out>, name=<optimized out>, path=0x7edf8fc4dfa8 "./truecaller_ugc/ugc_state_que", frmblob=frmblob@entry=0x7f3948c5d7e0, frmlen=frmlen@entry=0x7f3948c5d7f0, 
    hton=<error reading variable: Unhandled dwarf expression opcode 0xfa>) at /home/vagrant/mariadb-10.0.19/storage/tokudb/hatoku_hton.cc:1082
 
#12 0x00007edf84c9cf98 in tokudb_discover_table (hton=<optimized out>, thd=0x7edf8e698008, ts=0x7edf8fc4db20) at /home/vagrant/mariadb-10.0.19/storage/tokudb/hatoku_hton.cc:1036
 
#13 0x00007f39492ad3ea in discover_handlerton (thd=thd@entry=0x7edf8e698008, plugin=0x7ee10e399968, arg=arg@entry=0x7edf8fc4db20) at /home/vagrant/mariadb-10.0.19/sql/handler.cc:4810
 
#14 0x00007f394915abf0 in plugin_foreach_with_mask (thd=0x7edf8e698008, func=0x7f39492ad3a0 <discover_handlerton(THD*, plugin_ref, void*)>, type=<optimized out>, state_mask=4294967287, arg=0x7edf8fc4db20)
 
    at /home/vagrant/mariadb-10.0.19/sql/sql_plugin.cc:2340
 
#15 0x00007f39492b5209 in ha_discover_table (thd=<optimized out>, share=0x7edf8fc4db20) at /home/vagrant/mariadb-10.0.19/sql/handler.cc:4849
 
#16 0x00007f39491ee6ee in open_table_def (thd=0x7edf8e698008, share=0x7edf8fc4db20, flags=<optimized out>) at /home/vagrant/mariadb-10.0.19/sql/table.cc:583
 
#17 0x00007f394925af38 in tdc_acquire_share (thd=0x7edf8e698008, db=0x7edf8fc1f700 "truecaller_ugc", table_name=0x7edf8fc1f0d8 "ugc_state_que", key=<optimized out>, key_length=<optimized out>, hash_value=<optimized out>, flags=3, 
    out_table=0x7f3948c5dd50) at /home/vagrant/mariadb-10.0.19/sql/table_cache.cc:741
 
#18 0x00007f39491102f4 in open_table (thd=thd@entry=0x7edf8e698008, table_list=table_list@entry=0x7edf8fc1f120, mem_root=mem_root@entry=0x7f3948c5e3f0, ot_ctx=ot_ctx@entry=0x7f3948c5e3b0)
 
    at /home/vagrant/mariadb-10.0.19/sql/sql_base.cc:2371
 
#19 0x00007f3949111dfd in open_and_process_table (new_frm_mem=0x7f3948c5e3f0, ot_ctx=0x7f3948c5e3b0, has_prelocking_list=false, prelocking_strategy=0x7f3948c5e490, flags=1024, counter=0x7f3948c5e480, tables=0x7edf8fc1f120, 
    lex=0x7edf8e69b760, thd=0x7edf8e698008) at /home/vagrant/mariadb-10.0.19/sql/sql_base.cc:3988
 
#20 open_tables (thd=0x7edf8e698008, start=0x7f3948c5e478, counter=0x7f3948c5e480, flags=1024, prelocking_strategy=0x7f3948c5e490) at /home/vagrant/mariadb-10.0.19/sql/sql_base.cc:4522
 
#21 0x00007f39491b923c in open_tables (flags=1024, counter=0x7f3948c5e480, tables=0x7f3948c5e478, thd=0x7edf8e698008) at /home/vagrant/mariadb-10.0.19/sql/sql_base.h:470
 
#22 mysqld_show_create (thd=0x7edf8e698008, table_list=0x7edf8fc1f120) at /home/vagrant/mariadb-10.0.19/sql/sql_show.cc:1046
 
#23 0x00007f394914e17a in mysql_execute_command (thd=thd@entry=0x7edf8e698008) at /home/vagrant/mariadb-10.0.19/sql/sql_parse.cc:3271
 
#24 0x00007f3949152943 in mysql_parse (thd=0x7edf8e698008, parser_state=0x7f3948c60570, length=<optimized out>, rawbuf=<optimized out>) at /home/vagrant/mariadb-10.0.19/sql/sql_parse.cc:6529
 
#25 0x00007f3949154ea4 in dispatch_command (command=COM_QUERY, thd=0x7edf8e698008, packet=<optimized out>, packet_length=1240697280) at /home/vagrant/mariadb-10.0.19/sql/sql_parse.cc:1780
 
#26 0x00007f3949216b4b in do_handle_one_connection (thd_arg=thd_arg@entry=0x7edf8e541008) at /home/vagrant/mariadb-10.0.19/sql/sql_connect.cc:1375
 
#27 0x00007f3949216c27 in handle_one_connection (arg=arg@entry=0x7edf8e541008) at /home/vagrant/mariadb-10.0.19/sql/sql_connect.cc:1289
 
#28 0x00007f39495f3e1d in pfs_spawn_thread (arg=0x7edf8e4c7c08) at /home/vagrant/mariadb-10.0.19/storage/perfschema/pfs.cc:1860
 
#29 0x00007f39488f0b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
 
#30 0x00007f3947012a7d in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 
#31 0x0000000000000000 in ?? ()

Comment by Elena Stepanova [ 2015-06-18 ]

Hi,

Could you please paste the output of

command line

 
ls -l /data/db/mysql/db_ugc/*ugc_state_que*
 
ls -l /data/db/mysql/*ugc_state_que*

MySQL client

 
SHOW ENGINE TOKUDB STATUS;
 
--Hopefully this one won't hang:
 
SELECT ENGINE, COUNT(*) FROM INFORMATION_SCHEMA.TABLES GROUP BY ENGINE;

Please also attach your error log (or, if there is anything confidential in there, you can upload it to ftp.askmonty.org/private).

Thanks.

Comment by Kenny [ 2015-06-18 ]

I am sorry, I cannot share the full table list as it's contains confidential customer information, but this should get you going....

This is the table:

 12K -rw-r-----  1 mysql mysql 8.7K Sep  4  2013 ugc_state_que.frm
 
2.7G -rw-r-----  1 mysql mysql 2.7G Jun 18 12:11 ugc_state_que.ibd
 

 
 
MariaDB [(none)]> SELECT ENGINE, COUNT(*) FROM INFORMATION_SCHEMA.TABLES GROUP BY ENGINE;
 
+--------------------+----------+
 
| ENGINE             | COUNT(*) |
 
+--------------------+----------+
 
| NULL               |       81 |
 
| Aria               |       10 |
 
| CSV                |        2 |
 
| InnoDB             |      297 |
 
| MEMORY             |       66 |
 
| MyISAM             |       28 |
 
| PERFORMANCE_SCHEMA |       52 |
 
+--------------------+----------+
 
7 rows in set (0.05 sec)

Comment by Elena Stepanova [ 2015-06-18 ]

So, there is certainly nothing matching the table name in /data/db/mysql (not in the schema subdir)?
It's important because maybe, even though you don't have TokuDB tables now, there used to be one, and there are some remains of it that get TokuDB and the server confused.

What are these 81 tables with NULL instead of the engine, can you check it out?

Is TokuDB engine currently enabled, even though it's not used? If it is, it might still be useful to see SHOW ENGINE TOKUDB STATUS.

Comment by Kenny [ 2015-06-18 ]

The tables with `null` is coming from SYS schema, they are views:

MariaDB [(none)]> select engine, table_schema, count(*) from information_schema.TABLES where engine is null group by engine, table_schema;
 
+--------+--------------+----------+
 
| engine | table_schema | count(*) |
 
+--------+--------------+----------+
 
| NULL   | sys          |       81 |
 
+--------+--------------+----------+
 
1 row in set (0.04 sec)

These are the only tokudb related files:

   0 -rw-------  1 mysql mysql    0 Jun 17 13:32 __tokudb_lock_dont_delete_me_data
 
   0 -rw-------  1 mysql mysql    0 Jun 17 13:32 __tokudb_lock_dont_delete_me_environment
 
   0 -rw-------  1 mysql mysql    0 Jun 17 13:32 __tokudb_lock_dont_delete_me_logs
 
   0 -rw-------  1 mysql mysql    0 Jun 17 13:32 __tokudb_lock_dont_delete_me_recovery
 
   0 -rw-------  1 mysql mysql    0 Jun 17 13:32 __tokudb_lock_dont_delete_me_temp

I can give you `SHOW ENGINE TOKUDB STATUS`, however, that will not be accurate as we restarted the server since.

Comment by Kenny [ 2015-06-18 ]

I am not sure anymore, but it is possible that I have created a test tokudb table (empty) to demonstrate something.
Again, I am not sure about this, but I am not ruling it out.

Kenny

Comment by Elena Stepanova [ 2015-06-18 ]

How about server error log, can you provide it?

Comment by Kenny [ 2015-06-18 ]

Hi Elena, There's nothing in the error log during that time, just regular starting of MySQL

Comment by Elena Stepanova [ 2015-07-17 ]

gryp,
Did you ever experience the problem again, or was it a one-time thing?

Comment by Elena Stepanova [ 2015-08-17 ]

Closing for now as "Incompelete". If you have any new information, please comment to re-open the issue.

Comment by Kenny [ 2015-08-24 ]

Hi Elena, Yes it was an one time thing. I haven't worked with this customer for a couple of months, so I'm not aware if they had it again. I might see them again in a few months. Will report back if necessary.

Tnx

Kenny

Generated at Thu Feb 08 07:26:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.