[MDEV-16279] Spider crashes on CHECKSUM TABLE with spider_quick_mode=3 Created: 2018-05-23  Updated: 2020-08-25  Resolved: 2019-02-05

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.3
Fix Version/s: 10.4.3, 10.3.13

Type: Bug Priority: Major
Reporter: Mattias Jonsson Assignee: Kentoku Shiba (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

linux


Issue Links:
Duplicate
is duplicated by MDEV-16521 Server crash with spider_quick_mode =... Closed
Relates
relates to MDEV-16520 Out-Of-Memory running big aggregate q... Closed

 Description   

Using mariadb 10.3.7 (built from fe3bf136b6cf83aac5a6e21d3db1c4e821612017)

setting spider_quick_mode = 3 on the spider head and then running
CHECKSUM TABLE t1 EXTENDED;
crashes the spider node.
Setting spider_quick_mode = 0 does not crash, but is affected by MDEV-16249 with loading all the results into memory.

Stacktrace:
Server version: 10.3.7-MariaDB-log
key_buffer_size=805306368
read_buffer_size=131072
max_used_connections=21
max_threads=3002
thread_count=48
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4311648 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f17a80009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f1854744e80 thread_stack 0x49000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x7f19a3ae60ee]
mysys/stacktrace.c:270(my_print_stacktrace)[0x7f19a3571f07]
sigaction.c:0(__restore_rt)[0x7f19a29bd6d0]
spider/spd_db_mysql.cc:471(spider_db_mysql_row::store_to_tmp_table(TABLE*, spider_string*))[0x7f195a0ee960]
spider/spd_db_conn.cc:4010(spider_db_store_result(ha_spider*, int, TABLE*))[0x7f195a08a25b]
spider/ha_spider.cc:7761(ha_spider::rnd_next_internal(unsigned char*))[0x7f195a0dd34f]
spider/ha_spider.cc:7832(ha_spider::pre_rnd_next(bool))[0x7f195a0dd6fa]
/usr/local/mysql/bin/mysqld(+0xcc777c)[0x7f19a3ab577c]
sql/ha_partition.cc:7221(ha_partition::handle_pre_scan(bool, bool))[0x7f19a3ab8fc9]
sql/ha_partition.cc:5050(ha_partition::rnd_next(unsigned char*))[0x7f19a3576b81]
sql/handler.cc:2764(handler::ha_rnd_next(unsigned char*))[0x7f19a34265cf]
sql/sql_table.cc:10601(mysql_checksum_table(THD*, TABLE_LIST*, st_ha_check_opt*))[0x7f19a339ca26]
sql/sql_parse.cc:4544(mysql_execute_command(THD*))[0x7f19a33a36b2]
sql/sql_parse.cc:8019(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x7f19a33a61bf]
sql/sql_parse.cc:1846(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x7f19a33a702e]
sql/sql_parse.cc:1393(do_command(THD*))[0x7f19a347652a]
sql/sql_connect.cc:1402(do_handle_one_connection(CONNECT*))[0x7f19a347664d]
sql/sql_connect.cc:1310(handle_one_connection)[0x7f19a3a9960d]
pthread_create.c:0(start_thread)[0x7f19a29b5e25]
/lib64/libc.so.6(clone+0x6d)[0x7f19a1872bad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f17a8011620): checksum table test_table EXTENDED
Connection ID (thread ID): 210312
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file at ...

Some info from the core:
#3 0x00007f195a0ee960 in spider_db_mysql_row::store_to_tmp_table (this=0x7f17a8080a20, tmp_table=0x7f17a80991d0,
str=0x7f1854742660) at server/storage/spider/spd_db_mysql.cc:471
471 tmp_table->field[0]->set_notnull();
(gdb) p tmp_table
$1 = (TABLE *) 0x7f17a80991d0
(gdb) p tmp_table->field
$2 = (Field **) 0x7f17a809a1b0
(gdb) p tmp_table->field[0]
$3 = (Field *) 0x0
...
#4 0x00007f195a08a25b in spider_db_store_result (spider=spider@entry=0x7f17a801a9e0, link_idx=link_idx@entry=0,
table=0x7f17a80191d8) at server/storage/spider/spd_db_conn.cc:4010
4010 if ((error_num = row->store_to_tmp_table(tmp_tbl, &tmp_str)))
(gdb) p result_list
$5 = (SPIDER_RESULT_LIST *) 0x7f17a801af28
(gdb) p result_list->quick_mode
$6 = 3

From general log:
242 Connect root@localhost as anonymous on test_db
242 Query checksum table test_table EXTENDED
242 Query mysql test_server1.example.com set session transaction isolation level read committed;set session autocommit = 1
242 Query mysql test_server1.example.com select `id`,...

Spider variables:
spider_bka_engine
spider_block_size 16384
spider_conn_recycle_mode 0
spider_conn_recycle_strict 0
spider_conn_wait_timeout 10
spider_connect_error_interval 1
spider_connect_mutex OFF
spider_connect_retry_count 1000
spider_connect_retry_interval 1000
spider_direct_dup_insert 1
spider_dry_access OFF
spider_force_commit 1
spider_general_log ON
spider_index_hint_pushdown OFF
spider_internal_unlock OFF
spider_internal_xa OFF
spider_internal_xa_id_type 0
spider_internal_xa_snapshot 0
spider_local_lock_table OFF
spider_lock_exchange OFF
spider_log_result_error_with_sql 3
spider_log_result_errors 3
spider_max_connections 0
spider_net_read_timeout 86400
spider_net_write_timeout 86400
spider_ping_interval_at_trx_start 3600
spider_quick_mode 3
spider_remote_access_charset
spider_remote_default_database
spider_remote_time_zone
spider_same_server_link OFF
spider_semi_table_lock 1
spider_semi_trx ON
spider_support_xa ON
spider_sync_autocommit ON
spider_sync_time_zone OFF
spider_sync_trx_isolation ON
spider_table_crd_thread_count 10
spider_table_init_error_interval 1
spider_table_sts_thread_count 10
spider_udf_table_lock_mutex_count 20
spider_udf_table_mon_mutex_count 20
spider_use_all_conns_snapshot OFF
spider_use_consistent_snapshot OFF
spider_use_default_database ON
spider_use_flash_logs OFF
spider_use_snapshot_with_flush_tables 0
spider_version 3.3.13
spider_xa_register_mode 1



 Comments   
Comment by Kentoku Shiba (Inactive) [ 2018-11-19 ]

The fields of the temporary table were not created in create_tmp_table function. Because item->const_item() was true. But the temporary tables that is created by Spider are always used all columns. So Spider should call create_tmp_table function with TMP_TABLE_ALL_COLUMNS flag.
8a32b0e

Comment by Kentoku Shiba (Inactive) [ 2018-12-24 ]

Add the tests.
For 10.3 e7b4b91
For 10.4 43b8fe3

Comment by Kentoku Shiba (Inactive) [ 2018-12-24 ]

Sachin, please review this.

Comment by Sachin Setiya (Inactive) [ 2019-02-01 ]

Okay to push.

Comment by Kentoku Shiba (Inactive) [ 2019-02-05 ]

Merged to 10.3 and 10.4 tree.

Generated at Thu Feb 08 08:27:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.