[MDEV-16547] Spider Crash server in 10.3.7 Version Created: 2018-06-21  Updated: 2020-08-25  Resolved: 2018-09-11

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.3.7
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Antoine Girardi Assignee: Jacob Mathew (Inactive)
Resolution: Cannot Reproduce Votes: 2
Labels: None
Environment:

Windows

Attachments: File Aggregation-my.ini     File Aggregation_table_data.sql     File CPM-SERVER.err     File Shard01-my.ini     File Shard01_table_data.sql     File Shard02-my.ini     File Shard02_table_data.sql     File Shard03-my.ini     File Shard03_table_data.sql     Microsoft Word server_topology.xlsx    

 Description   

Hi,

we have upgraded to 10.3.7 because of some crashes in 10.1.25
but still have crashes

180621 13:02:30 [ERROR] mysqld got exception 0xc0000005 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.7-MariaDB-log
 
key_buffer_size=536870912
 
read_buffer_size=131072
 
max_used_connections=3
 
max_threads=65537
 
thread_count=29
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 540890 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x5b244e6468
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
mysqld.exe!??_9Item@@$BDFA@AA()
 
mysqld.exe!?get_new_handler@@YAPEAVhandler@@PEAUTABLE_SHARE@@PEAUst_mem_root@@PEAUhandlerton@@@Z()
 
mysqld.exe!?ha_delete_table@@YAHPEAVTHD@@PEAUhandlerton@@PEBDPEBUst_mysql_const_lex_string@@3_N@Z()
 
mysqld.exe!?mysql_rm_table_no_locks@@YAHPEAVTHD@@PEAUTABLE_LIST@@_N22222@Z()
 
mysqld.exe!?mysql_rm_table@@YA_NPEAVTHD@@PEAUTABLE_LIST@@_N22@Z()
 
mysqld.exe!?mysql_execute_command@@YAHPEAVTHD@@@Z()
 
mysqld.exe!?mysql_parse@@YAXPEAVTHD@@PEADIPEAVParser_state@@_N3@Z()
 
mysqld.exe!?dispatch_command@@YA_NW4enum_server_command@@PEAVTHD@@PEADI_N3@Z()
 
mysqld.exe!?do_command@@YA_NPEAVTHD@@@Z()
 
mysqld.exe!?pool_of_threads_scheduler@@YAXPEAUscheduler_functions@@PEAKPEAI@Z()
 
mysqld.exe!?tp_callback@@YAXPEAUTP_connection@@@Z()
 
ntdll.dll!RtlDosPathNameToNtPathName_U()
 
ntdll.dll!RtlCompareUnicodeString()
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x5b2457bb80): DROP TABLE IF EXISTS DOCUMENT_ARCHIVE
 
Connection ID (thread ID): 32
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

---------------------------

Server version: 10.3.7-MariaDB
 
key_buffer_size=536870912
 
read_buffer_size=131072
 
max_used_connections=7
 
max_threads=65537
 
thread_count=32
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 540890 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0xe7a95e0e28
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
ha_spider.dll!spider_ping_table_init()
 
ha_spider.dll!spider_ping_table_init()
 
mysqld.exe!?execute@Pushdown_query@@QEAAHPEAVJOIN@@@Z()
 
mysqld.exe!?disjoin@?$List@VItem@@@@QEAAXPEAV1@@Z()
 
mysqld.exe!?exec_inner@JOIN@@QEAAXXZ()
 
mysqld.exe!?exec@JOIN@@QEAAXXZ()
 
mysqld.exe!?mysql_select@@YA_NPEAVTHD@@PEAUTABLE_LIST@@IAEAV?$List@VItem@@@@PEAVItem@@IPEAUst_order@@434_KPEAVselect_result@@PEAVst_select_lex_unit@@PEAVst_select_lex@@@Z()
 
mysqld.exe!?handle_select@@YA_NPEAVTHD@@PEAULEX@@PEAVselect_result@@K@Z()
 
mysqld.exe!?execute_init_command@@YAXPEAVTHD@@PEAUst_mysql_lex_string@@PEAUst_mysql_rwlock@@@Z()
 
mysqld.exe!?mysql_execute_command@@YAHPEAVTHD@@@Z()
 
mysqld.exe!?mysql_parse@@YAXPEAVTHD@@PEADIPEAVParser_state@@_N3@Z()
 
mysqld.exe!?dispatch_command@@YA_NW4enum_server_command@@PEAVTHD@@PEADI_N3@Z()
 
mysqld.exe!?do_command@@YA_NPEAVTHD@@@Z()
 
mysqld.exe!?pool_of_threads_scheduler@@YAXPEAUscheduler_functions@@PEAKPEAI@Z()
 
mysqld.exe!?tp_callback@@YAXPEAUTP_connection@@@Z()
 
ntdll.dll!RtlDosPathNameToNtPathName_U()
 
ntdll.dll!RtlCompareUnicodeString()
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0xe7ba755030): SELECT 1 FROM DOCUMENT_ARCHIVE LIMIT 1
 
Connection ID (thread ID): 53
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=

------------------

2018-06-21 12:46:19 28 [Warning] .\gedazur\doc_crit_archive.frm is inconsistent: engine typecode 44, engine name SPIDER (45)
 
2018-06-21 12:46:21 28 [Warning] .\gedazur\doc_memo_archive.frm is inconsistent: engine typecode 44, engine name SPIDER (45)
 
2018-06-21 12:46:32 28 [Warning] .\gedazur\document_archive.frm is inconsistent: engine typecode 44, engine name SPIDER (45)
 
2018-06-21 12:57:44 64 [Warning] .\gedazur\doc_arbo_adr_archive.frm is inconsistent: engine typecode 44, engine name SPIDER (45)
 
2018-06-21 12:57:44 64 [Warning] .\gedazur\document_archive.frm is inconsistent: engine typecode 44, engine name SPIDER (45)
 
2018-06-21 12:57:44 64 [Warning] .\gedazur\doc_crit_archive.frm is inconsistent: engine typecode 44, engine name SPIDER (45)



 Comments   
Comment by PALMA Gabin [ 2018-06-22 ]

We reproduce systematically the crash when select 1 from a table sipder is execute. on 10.3.7

Comment by Jacob Mathew (Inactive) [ 2018-08-30 ]

Note that the two stack traces are very different. The first stack trace does not go through Spider and indicates that a DROP TABLE statement was being executed. The second stack trace is an impossible stack trace that appears to go through Spider. It is an impossible stack trace because the function spider_ping_table_init() does not call itself.

Comment by Jacob Mathew (Inactive) [ 2018-08-30 ]

The customer's information files were not added to this bug. I have now copied them from the support case and added them to this bug as attachments.

Comment by Jacob Mathew (Inactive) [ 2018-08-30 ]

There are numerous errors in the Spider node server's event log that indicate that the customer did not run mysql_upgrade. The customer needs to run mysql_upgrade on the Spider node and then retry.

Comment by Jacob Mathew (Inactive) [ 2018-08-30 ]

The spider node server's event log contains numerous error messages regarding network connections. In order to use Spider, the network needs to be reliable.

Comment by Jacob Mathew (Inactive) [ 2018-08-30 ]

We also need to see the customer's CREATE SERVER statements that were entered on the Spider node. There are numerous errors in the Spider node server's event log regarding access denied for the 'spider' user.

Comment by Jacob Mathew (Inactive) [ 2018-09-05 ]

The customer replied that they enter the following CREATE SERVER statements on each respective data node:

create server shard01 foreign data wrapper mysql options
 
(host 'CPM-Shard-01', user 'spider', password 'spider', port 3306);
 
create server shard02 foreign data wrapper mysql options
 
(host 'CPM-Shard-02', user 'spider', password 'spider', port 3306);
 
create server shard03 foreign data wrapper mysql options
 
(host 'CPM-Shard-03', user 'spider', password 'spider', port 3306);

, and that they enter the following CREATE SERVER statement on the Spider node:

create server mon foreign data wrapper mysql options
 
(host 'CPM-Server', user 'spider', password 'spider', port 3306);

This would seem to indicate that the customer does not understand how to set up Spider. The customer needs to read the following KB page before proceeding:

https://mariadb.com/kb/en/library/spider-storage-engine-overview/

The CREATE SERVER statements are to be executed only on the Spider node. Each CREATE SERVER statement defines a connection from the Spider node to one of the data nodes. So their first three CREATE SERVER statements need to be executed on the Spider node. The 4th CREATE SERVER statement should be removed.

The 'spider' user on each data node needs to be set up so that the CREATE SERVER connections from the Spider node can log in remotely from the Spider node. Otherwise, those network connections from the Spider node will be aborted because access is denied.

The syntax the customer is using on the CREATE TABLE statements is correct if they want the full table duplicated on each data node. It is NOT correct if they want to shard the table across the 3 data nodes. Refer to the KB page for the correct syntax for sharding. Partitioning is necessary for sharding.

Only the DDL statements should be entered on every server. The DML statements, i.e. INSERT, UPDATE, DELETE and SELECT should be entered only on the Spider node.

Comment by Jacob Mathew (Inactive) [ 2018-09-11 ]

Per Satoru Goto, the partner (Smart Style) and the end customer misunderstood the architecture of Spider; the end customer uses Spider just for HA monitoring. Satoru Goto has asked for the issue to be closed.

Comment by Jacob Mathew (Inactive) [ 2018-09-11 ]

Closing the issue as requested by Satoru Goto. I could not reproduce it.

Generated at Thu Feb 08 08:29:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.