[MDEV-21824] 0xc0000005 crash Created: 2020-02-27  Updated: 2020-03-16  Resolved: 2020-03-16

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.5.1
Fix Version/s: 10.5.2

Type: Bug Priority: Major
Reporter: Moshe L Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None
Environment:

Windows 10 x64

Issue Links:
Relates
relates to MDEV-21919 Crash on mariadb:10.5.1 (docker) when... Closed

 Description   

I am trying the 10.5.1 on a dev machine.

200227  9:34:30 [ERROR] mysqld got exception 0xc0000005 ;
 
 
 
Thread pointer: 0x27b25b329b8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
mysqld.exe!my_wc_to_printable_generic()[ctype.c:1055]
 
mysqld.exe!my_convert_using_func()[ctype.c:1141]
 
mysqld.exe!convert_error_message()[sql_error.cc:959]
 
mysqld.exe!net_send_error_packet()[protocol.cc:456]
 
mysqld.exe!Protocol::end_statement()[protocol.cc:599]
 
mysqld.exe!do_command()[sql_parse.cc:1258]
 
mysqld.exe!threadpool_process_request()[threadpool_common.cc:369]
 
mysqld.exe!tp_callback()[threadpool_common.cc:196]
 
ntdll.dll!RtlInitializeCriticalSection()
 
ntdll.dll!RtlReleaseSRWLockExclusive()
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()
 
 
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): 
Connection ID (thread ID): 704
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

Sometimes with a specific popular SQL query. crashed also if the query table did not exists.
I was tried to execute the query manualy with no problem. sometimes it crashes.

other strange thing: this code is updating (or inserting) one row, but I see in my GUI:
2 row(s) affected

What I was tried:

  • Run the query from a GUI - works (~99%).

the query:
insert into online set o_user=1, o_ip='127.0.0.1', o_url='', o_lastDate=now(), o_source=0 on duplicate key update o_user=1, o_ip='127.0.0.1', o_url='', o_lastDate=now(), o_source=0;

the table:

CREATE TABLE `online` (
 
  `o_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
 
  `o_ip` varchar(35) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
 
  `o_user` int(11) NOT NULL DEFAULT 0,
 
  `o_lastDate` datetime NOT NULL DEFAULT '1987-11-13 00:00:00',
 
  `o_url` varchar(155) NOT NULL DEFAULT '',
 
  `o_source` int(11) NOT NULL DEFAULT 0,
 
  `o_imstate` datetime NOT NULL DEFAULT '1987-11-13 00:00:00',
 
  PRIMARY KEY (`o_id`),
 
  UNIQUE KEY `user` (`o_user`) USING BTREE,
 
  KEY `date` (`o_lastDate`,`o_user`) USING BTREE
 
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8 MAX_ROWS=10000;

the original ENGINE was MEMORY, I was changed it to INNODB to remove the option that the MEMORY engins throw this bug.

  • Remove the table - still a chance for crash
  • Tried to run other code on the server - works
  • Tried to run the query 100 times - works

I suspect the query isn't the problem and it just a big chance for it to run on the server crash, but when I was disabled it from the code side, the chance to crash was lowered from minutes to hours.

Thanks !

 Comments   
Comment by Elena Stepanova [ 2020-02-29 ]

You said
> I was tried to execute the query manualy with no problem. sometimes it crashes
and
> Run the query from a GUI - works (~99%)

So, how do you execute it when it crashes? Are you using some connector? And what is the difference between "execute the query manually" and "run the query from a GUI" – do you mean command-line client as "manually" and HeidiSQL as "GUI"?
Or does it just mean that no matter how you execute it, it works 99% of time, and fails once in a while?

Comment by Moshe L [ 2020-02-29 ]

GUI is MySQL Workbench
I tried to run it also with mysql commandline and from the Website via .NET MySqlConnector.

I think it failed more from the web, but don't sure abut this. I tried the same back to 10.4.12 and it is working without problems.

(the original code was wrote in ~ 2008 and it works on every MySQL/MariaDB version from 5.0 and up....)

Comment by Moshe L [ 2020-03-01 ]

I has some more time to debug this problem.
the first was tried to drop and create the table with ENGINEs. the server is not crashed right now.

other problem still here: "2 rows effected", but effected only one.

Comment by Elena Stepanova [ 2020-03-08 ]

2 rows affected is not a problem, it's expected and documented behavior, also in MySQL (assuming your INSERT actually updates the row):

With ON DUPLICATE KEY UPDATE, the affected-rows value per row is 1 if the row is inserted as a new row, 2 if an existing row is updated, and 0 if an existing row is set to its current values.

Comment by Vladislav Vaintroub [ 2020-03-08 ]

bar, this seems to be from MDEV-8844 Unreadable control characters printed as is in warnings.
Apparently, I've no idea how to reproduce it, but from looking at MDEV-8884 diff, it seems that in the past client charset = NULL was taken care of, and now it is not
https://github.com/mariadb/server/commit/3c6065a270f547d160e11bbcaa653930ec5ff90f#diff-607618e4dfd23909032710ae5a80526eL939

Whether this is a problem, or whether it is a valid reason, is hard to tell. Apparently, I think it is not an actual query is causing this, due to query in the output. being 0x0. This might be ping or something like that.

Comment by Vladislav Vaintroub [ 2020-03-08 ]

The actual line that is causing trouble must be 

    int mblen= cs->cset->wc_mb(cs, wc, str, end);

since the crash callstack does not jump into wc_mb(), I believe that only referencing pointers might be a problem
e.g either of cs, cs->cset, or cs->cset->wc_mb is NULL or junk

Comment by Moshe L [ 2020-03-09 ]

I am not a SQL debugger, but I suspect the COM_RESET_CONNECTION action used by the MySQLConnector when reusing pooled connection as a new one.
Maybe it is the problem, but I do not know how to repro it.

Comment by Moshe L [ 2020-03-10 ]

after 4 days of uptime, it crashed again

With other simple UPDATE query, the first one after reset connection
same stacktrace.

Comment by Vladislav Vaintroub [ 2020-03-10 ]

I think it is not "after" reset connection, it is "during" reset connection.
Do you have mysqld.dmp saved in the data directory? Maybe you can upload this.

Comment by Moshe L [ 2020-03-10 ]

update: I was wrong here, the code make the crash used Oracle MySQL .NET Connector 6.9.12 from 2015 that did not use the COM_RESET_CONNECTION.

So, I upgraded the connector to MySQLConnector that use COM_RESET_CONNECTION and for now the problem was solved.

if you want, I can test a debug-able version of MariaDB on this system. I did not have a mysqld.dmp.

Generated at Thu Feb 08 09:10:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.