[MDEV-10498]  10.1.9-MariaDB-log MariaDB Server crashed while changing Query cache Created: 2016-08-05  Updated: 2017-09-04  Resolved: 2017-09-04

Status: Closed
Project: MariaDB Server
Component/s: Query Cache
Affects Version/s: 10.1.9
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: suraj chauhan Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

Red Hat Enterprise Linux Server release 7.1 (Maipo)
10.1.9-MariaDB-log MariaDB Server
40 cores,252 RAM

Attachments: File Mariadb-LiveDB02.err    
Issue Links:
Relates
relates to MDEV-10826 Assertion `thd->get_stmt_da()->is_eof... Confirmed
relates to MDEV-11622 Mariadb= 10.1.17-MariaDB get crashed... Closed
relates to MDEV-12156 10.1.21 crash Closed

 Description   

Today one of our production database (10.1.9-MariaDB-log MariaDB Server) has been crashed while reducing query cache size. It this bug?

Error logs start from:

160805 13:08:06 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.9-MariaDB-log
 
key_buffer_size=16777216
 
read_buffer_size=2097152
 
max_used_connections=231
 
max_threads=2511
 
thread_count=99
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18066490 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7ee1d73d3008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went


....
 
.....
 
....


2016-08-05 13:09:33 140389545150336 [Note] InnoDB: Starting an apply batch of log records to the database...
 
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
 
InnoDB: Apply batch completed
 
InnoDB: Last MySQL binlog file position 0 88325172, file name /logs/binarylogs/binary-log-174.004436
 
2016-08-05 13:09:56 140389545150336 [Note] InnoDB: 128 rollback segment(s) are active.
 
2016-08-05 13:09:56 140389545150336 [Note] InnoDB: Waiting for purge to start
 
2016-08-05 13:09:56 140389545150336 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 3717164310403
 
2016-08-05 13:10:13 140206342641408 [Note] InnoDB: Dumping buffer pool(s) not yet started
 
2016-08-05 13:10:13 140389545150336 [Note] Plugin 'FEEDBACK' is disabled.
 
160805 13:10:13 server_audit: MariaDB Audit Plugin version 1.3.0 STARTED.
 
160805 13:10:13 server_audit: logging started to the file /logs/auditlogs/server_audit.log.
 
2016-08-05 13:10:13 140389545150336 [Note] Recovering after a crash using /logs/binarylogs/binary-log-174
 
2016-08-05 13:10:13 140389545150336 [Note] Starting crash recovery...
 
2016-08-05 13:10:13 140389545150336 [Note] Crash recovery finished.
 
2016-08-05 13:10:13 140389545150336 [Note] Server socket created on IP: '::'.
 
2016-08-05 13:10:13 140389545150336 [Note] Server socket created on IP: '::'.
 
2016-08-05 13:10:13 140389545150336 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired
 
2016-08-05 13:10:13 140389545150336 [Warning] Checking table:   './mysql/user'
 
2016-08-05 13:10:13 140389545150336 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly
 
2016-08-05 13:10:13 140389545150336 [ERROR] mysqld: Table './mysql/tables_priv' is marked as crashed and should be repaired
 
2016-08-05 13:10:13 140389545150336 [Warning] Checking table:   './mysql/tables_priv'
 
2016-08-05 13:10:13 140389545150336 [ERROR] mysql.tables_priv: 1 client is using or hasn't closed the table properly
 
2016-08-05 13:10:13 140389545150336 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mariadb-relay-bin' to avoid this problem.
 
2016-08-05 13:10:13 140389545150336 [Note] /opt/mariadb/bin/mysqld: ready for connections.
 
Version: '10.1.9-MariaDB-log'  socket: '/data/mariadb-data/mariadb.sock'  port: 3310  MariaDB Server



 Comments   
Comment by Elena Stepanova [ 2016-08-05 ]

It is quite certainly a bug, unless of course somebody killed the server intentionally with SIGSEGV, which is rather unlikely. 10.1.9 is quite old, but still it would be interesting to see what the problem was. However, the most important part of the error log is the one that you skipped (replaced with '....'). Could you please paste or attach the unabridged version?

Thanks.

Comment by suraj chauhan [ 2016-08-05 ]

thanks Elena, Here is the complete error log Mariadb-LiveDB02.err

Comment by Elena Stepanova [ 2016-08-24 ]

Stack trace from the error log (to make it searchable in JIRA):

160805 13:11:05 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.9-MariaDB-log
 
key_buffer_size=16777216
 
read_buffer_size=2097152
 
max_used_connections=108
 
max_threads=2511
 
thread_count=91
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10352698 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f848e216008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fa451ca2e88 thread_stack 0x48000
 
mysys/stacktrace.c:247(my_print_stacktrace)[0xbf822e]
 
sql/signal_handler.cc:160(handle_fatal_signal)[0x760e24]
 
/lib64/libpthread.so.0(+0xf130)[0x7faefcb73130]
 
sql/sql_cache.cc:849(Query_cache_block::data())[0x5733ef]
 
sql/sql_parse.cc:7249(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5b7cc2]
 
sql/sql_parse.cc:1487(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x5baef7]
 
sql/sql_parse.cc:1111(do_command(THD*))[0x5bb2f7]
 
sql/sql_connect.cc:1349(do_handle_one_connection(THD*))[0x689883]
 
sql/sql_connect.cc:1263(handle_one_connection)[0x689aa2]
 
/lib64/libpthread.so.0(+0x7df5)[0x7faefcb6bdf5]
 
/lib64/libc.so.6(clone+0x6d)[0x7faefb81a1ad]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f84ac432020): is an invalid pointer
 
Connection ID (thread ID): 65
 
Status: NOT_KILLED

Comment by Elena Stepanova [ 2016-10-24 ]

suraj_mariadb,

Do you happen to remember how exactly you were changing the query cache size? Were you just running SET GLOBAL query_cache_size = ...., or were you disabling the query cache first, or did you something else?

I've run some experiments for this and similar reported issues, but so far I haven't got the crash upon just changing the size. When I change query_cache_type, I'm getting a problem easily enough; but unless you also did it, it might be a different problem.

Comment by Elena Stepanova [ 2017-06-25 ]

There have been complaints from different people about this a similar issue, but we are still struggling to reproduce it. Whoever else encountered it and/or has any potentially useful information (stack traces, configurations, and best of all general logs) – please add them here.

Comment by Sergei Golubchik [ 2017-09-04 ]

No feedback, closing. (issue can be reopened if more information is provided)

Generated at Thu Feb 08 07:42:40 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.