During repeated runs of a migration script in a single connection on an otherwise idle server, InnoDB crashes on an assert:
2017-07-14 10:17:09 0x700004690000 InnoDB: Assertion failure in file /tmp/mariadb-20170712-4418-z03ns4/mariadb-10.2.7/storage/innobase/dict/dict0dict.cc line 1760
|
InnoDB: Failing assertion: dict_sys->size > 0
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
|
InnoDB: about forcing recovery.
|
170714 10:17:09 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.2.7-MariaDB
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=1
|
max_threads=153
|
thread_count=7
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467207 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7fe90f10c408
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
...skipping...
|
0 mysqld 0x000000010e450c33 _Z11mysql_parseP3THDPcjP12Parser_statebb + 649
|
0 mysqld 0x000000010e44e862 _Z16dispatch_command19enum_server_commandP3THDPcjbb + 5485
|
0 mysqld 0x000000010e44fd64 _Z10do_commandP3THD + 892
|
0 mysqld 0x000000010e51f6de _Z24do_handle_one_connectionP7CONNECT + 547
|
0 mysqld 0x000000010e51f4ae handle_one_connection + 56
|
0 libsystem_pthread.dylib 0x00007fffbd98393b _pthread_body + 180
|
0 libsystem_pthread.dylib 0x00007fffbd983887 _pthread_body + 0
|
0 libsystem_pthread.dylib 0x00007fffbd98308d thread_start + 13
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7fe910064e20): is an invalid pointer
|
Connection ID (thread ID): 35
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
|
We think the query pointer is invalid, but we will try to print it anyway.
|
Query: ALTER TABLE `instances` ADD COLUMN `resurrection_paused` tinyint(1)
|
|
This was brought to my attention due to a similar crash on RDS on a MySQL 5.7.17 instances. AWS support pointed to this MySQL bug:
https://bugs.mysql.com/bug.php?id=84672
That was not reproducible apparently and hand-waved away as hardware troubles.
This crash similarly occurs on Percona Server 5.7. This has been encountered in multiple environments, and this sample is from my local workstation.
To reproduce I use the attached sql script and run:
#!/bin/bash
|
|
set -e
|
|
while true; do
|
mysql < migration_crasher.sql
|
done
|
This fails after approximately 15 seconds on my local environment.
I additionally inspected data dictionary memory concurrently and found strange output:
while true;do mysql -sse 'SHOW ENGINE INNODB STATUS\G' | egrep '^Dictionary memory allocated';sleep 0.1;done
|
...
|
Dictionary memory allocated 14342
|
Dictionary memory allocated 9811
|
Dictionary memory allocated 211
|
Dictionary memory allocated 20302
|
Dictionary memory allocated 18446744073709551123
|
Dictionary memory allocated 3955
|
Dictionary memory allocated 3334
|
Dictionary memory allocated 9997
|
Dictionary memory allocated 18446744073709540819
|
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (61)
|
I was not able to reproduce this under MariaDB 10.1. Additionally, after enabling old_alter_table I could not reproduce the problem.
- relates to
-
MDEV-17933
slow server status - dict_sys_get_size()
-
-
Closed
{"report":{"fcp":1543.3000001907349,"ttfb":722.3999996185303,"pageVisibility":"visible","entityId":62272,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"1ed1e918-ee67-4cb0-9f05-6156dd57a4ca","navigationType":0,"readyForUser":1640.5,"redirectCount":0,"resourceLoadedEnd":1852.5999994277954,"resourceLoadedStart":733.6999998092651,"resourceTiming":[{"duration":271.19999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":733.6999998092651,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":733.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1004.8999996185303,"responseStart":0,"secureConnectionStart":0},{"duration":271.1000003814697,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":734.0999994277954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":734.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1005.1999998092651,"responseStart":0,"secureConnectionStart":0},{"duration":336.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":734.1999998092651,"connectEnd":734.1999998092651,"connectStart":734.1999998092651,"domainLookupEnd":734.1999998092651,"domainLookupStart":734.1999998092651,"fetchStart":734.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":734.1999998092651,"responseEnd":1070.3999996185303,"responseStart":1070.3999996185303,"secureConnectionStart":734.1999998092651},{"duration":407.1000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":734.3999996185303,"connectEnd":734.3999996185303,"connectStart":734.3999996185303,"domainLookupEnd":734.3999996185303,"domainLookupStart":734.3999996185303,"fetchStart":734.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":734.3999996185303,"responseEnd":1141.5,"responseStart":1141.5,"secureConnectionStart":734.3999996185303},{"duration":410.9000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":734.5999994277954,"connectEnd":734.5999994277954,"connectStart":734.5999994277954,"domainLookupEnd":734.5999994277954,"domainLookupStart":734.5999994277954,"fetchStart":734.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":734.5999994277954,"responseEnd":1145.5,"responseStart":1145.5,"secureConnectionStart":734.5999994277954},{"duration":411.29999923706055,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":734.8000001907349,"connectEnd":734.8000001907349,"connectStart":734.8000001907349,"domainLookupEnd":734.8000001907349,"domainLookupStart":734.8000001907349,"fetchStart":734.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":734.8000001907349,"responseEnd":1146.0999994277954,"responseStart":1146.0999994277954,"secureConnectionStart":734.8000001907349},{"duration":411.5999994277954,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":735,"connectEnd":735,"connectStart":735,"domainLookupEnd":735,"domainLookupStart":735,"fetchStart":735,"redirectEnd":0,"redirectStart":0,"requestStart":735,"responseEnd":1146.5999994277954,"responseStart":1146.5999994277954,"secureConnectionStart":735},{"duration":486.9000005722046,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":735.0999994277954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":735.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1222,"responseStart":0,"secureConnectionStart":0},{"duration":413.1000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":735.1999998092651,"connectEnd":735.1999998092651,"connectStart":735.1999998092651,"domainLookupEnd":735.1999998092651,"domainLookupStart":735.1999998092651,"fetchStart":735.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":735.1999998092651,"responseEnd":1148.3000001907349,"responseStart":1148.3000001907349,"secureConnectionStart":735.1999998092651},{"duration":486.69999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":735.3999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":735.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1222.0999994277954,"responseStart":0,"secureConnectionStart":0},{"duration":414.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":735.5999994277954,"connectEnd":735.5999994277954,"connectStart":735.5999994277954,"domainLookupEnd":735.5999994277954,"domainLookupStart":735.5999994277954,"fetchStart":735.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":735.5999994277954,"responseEnd":1149.8999996185303,"responseStart":1149.8999996185303,"secureConnectionStart":735.5999994277954},{"duration":1086.5999994277954,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":740.8000001907349,"connectEnd":740.8000001907349,"connectStart":740.8000001907349,"domainLookupEnd":740.8000001907349,"domainLookupStart":740.8000001907349,"fetchStart":740.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":740.8000001907349,"responseEnd":1827.3999996185303,"responseStart":1827.3999996185303,"secureConnectionStart":740.8000001907349},{"duration":1111.6999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":740.8999996185303,"connectEnd":740.8999996185303,"connectStart":740.8999996185303,"domainLookupEnd":740.8999996185303,"domainLookupStart":740.8999996185303,"fetchStart":740.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":740.8999996185303,"responseEnd":1852.5999994277954,"responseStart":1852.5999994277954,"secureConnectionStart":740.8999996185303},{"duration":610.0999994277954,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1234.3000001907349,"connectEnd":1234.3000001907349,"connectStart":1234.3000001907349,"domainLookupEnd":1234.3000001907349,"domainLookupStart":1234.3000001907349,"fetchStart":1234.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":1234.3000001907349,"responseEnd":1844.3999996185303,"responseStart":1844.3999996185303,"secureConnectionStart":1234.3000001907349},{"duration":332.9000005722046,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1534.0999994277954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1534.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1867,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":560,"responseStart":723,"responseEnd":735,"domLoading":731,"domInteractive":1873,"domContentLoadedEventStart":1873,"domContentLoadedEventEnd":1915,"domComplete":2747,"loadEventStart":2747,"loadEventEnd":2749,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1854.8000001907349},{"name":"bigPipe.sidebar-id.end","time":1855.5999994277954},{"name":"bigPipe.activity-panel-pipe-id.start","time":1855.6999998092651},{"name":"bigPipe.activity-panel-pipe-id.end","time":1858.0999994277954},{"name":"activityTabFullyLoaded","time":1922.5999994277954}],"measures":[],"correlationId":"5639d69609c08d","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":95,"dbReadsTimeInMs":12,"dbConnsTimeInMs":21,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}