# my_print_defaults mysqld
|
--performance_schema=on
|
--datadir=/opt/data/mysql
|
--tmpdir=/tmp/
|
--log_error=/opt/data/mysql/error.log
|
--log_warnings=2
|
--query_cache_size=0
|
--query_cache_type=0
|
--log_bin
|
--log_slave_updates
|
--binlog_format=ROW
|
--innodb_log_file_size=128M
|
--innodb_buffer_pool_size=2G
|
--innodb_flush_method=O_DIRECT
|
--innodb_flush_log_at_trx_commit=1
|
--innodb_autoinc_lock_mode=2
|
--wsrep_cluster_name=GaleraPOC
|
--wsrep_cluster_address=gcomm://mariadb001,mariadb002,mariadb003
|
--wsrep_provider=/usr/lib64/galera/libgalera_smm.so
|
--wsrep_provider_options=gcache.size=512M
|
--wsrep_sst_method=xtrabackup-v2
|
--wsrep_sst_auth=sst:sstpass
|
--server_id=1
|
--wsrep_node_name=mariadb001
|
--wsrep_node_address=mariadb001
|
--wsrep_sst_receive_address=mariadb001
|
* 10.0.12-MariaDB-wsrep-log MariaDB Server, wsrep_25.10.r4002
|
* wsrep_provider_version=25.3.5(rXXXX)
|
# ./node1.sh
|
.............ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
|
ERROR with query
|
The server on node1 crashes:
140828 15:34:49 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.0.12-MariaDB-wsrep-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=7
|
max_threads=153
|
thread_count=8
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467215 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0x7f2d7ce8f008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f2e43e56ce0 thread_stack 0x48000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb914fb]
|
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x743318]
|
/lib64/libpthread.so.0(+0xf710)[0x7f2e43b1c710]
|
/lib64/libc.so.6(+0x83742)[0x7f2e421cc742]
|
/usr/sbin/mysqld(_ZNK19rpl_sql_thread_info22cached_charset_compareEPc+0x20)[0x699bf0]
|
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x837)[0x800817]
|
/usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0x525)[0x6f05e5]
|
/usr/lib64/galera/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xb1)[0x7f2e3eb
|
4c2c1]
|
/usr/lib64/galera/libgalera_smm.so(+0x1aaf95)[0x7f2e3eb83f95]
|
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10replay_trxEPNS_9TrxHandleEPv+0x12e)[0x7f2e3eb8485e]
|
/usr/lib64/galera/libgalera_smm.so(galera_replay_trx+0x5c)[0x7f2e3eb9845c]
|
/usr/sbin/mysqld(_Z24wsrep_replay_transactionP3THD+0x2de)[0x6f217e]
|
/usr/sbin/mysqld[0x5e1120]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x16d0)[0x5e28a0]
|
/usr/sbin/mysqld(_Z10do_commandP3THD+0x132)[0x5e3072]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x54b)[0x6a190b]
|
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6a1a02]
|
/lib64/libpthread.so.0(+0x79d1)[0x7f2e43b149d1]
|
/lib64/libc.so.6(clone+0x6d)[0x7f2e42231b5d]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f2e1b631251): is an invalid pointer
|
Connection ID (thread ID): 331
|
Status: NOT_KILLED
|
I spoke with Teemu from Codership and he mentioned this is likely a bug specific to MariaDB.
I tested this on:
- 5.5.38: no crash
- PXC 5.6.19: no crash
12:22 < gryp> I have crash of MGC-10.0.12: https://gist.github.com/grypyrg/22b0512cb59b5e32a538, seems to be similar to
knielsen's fix of https://mariadb.atlassian.net/browse/MDEV-6156 . Anybody got any ideas?
12:27 < knielsen> gryp: The code path involved is completely different (Galera replication vs. MariaDB parallel replication).
But it is possible that Galera has a similar bug (it also does parallel replication), might help the
developers track it down...
12:29 < knielsen> Interesting that Galera seems to be calling Query_log_event::do_apply_event() ... I suppose this is to
handle DDL perhaps, which shouldn't run in parallel
12:29 < knielsen> well, not that I know much of what Galera code does
12:29 < gryp> indeed. it's during some DDL's. I'm still working on getting a cleaner test case.
12:31 < gryp> I'm also running with wsrep_slave_threads=1 and the DDL is executed on the crashing node.
12:31 < gryp> knielsen: I'll talk some more with Codership, see what they think based on your feedback. TNx
12:33 < knielsen> gryp: there's more discussions of this bug on maria-developers@ (google should know), in case it helps
understand the issue. Though it could be something different, hard to say
12:33 < gryp> Yep, have found that as well. tnx. (https://www.mail-archive.com/maria-developers@lists.launchpad.net/msg06846.html)
{"report":{"fcp":854.9000000953674,"ttfb":157.5,"pageVisibility":"visible","entityId":43907,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"47f0ab85-7719-4393-8a51-cd9c5c3b9593","navigationType":0,"readyForUser":932,"redirectCount":0,"resourceLoadedEnd":1202.9000000953674,"resourceLoadedStart":162.90000009536743,"resourceTiming":[{"duration":262.7999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":162.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":162.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":425.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":262.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":163.10000014305115,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":163.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":425.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":273.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":163.20000004768372,"connectEnd":163.20000004768372,"connectStart":163.20000004768372,"domainLookupEnd":163.20000004768372,"domainLookupStart":163.20000004768372,"fetchStart":163.20000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":163.20000004768372,"responseEnd":436.90000009536743,"responseStart":436.90000009536743,"secureConnectionStart":163.20000004768372},{"duration":299.10000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":163.29999995231628,"connectEnd":163.29999995231628,"connectStart":163.29999995231628,"domainLookupEnd":163.29999995231628,"domainLookupStart":163.29999995231628,"fetchStart":163.29999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":163.29999995231628,"responseEnd":462.40000009536743,"responseStart":462.40000009536743,"secureConnectionStart":163.29999995231628},{"duration":302.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":163.40000009536743,"connectEnd":163.40000009536743,"connectStart":163.40000009536743,"domainLookupEnd":163.40000009536743,"domainLookupStart":163.40000009536743,"fetchStart":163.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":163.40000009536743,"responseEnd":466,"responseStart":466,"secureConnectionStart":163.40000009536743},{"duration":302.90000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":163.5,"connectEnd":163.5,"connectStart":163.5,"domainLookupEnd":163.5,"domainLookupStart":163.5,"fetchStart":163.5,"redirectEnd":0,"redirectStart":0,"requestStart":163.5,"responseEnd":466.40000009536743,"responseStart":466.40000009536743,"secureConnectionStart":163.5},{"duration":303.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":163.60000014305115,"connectEnd":163.60000014305115,"connectStart":163.60000014305115,"domainLookupEnd":163.60000014305115,"domainLookupStart":163.60000014305115,"fetchStart":163.60000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":163.60000014305115,"responseEnd":466.7000000476837,"responseStart":466.7000000476837,"secureConnectionStart":163.60000014305115},{"duration":303.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":163.70000004768372,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":163.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":467.2999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":303.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":163.79999995231628,"connectEnd":163.79999995231628,"connectStart":163.79999995231628,"domainLookupEnd":163.79999995231628,"domainLookupStart":163.79999995231628,"fetchStart":163.79999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":163.79999995231628,"responseEnd":467.2999999523163,"responseStart":467.2999999523163,"secureConnectionStart":163.79999995231628},{"duration":303.7999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":163.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":163.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":467.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":303.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":164,"connectEnd":164,"connectStart":164,"domainLookupEnd":164,"domainLookupStart":164,"fetchStart":164,"redirectEnd":0,"redirectStart":0,"requestStart":164,"responseEnd":467.7999999523163,"responseStart":467.7999999523163,"secureConnectionStart":164},{"duration":473,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":164.70000004768372,"connectEnd":164.70000004768372,"connectStart":164.70000004768372,"domainLookupEnd":164.70000004768372,"domainLookupStart":164.70000004768372,"fetchStart":164.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":164.70000004768372,"responseEnd":637.7000000476837,"responseStart":637.7000000476837,"secureConnectionStart":164.70000004768372},{"duration":1038.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":164.70000004768372,"connectEnd":164.70000004768372,"connectStart":164.70000004768372,"domainLookupEnd":164.70000004768372,"domainLookupStart":164.70000004768372,"fetchStart":164.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":164.70000004768372,"responseEnd":1202.9000000953674,"responseStart":1202.9000000953674,"secureConnectionStart":164.70000004768372},{"duration":465.7999999523163,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":663.1000001430511,"connectEnd":663.1000001430511,"connectStart":663.1000001430511,"domainLookupEnd":663.1000001430511,"domainLookupStart":663.1000001430511,"fetchStart":663.1000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":663.1000001430511,"responseEnd":1128.9000000953674,"responseStart":1128.9000000953674,"secureConnectionStart":663.1000001430511},{"duration":379.90000009536743,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":834.2000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":834.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1214.1000001430511,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":15,"responseStart":158,"responseEnd":159,"domLoading":161,"domInteractive":1223,"domContentLoadedEventStart":1223,"domContentLoadedEventEnd":1260,"domComplete":1989,"loadEventStart":1989,"loadEventEnd":1991,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1204.5},{"name":"bigPipe.sidebar-id.end","time":1205.2999999523163},{"name":"bigPipe.activity-panel-pipe-id.start","time":1205.4000000953674},{"name":"bigPipe.activity-panel-pipe-id.end","time":1207.6000001430511},{"name":"activityTabFullyLoaded","time":1266.5}],"measures":[],"correlationId":"87368eac9101ce","effectiveType":"4g","downlink":9.7,"rtt":0,"serverDuration":80,"dbReadsTimeInMs":13,"dbConnsTimeInMs":21,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}