If you have an entry in your configuration file like this:
innodb_data_home_dir=/innodbdata
Then rsync SSTs will fail.
The first time a joiner attempts to SST, the node will see log entries like this, showing that a new InnoDB data file is being created after the SST:
2016-09-06 15:47:41 140177158698752 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.22.174' --datadir '/var/lib/mysql/' --parent '1957' --binlog '/var/mysql/binlogs/mariadb-bin' '
2016-09-06 15:47:41 140177453222656 [Note] WSREP: Prepared SST request: rsync|172.31.22.174:4444/rsync_sst
2016-09-06 15:47:41 140177453222656 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-09-06 15:47:41 140177453222656 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-09-06 15:47:41 140177231623936 [Note] WSREP: Service thread queue flushed.
2016-09-06 15:47:41 140177453222656 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2016-09-06 15:47:41 140177231623936 [Note] WSREP: Service thread queue flushed.
2016-09-06 15:47:41 140177453222656 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (71de705d-6faa-11e6-9a78-8295746971ef) does not match group state UUID (bae8d70c-746a-11e6-869f-632ad20bfb5c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-09-06 15:47:41 140177175471872 [Note] WSREP: Member 1.0 () requested state transfer from '*any*'. Selected 0.0 (ip-172-31-19-192.us-west-2.compute.internal)(SYNCED) as donor.
2016-09-06 15:47:41 140177175471872 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2016-09-06 15:47:41 140177453222656 [Note] WSREP: Requesting state transfer: success, donor: 0
2016-09-06 15:47:43 140177183864576 [Note] WSREP: (c52d1933, 'tcp://0.0.0.0:4567') turning message relay requesting off
2016-09-06 15:47:44 140177175471872 [Note] WSREP: 0.0 (ip-172-31-19-192.us-west-2.compute.internal): State transfer to 1.0 () complete.
2016-09-06 15:47:44 140177175471872 [Note] WSREP: Member 0.0 (ip-172-31-19-192.us-west-2.compute.internal) synced with group.
WSREP_SST: [INFO] Extracting binlog files: (20160906 15:47:44.804)
mariadb-bin.000006
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 1997 (20160906 15:47:44.815)
WSREP_SST: [INFO] Joiner cleanup done. (20160906 15:47:45.320)
2016-09-06 15:47:45 140177453537408 [Note] WSREP: SST complete, seqno: 0
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: The InnoDB memory heap is disabled
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Memory barrier is not used
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Compressed tables use zlib 1.2.7
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Using Linux native AIO
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Using SSE crc32 instructions
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Completed initialization of buffer pool
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: The first specified data file /innodbdata/ibdata1 did not exist: a new database to be created!
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Setting file /innodbdata/ibdata1 size to 12 MB
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Database physically writes the file full: wait...
2016-09-06 15:47:45 140177453537408 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2016-09-06 15:47:46 140177453537408 [ERROR] InnoDB: Cannot create ./ib_logfile1
2016-09-06 15:47:46 140177453537408 [ERROR] Plugin 'InnoDB' init function returned error.
2016-09-06 15:47:46 140177453537408 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-09-06 15:47:46 140177453537408 [Note] Plugin 'FEEDBACK' is disabled.
2016-09-06 15:47:46 140177453537408 [ERROR] Unknown/unsupported storage engine: InnoDB
2016-09-06 15:47:46 140177453537408 [ERROR] Aborting
And then every time the joiner performs an SST after the first, the error log will show issues with the InnoDB tablespace:
2016-09-06 15:51:15 139831426414336 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.22.174' --datadir '/var/lib/mysql/' --parent '2092' --binlog '/var/mysql/binlogs/mariadb-bin' '
2016-09-06 15:51:15 139831721270016 [Note] WSREP: Prepared SST request: rsync|172.31.22.174:4444/rsync_sst
2016-09-06 15:51:15 139831721270016 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-09-06 15:51:15 139831721270016 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-09-06 15:51:15 139831499339520 [Note] WSREP: Service thread queue flushed.
2016-09-06 15:51:15 139831721270016 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2016-09-06 15:51:15 139831499339520 [Note] WSREP: Service thread queue flushed.
2016-09-06 15:51:15 139831721270016 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bae8d70c-746a-11e6-869f-632ad20bfb5c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-09-06 15:51:15 139831443187456 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 (ip-172-31-19-192.us-west-2.compute.internal)(SYNCED) as donor.
2016-09-06 15:51:15 139831443187456 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2016-09-06 15:51:15 139831721270016 [Note] WSREP: Requesting state transfer: success, donor: 1
2016-09-06 15:51:17 139831443187456 [Note] WSREP: 1.0 (ip-172-31-19-192.us-west-2.compute.internal): State transfer to 0.0 () complete.
2016-09-06 15:51:17 139831443187456 [Note] WSREP: Member 1.0 (ip-172-31-19-192.us-west-2.compute.internal) synced with group.
2016-09-06 15:51:17 139831451580160 [Note] WSREP: (44e8acf0, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] Extracting binlog files: (20160906 15:51:18.363)
mariadb-bin.000007
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2132 (20160906 15:51:18.369)
WSREP_SST: [INFO] Joiner cleanup done. (20160906 15:51:18.874)
2016-09-06 15:51:18 139831721584768 [Note] WSREP: SST complete, seqno: 0
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: The InnoDB memory heap is disabled
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Memory barrier is not used
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Compressed tables use zlib 1.2.7
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Using Linux native AIO
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Using SSE crc32 instructions
2016-09-06 15:51:18 139831721584768 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-09-06 15:51:19 139831721584768 [Note] InnoDB: Completed initialization of buffer pool
2016-09-06 15:51:19 139831721584768 [Note] InnoDB: Restoring page 0 of tablespace 0
2016-09-06 15:51:19 139831721584768 [Warning] InnoDB: Doublewrite does not have page_no=0 of space: 0
2016-09-06 15:51:19 139831721584768 [ERROR] InnoDB: space header page consists of zero bytes in data file /innodbdata/ibdata1
2016-09-06 15:51:19 139831721584768 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2016-09-06 15:51:19 139831721584768 [ERROR] Plugin 'InnoDB' init function returned error.
made changes -
2016-09-06 20:01
Field
Original Value
New Value
Assignee
Nirbhay Choubey
[ nirbhay_c
]
made changes -
2017-02-25 21:34
Assignee
Nirbhay Choubey
[ nirbhay_c
]
made changes -
2018-05-02 10:59
Assignee
Sachin Setiya
[ sachin.setiya.007
]
Jan Lindström
[ jplindst
]
made changes -
2018-07-03 12:57
Status
Open
[ 1
]
In Progress
[ 3
]
made changes -
2018-07-24 12:15
Status
In Progress
[ 3
]
In Review
[ 10002
]
made changes -
2018-07-24 12:15
Assignee
Julius Goryavsky
[ sysprg
]
Jan Lindström
[ jplindst
]
made changes -
2018-07-24 12:24
Assignee
Jan Lindström
[ jplindst
]
Julius Goryavsky
[ sysprg
]
Status
In Review
[ 10002
]
Stalled
[ 10000
]
made changes -
2018-08-09 02:31
Status
Stalled
[ 10000
]
In Progress
[ 3
]
made changes -
2018-08-09 02:32
Assignee
Julius Goryavsky
[ sysprg
]
Jan Lindström
[ jplindst
]
Status
In Progress
[ 3
]
In Review
[ 10002
]
made changes -
2018-08-21 12:59
issue.field.resolutiondate
2018-08-21 12:59:01.0
2018-08-21 12:59:01.043
made changes -
2018-08-21 12:59
Fix Version/s
10.1.36
[ 23117
]
Fix Version/s
10.2.18
[ 23112
]
Fix Version/s
10.3.11
[ 23141
]
Fix Version/s
10.1
[ 16100
]
Resolution
Fixed
[ 1
]
Status
In Progress
[ 3
]
Closed
[ 6
]
made changes -
2019-02-14 09:04
NRE Projects
AC-2328 AC-3299
made changes -
2021-12-06 21:43
Workflow
MariaDB v3
[ 76933
]
MariaDB v4
[ 150878
]
{"report":{"fcp":1849,"ttfb":658.4000000953674,"pageVisibility":"visible","entityId":57928,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"ed32151b-b514-4a58-8b6f-61b565d7b88b","navigationType":0,"readyForUser":1973.0999999046326,"redirectCount":0,"resourceLoadedEnd":2624.5,"resourceLoadedStart":664.5999999046326,"resourceTiming":[{"duration":356,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":664.5999999046326,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":664.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1020.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":355.8999996185303,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":664.9000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":664.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1020.7999997138977,"responseStart":0,"secureConnectionStart":0},{"duration":643.3000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":665.0999999046326,"connectEnd":665.0999999046326,"connectStart":665.0999999046326,"domainLookupEnd":665.0999999046326,"domainLookupStart":665.0999999046326,"fetchStart":665.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":665.0999999046326,"responseEnd":1308.4000000953674,"responseStart":1308.4000000953674,"secureConnectionStart":665.0999999046326},{"duration":820.1000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":665.2999997138977,"connectEnd":665.2999997138977,"connectStart":665.2999997138977,"domainLookupEnd":665.2999997138977,"domainLookupStart":665.2999997138977,"fetchStart":665.2999997138977,"redirectEnd":0,"redirectStart":0,"requestStart":665.2999997138977,"responseEnd":1485.4000000953674,"responseStart":1485.4000000953674,"secureConnectionStart":665.2999997138977},{"duration":827.8999996185303,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":665.4000000953674,"connectEnd":665.4000000953674,"connectStart":665.4000000953674,"domainLookupEnd":665.4000000953674,"domainLookupStart":665.4000000953674,"fetchStart":665.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":665.4000000953674,"responseEnd":1493.2999997138977,"responseStart":1493.2999997138977,"secureConnectionStart":665.4000000953674},{"duration":828.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":665.6999998092651,"connectEnd":665.6999998092651,"connectStart":665.6999998092651,"domainLookupEnd":665.6999998092651,"domainLookupStart":665.6999998092651,"fetchStart":665.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":665.6999998092651,"responseEnd":1494.2999997138977,"responseStart":1494.2999997138977,"secureConnectionStart":665.6999998092651},{"duration":829.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":665.9000000953674,"connectEnd":665.9000000953674,"connectStart":665.9000000953674,"domainLookupEnd":665.9000000953674,"domainLookupStart":665.9000000953674,"fetchStart":665.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":665.9000000953674,"responseEnd":1495.4000000953674,"responseStart":1495.4000000953674,"secureConnectionStart":665.9000000953674},{"duration":832,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":666,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":666,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1498,"responseStart":0,"secureConnectionStart":0},{"duration":830.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":666.1999998092651,"connectEnd":666.1999998092651,"connectStart":666.1999998092651,"domainLookupEnd":666.1999998092651,"domainLookupStart":666.1999998092651,"fetchStart":666.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":666.1999998092651,"responseEnd":1496.7999997138977,"responseStart":1496.7999997138977,"secureConnectionStart":666.1999998092651},{"duration":831.7999997138977,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":666.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":666.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1498.1999998092651,"responseStart":0,"secureConnectionStart":0},{"duration":831.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":666.5999999046326,"connectEnd":666.5999999046326,"connectStart":666.5999999046326,"domainLookupEnd":666.5999999046326,"domainLookupStart":666.5999999046326,"fetchStart":666.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":666.5999999046326,"responseEnd":1498,"responseStart":1498,"secureConnectionStart":666.5999999046326},{"duration":1728.6999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":673.0999999046326,"connectEnd":673.0999999046326,"connectStart":673.0999999046326,"domainLookupEnd":673.0999999046326,"domainLookupStart":673.0999999046326,"fetchStart":673.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":673.0999999046326,"responseEnd":2401.7999997138977,"responseStart":2401.7999997138977,"secureConnectionStart":673.0999999046326},{"duration":1949.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":674.6999998092651,"connectEnd":674.6999998092651,"connectStart":674.6999998092651,"domainLookupEnd":674.6999998092651,"domainLookupStart":674.6999998092651,"fetchStart":674.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":674.6999998092651,"responseEnd":2624.5,"responseStart":2624.5,"secureConnectionStart":674.6999998092651},{"duration":930.1000003814697,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1470.7999997138977,"connectEnd":1470.7999997138977,"connectStart":1470.7999997138977,"domainLookupEnd":1470.7999997138977,"domainLookupStart":1470.7999997138977,"fetchStart":1470.7999997138977,"redirectEnd":0,"redirectStart":0,"requestStart":1470.7999997138977,"responseEnd":2400.9000000953674,"responseStart":2400.9000000953674,"secureConnectionStart":1470.7999997138977},{"duration":903,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1841.1999998092651,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1841.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":2744.199999809265,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":412,"responseStart":659,"responseEnd":672,"domLoading":663,"domInteractive":2667,"domContentLoadedEventStart":2667,"domContentLoadedEventEnd":2740,"domComplete":3459,"loadEventStart":3459,"loadEventEnd":3460,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":2627.699999809265},{"name":"bigPipe.sidebar-id.end","time":2628.699999809265},{"name":"bigPipe.activity-panel-pipe-id.start","time":2628.7999997138977},{"name":"bigPipe.activity-panel-pipe-id.end","time":2634},{"name":"activityTabFullyLoaded","time":2765}],"measures":[],"correlationId":"e53a57e0339871","effectiveType":"4g","downlink":9.7,"rtt":0,"serverDuration":175,"dbReadsTimeInMs":40,"dbConnsTimeInMs":52,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Major