Increase the parallelism, MDEV-32216 using --tab option we can able to backup only one database at a time.
In mariadb-dump --tab, the binlog positions are not stored in a separate file. It is displayed in stdout
The big tables were split into multiple data files which will help faster restore.
this should be done when bulk-load in multiple connection becomes faster
Other features that could be added to mariadb-dump that would make it more useful:
Compression of the dump (especially with --tab). This also implies that we would implement de-compression of the files ion the server (MDEV-28395).
Dump tables not alphabetically, but in inverse size order (biggest tables first). This will help ensuring that the user does not have to wait for the big tables to finish at the end of the backup (in most cases). See benchmarks in MDEV-32216.
Restore:
Using mariadb-import can restore only one database at a time and the user need to restore the table structure and data files separately. We should enhance mariadb-import with an option to re-create the table structure first and then start the data import.
In case of customers having 100+ databases mariadb-dump (with --tab) and mariadb-import are not user-friendly. (Not hard to fix)
Have mariadb-import in parallel mode start importing the tables in inverse size order (biggest tables first). This will help ensuring that the user does not have to wait for the big tables to finish at the end of the backup (in most cases).
We need to improve the functionality of mariadb-dump and mariadb-import to allow for the backup and restoration of multiple databases using a single command.
Description of the intended design
allow multiple databases with a new switch --dir (suggestions for a better name welcome, as I got no good idea at the moment)
With this set, mariadb-dump creates a directory tree structure in the given path, <path>/dbname/ for each db, and <tablename>.txt (tab-separated data, created using SELECT INTO OUTFILE) and tablename.sql with DDL . for all tables under the corresponding db directory.
The files are almost exactly the same as before with --tab, except for directory tree
mariadb-import also takes new --dir parameter, that points to directory tree created by dump, executes all *.sql files to create tables (and databases if not exist), loads the data using "LOAD DATA INFILE" for the .txt files
Big tables won't be split into small tables now, and I do not think they should be in the future. If required, threading should be handled transparently by LOAD DATA INFILE, i.e by the server, rather than loading single table from 2 different connections. LOAD DATA is our officiaL bulk-loading interface, and we better improve it in the server, rather than building workarounds.
Attachments
Issue Links
is blocked by
MDEV-34719Disable purge for LOAD DATA INFILE into empty table
Closed
MDEV-34740mariadb-import: delay creation of secondary indexes until after data load
Closed
relates to
MDEV-34703Innodb bulk load : high IO, crashes on Linux, OOM on Linux, as tested with LOAD DATA INFILE
Closed
MDEV-34739Implement DISABLE KEYS/ENABLE KEYS in Innodb
Open
MDEV-34832Support adding AUTO_INCREMENT flag to existing numeric using INPLACE
--single-transaction can be emulated with a global read lock, I suppose.
--hex-blob is only defined for SQL dumps. And --tab uses SELECT ... INTO OUTFILE. This needs to be implemented in the server. And for LOAD DATA too.
Sergei Golubchik
added a comment - --single-transaction can be emulated with a global read lock, I suppose.
--hex-blob is only defined for SQL dumps. And --tab uses SELECT ... INTO OUTFILE . This needs to be implemented in the server. And for LOAD DATA too.
Thanks for the updated information. I have no idea how complex it would be, to add support for --hex-blob. I knew it wasn't just one line of code!
I have just looked at performance charts for mydumper and other alternatives, and it's quite massive compared to plain old SQL-dumps.
Johan Eklund
added a comment - Thanks for the updated information. I have no idea how complex it would be, to add support for --hex-blob. I knew it wasn't just one line of code!
I have just looked at performance charts for mydumper and other alternatives, and it's quite massive compared to plain old SQL-dumps.
I'm unsure that AUTO_INCREMENT is better to add after loading data. Why so? When we load data, we define all fields. We do not calculate autoincrement. If there is still overhead, even when autoinc field is defined, it needs to be solved by storage engines.
Vladislav Vaintroub
added a comment - - edited I'm unsure that AUTO_INCREMENT is better to add after loading data. Why so? When we load data, we define all fields. We do not calculate autoincrement. If there is still overhead, even when autoinc field is defined, it needs to be solved by storage engines.
People
Vladislav Vaintroub
Pandikrishnan Gurusamy
Votes:
0Vote for this issue
Watchers:
13Start watching this issue
Dates
Created:
Updated:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":892.5,"ttfb":283.2999999523163,"pageVisibility":"visible","entityId":127172,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"d6909da6-ee59-4482-b20f-e0491700ac7e","navigationType":0,"readyForUser":957.2999999523163,"redirectCount":0,"resourceLoadedEnd":993.0999999046326,"resourceLoadedStart":288.5,"resourceTiming":[{"duration":127.20000004768372,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":288.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":288.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":415.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":127.10000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":288.89999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":288.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":416,"responseStart":0,"secureConnectionStart":0},{"duration":143.79999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":289,"connectEnd":289,"connectStart":289,"domainLookupEnd":289,"domainLookupStart":289,"fetchStart":289,"redirectEnd":0,"redirectStart":0,"requestStart":289,"responseEnd":432.7999999523163,"responseStart":432.7999999523163,"secureConnectionStart":289},{"duration":243.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":289.2000000476837,"connectEnd":289.2000000476837,"connectStart":289.2000000476837,"domainLookupEnd":289.2000000476837,"domainLookupStart":289.2000000476837,"fetchStart":289.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":289.2000000476837,"responseEnd":532.2999999523163,"responseStart":532.2999999523163,"secureConnectionStart":289.2000000476837},{"duration":246.20000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":289.2999999523163,"connectEnd":289.2999999523163,"connectStart":289.2999999523163,"domainLookupEnd":289.2999999523163,"domainLookupStart":289.2999999523163,"fetchStart":289.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":289.2999999523163,"responseEnd":535.5,"responseStart":535.5,"secureConnectionStart":289.2999999523163},{"duration":246.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":289.5,"connectEnd":289.5,"connectStart":289.5,"domainLookupEnd":289.5,"domainLookupStart":289.5,"fetchStart":289.5,"redirectEnd":0,"redirectStart":0,"requestStart":289.5,"responseEnd":536,"responseStart":536,"secureConnectionStart":289.5},{"duration":246.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":289.7000000476837,"connectEnd":289.7000000476837,"connectStart":289.7000000476837,"domainLookupEnd":289.7000000476837,"domainLookupStart":289.7000000476837,"fetchStart":289.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":289.7000000476837,"responseEnd":536.2999999523163,"responseStart":536.2999999523163,"secureConnectionStart":289.7000000476837},{"duration":285.2000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":289.89999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":289.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":575.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":246.70000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":290.09999990463257,"connectEnd":290.09999990463257,"connectStart":290.09999990463257,"domainLookupEnd":290.09999990463257,"domainLookupStart":290.09999990463257,"fetchStart":290.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":290.09999990463257,"responseEnd":536.7999999523163,"responseStart":536.7999999523163,"secureConnectionStart":290.09999990463257},{"duration":284.90000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":290.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":290.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":575.2000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":247,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":290.39999985694885,"connectEnd":290.39999985694885,"connectStart":290.39999985694885,"domainLookupEnd":290.39999985694885,"domainLookupStart":290.39999985694885,"fetchStart":290.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":290.39999985694885,"responseEnd":537.3999998569489,"responseStart":537.3999998569489,"secureConnectionStart":290.39999985694885},{"duration":648.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":296.7999999523163,"connectEnd":296.7999999523163,"connectStart":296.7999999523163,"domainLookupEnd":296.7999999523163,"domainLookupStart":296.7999999523163,"fetchStart":296.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":296.7999999523163,"responseEnd":945.2999999523163,"responseStart":945.2000000476837,"secureConnectionStart":296.7999999523163},{"duration":648.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":296.89999985694885,"connectEnd":296.89999985694885,"connectStart":296.89999985694885,"domainLookupEnd":296.89999985694885,"domainLookupStart":296.89999985694885,"fetchStart":296.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":296.89999985694885,"responseEnd":945.7000000476837,"responseStart":945.7000000476837,"secureConnectionStart":296.89999985694885},{"duration":152,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":587.5999999046326,"connectEnd":587.5999999046326,"connectStart":587.5999999046326,"domainLookupEnd":587.5999999046326,"domainLookupStart":587.5999999046326,"fetchStart":587.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":587.5999999046326,"responseEnd":739.5999999046326,"responseStart":739.5999999046326,"secureConnectionStart":587.5999999046326},{"duration":148.39999985694885,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":844.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":844.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":992.8999998569489,"responseStart":0,"secureConnectionStart":0},{"duration":148,"initiatorType":"link","name":"https://jira.mariadb.org/s/50bc9be5bfead1a25e72c1a9338c94f6-CDN/lu2cib/820016/12ta74/03ec8e95fb0700cf309083e2358eb54b/_/download/contextbatch/css/com.atlassian.jira.plugins.jira-development-integration-plugin:0,com.atlassian.jira.plugins.jira-quicksearch-plugin:5,-_super,-jira.view.issue,-jira.global,-jira.general,-jira.browse.project,-project.issue.navigator,-atl.general/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":845.0999999046326,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":845.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":993.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":104.60000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":845.8999998569489,"connectEnd":845.8999998569489,"connectStart":845.8999998569489,"domainLookupEnd":845.8999998569489,"domainLookupStart":845.8999998569489,"fetchStart":845.8999998569489,"redirectEnd":0,"redirectStart":0,"requestStart":845.8999998569489,"responseEnd":950.5,"responseStart":950.5,"secureConnectionStart":845.8999998569489},{"duration":114.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":846.2999999523163,"connectEnd":846.2999999523163,"connectStart":846.2999999523163,"domainLookupEnd":846.2999999523163,"domainLookupStart":846.2999999523163,"fetchStart":846.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":846.2999999523163,"responseEnd":960.3999998569489,"responseStart":960.3999998569489,"secureConnectionStart":846.2999999523163},{"duration":115.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d44aa1ee0c678dcc675ac6ce6f14e4a-CDN/lu2cib/820016/12ta74/03ec8e95fb0700cf309083e2358eb54b/_/download/contextbatch/js/com.atlassian.jira.plugins.jira-development-integration-plugin:0,com.atlassian.jira.plugins.jira-quicksearch-plugin:5,-_super,-jira.view.issue,-jira.global,-jira.general,-jira.browse.project,-project.issue.navigator,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":846.5999999046326,"connectEnd":846.5999999046326,"connectStart":846.5999999046326,"domainLookupEnd":846.5999999046326,"domainLookupStart":846.5999999046326,"fetchStart":846.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":846.5999999046326,"responseEnd":962.0999999046326,"responseStart":962.0999999046326,"secureConnectionStart":846.5999999046326},{"duration":158.30000019073486,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":886.3999998569489,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":886.3999998569489,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1044.7000000476837,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":16,"domainLookupEnd":41,"connectStart":41,"connectEnd":62,"secureConnectionStart":50,"requestStart":62,"responseStart":283,"responseEnd":295,"domLoading":286,"domInteractive":1090,"domContentLoadedEventStart":1090,"domContentLoadedEventEnd":1150,"domComplete":1408,"loadEventStart":1408,"loadEventEnd":1410,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1048.5999999046326},{"name":"bigPipe.sidebar-id.end","time":1049.3999998569489},{"name":"bigPipe.activity-panel-pipe-id.start","time":1049.5999999046326},{"name":"bigPipe.activity-panel-pipe-id.end","time":1051.8999998569489},{"name":"activityTabFullyLoaded","time":1167.5}],"measures":[],"correlationId":"b88a5e201c3ab4","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":108,"dbReadsTimeInMs":18,"dbConnsTimeInMs":27,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
--single-transaction can be emulated with a global read lock, I suppose.
--hex-blob is only defined for SQL dumps. And --tab uses SELECT ... INTO OUTFILE. This needs to be implemented in the server. And for LOAD DATA too.