[MCOL-1781] Faster way to load old innodb tables Created: 2018-10-07 Updated: 2024-01-04 Resolved: 2018-10-10 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | MariaDB Server |
| Affects Version/s: | 1.1.6 |
| Fix Version/s: | N/A |
| Type: | New Feature | Priority: | Major |
| Reporter: | thomas lloancy | Assignee: | Todd Stoffel (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Linux |
||
| Description |
|
I don't know if it's more of a question or a suggestion but here is the thing: If you have to load a huge amount of datas (let's say 200go on a daily basis) to mariadb server from another, it is way more efficient to copy from source and replace the destination /var/lib/mysql directory (Server may take 5min to start) than dump and then import a gigantic sql file. when you try to do that in columnstore, you can see tables appear in target db, but when you want to read them, you see "table doesn't exist as columnstore engine" or something like, probably for the same reason you can't alter table innodbt engine=columnstore. So bask to that, the only way seems to be dump from row mariadb server and import sql file into columnstore server, then create table with engine columnstore equivalent tables but without indexes (show create table can be useful since mariadb columnstore now cover almost all types existing in mariadb) and then do something like insert into columnstore_table select * from row_table. I read a few about cpimport, but is it something to use with select into outfile? Is there a real good method to go from row to column here ? Thank you for your time Thomas |
| Comments |
| Comment by David Thompson (Inactive) [ 2018-10-07 ] |
|
So yes you can copy the mysql db files, however the actual columnar data files are stored seperately (and on different servers should you have a multi node deployment). So if you also copy the data1 - dataN directories (under /usr/local/mariadb/columnstore to the target system as well you should have a working system (for a single node it would all just be on one server. You can also use cpimport combined with a mysql command output: |
| Comment by thomas lloancy [ 2018-10-08 ] |
|
Yep that's right but from one mariadb columnstore server to another. What if you want to do that from normal mariadb server to mariadb columnstore server ? Thomas |
| Comment by David Thompson (Inactive) [ 2018-10-08 ] |
|
you can't get away from re-populating the columnstore table since it's a different storage format. So the most straightforward is the mysql client piped into cpimport above as the source doesn't have to be a columnstore table. It may be possible to copy over the innodb table files first and do insert select with autocommit (which maps to cpimport internally) but there's a few things that you can screw up there especially on a live system with existing tables. |
| Comment by thomas lloancy [ 2018-10-09 ] |
|
Ok thank your for that Thomas |