info.zip contains info log for the redistribute logs.

Before the resdistribute for the largest table, the extents per dbroot were (order by dbroot followed by total):

ros_fin_mkt_insm	6203	6207	6202	6192	6200	31004

After the redistribute:

ros_fin_mkt_insm	12410	6202	6192	6200		31004

I think there are a couple of bugs causing the move to dbroot1 for most partitions.

For the first case in RedistributeControlThread::makeRedistributePlan():

• L308 :if (dbPartVec[*j].size() > average) In this case every dbroot is less than the calculated average of the target set (7751) so nothing ends up in the first pass algorithm.
• L397: all of the dbroot2 partitions are falling into this logic but it's only really designed to handle 1 lagging entry so it always picks 1 dbroot to move these too, in this case dbroot1.

Probably only one of these loops needs fixing.

I can't yet reason why a subsequent redistribute does nothing though as dbroot1 very clearly has more partitions than the others and there would be no remove to confuse the algorithm.

To reproduce both cases use the following to generate a large amount of test data:

rm -f t.tbl

for m in {1..12}; do

    for i in {1..31}; do 

       for j in {1..10}; do echo "2017-$(printf %02d $m)-$(printf %02d $i)|$j"; done > t0.tbl

       for k in {1..5}; do  

         cat t0.tbl t0.tbl t0.tbl t0.tbl t0.tbl t0.tbl t0.tbl t0.tbl t0.tbl t0.tbl  > t1.tbl 

         rm -f t0.tbl; cp t1.tbl t0.tbl

       done

       cat t0.tbl >> t.tbl; rm -f t0.tbl t1.tbl

    done

done

create an instance with multiple dbroots, e.g. 4. Then cpimport this into:

create table t1(dt date, i int) engine=columnstore;

it will complain about a few invalid values because the data generator is not month aware so it generates 2017-02-31 etc.

Now do a redistribute remove:

mcsadmin 

> redistributeData start remove 2

- redistributeData status

After this you'll see that dbroot1 has all the entries from dbroot2:

MariaDB [information_schema]> select object_id, dbroot, count(*) from columnstore_extents where object_id > 3000 group by 1,2;

+-----------+--------+----------+

| object_id | dbroot | count(*) |

+-----------+--------+----------+

|      3001 |      1 |       23 |

|      3001 |      3 |       11 |

|      3001 |      4 |       11 |

|      3002 |      1 |       23 |

|      3002 |      3 |       11 |

|      3002 |      4 |       11 |

+-----------+--------+----------+

6 rows in set (0.02 sec)

Now unassign dbroot2 and do another redistribute:

mcsadmin

> stopsystem

> unassignDbrootPmConfig 2 pm1

> startsystem

> redistributeData start

Nothing happens, and the dbroot info schema report is the same.

Build tested: Github source 1.1.0

[root@localhost mariadb-columnstore-server]# git show
commit 60f2f261f81d994307762d6d93380873513a0be8
Author: david hill <david.hill@mariadb.com>
Date: Tue Jun 20 16:05:32 2017 -0500

[root@localhost mariadb-columnstore-server]# cd mariadb-columnstore-engine/
[root@localhost mariadb-columnstore-engine]# git show
commit b424a02c2e431be6f4cf39c12f0c7fd0155c23f5
Author: david hill <david.hill@mariadb.com>
Date: Mon Jun 26 09:58:38 2017 -0500

Reproduced the issue with 100gb dbt3 database.
Also noticed an issue where redistributedata (without the remove option) did not redistribute the dta evenly.

before

[root@localhost columnstore]# du -sh *
111M bin
1.2M data
19G data1
15G data2
15G data3
15G data4
17G data5
176K etc

mcsadmin> redistributedata start remove 2
redistributedata Tue Jun 27 19:17:20 2017
redistributeData START Removing dbroots: 2
Source dbroots: 1 2 3 4 5
Destination dbroots: 1 3 4 5

mcsadmin> redistributedata status
redistributedata Tue Jun 27 19:21:38 2017
WriteEngineServer returned status 2: Redistribute is in progress: total 5 logical partitions are planned to move

after

[root@localhost columnstore]# du -sh *
111M bin
1.2M data
33G data1
8.0K data2
15G data3
15G data4
17G data5

mcsadmin> redistributedata start
redistributedata Tue Jun 27 19:23:36 2017
redistributeData START
Source dbroots: 1 2 3 4 5
Destination dbroots: 1 2 3 4 5

WriteEngineServer returned status 1: Cleared.
WriteEngineServer returned status 2: Redistribute is started.
mcsadmin> redistributedata status
redistributedata Tue Jun 27 19:23:42 2017
WriteEngineServer returned status 2: Redistribute is in progress: total 1 logical partition is planned to move.

[root@localhost columnstore]# du -sh *
111M bin
1.2M data
33G data1
4.2G data2
15G data3
15G data4
13G data5

Given the example after the REDISTRIBUTE REMOVE action, further redistributes do nothing because the partitions are evenly distributed:

MariaDB [information_schema]> select object_id, dbroot, count(DISTINCT partition_id) from information_schema.columnstore_extents where object_id > 3000 group by 1,2;
---------------------------------------------

---------------------------------------------

---------------------------------------------
6 rows in set (0.08 sec)

Right, the point is the first redistribute doesn't actually rebalance and loads up dbroot1. The second point is there is no workaround to do another redistribute to fix it. Behavior is still the same in 1.1.0

Remember to merge before testing.

Build verified: 1.1.1-1 package

Verified on a 1um4pm stack, with 1 dbroot in each PM.

after loading 10g source twice

Before redistribute

[root@localhost columnstore]# du -sh data1
9.5G data1
[root@localhost columnstore]# du -sh data2
9.3G data2
[root@localhost columnstore]# du -sh data3
8.8G data3
[root@localhost columnstore]# du -sh data4
9.4G data4

After redistribute

[root@localhost columnstore]# du -sh data1
16G data1
[root@localhost columnstore]# du -sh data2
120K data2
[root@localhost columnstore]# du -sh data3
11G data3
[root@localhost columnstore]# du -sh data4
11G data4

after loaded another 100g source, partsupp did not load due to an error

before redistribute

[root@localhost columnstore]# du -sh data1
29G data1
[root@localhost columnstore]# du -sh data2
17G data2
[root@localhost columnstore]# du -sh data3
24G data3
[root@localhost columnstore]# du -sh data4
25G data4

after redistribute

[root@localhost columnstore]# du -sh data1
35G data1
[root@localhost columnstore]# du -sh data2
120K data2
[root@localhost columnstore]#
[root@localhost columnstore]# du -sh data3
29G data3
[root@localhost columnstore]# du -sh data4
30G data4