[MCOL-5089] Merge RBTree-based Extent Map with EM Index to remove scaleability slow-downs -develop5 Created: 2022-05-16  Updated: 2022-08-17  Resolved: 2022-07-29

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 5.6.5
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: Roman Assignee: David Hall (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
includes MCOL-5090 Up-merge EMIndex + RBTree-based EM in... Closed
includes MCOL-5091 Up-merge RBTree-based EM into develop Closed
is part of MCOL-4912 MCS bulk insertion is slow Closed
is part of MCOL-4917 Certain Extent Map operations doesn't... Closed
Sprint: 2021-17

 Description   

This is a final merge of the two features introduced to reduce ExtentMap operations slowdowns caused by a significant Extent Map.
In the end MCS operations, e.g. INSERT, bulk ingestion, SELECT must operate almost as fast as they did with an empty ExtentMap.

4 QA There are two aspects of this project:

  • functionality(there must be no regression) so you need to test overal MCS functionality(pay attention to bulk ingestion running in different modes and rollbacks).
  • overall speedup when used with Extent Maps(> 50MB). There is compressed extent map image(BRM_saved_em.tar.bz2) from one of customers I will share that should be used for testing. One needs to replace empty EM file with the image and compare timings for a variety of operations: INSERT, bulk ingestion, rollback, deletes. Plz note that the previous stable MCS 5 version already contains the subset of the functionality. There is another important detail, namely with the current code Extent Map takes about 5 times more then previously in shared memory, e.g. 300 MB image will now takes 1.2GB for Extent Map + 400 MB for index. We will address the extent map size after this patch.

 Comments   
Comment by Roman [ 2022-05-16 ]

Previously MCS5 w/o MCOL-4912, MCOL-4917 test000 converges in roughly 30 minutes. With MCOL-4917 + MCOL-4912 it takes 1 hour and 30 minutes(rbtree traversal takes more then simple array traversal). With the merge patch it takes 4 minutes, 40 seconds as usually. All runs were done are with EM image I mentioned.

Comment by Roman [ 2022-05-17 ]

This issue says nothing about develop or develop-6 so develop-5.

Comment by Daniel Lee (Inactive) [ 2022-05-19 ]

Build tested: 5.6.6-1 (Drone 4443)

Unabled to reproduce the slowness using an older release 5.6.4-2 using the provided EM file. MCS is unable to load the large EM file. Therefore, got base line timing for the release when EM is virtually empty.

With 5.6.6-1, timings for both empty and large EMs are virutally the same, which is also comparable to the timing to 5.6.4-2. It has been verified that a large EM no longer slow down LDI, query, and DMLs.

5.6.6-1
-------
----- empty dbrm

[centos8:root~]# time /data/qa/autopilot/databases/dbt3/sh/buildDatabase.sh tpch10 columnstore 10g

real 1m20.191s
user 3m40.372s
sys 0m6.609s

[centos8:root~]# time mariadb tpch10 < /data/qa/autopilot/performance/dbt3/sql/10g/19.sql
revenue
37135011.7543

real 0m41.344s
user 0m0.008s
sys 0m0.020s
[centos8:root~]#

MariaDB [tpch10]> update lineitem set l_orderkey=l_partkey, l_shipdate = l_commitdate;
Query OK, 59986052 rows affected (4 min 26.296 sec)
Rows matched: 59986052 Changed: 59986052 Warnings: 0

MariaDB [tpch10]> delete from orders;
Query OK, 15000000 rows affected (56.704 sec)

----- large dbrm

[centos8:root~]# time /data/qa/autopilot/databases/dbt3/sh/buildDatabase.sh tpch10 columnstore 10g

real 1m21.478s
user 3m39.537s
sys 0m6.737s

[centos8:root~]# time mariadb tpch10 < /data/qa/autopilot/performance/dbt3/sql/10g/19.sql
revenue
37135011.7543

real 0m40.729s
user 0m0.025s
sys 0m0.005s

MariaDB [tpch10]> update lineitem set l_orderkey=l_partkey, l_shipdate = l_commitdate;
Query OK, 59986052 rows affected (4 min 17.451 sec)
Rows matched: 59986052 Changed: 59986052 Warnings: 0

MariaDB [tpch10]> delete from orders;
Query OK, 15000000 rows affected (1 min 3.001 sec)

5.6.4-2
-------
----- empty dbrm

time /data/qa/autopilot/databases/dbt3/sh/buildDatabase.sh tpch10 columnstore 10g

real 1m23.342s
user 3m43.255s
sys 0m6.843s

[centos8:root~]# time mariadb tpch10 < /data/qa/autopilot/performance/dbt3/sql/10g/19.sql
revenue
37135011.7543

real 0m42.270s
user 0m0.020s
sys 0m0.009s
[centos8:root~]#

MariaDB [tpch10]> update lineitem set l_orderkey=l_partkey, l_shipdate = l_commitdate;
Query OK, 59986052 rows affected (4 min 21.257 sec)
Rows matched: 59986052 Changed: 59986052 Warnings: 0

MariaDB [tpch10]> delete from orders;
Query OK, 15000000 rows affected (1 min 4.749 sec)

Comment by David Hall (Inactive) [ 2022-06-03 ]

The 5.6.6-1 beta test build can be found at:
https://es-repo.mariadb.net/jenkins/ENTERPRISE/bb-10.5.16-11-cs-6.5.6-1/cee626c87b62b893dc539cf2af39a5521f40a27e/

Comment by Roman [ 2022-06-10 ]

Plz review the bug fix.

Comment by Roman [ 2022-06-10 ]

David.Hall Could you make a package from develop-5?

Generated at Thu Feb 08 02:55:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.