[MCOL-4439] Installation with S3 storage is much slower then the last release Created: 2020-12-10  Updated: 2020-12-11  Resolved: 2020-12-11

Status: Closed
Project: MariaDB ColumnStore
Component/s: Storage Manager
Affects Version/s: 5.5.1
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Daniel Lee (Inactive) Assignee: Ben Thompson (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None

Attachments: Text File AWS_test.txt    

 Description   

Build tested: 5.5.1 (Drone 1313)

Installation is successful, but create table and LDI are much slower than 5.4.3-1. There is close to 2x of the data in the S3 bucket.

5.5.1-1

Total test run time:
real 19m57.116s
user 0m6.294s
sys 0m3.207s

Create table time:
s1pm1: Query OK, 0 rows affected (1 min 10.077 sec) – orders
s1pm1: Query OK, 0 rows affected (40.523 sec) – lineitem

LDI of 1gb lineitem:
s1pm1: Query OK, 6001215 rows affected (14 min 36.892 sec)
s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

S3 Bucket usage:
AWS S3 storage is used, bucket = dleeqadbroot1, objects = 153, total size = 304130229 (290 MB)

5.4.3-1

Total test run time:
real 11m14.021s
user 0m5.488s
sys 0m2.585s

Create table time:
s1pm1: Query OK, 0 rows affected (1.996 sec) – orders
s1pm1: Query OK, 0 rows affected (0.198 sec) – lineitem

LDI of 1gb lineitem:

s1pm1: Query OK, 6001215 rows affected (6 min 50.879 sec)
s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

S3 bucket usage:
AWS S3 storage is used, bucket = dleeqadbroot1, objects = 131, total size = 184797331 (176 MB)

Detail info for both releases compared:

5.5.1-1

[dlee@aloha tests]$ /data/qa/installtest/test/installtest.sh s1 5.5.1-1 centos8 rpm 1pm.s3cloud.xml n root 1
Using local setup file /home/dlee/.installtest/localSetup.xml
Using local setup file /home/dlee/.installtest/localSetup.xml
Thu Dec 10 11:43:42 CST 2020 Extracting test parameters from test file
Thu Dec 10 11:43:42 CST 2020 Generating test scripts
Using local setup file /home/dlee/.installtest/localSetup.xml
Executing installation test......
StackName=s1 Release=5.5.1-1 OS=centos8 Package=rpm Test=/data/qa/installtest/ctrl/test/5.5.1-1/1pm.s3cloud.xml installMode=n TestUser=root
Release type: Enterprise
Thu Dec 10 11:43:43 CST 2020 Starting test using Vagrant boxes

real 19m57.116s
user 0m6.294s
sys 0m3.207s
s1pm1: ----------
s1pm1: 1 row in set (0.070 sec)
s1pm1:
s1pm1: Bye
s1pm1: No tables are locked in the database.
s1pm1: end runDefaultTest.sh
s1pm1: Thu Dec 10 18:03:38 UTC 2020
s1pm1: Exit status 0
AWS S3 storage is used, bucket = dleeqadbroot1, objects = 153, total size = 304130229 (290 MB)
Thu Dec 10 12:03:41 CST 2020
Correct row counts: 6001215, 1500000
Software versions matched
Expected versions: 10.5.9, 5.5.1
Tested versions= s1pm1: Server version: 10.5.9-MariaDB MariaDB Server, s1pm1: | Columnstore_version | 5.5.1 |
Passed /home/dlee/tests/5.5.1-1/centos8/root.rpm.1pm.s3.n.s1
Thu Dec 10 12:03:41 CST 2020 Pau......
[dlee@aloha tests]$

s1pm1: create table if not exists orders (
s1pm1: o_orderkey int,
s1pm1: o_custkey int,
s1pm1: o_orderstatus char (1),
s1pm1: o_totalprice decimal(12,2),
s1pm1: o_orderdate date,
s1pm1: o_orderpriority char (15),
s1pm1: o_clerk char (15),
s1pm1: o_shippriority int,
s1pm1: o_comment varchar (79)
s1pm1: ) engine=Columnstore
s1pm1: --------------
s1pm1:
s1pm1: Query OK, 0 rows affected (1 min 10.077 sec)
s1pm1:
s1pm1: --------------
s1pm1: create table if not exists lineitem (
s1pm1: l_orderkey int,
s1pm1: l_partkey int,
s1pm1: l_suppkey int,
s1pm1: l_linenumber bigint,
s1pm1: l_quantity decimal(12,2),
s1pm1: l_extendedprice decimal(12,2),
s1pm1: l_discount decimal(12,2),
s1pm1: l_tax decimal(12,2),
s1pm1: l_returnflag char (1),
s1pm1: l_linestatus char (1),
s1pm1: l_shipdate date,
s1pm1: l_commitdate date,
s1pm1: l_receiptdate date,
s1pm1: l_shipinstruct char (25),
s1pm1: l_shipmode char (10),
s1pm1: l_comment varchar (44)
s1pm1: ) engine=columnstore
s1pm1: --------------
s1pm1:
s1pm1: Query OK, 0 rows affected (40.523 sec)
s1pm1:
s1pm1: --------------
s1pm1: load data infile
s1pm1:
s1pm1: "/data/qa/source/dbt3/1g/lineitem.tbl" into table lineitem fields terminated by "|"
s1pm1: --------------
s1pm1:
s1pm1: Query OK, 6001215 rows affected (14 min 36.892 sec)
s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

5.4.3-1

[dlee@aloha tests]$ /data/qa/installtest/test/installtest.sh s1 5.4.3-1 centos8 rpm 1pm.s3cloud.xml n root 1
Using local setup file /home/dlee/.installtest/localSetup.xml
Using local setup file /home/dlee/.installtest/localSetup.xml
Thu Dec 10 11:27:56 CST 2020 Extracting test parameters from test file
Thu Dec 10 11:27:57 CST 2020 Generating test scripts
Using local setup file /home/dlee/.installtest/localSetup.xml
Executing installation test......
StackName=s1 Release=5.4.3-1 OS=centos8 Package=rpm Test=/data/qa/installtest/ctrl/test/5.4.3-1/1pm.s3cloud.xml installMode=n TestUser=root
Release type: hotfix
Thu Dec 10 11:27:57 CST 2020 Starting test using Vagrant boxes

real 11m14.021s
user 0m5.488s
sys 0m2.585s
s1pm1: ----------
s1pm1: 1 row in set (0.094 sec)
s1pm1:
s1pm1: Bye
s1pm1: No tables are locked in the database.
s1pm1: end runDefaultTest.sh
s1pm1: Thu Dec 10 17:39:09 UTC 2020
s1pm1: Exit status 0
AWS S3 storage is used, bucket = dleeqadbroot1, objects = 131, total size = 184797331 (176 MB)
Thu Dec 10 11:39:13 CST 2020
Correct row counts: 6001215, 1500000
Software versions matched
Expected versions: 10.5.6-4, 5.4.3
Tested versions= s1pm1: Server version: 10.5.6-4.5.4.3-MariaDB-enterprise MariaDB Enterprise Server, s1pm1: | Columnstore_version | 5.4.3 |
Passed /home/dlee/tests/5.4.3-1/centos8/root.rpm.1pm.s3.n.s1
Thu Dec 10 11:39:13 CST 2020 Pau......

create table if not exists orders (
s1pm1: o_orderkey int,
s1pm1: o_custkey int,
s1pm1: o_orderstatus char (1),
s1pm1: o_totalprice decimal(12,2),
s1pm1: o_orderdate date,
s1pm1: o_orderpriority char (15),
s1pm1: o_clerk char (15),
s1pm1: o_shippriority int,
s1pm1: o_comment varchar (79)
s1pm1: ) engine=Columnstore
s1pm1: --------------
s1pm1:
s1pm1: Query OK, 0 rows affected (1.996 sec)
s1pm1:
s1pm1: --------------
s1pm1: create table if not exists lineitem (
s1pm1: l_orderkey int,
s1pm1: l_partkey int,
s1pm1: l_suppkey int,
s1pm1: l_linenumber bigint,
s1pm1: l_quantity decimal(12,2),
s1pm1: l_extendedprice decimal(12,2),
s1pm1: l_discount decimal(12,2),
s1pm1: l_tax decimal(12,2),
s1pm1: l_returnflag char (1),
s1pm1: l_linestatus char (1),
s1pm1: l_shipdate date,
s1pm1: l_commitdate date,
s1pm1: l_receiptdate date,
s1pm1: l_shipinstruct char (25),
s1pm1: l_shipmode char (10),
s1pm1: l_comment varchar (44)
s1pm1: ) engine=columnstore
s1pm1: --------------
s1pm1:
s1pm1: Query OK, 0 rows affected (0.198 sec)
s1pm1:
s1pm1: --------------
s1pm1: load data infile "/data/qa/source/dbt3/1
s1pm1: g
s1pm1: /lineitem.tbl" into table lineitem fields terminated by "|"
s1pm1: --------------
s1pm1:
s1pm1: Query OK, 6001215 rows affected (6 min 50.879 sec)
s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0



 Comments   
Comment by Todd Stoffel (Inactive) [ 2020-12-11 ]

As Ben pointed out this was likely related to VM connections to remote S3 bucket. Closing at not a bug.

Generated at Thu Feb 08 02:50:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.