Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4439

Installation with S3 storage is much slower then the last release

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Not a Bug
    • Affects Version/s: 5.5.1
    • Fix Version/s: N/A
    • Component/s: Storage Manager
    • Labels:
      None

      Description

      Build tested: 5.5.1 (Drone 1313)

      Installation is successful, but create table and LDI are much slower than 5.4.3-1. There is close to 2x of the data in the S3 bucket.

      5.5.1-1

      Total test run time:
      real 19m57.116s
      user 0m6.294s
      sys 0m3.207s

      Create table time:
      s1pm1: Query OK, 0 rows affected (1 min 10.077 sec) – orders
      s1pm1: Query OK, 0 rows affected (40.523 sec) – lineitem

      LDI of 1gb lineitem:
      s1pm1: Query OK, 6001215 rows affected (14 min 36.892 sec)
      s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

      S3 Bucket usage:
      AWS S3 storage is used, bucket = dleeqadbroot1, objects = 153, total size = 304130229 (290 MB)

      5.4.3-1

      Total test run time:
      real 11m14.021s
      user 0m5.488s
      sys 0m2.585s

      Create table time:
      s1pm1: Query OK, 0 rows affected (1.996 sec) – orders
      s1pm1: Query OK, 0 rows affected (0.198 sec) – lineitem

      LDI of 1gb lineitem:

      s1pm1: Query OK, 6001215 rows affected (6 min 50.879 sec)
      s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

      S3 bucket usage:
      AWS S3 storage is used, bucket = dleeqadbroot1, objects = 131, total size = 184797331 (176 MB)

      Detail info for both releases compared:

      5.5.1-1

      [dlee@aloha tests]$ /data/qa/installtest/test/installtest.sh s1 5.5.1-1 centos8 rpm 1pm.s3cloud.xml n root 1
      Using local setup file /home/dlee/.installtest/localSetup.xml
      Using local setup file /home/dlee/.installtest/localSetup.xml
      Thu Dec 10 11:43:42 CST 2020 Extracting test parameters from test file
      Thu Dec 10 11:43:42 CST 2020 Generating test scripts
      Using local setup file /home/dlee/.installtest/localSetup.xml
      Executing installation test......
      StackName=s1 Release=5.5.1-1 OS=centos8 Package=rpm Test=/data/qa/installtest/ctrl/test/5.5.1-1/1pm.s3cloud.xml installMode=n TestUser=root
      Release type: Enterprise
      Thu Dec 10 11:43:43 CST 2020 Starting test using Vagrant boxes

      real 19m57.116s
      user 0m6.294s
      sys 0m3.207s
      s1pm1: ----------
      s1pm1: 1 row in set (0.070 sec)
      s1pm1:
      s1pm1: Bye
      s1pm1: No tables are locked in the database.
      s1pm1: end runDefaultTest.sh
      s1pm1: Thu Dec 10 18:03:38 UTC 2020
      s1pm1: Exit status 0
      AWS S3 storage is used, bucket = dleeqadbroot1, objects = 153, total size = 304130229 (290 MB)
      Thu Dec 10 12:03:41 CST 2020
      Correct row counts: 6001215, 1500000
      Software versions matched
      Expected versions: 10.5.9, 5.5.1
      Tested versions= s1pm1: Server version: 10.5.9-MariaDB MariaDB Server, s1pm1: | Columnstore_version | 5.5.1 |
      Passed /home/dlee/tests/5.5.1-1/centos8/root.rpm.1pm.s3.n.s1
      Thu Dec 10 12:03:41 CST 2020 Pau......
      [dlee@aloha tests]$

      s1pm1: create table if not exists orders (
      s1pm1: o_orderkey int,
      s1pm1: o_custkey int,
      s1pm1: o_orderstatus char (1),
      s1pm1: o_totalprice decimal(12,2),
      s1pm1: o_orderdate date,
      s1pm1: o_orderpriority char (15),
      s1pm1: o_clerk char (15),
      s1pm1: o_shippriority int,
      s1pm1: o_comment varchar (79)
      s1pm1: ) engine=Columnstore
      s1pm1: --------------
      s1pm1:
      s1pm1: Query OK, 0 rows affected (1 min 10.077 sec)
      s1pm1:
      s1pm1: --------------
      s1pm1: create table if not exists lineitem (
      s1pm1: l_orderkey int,
      s1pm1: l_partkey int,
      s1pm1: l_suppkey int,
      s1pm1: l_linenumber bigint,
      s1pm1: l_quantity decimal(12,2),
      s1pm1: l_extendedprice decimal(12,2),
      s1pm1: l_discount decimal(12,2),
      s1pm1: l_tax decimal(12,2),
      s1pm1: l_returnflag char (1),
      s1pm1: l_linestatus char (1),
      s1pm1: l_shipdate date,
      s1pm1: l_commitdate date,
      s1pm1: l_receiptdate date,
      s1pm1: l_shipinstruct char (25),
      s1pm1: l_shipmode char (10),
      s1pm1: l_comment varchar (44)
      s1pm1: ) engine=columnstore
      s1pm1: --------------
      s1pm1:
      s1pm1: Query OK, 0 rows affected (40.523 sec)
      s1pm1:
      s1pm1: --------------
      s1pm1: load data infile
      s1pm1:
      s1pm1: "/data/qa/source/dbt3/1g/lineitem.tbl" into table lineitem fields terminated by "|"
      s1pm1: --------------
      s1pm1:
      s1pm1: Query OK, 6001215 rows affected (14 min 36.892 sec)
      s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

      5.4.3-1

      [dlee@aloha tests]$ /data/qa/installtest/test/installtest.sh s1 5.4.3-1 centos8 rpm 1pm.s3cloud.xml n root 1
      Using local setup file /home/dlee/.installtest/localSetup.xml
      Using local setup file /home/dlee/.installtest/localSetup.xml
      Thu Dec 10 11:27:56 CST 2020 Extracting test parameters from test file
      Thu Dec 10 11:27:57 CST 2020 Generating test scripts
      Using local setup file /home/dlee/.installtest/localSetup.xml
      Executing installation test......
      StackName=s1 Release=5.4.3-1 OS=centos8 Package=rpm Test=/data/qa/installtest/ctrl/test/5.4.3-1/1pm.s3cloud.xml installMode=n TestUser=root
      Release type: hotfix
      Thu Dec 10 11:27:57 CST 2020 Starting test using Vagrant boxes

      real 11m14.021s
      user 0m5.488s
      sys 0m2.585s
      s1pm1: ----------
      s1pm1: 1 row in set (0.094 sec)
      s1pm1:
      s1pm1: Bye
      s1pm1: No tables are locked in the database.
      s1pm1: end runDefaultTest.sh
      s1pm1: Thu Dec 10 17:39:09 UTC 2020
      s1pm1: Exit status 0
      AWS S3 storage is used, bucket = dleeqadbroot1, objects = 131, total size = 184797331 (176 MB)
      Thu Dec 10 11:39:13 CST 2020
      Correct row counts: 6001215, 1500000
      Software versions matched
      Expected versions: 10.5.6-4, 5.4.3
      Tested versions= s1pm1: Server version: 10.5.6-4.5.4.3-MariaDB-enterprise MariaDB Enterprise Server, s1pm1: | Columnstore_version | 5.4.3 |
      Passed /home/dlee/tests/5.4.3-1/centos8/root.rpm.1pm.s3.n.s1
      Thu Dec 10 11:39:13 CST 2020 Pau......

      create table if not exists orders (
      s1pm1: o_orderkey int,
      s1pm1: o_custkey int,
      s1pm1: o_orderstatus char (1),
      s1pm1: o_totalprice decimal(12,2),
      s1pm1: o_orderdate date,
      s1pm1: o_orderpriority char (15),
      s1pm1: o_clerk char (15),
      s1pm1: o_shippriority int,
      s1pm1: o_comment varchar (79)
      s1pm1: ) engine=Columnstore
      s1pm1: --------------
      s1pm1:
      s1pm1: Query OK, 0 rows affected (1.996 sec)
      s1pm1:
      s1pm1: --------------
      s1pm1: create table if not exists lineitem (
      s1pm1: l_orderkey int,
      s1pm1: l_partkey int,
      s1pm1: l_suppkey int,
      s1pm1: l_linenumber bigint,
      s1pm1: l_quantity decimal(12,2),
      s1pm1: l_extendedprice decimal(12,2),
      s1pm1: l_discount decimal(12,2),
      s1pm1: l_tax decimal(12,2),
      s1pm1: l_returnflag char (1),
      s1pm1: l_linestatus char (1),
      s1pm1: l_shipdate date,
      s1pm1: l_commitdate date,
      s1pm1: l_receiptdate date,
      s1pm1: l_shipinstruct char (25),
      s1pm1: l_shipmode char (10),
      s1pm1: l_comment varchar (44)
      s1pm1: ) engine=columnstore
      s1pm1: --------------
      s1pm1:
      s1pm1: Query OK, 0 rows affected (0.198 sec)
      s1pm1:
      s1pm1: --------------
      s1pm1: load data infile "/data/qa/source/dbt3/1
      s1pm1: g
      s1pm1: /lineitem.tbl" into table lineitem fields terminated by "|"
      s1pm1: --------------
      s1pm1:
      s1pm1: Query OK, 6001215 rows affected (6 min 50.879 sec)
      s1pm1: Records: 6001215 Deleted: 0 Skipped: 0 Warnings: 0

        Attachments

          Activity

            People

            Assignee:
            ben.thompson Ben Thompson
            Reporter:
            dleeyh Daniel Lee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.