[MCOL-454] columnstore_info's total_usage() and table_usage() reported 0 usage on multi-node configuration Created: 2016-12-08  Updated: 2023-10-26  Resolved: 2017-01-20

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 1.0.6
Fix Version/s: 1.0.7

Type: Bug Priority: Minor
Reporter: Daniel Lee (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

Sprint: 2016-24, 2016-25, 2017-01, 2017-2

 Description   

Build tested: 1.0.6-1

mcsadmin> getsoft
getsoftwareinfo Thu Dec 8 15:32:44 2016

Name : mariadb-columnstore-platform
Version : 1.0.6
Release : 1
Architecture: x86_64
Install Date: Thu 08 Dec 2016 03:21:18 PM UTC
Group : Applications/Databases
Size : 10017001
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.
Signature : (none)
Source RPM : mariadb-columnstore-platform-1.0.6-1.src.rpm
Build Date : Wed 07 Dec 2016 07:08:09 PM UTC

I noticed the issue yesterday. Today I did more testing starting with brand new installations and found that the issue occurs on multi-node installations. That makes me wonder if the info used for the report is actually on the PM1. It works for single installation because both UM and PM functionalities are on the same server.

On 1UM2PM installation (It was on 1UM4PM yesterday)

MariaDB [columnstore_info]> call total_usage();
---------------------------------+

TOTAL_DATA_SIZE TOTAL_DISK_USAGE

---------------------------------+

1.03 GB 0 Bytes

---------------------------------+
1 row in set (0.02 sec)

Query OK, 0 rows affected (0.02 sec)

MariaDB [columnstore_info]> call table_usage(NULL,NULL);
-----------------------------------------------------------------

TABLE_SCHEMA TABLE_NAME DATA_DISK_USAGE DICT_DISK_USAGE TOTAL_USAGE

-----------------------------------------------------------------

mytest orders 0 Bytes 0.00 Bytes 0 Bytes
tpch1c customer 0 Bytes 0.00 Bytes 0 Bytes
tpch1c lineitem 0 Bytes 0.00 Bytes 0 Bytes
tpch1c nation 0 Bytes 0.00 Bytes 0 Bytes
tpch1c orders 0 Bytes 0.00 Bytes 0 Bytes
tpch1c part 0 Bytes 0.00 Bytes 0 Bytes
tpch1c partsupp 0 Bytes 0.00 Bytes 0 Bytes
tpch1c region 0 Bytes 0.00 Bytes 0 Bytes
tpch1c supplier 0 Bytes 0.00 Bytes 0 Bytes

-----------------------------------------------------------------
9 rows in set (0.31 sec)

Query OK, 0 rows affected (0.31 sec)

It works on a single node installation

MariaDB [columnstore_info]> call total_usage();
---------------------------------+

TOTAL_DATA_SIZE TOTAL_DISK_USAGE

---------------------------------+

4.78 GB 2.84 GB

---------------------------------+
1 row in set (0.07 sec)

Query OK, 0 rows affected (0.07 sec)

MariaDB [columnstore_info]> call table_usage(NULL,NULL);
-----------------------------------------------------------------

TABLE_SCHEMA TABLE_NAME DATA_DISK_USAGE DICT_DISK_USAGE TOTAL_USAGE

-----------------------------------------------------------------

mytest orders 392.07 MB 194.02 MB 586.09 MB
tpch1c customer 14.06 MB 258.04 MB 272.10 MB
tpch1c lineitem 720.12 MB 196.02 MB 916.15 MB
tpch1c nation 6.03 MB 4.02 MB 10.05 MB
tpch1c orders 392.07 MB 194.02 MB 586.09 MB
tpch1c part 16.07 MB 136.05 MB 152.12 MB
tpch1c partsupp 224.04 MB 128.02 MB 352.05 MB
tpch1c region 5.02 MB 4.02 MB 9.05 MB
tpch1c supplier 12.05 MB 8.03 MB 20.09 MB

-----------------------------------------------------------------
9 rows in set (0.22 sec)

 Comments   
Comment by Andrew Hutchings (Inactive) [ 2016-12-14 ]

Confirmed, the path retrieval isn't working properly for remote nodes so we aren't requesting the file size for the right paths.

We are also requesting file sizes the wrong way.

Comment by Andrew Hutchings (Inactive) [ 2016-12-14 ]

This now uses a remote command to WriteEngine for the information_schema.columnstore_files table and correctly builds the file name.

Comment by Daniel Lee (Inactive) [ 2017-01-19 ]

mcsadmin> getcal
getcal: Unknown Command, type help for list of commands

mcsadmin> getsoft
getsoftwareinfo Thu Jan 19 16:19:08 2017

Name : mariadb-columnstore-platform
Version : 1.0.7
Release : 1
Architecture: x86_64
Install Date: Thu 19 Jan 2017 03:38:08 PM UTC

Started with some data:

MariaDB [columnstore_info]> call total_usage();
---------------------------------+

TOTAL_DATA_SIZE TOTAL_DISK_USAGE

---------------------------------+

167.77 MB 1.02 GB

---------------------------------+
1 row in set (0.12 sec)

I then cpimported a 7.3gb lineitem source file.

[root@localhost ~]# /usr/local/mariadb/columnstore/bin/cpimport mytest lineitem /data/qa/source/dbt3/10g/lineitem.tbl
2017-01-19 15:53:29 (14913) INFO : Running distributed import (mode 1) on all PMs...
2017-01-19 15:56:10 (14913) INFO : For table mytest.lineitem: 59986052 rows processed and 59986052 rows inserted.
2017-01-19 15:56:10 (14913) INFO : Bulk load completed, total run time : 161.068 seconds

[root@localhost ~]# ls -alh /data/qa/source/dbt3/10g/lineitem.tbl
rw-rr-. 1 vagrant vagrant 7.3G Jul 9 2016 /data/qa/source/dbt3/10g/lineitem.tbl

total_usage() reported 78.6 gb for total_data_size

MariaDB [columnstore_info]> call total_usage();
---------------------------------+

TOTAL_DATA_SIZE TOTAL_DISK_USAGE

---------------------------------+

78.60 GB 5.51 GB

---------------------------------+
1 row in set (1.30 sec)

Comment by Andrew Hutchings (Inactive) [ 2017-01-19 ]

Additional pull request to cover the case in I_S.columnstore_extents where a dict column can have a block offset > 0 and HWM = 0 which breaks the DATA_SIZE calculation.

This fixes the addition problem Daniel observed.

Comment by Daniel Lee (Inactive) [ 2017-01-20 ]

Build verified: 1.0-7-1

mcsadmin> getsoft
getsoftwareinfo Fri Jan 20 22:59:24 2017

Name : mariadb-columnstore-platform
Version : 1.0.7
Release : 1
Architecture: x86_64
Install Date: Fri 20 Jan 2017 10:46:12 PM UTC
Group : Applications/Databases
Size : 10001348
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.
Signature : (none)

retested my last scenario and got:

MariaDB [columnstore_info]> call total_usage();
---------------------------------+

TOTAL_DATA_SIZE TOTAL_DISK_USAGE

---------------------------------+

2.48 GB 5.51 GB

---------------------------------+
1 row in set (0.22 sec)

Generated at Thu Feb 08 02:21:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.