[MCOL-3948] 1.4 install and upgrade guides needs more details on S3 setup Created: 2020-04-16  Updated: 2021-01-16  Resolved: 2021-01-16

Status: Closed
Project: MariaDB ColumnStore
Component/s: Documentation
Affects Version/s: None
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: David Hill (Inactive) Assignee: Geoff Montee (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None


 Description   

Currently in the multi-node guides it mentions that there is additional setup required to use Storage Manager

"If you would like to use any sort of storage manager, you need to install and configure it prior to installing MariaDB ColumnStore."

Need to document what this is or a link to what needs to be setup. I didn't do anything for this step and when I ran postConfigure to select S3, I got this in the description.

4 - S3-compatible cloud storage *. Note: that should be configured
before running postConfigure (see storagemanager.cnf)

So I assume setting up this file, storagemanager.cnf, is what is meant by the first statement.

In searching for this cnf file, I only found info about it in the release notes. Should it be somewhere else like in a setup or configure Columnstore system or having it just in the release notes good enough.

https://mariadb.com/docs/release-notes/mariadb-columnstore-1-4-2-release-notes/#s3-storage-manager

And should the install/upgrade docs point back to the release notes?

Also this is in the release notes, if pointing here, then this requires a specific location where the file resides

"Before running postConfigure, edit the storagemanager.cnf configuration file to specify the S3"

 Comments   
Comment by David Hill (Inactive) [ 2020-04-16 ]

location

sudo find // -name storagemanager.cnf
//etc/columnstore/storagemanager.cnf

Comment by David Hill (Inactive) [ 2020-04-16 ]

Here is the S3 parameters in the file that need to be setup. I believe we need to document or provide links to what the user needs to do to create or setup S3 storage to where they would get the bucket_id. And also a link to where to get the HMAC access keys.

[S3]

  1. These should be self-explanatory. Region can be blank or commented
  2. if using a private cloud storage system. Bucket has to be set to
  3. something though. Obviously, do not change these after running
  4. postConfigure, or SM will not be able to find your data.
    region = some_region
    bucket = some_bucket
  1. Specify the endpoint to connect to if using an S3 compatible object
  2. store like Google Cloud Storage or IBM's Cleversafe.
  3. endpoint = storage.googleapis.com
  1. For the best performance do not specify a prefix. It may be useful,
  2. however, if you must use a bucket with other data in it. Because
  3. of the way object stores route data and requests, keep the
  4. prefix as short as possible for performance reasons.
  5. prefix = cs/
  1. Put your HMAC access keys here. Keys can also be set through the
  2. environment vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  3. If set, SM will use these values and ignore the envvars.
  4. aws_access_key_id =
  5. aws_secret_access_key =

So from this, I do believe we need a separate ColumnStore S3 configuration documentation that goes into more details on how to set this up. Then all install and upgrade Docs would point to it.

Comment by David Hill (Inactive) [ 2020-04-23 ]

Also need to add details on setting up a multi PM system. Is there a separate S3 storage for each PM or is there 1 that is shared across all PMs? Probably its separate then need to document that each PM storeagemanager.cnf will need to be updated.

Comment by Geoff Montee (Inactive) [ 2020-07-22 ]

The S3 storage manager configuration instructions are present in the ColumnStore 1.5 deployment instructions. We can definitely add the same content to the 1.4 deployment instructions.

Comment by Todd Stoffel (Inactive) [ 2021-01-16 ]

Obsoleted by newer versions.

Generated at Thu Feb 08 02:46:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.