Details
-
New Feature
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
BB V1.01
-
Sprint 3 (10.02.2025)
Description
Per MDEV-23662 and MDEV-23650 we're soon going to be able to test s3 in buildbot.
I expect to be writing a few issues and getting some tests masked in addition to bugs like MDEV-23648.
This will need the server minio (https://dl.min.io/server/minio/release/linux-amd64/ and relative links), and the client mc https://dl.min.io/client/mc/release/linux-amd64/. These are golang built static executables.
These at least for me were painfully slow to download.
How to run the tests is listed:
https://jira.mariadb.org/browse/MDEV-23662?focusedCommentId=165200&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-165200 contains how to create the service on port 9000 internally for test.
Attachments
Issue Links
Activity
... and also because for regression testing purposes it is better to have a fixed version of a 3rd-party tool.
Or run minio containers linked to the bb containers - https://docs.min.io/docs/minio-docker-quickstart-guide.html
according to monty it should also be in branch protection (these tests are fast), because the recent merge made the server unusable as S3 crashed on load.
On implementation the following environment variables control the MTR test of S3 (if a port 9000 service locally doesn't exist
mysql-test/suite/s3/suite.pm
$ENV{'S3_HOST_NAME'} = "127.0.0.1";
|
$ENV{'S3_PORT'} = 9000;
|
$ENV{'S3_BUCKET'} = "storage-engine";
|
$ENV{'S3_ACCESS_KEY'} = "minio";
|
$ENV{'S3_SECRET_KEY'} = "minioadmin";
|
$ENV{'S3_REGION'} = "";
|
$ENV{'S3_PROTOCOL_VERSION'} = "Auto";
|
$ENV{'S3_USE_HTTP'} = "ON";
|
S3_SSL_NO_VERIFY
|
S3_PROVIDER
|
So we could run a single S3 service in Hetzner where access control is IP limited (environment variables can be exposed by a PR).
Buildbot can change the S3 bucket environment to a unique job/worker and then concurrent s3 mtr jobs can occur without interaction.
MTR Tests cleanup the bucket by virtue of DROP TABLE in tests. Even in the eventually of a failing test leaking data a unique bucket should have only a slow storage growth. A restart/clearing on a master deployment or cleanup of the bucket as build step is possible.
As an option - create the user as a pre-test step and remove it post step
https://min.io/docs/minio/linux/administration/identity-access-management/minio-user-management.html#create-a-user
See https://github.com/MariaDB/buildbot/pull/693
Thanks for fixing s3.mysqldump on Main !
S3 tests now run for a protected builder too.
See example: https://buildbot.mariadb.org/#/builders/369/builds/28252
note minio versions are mirrored here ftp://ftp.mariadb.org/public/minio/ because the upstream mirror is so slow.