[MDBF-299] Minimum Viable Product - MariaDB Kubernetes Operator for traditional replication Created: 2021-12-15  Updated: 2022-06-22  Resolved: 2022-06-22

Status: Closed
Project: MariaDB Foundation Development
Component/s: Kubernetes
Affects Version/s: None
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Anel Husakovic Assignee: Anel Husakovic
Resolution: Fixed Votes: 0
Labels: None
Σ Remaining Estimate: 55d 7.5h Remaining Estimate: Not Specified
Σ Time Spent: 4d 0.5h Time Spent: Not Specified
Σ Original Estimate: 84d Original Estimate: Not Specified

Issue Links:
PartOf
is part of MDBF-298 MariaDB and Kubernetes Open
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MDBF-369 Create scaffold of resource API in Go... Technical task Closed Anel Husakovic  
MDBF-370 Create Specification and Status for M... Technical task Closed Anel Husakovic  
MDBF-371 Create deployment - for stateless app... Technical task Closed Anel Husakovic  
MDBF-372 Create service for the deployment Technical task Closed Anel Husakovic  
MDBF-373 Add miscellaneous features Technical task Closed Anel Husakovic  
MDBF-374 Statefulset part M/S - create initcon... Technical task Closed Anel Husakovic  
MDBF-375 Statefulset part M/S - create containers Technical task Closed Anel Husakovic  
MDBF-376 Statefulset part M/S - create Services Technical task Closed Anel Husakovic  
MDBF-377 Statefulset part M/S - create proper ... Technical task Closed Anel Husakovic  
MDBF-378 Statefulset part M/S - create Secrets Technical task Closed Anel Husakovic  
MDBF-380 Test each subtask / blog coverage Technical task Open  
MDBF-381 Create and learn about replication us... Technical task Open  
MDBF-392 Documentation of K8s features Technical task Open Ian Gilfillan  
MDBF-393 Communication during development Technical task Open Kaj Arnö  
MDBF-394 Review of K8s features Technical task Closed Daniel Black  
MDBF-395 Weekly review of K8s features Technical task Open Vicențiu Ciorbaru  
MDBF-434 Learn Golang Technical task Open  

 Description   

About K8s operator

A Kubernetes Operator is a controller that encodes human operational knowledge: how do I run and manage a specific piece of complex software.

Operator is an custom resources in K8s involving specific knowledge of application (in our case MariaDB) in order to automate lifecycle.
This task should create MariaDB operator used to create the MariaDB statefull application in a cluster.

Project scope

The goal of this project is to create a Kubernetes Operator. As a Minimum Viable Product (MVP), the operator needs to know how to:

  1. Start a MariaDB Server
  2. Set up the initial database (mariadb_install_db, authentication methods, networking ports)
    1. Set up a complete and secure networking configuration (service) between nodes.
    2. Configuration method for specifying which machines to be used as nodes.
    3. Configuration method for specifying data directory locations.
  3. Set up a replication topology - primary server (master) & at least 2 replicas (slaves) (Asynchronous replication)
  4. Provides HA.
    1. Monitor the status of all MariaDB nodes in the cluster and restart in case nodes go down.
  5. Easy management of cluster; Change the size parameter to add/remove members from cluster

Things that are not part of MVP, but are to be considered as future development:

  1. Ability to take backups (on demand or scheduled backup - locally or to object storage -S3, restore DB from existing backup)
  2. Provisioning of slaves from backups.
  3. Ability to fine tune which machines get assigned to which nodes. (For example master node might need to have more CPUs).
  4. Detect master failure and automatically promote one of the slaves to become the new master.
  5. Proxying. Automatically route writes to Primary/ies and distribute reads between all members (example haproxy, proxysql).
  6. Encryption in transit and Data-At-Rest
  7. Usage of private docker registries
  8. Helm chart to deploy the MariaDB operator (instead of editing yaml file).
  9. Change from master-slave replication to Galera replication topology.


 Comments   
Comment by Anel Husakovic [ 2021-12-15 ]

Current state of the patch in directory mariadb-k8s:
Scaffolding:

$ operator-sdk init --domain mariadb.org --repo github.com/mariadb/mariadb.org-tools/mariadb-operator
$ go mod tidy
$ operator-sdk create api --group mariak8g --version v1alpha1 --kind MariaDB --resource
$ make manifests
$ make install # the same as `kubebuilder create -f config/crd/bases`
$ kubectl get crds|grep maria
mariadbs.mariak8g.mariadb.org                 2021-12-14T11:02:33Z
 
$ kubectl get crd mariadbs.mariak8g.mariadb.org
NAME                            CREATED AT
mariadbs.mariak8g.mariadb.org   2021-12-14T11:02:33Z

  • CRDs obtained:

    # The same output as in file: `config/crd/bases/mariak8g.mariadb.org_mariadbs.yaml`
    $ kubectl get crd mariadbs.mariak8g.mariadb.org -o yaml
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      annotations:
        controller-gen.kubebuilder.io/version: v0.7.0
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.7.0"},"creationTimestamp":null,"name":"mariadbs.mariak8g.mariadb.org"},"spec":{"group":"mariak8g.mariadb.org","names":{"kind":"MariaDB","listKind":"MariaDBList","plural":"mariadbs","singular":"mariadb"},"scope":"Namespaced","versions":[{"name":"v1alpha1","schema":{"openAPIV3Schema":{"description":"MariaDB is the Schema for the mariadbs API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"MariaDBSpec defines the desired state of MariaDB","properties":{"foo":{"description":"Foo is an example field of MariaDB. Edit mariadb_types.go to remove/update","type":"string"}},"type":"object"},"status":{"description":"MariaDBStatus defines the observed state of MariaDB","type":"object"}},"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
      creationTimestamp: "2021-12-14T11:02:33Z"
      generation: 1
      name: mariadbs.mariak8g.mariadb.org
      resourceVersion: "194510"
      uid: f0201fb9-1a86-4ae5-94c3-c776d8bce470
    spec:
      conversion:
        strategy: None
      group: mariak8g.mariadb.org
      names:
        kind: MariaDB
        listKind: MariaDBList
        plural: mariadbs
        singular: mariadb
      scope: Namespaced
      versions:
      - name: v1alpha1
        schema:
          openAPIV3Schema:
            description: MariaDB is the Schema for the mariadbs API
            properties:
              apiVersion:
                description: 'APIVersion defines the versioned schema of this representation
                  of an object. Servers should convert recognized schemas to the latest
                  internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
                type: string
              kind:
                description: 'Kind is a string value representing the REST resource this
                  object represents. Servers may infer this from the endpoint the client
                  submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
                type: string
              metadata:
                type: object
              spec:
                description: MariaDBSpec defines the desired state of MariaDB
                properties:
                  foo:
                    description: Foo is an example field of MariaDB. Edit mariadb_types.go
                      to remove/update
                    type: string
                type: object
              status:
                description: MariaDBStatus defines the observed state of MariaDB
                type: object
            type: object
        served: true
        storage: true
        subresources:
          status: {}
    status:
      acceptedNames:
        kind: MariaDB
        listKind: MariaDBList
        plural: mariadbs
        singular: mariadb
      conditions:
      - lastTransitionTime: "2021-12-14T11:02:33Z"
        message: no conflicts found
        reason: NoConflicts
        status: "True"
        type: NamesAccepted
      - lastTransitionTime: "2021-12-14T11:02:33Z"
        message: the initial names have been accepted
        reason: InitialNamesAccepted
        status: "True"
        type: Established
      storedVersions:
      - v1alpha1
    

  • Creating custom resource:

    # Create custom resource (MariaDB)
    # Not good, since some fields are requried - seems like I can add empty sample, what is probably not good @todo-1!
    $ kubectl create -f config/samples/mariak8g_v1alpha1_mariadb.yaml 
    mariadb.mariak8g.mariadb.org/mariadb-sample created
     
    $ kubectl get mariadbs
    NAME             MARIADB STATE   PORT
    mariadb-sample 
    

  • Validate markers in kubebuilder:

    # When adding {{port:-1}}, error is raised,but related to required fields only:
     
    $ kubectl create -f config/samples/mariak8g_v1alpha1_mariadb.yaml 
    error: error validating "config/samples/mariak8g_v1alpha1_mariadb.yaml": 
    error validating data: 
    [
    ValidationError(MariaDB.spec): missing required field "dataStoragePath" in org.mariadb.mariak8g.v1alpha1.MariaDB.spec, 
    ValidationError(MariaDB.spec): missing required field "database" in org.mariadb.mariak8g.v1alpha1.MariaDB.spec, 
    ValidationError(MariaDB.spec): missing required field "image" in org.mariadb.mariak8g.v1alpha1.MariaDB.spec, 
    ValidationError(MariaDB.spec): missing required field "password" in org.mariadb.mariak8g.v1alpha1.MariaDB.spec, 
    ValidationError(MariaDB.spec): missing required field "rootpwd" in org.mariadb.mariak8g.v1alpha1.MariaDB.spec, 
    ValidationError(MariaDB.spec): missing required field "username" in org.mariadb.mariak8g.v1alpha1.MariaDB.spec
    ]; 
    if you choose to ignore these errors, turn validation off with --validate=false
    

    Conclusion: if empty yaml is supplied than resource can be created (need to search why is this happening?) otherwise, required takes precedence - ok!

  • Validate marker in kubebuilder - wrong port

    # After specifying required fields we got the expected error for the wrong port - good
    $ kubectl create -f config/samples/mariak8g_v1alpha1_mariadb.yaml 
    The MariaDB "mariadb-sample" is invalid: 
    spec.port: Invalid value: -1: 
    spec.port in body should be greater than or equal to 0
    

  • Validate defaults - type vs pointer

    # When set to 0
    $ kubectl get mariadb
    NAME             MARIADB STATE   PORT
    mariadb-sample                   0
    # Not in line with the premise that: 
    # setting 0 to the integer type should be the same as unset (where pointers should be used),
    # and the default value should be set (I would expect default value 3306)
    

  • How to get the default value:

    # However when port is not used, default value from the marker is picked
    $ kubectl get mariadb
    NAME             MARIADB STATE   PORT
    mariadb-sample                   3306
    

  • Successfull reconcilation, during creation of an object - problem with logr - added in patch

    $ kubectl create -f config/samples/mariak8g_v1alpha1_mariadb.yaml 
    

    Output from the controller:

    2021-12-14T05:25:23.053-0800    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
    2021-12-14T05:25:23.053-0800    INFO    setup   starting manager
    2021-12-14T05:25:23.053-0800    INFO    starting metrics server {"path": "/metrics"}
    2021-12-14T05:25:23.053-0800    INFO    controller.mariadb      Starting EventSource    {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
    2021-12-14T05:25:23.053-0800    INFO    controller.mariadb      Starting Controller     {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB"}
    2021-12-14T05:25:23.155-0800    INFO    controller.mariadb      Starting workers        {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "worker count": 1}
    2021-12-14T05:26:04.266-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample"}
    2021-12-14T05:26:04.266-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample"}
    

  • Delete the object/resourece:

    $ kubectl delete -f config/samples/mariak8g_v1alpha1_mariadb.yaml 
    mariadb.mariak8g.mariadb.org "mariadb-sample" deleted
     
    2021-12-14T05:30:45.007-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample"}
    2021-12-14T05:30:45.007-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind after delete    {"MariaDB: ": "default/mariadb-sample"}
    

  • If the resource exists and controller is started on non-empty object, the reconcilation is done and the output is following:

    2021-12-14T05:31:57.340-0800    INFO    controller.mariadb      Starting EventSource    {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
    2021-12-14T05:31:57.340-0800    INFO    controller.mariadb      Starting Controller     {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB"}
    2021-12-14T05:31:57.441-0800    INFO    controller.mariadb      Starting workers        {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "worker count": 1}
    2021-12-14T05:31:57.441-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample"}
    2021-12-14T05:31:57.441-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample"}
    

  • Added the same + message for resource name

    2021-12-14T06:03:07.181-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
    2021-12-14T06:03:07.181-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample"}
    

Comment by Anel Husakovic [ 2021-12-21 ]

Problem with import after changing the desired state and importing core module apps "k8s.io/api/apps/v1"

$ make manifests
/home/anel/operators/mariadb/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
main.go:35:2: package github.com/mariadb/mariadb.org-tools/mariadb-operator/controllers imports k8s.io/api/core/v1 from implicitly required module; to add missing requirements, run:
	go get k8s.io/api@v0.22.1
Error: not all generators ran successfully

When looking into updating the go module, seems that Makefile should be updated:

$ make manifests
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.7.0
go get: installing executables with 'go get' in module mode is deprecated.
	To adjust and download dependencies of the current module, use 'go get -d'.
	To install using requirements of the current module, use 'go install'.
	To install ignoring the current module, use 'go install' with a version,
	like 'go install example.com/cmd@latest'.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.12.0
go get: added github.com/go-logr/logr v0.4.0
go get: added github.com/gobuffalo/flect v0.2.3
go get: added github.com/gogo/protobuf v1.3.2
go get: added github.com/google/go-cmp v0.5.6
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.11
go get: added github.com/mattn/go-colorable v0.1.8
go get: added github.com/mattn/go-isatty v0.0.12
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/spf13/cobra v1.2.1
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.4.2
go get: added golang.org/x/net v0.0.0-20210520170846-37e1c6afe023
go get: added golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
go get: added golang.org/x/text v0.3.6
go get: added golang.org/x/tools v0.1.5
go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.4.0
go get: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go get: added k8s.io/api v0.22.2   # NOTE HERE IT IS ADDED THE MODULE WE WANT
go get: added k8s.io/apiextensions-apiserver v0.22.2
go get: added k8s.io/apimachinery v0.22.2
go get: added k8s.io/klog/v2 v2.9.0
go get: added k8s.io/utils v0.0.0-20210819203725-bdf08cb9a70a
go get: added sigs.k8s.io/controller-tools v0.7.0
go get: added sigs.k8s.io/structured-merge-diff/v4 v4.1.2
go get: added sigs.k8s.io/yaml v1.2.0
/home/anel/operators/mariadb/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

So the line should be sufficient:

$ go get -d k8s.io/api@v0.22.1

Package https://pkg.go.dev/k8s.io/api/apps/v1 used for defining the deployments.
Package https://pkg.go.dev/k8s.io/api/core/v1 used for defining the containers
Example of CreateOrUpdate function https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil#CreateOrUpdate

  • First result with deployment

    $ kubectl get deploy 
    NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
    mariadb-sample-server   0/1     1            0           59s
    

  • todo:
  • add custom image/image version,
  • number of replicas
  • validate results in controller
Comment by Anel Husakovic [ 2021-12-22 ]

Obtained deployment with custom image/image version for POD and desired number of replicas:

$ kubectl get deploy
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
mariadb-sample-server   0/2     2            0           19s
 
$ kubectl describe deploy mariadb-sample-server
Name:                   mariadb-sample-server
Namespace:              default
CreationTimestamp:      Wed, 22 Dec 2021 01:36:58 -0800
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               mariadb=mariadb-sample
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  mariadb=mariadb-sample
  Containers:
   mariadb:
    Image:      quay.io/mariadb-foundation/mariadb-devel:10.5
    Port:       3306/TCP
    Host Port:  0/TCP
    Environment:
      MARIADB_ALLOW_EMPTY_ROOT_PASSWORD:  1
      MARIADB_ROOT_PASSWORD:              my-secret-pw
      MARIADB_USER:                       example-user
      MARIADB_PASSWORD:                   my_cool_secret
    Mounts:                               <none>
  Volumes:                                <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   mariadb-sample-server-6945bcb56 (2/2 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  37s   deployment-controller  Scaled up replica set mariadb-sample-server-6945bcb56 to 2
 
# Deployment with 2 replicas started
$ kubectl get deploy
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
mariadb-sample-server   2/2     2            2           45s
 
# Checking the pods
$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
mariadb-sample-server-6945bcb56-mbbjg   1/1     Running   0          70s
mariadb-sample-server-6945bcb56-v6sls   1/1     Running   0          70s
 
# Checking the resource created from operator and its state
$ kubectl get mariadb
NAME             MARIADB STATE   PORT
mariadb-sample   RUNNING         3306
 
# Connecting to the POD with credentials of root user
$ kctl exec -it mariadb-sample-server-6945bcb56-mbbjg -- mariadb -uroot -pmy-secret-pw -e "select version();"
+----------------------------------------------------------+
| version()                                                |
+----------------------------------------------------------+
| 10.5.14-MariaDB-2776635cb98d35867447d375fdc04a44ef11a697 |
+----------------------------------------------------------+

Result validated in controller:

$ make run
/home/anel/operators/mariadb/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/anel/operators/mariadb/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
2021-12-22T01:36:58.771-0800    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2021-12-22T01:36:58.772-0800    INFO    setup   starting manager
2021-12-22T01:36:58.772-0800    INFO    starting metrics server {"path": "/metrics"}
2021-12-22T01:36:58.772-0800    INFO    controller.mariadb      Starting EventSource    {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
2021-12-22T01:36:58.773-0800    INFO    controller.mariadb      Starting EventSource    {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
2021-12-22T01:36:58.773-0800    INFO    controller.mariadb      Starting Controller     {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB"}
2021-12-22T01:36:58.875-0800    INFO    controller.mariadb      Starting workers        {"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "worker count": 1}
2021-12-22T01:36:58.875-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:36:58.889-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:36:58.892-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:36:58.907-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:36:58.907-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:36:58.934-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:36:58.934-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:36:58.970-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:36:58.970-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:36:58.979-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:37:26.757-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:37:26.771-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:37:26.772-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:37:26.791-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:37:28.794-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:37:28.821-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
2021-12-22T01:37:28.821-0800    INFO    controllers.MariaDB1    Reconciling MariaDB kind        {"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
2021-12-22T01:37:28.833-0800    INFO    controllers.MariaDB1    Reconciled MariaDB kind {"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}

Commit - https://github.com/an3l/mariadb-k8s/commit/512d8e8633f543e3b334d14581bb75990b3cb705

Comment by Anel Husakovic [ 2021-12-22 ]

Occasionally problem may occur to not be able to start the controller:

$ make run
/home/anel/operators/mariadb/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/anel/operators/mariadb/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
2021-12-22T02:28:34.788-0800	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-12-22T02:28:34.788-0800	ERROR	controller-runtime.metrics	metrics server failed to listen. You may want to disable the metrics server or use another port if it is due to conflicts	{"error": "error listening on :8080: listen tcp :8080: bind: address already in use"}
runtime.main
	/usr/local/go/src/runtime/proc.go:255
2021-12-22T02:28:34.788-0800	ERROR	setup	unable to start manager	{"error": "error listening on :8080: listen tcp :8080: bind: address already in use"}
exit status 1
Makefile:97: recipe for target 'run' failed
make: *** [run] Error 1

Solution, find the port and PID and kill the process

$ sudo lsof -i -P -n|grep 8080
main      33517            anel    7u  IPv6 193406      0t0  TCP *:8080 (LISTEN)

Or kill the name "main" $ kill -9 `pgrep -x main`

Comment by Anel Husakovic [ 2021-12-22 ]
  • Service created

    $ kubectl get svc
    NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
    kubernetes       ClusterIP      10.96.0.1      <none>        443/TCP          54d
    mariadb-sample   LoadBalancer   10.105.71.42   <pending>     3306:31193/TCP   14s
    

  • Result from controller:

    $ make run
    /home/anel/operators/mariadb/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
    /home/anel/operators/mariadb/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
    go fmt ./...
    go vet ./...
    go run ./main.go
    2021-12-22T02:33:51.198-0800	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
    2021-12-22T02:33:51.198-0800	INFO	setup	starting manager
    2021-12-22T02:33:51.198-0800	INFO	starting metrics server	{"path": "/metrics"}
    2021-12-22T02:33:51.199-0800	INFO	controller.mariadb	Starting EventSource	{"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
    2021-12-22T02:33:51.199-0800	INFO	controller.mariadb	Starting EventSource	{"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
    2021-12-22T02:33:51.199-0800	INFO	controller.mariadb	Starting EventSource	{"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "source": "kind source: /, Kind="}
    2021-12-22T02:33:51.199-0800	INFO	controller.mariadb	Starting Controller	{"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB"}
    2021-12-22T02:33:51.300-0800	INFO	controller.mariadb	Starting workers	{"reconciler group": "mariak8g.mariadb.org", "reconciler kind": "MariaDB", "worker count": 1}
    2021-12-22T02:33:51.300-0800	INFO	controllers.MariaDB1	Reconciling MariaDB kind	{"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
    2021-12-22T02:33:51.332-0800	INFO	controllers.MariaDB1	Reconciled MariaDB kind	{"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
    2021-12-22T02:33:51.332-0800	INFO	controllers.MariaDB1	Reconciling MariaDB kind	{"MariaDB: ": "default/mariadb-sample", "mariadb": ""}
    2021-12-22T02:33:51.364-0800	INFO	controllers.MariaDB1	Reconciled MariaDB kind	{"MariaDB: ": "default/mariadb-sample", "mariadb": "mariadb-sample", "status": {"desiredReplicas":0,"lastMessage":"","dbState":"RUNNING"}}
    

Comment by Anel Husakovic [ 2022-02-22 ]

Commit: https://github.com/an3l/mariadb-k8s/commit/9fb056cd71ba80c84a6e574d676078bebebf1cde

$ kubectl get mariadbs -w
NAME             MARIADB STATE   PORT   AGE
mariadb-sample                   3306   0s
$ kubectl get mariadbs -w -o wide
NAME             MARIADB STATE   PORT   IMAGE                                           AGE
mariadb-sample                   3306   quay.io/mariadb-foundation/mariadb-devel:10.5   9s
 
 
Note that spec attributes are not expected to change during reconcilation - Spec.ShowState
 
 
$ kubectl get all
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/mariadb-sample-server-68bc5c6f9f-45v94              1/1     Running   0          6m9s
pod/mariadb-sample-server-68bc5c6f9f-dm6b7              1/1     Running   0          6m9s
pod/mariadb-sample-server-deployment-68bc5c6f9f-lf84g   1/1     Running   0          4s
pod/mariadb-sample-server-deployment-68bc5c6f9f-mntlm   1/1     Running   0          4s
 
NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                      ClusterIP   10.96.0.1       <none>        443/TCP    105d
service/mariadb-sample                  ClusterIP   10.106.75.88    <none>        3306/TCP   6m9s
service/mariadb-sample-server-service   ClusterIP   10.110.35.216   <none>        3306/TCP   4s
service/mariadb-sample-service          ClusterIP   10.103.95.33    <none>        3306/TCP   2m8s
 
NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mariadb-sample-server              2/2     2            2           6m9s
deployment.apps/mariadb-sample-server-deployment   2/2     2            2           4s
 
NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/mariadb-sample-server-68bc5c6f9f              2         2         2       6m9s
replicaset.apps/mariadb-sample-server-deployment-68bc5c6f9f   2         2         2       4s
 
 
$ kubectl exec -it svc/mariadb-sample-server-service -- mariadb -uexample-user -pmy_cool_secret -e "show databases;"
+--------------------+
| Database           |
+--------------------+
| information_schema |
|    testDB-operator |
+--------------------+

Note for Anel: spec attributes are not expected to change during reconcilation - Spec.ShowState

Comment by Anel Husakovic [ 2022-03-07 ]

Added secret creation
https://github.com/an3l/mariadb-k8s/commits/mariadb-operator-secrets
Todo: Edit secrets should work on the fly, find out about API call for this

Comment by Anel Husakovic [ 2022-06-14 ]

Switching from go 1.16 to go 1.18 I'm facing the problem that cannot start the operator and had to install in GOBIN relative to project (although GOBIN doesn't accept relative paths) controller-gen and kustomize.

$ GOBIN=/home/anel/mariadb/k8s/mariadb-k8s/bin go install sigs.k8s.io/kustomize/kustomize/v4@latest
$  GOBIN=/home/anel/mariadb/k8s/mariadb-k8s/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest # note here I used @v0.3.0 that couldn't install the manifests
$ make manifests
$ make install
# Custom resource definitions CRD installed
$ kubectl get crds
NAME                            CREATED AT
mariadbs.mariak8g.mariadb.org   2022-06-14T12:25:33Z
 
# Apply sample to create custom resource CR mariadb
$ kubectl apply -f config/samples/mariak8g_v1alpha1_mariadb.yaml 
mariadb.mariak8g.mariadb.org/mariadb-sample created
 
$ kubectl get mariadb
NAME             MARIADB STATE   PORT   AGE
mariadb-sample                   3306   39s

Comment by Anel Husakovic [ 2022-06-22 ]

Done with https://github.com/an3l/mariadb-k8s/tree/mariadb-refactoring

Future work will be contribution to the Percona operator
https://github.com/percona/percona-server-mysql-operator

Generated at Thu Feb 08 03:36:52 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.