Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-5630

Fix race condition in rocky8 multi node MTR.

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • None
    • None
    • CI, cmapi, Docker
    • None

    Description

      Sometimes mtr stage (multi node stage using docker images) for rocky 8 failed in CI.

      The issue is in combination of code in drone.jsonnet file (engine repo) and in provision file (docker repo).

      In multi_mtr step (drone.jsonnet) we invoking docker compose up -d right before invoking provision command on mcs1 node. At the same time, while "provisioning" we never checked if cmapi finished starting or not.
      So mcs cluster set-apikey command invoked before CMAPI finished its initialization.

      Attachments

        Activity

          alan.mologorsky Alan Mologorsky created issue -
          alan.mologorsky Alan Mologorsky made changes -
          Field Original Value New Value
          Summary Fix race condition in rocky8 multi node MTR Fix race condition in rocky8 multi node MTR.
          alan.mologorsky Alan Mologorsky made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          alan.mologorsky Alan Mologorsky made changes -
          Description Sometimes mtr stage (multi node stage using docker images) for rocky 8 failed in CI.

          The issue is in combination of code in `drone.jsonnet` file (engine repo) and in `provision` file (docker repo).

          In `multi_mtr` step (drone.jsonnet) we invoking `docker compose up -d` right before invoking provision command on mcs1 node. At the same time, while "provisioning" we never checked if cmapi finished starting or not.
          So `mcs cluster set-apikey` command invoked before CMAPI finished its initialization.

          Sometimes mtr stage (multi node stage using docker images) for rocky 8 failed in CI.

          The issue is in combination of code in *drone.jsonnet* file (engine repo) and in *provision* file (docker repo).

          In _multi_mtr_ step (*drone.jsonnet*) we invoking _docker compose up -d_ right before invoking provision command on mcs1 node. At the same time, while "provisioning" we never checked if cmapi finished starting or not.
          So _mcs cluster set-apikey_ command invoked before CMAPI finished its initialization.

          alan.mologorsky Alan Mologorsky made changes -
          Priority Critical [ 2 ] Major [ 3 ]
          alan.mologorsky Alan Mologorsky made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          alan.mologorsky Alan Mologorsky made changes -
          Priority Blocker [ 1 ] Critical [ 2 ]
          alan.mologorsky Alan Mologorsky made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]

          People

            alan.mologorsky Alan Mologorsky
            alan.mologorsky Alan Mologorsky
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.