How to recover from a canceled coordinated pipeline?

How to recover from a canceled coordinated pipeline? #

NOTE: This should be resolved as of January 2022: https://gitlab.com/gitlab-org/gitlab/-/issues/341561

With the current GitLab stable version - 14.3 at the moment of this guide - there’s no straightforward way to restart a canceled coordinated pipeline. Canceling a multi-project pipeline, such as the coordinated pipeline, causes the bridge jobs not to be initiated, recovering from this situation is complicated and time consuming. Because of this, it’s strongly advise for Release Managers to:

1. Wait for the next coordinated pipeline to be created, or to,
2. Manually create one by triggering the prepare task.

In the strange case the Release Manager requires to recover the coordinated pipeline, there are some steps they can follow depending on the scenario.

Continue with the deployments of the canceled pipeline #

Bridge jobs, or deployment jobs, of a canceled coordinated pipeline are mark as skipped. To initiate them Release Managers need to retry the previous job:

Recreate the same coordinated pipeline. #

If a Release Manager requires to create another coordinated pipeline based on the previous auto-deploy tag, these are the steps they can follow:

1. Go to the pipelines page on the release/tools project in Ops.
2. Click on Run Pipeline.
3. On Run for branch or tag search for the auto-deploy tag.
   • E.g. In the case of 14.4.202110081021-f1930dccb0f.1b5dda26e7dp the auto-deploy tag would be 14.4.202110081021.
4. Click on Run Pipeline.

This would generate a new coordinated pipeline using the specific auto-deploy tag. Two items to notice:

1. auto_deploy:wait:* jobs can be unscheduled and executed at will since the auto-deploy package should already exist on dev.
2. Only the coordinated pipeline notification will be sent to Slack, packages notification won’t be sent again.