Overview #

Sometimes, we have risky changes being merged, which we would like to deploy in an isolated manner so that the cause of any failure can be quickly identified.

Examples of such changes:

• https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65201
• https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69655

Process to follow for deploying risky MRs in isolation #

1. Determine the DRI (and team) for the change and coordinate the time for merging. Merging can be performed by a release manager.

2. Deactivate auto-deploy tasks with /chatops run auto_deploy pause.

3. Merge the MR and set labels pick into autodeploy and severity::2.

4. Manually trigger the auto_deploy:pick scheduled pipeline. You might want to check this is the only MR with the Pick into auto-deploy label here.

5. Enable auto_deploy_tag_latest feature flag.

6. Manually trigger the auto_deploy:tag scheduled pipeline.

7. Disable auto_deploy_tag_latest feature flag.

8. Inform the EOC and the DRI about the coordinated pipeline deploying the change.

9. Wait for a successful pipeline on the current auto-deploy branch. Since we enabled the auto_deploy_tag_latest feature flag, we tagged a commit regardless of the pipeline status. After disabling auto_deploy_tag_latest, the auto-deploy:tag job will only tag commits with successful pipelines. In order to remove the possibility of a previous commit being tagged by the next run of the auto-deploy:tag job, we wait for a successful pipeline before unpausing auto_deploy.

10. Once the pipeline is successful, enable auto-deploy tasks with /chatops run auto_deploy unpause.

11. Make sure that the last coordinated pipeline before the one containing the MR is making it into gprd, so that we don’t have any other undeployed changes.

12. Watch the coordinated pipeline and ping the DRI and/or the team responsible in case of any problems.