Auto-deploy

Overview

GitLab auto-deploy deployments allow us to increase the frequency of deployments to GitLab.com from the default branch of gitlab, gitaly, and omnibus-gitlab.

Current terminology and the release cadence for all types of release are in the releases handbook page.

Process overview

Auto-deploy branches

At a cadence defined in the releases handbook page, up until the release date on the 22nd, auto-deploy branches are created with the following format:

<MAJOR>-<MINOR>-auto-deploy-<TIMESTAMP>

example: 11-10-auto-deploy-20190702

  • MAJOR.MINOR: This comes from the currently active release. We check the currently active release in this section of release tools from which we extract the version numbers.
  • TIMESTAMP: The day the branch was created, in YYYYmmddHHMM format.

Auto-deploy branches are created for the following projects:

These branches are created from the latest commit on the default branch at the moment of branch creation.

After an auto-deploy branch is created, the latest green (passing) commit on the branch is tagged. If the pipeline for a commit is still running on the auto-deploy branch, but the commit is green on the default branch, the commit is also tagged. This means that the HEAD of the auto-deploy branch won’t necessarily be the one that is a candidate for deployment. This is done to ensure that we always deploy commits with some confidence.

Auto-deployment is driven by automated tasks in the release-tools project. Releases handbook page describes how scheduled pipelines execute specific task.

Auto-deploy branches are protected branches, meaning that they require special permission for merging and pushing, and are also automatically mirrored to dev.gitlab.org. The permissions for merging and pushing are only granted to the release managers group.

Auto-deploy tagging

For every deployment created using the auto-deploy process, there is a git tag that matches the version of what is deployed. The auto-deploy tag varies depending on the project for which is being tagged.

For:

  • omnibus-gitlab
  • deployer

It has the following format:

<MAJOR>.<MINOR>.<TIMESTAMP>+<gitlab sha>.<omnibus-gitlab sha>

For:

  • CNG
  • charts/gitlab

It has the following format:

<MAJOR>.<MINOR>.<TIMESTAMP>+<gitlab sha>
  • MAJOR.MINOR: This is the currently active milestone in the gitlab-org group on gitlab.com and follows the same requirements for the auto-deploy branch (see above).
  • TIMESTAMP: Timestamp of the commit in omnibus-gitlab project. This value is guaranteed to increment when a new commit is made in omnibus-gitlab, which will happen at minimum when something changes the GitLab Rails project.
  • gitlab sha: The sha of gitlab for auto-deploy, it corresponds to a green (passing) commit on the gitlab auto-deploy branch.
  • omnibus-gitlab sha: The sha of omnibus-gitlab that will be used for the next auto-deploy, it corresponds to a green (passing) ref on the omnibus-gitlab auto-deploy branch

The discrepancy between the auto-deploy tags, stems from iterating as we work towards a much more refined method of maintaining version information across all repos that contribute to a GitLab package, regardless of what builds that package. The progress for this work is to be tracked here: https://gitlab.com/gitlab-com/gl-infra/delivery/issues/690

As mentioned previously, the omnibus-gitlab tag will trigger a deployment to a non-production environment.

Auto-deploy schedule

Using an example of a non existent release 11.12, let’s assume that a milestone is created with a start date of the 23rd and a due date of the 22nd.

The table below gives an overview of the schedule:

DayDescription
23rd
  • The 11.12 milestone is active.
daily at the [times listed in the handbook](https://about.gitlab.com/handbook/engineering/releases/#gitlabcom-releases-1)An automated CI job creates a branch named **11-12-auto-deploy-YYYYMMDDHHMM** in GitLab EE, omnibus-gitlab, CNG, and Helm from each project's default branch.
  • The latest green commit of gitlab is used to update versions of GitLab components in the omnibus-gitlab repository.
  • The commits in gitlab and omnibus-gitlab are tagged with 11.12.YYYYMMDDHHMM+aaaa.ffff.
  • The resulting auto-deploy package is deployed to staging canary, staging, and then production canary
  • In case an issue is discovered after deployment, one or many MRs need to be created resolving the issue.
  • MRs are reviewed and merged to the default branch using the regular process.
  • In order for the MR to be included in auto-deploy branch, a label Pick into auto-deploy needs to be applied. See here for up-to-date information on how to ensure your MR is picked.
  • A CI job checks the MRs for auto-deploy
    • Labeled MRs are picked automatically into the 11-12-auto-deploy-YYYYMMDDHHMM branch
    • The merge train ensures that CE is merged into EE
    • MRs are updated with a comment to let the MR author know that the change has been prepped for next deployment.
  • The next green commit on the auto-deploy branch is tagged with 11.12.YYYYMMDDHHMM+bbbb.ffff
    • The resulting auto-deploy package is deployed to staging canary, staging, and then production canary
    • In case the latest green commit has already been deployed, no action is taken.
  • This process is repeated on a frequent interval, with multiple deployments to the staging canary, staging and production canary environments
  • Other MRs (ones without the label) merged to the default branch are not included until the next auto-deploy branch, 11-12-auto-deploy-YYYYMMDDHHMM
22nd RELEASE DAY
  • The commit that is currently deployed to production will be used for the official release.
  • Changes that are released on production will be part of the release blog post.
  • If the MR introducing a change is picked into 11-12-auto-deploy-00004 but not deployed to production, the change in question is not going to be included into the final self-managed release.
  • 11.12 is published.
23rd
  • A new milestone, 11.13 is active with a start date of the 23rd and a due date of the 22nd
  • The process auto-deploy cycle repeats

General overview of the complete timeline is described in the engineering handbook.

Auto-deploy status

We’ve created an auto_deploy status ChatOps command that can be used to see the deployment status of all of our environments (staging canary, staging, production canary, production).

In #releases Slack channel:

/chatops run auto_deploy status

This command will output the version, revision, and auto-deploy branch of each environment, if available.

Status of a merged MR or a commit

The Merge Request widget shows the environment and the time of deployment in every Merge Request. This should provide sufficient information to understand where in the deployment process any merge request is:

  • release is for the final version that is going out for self-managed users. When an MR is deployed to release, the released::published label is applied to the MR to indicate that it will be included in the upcoming self-managed release.
  • pre is for release candidates and versions that are used as a preparation for a final release for self-managed users. When an MR is deployed to pre, the released::candidate label is applied to the MR to indicate that it will most likely be included in the upcoming self-managed release.
  • gstg-cny is the canary stage of the GitLab SaaS staging environment
  • gprd-cny is the canary stage of the GitLab SaaS production environment
  • gstg is the staging environment for GitLab Saas - staging.gitlab.com
  • gprd is the production environment for GitLab SaaS - GitLab.com
  • db/gstg indicates post migrations included in the merge request have been executed in the staging environment.
  • db/gprd indicates post migrations included in the merge request have been executed in the prodcution environment.
  • No environment in the widget means that the MR is not deployed to any environment yet

MR widget

For GitLab team-members, you can provide the SHA of a commit to see which environment (if any) that commit has been deployed to by running the following chatops command in Slack #releases channel:

/chatops run auto_deploy status c11d293537748555a0b64a371f7d72629b3dfe9e

This command will output information about the commit, as well as any environments it’s currently deployed to.

If you do not have access to ChatOps, see how you can obtain access.

Status of a merged MR with respect to monthly releases

There is a chatops command that can be used to check the status of an MR: /chatops run release check <MR URL> <upcoming release version (optional)>

For example: /chatops run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/12345 14.4. In this example, https://gitlab.com/gitlab-org/gitlab/-/merge_requests/12345 is the MR to be checked, and 14.4 is the upcoming monthly release. MRs from the gitlab-org/gitlab, gitlab-org/security/gitlab, gitlab-org/omnibus-gitlab and gitlab-org/security/omnibus-gitlab projects are supported by this command.

Two scenarios where this command can be useful:

  1. You can check which version an MR was first released in using: /chatops run release check <MR URL>. When the version is omitted from the command, it will only check if the MR has already been released.

  2. You can check the likelihood of an MR being included in the upcoming monthly release using: /chatops run release check <MR URL> <upcoming release version>.

The command uses the following steps to check the status of an MR:

  1. The command checks if the MR has already been released in a past release. If it has been released, the command will print the version in which the MR was first released. Since releases are incremental, all versions after that will also contain the MR. If a version was not specified in the command, the command will terminate here.

  2. If the MR has not yet been released, it will check if the stable branch has been created for the given release version, and if the MR has been included in the stable branch.

    1. If the stable branch has been created and the MR has been included in it, the MR will be released in the given release version.

    2. If the stable branch has been created, but the MR has not been included in it, the MR will not be released in the given release version.

  3. If the stable branch has not yet been created, it will check if the MR has been deployed to production.

    1. If the MR has been deployed to production, it is highly likely that the MR will be included in the stable branch whenever it is created. However, it is also possible for an older production deploy to be selected for the monthly release if the latest one has any problems.

    2. An MR that has not been deployed to production cannot be included in the monthly release.

It would be nice to have suggestions for additional features as well as feedback on what we currently have. Please add to https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2019.

FAQ

How often do we auto-deploy?

The automated pipelines are running on a 1 hour schedule and will create a new auto-deploy pipeline that will extend from staging canary to the production canary environment. The promotion of the auto-deploy build to production is manual, either through a GitLab Chatops command or a manual action on the pipeline.

The following two conditions must be met in order for an automatic deployment to proceed to production canary:

  • QA smoke tests must pass on the staging environment
  • We must not be in a change-lock window, which is current configured for multiple periods in the change-lock configuration file.

Since the deployment pipeline only looks for a green (passing) CI build, it is important that all changes merged to the default branch also pass all CI tests in the auto-deploy branch.

Example:

From the moment the automated cherry-picker leaves a message that the change is picked, the CI pipeline will run for that specific commit in the branch. Depending on the time the specs passed, the change might be deployed to the first non-production environment anywhere from 3-6 hours.

Note: The frequency of the auto-deployment is set conservatively at the beginning while we measure the impact on environment stability as well as process transition. As we get more comfortable, the goal is to have even more frequent deployment to production environments.

Do we deploy on weekends or holidays?

We generally do not deploy on weekends and some important holidays as we have reduced SRE cover. To enforce this, the pipeline schedule responsible for creating auto_deploy branches is excluding weekends and we have defined Production Change Lock (PCL) periods for weekends and some holidays, which are preventing deployments to get started during those times.

We also do not deploy on Family and Friends days or other dates with Production Change Locks. But as these days are defined on short notice, release managers should manually pause deployments using chatops on the evening before a F&F day and unpause on the evening before the next business day:

/chatops run auto_deploy pause
/chatops run auto_deploy unpause

To not forget about those manual steps, please also make a note in the current release issue (see this example).

What about registry, workhorse, pages and other components?

All GitLab created components such as gitlab-workhorse, gitlab-pages and similar have a VERSION file inside of the GitLab EE repository. For the time being, this remains unchanged and these files are the source of truth for the deployment pipelines.

Components such as Registry have the version set inside of the omnibus-gitlab project, and that is the source of truth for the deployment pipelines for the time being.

What will happen with security patches?

Process for P1/S1 security patches remains the same. Security patches included in the regular monthly security release will be deployed to GitLab.com prior to publishing by merging the changes into the default branch, and manually including them in the active auto-deploy branch. This change will only affect release managers, developers and other process stakeholders won’t have a process change for the time being.

How do I mark an MR so that will be picked into the auto-deploy branch?

In most cases, you won’t need to apply an additional label to your MR because your change will be deployed with the new auto-deploy branch after the MR has been merged. Please refer to the handbook for the rules about applying the Pick into auto-deploy label.

In cases where the MR was not able to pick cleanly to the auto-deploy branch, you will need to create a MR targeting the currently active auto-deploy branch and assign to all active release managers. The auto-deploy branch name will be posted in the comment created by the cherry-pick bot.

Does auto-deploy change how we release software to the self-managed users?

Yes and no. The stable branches are created closer to the 22nd, and the stable branches become backport branches for any patch releases that needs to happen in future. The self-managed users will still receive the next release on the 22nd of the month and will receive patch releases as needed, as was the case up until now.

How are auto-deploy branches different than the stable branch?

The auto-deploy branches are shorter lived and are created from the default branch branch more frequently. You can consider any auto-deploy branch as a “slowed down” default branch branch than a long running stable branch. The auto-deploy branches are intended to have a short lifespan.

Why are we not deploying directly from the default branch?

The default branch (e.g master or main) receives a large number of changes daily. While we do have a large test coverage covering individual pieces, end to end test coverage is still being improved. We are also still lacking the large enough data set for developers to test their changes at scale and number of issues are still discovered in staging or canary, and in some cases in production too. By the time an issue is discovered, the default branch received hundreds of different changes and it is pretty difficult to isolate the problem without introducing another one that would block further deployments. Blocking deployments has a worse effect; once the deployments are enabled again we would be introducing an even larger data set. The auto-deploy branches are giving us some time to discover issues and fix them without the pressure.

How do I query what ref is running on gitlab.com or one of the other environments?

On GitLab.com you can check the /help page when logged in and see the sha of deployed commit. You can also use /chatops run auto_deploy status <sha> to check whether ref is running on Gitlab.com or not

How will marketing teams know what features to put in the release blog post?

The release-post managers and product management will have to change the way the release blog post is created. The release blog post will have to be composed of items that are deployed on GitLab.com as opposed to what was planned for delivery. Release on the 22nd is the snapshot of what is active on GitLab.com.