Security Releases as a Release Manager

Overview

As noted in the general process document, release managers are responsible for handling regular and critical security releases. Before reading the specifics for each, please read through the general information below which applies to both.

When working on security releases, it’s important that all work happens only on Security repositories so nothing is disclosed publicly before we intend it. The release manager also makes sure that all the deadlines are respected. Any delay needs to be escalated to Sr. Manager of Infrastructure. We normally try to avoid releasing security packages to the community on Friday in order to avoid situations where a customer may be exploited during the weekend.

AppSec maintains a schedule of Security Release support engineers listed on https://about.gitlab.com/community/release-managers/. Note that as security releases come after the monthly release the current Security Release engineer will be on the line below the highlighted one. If multiple Security Release support engineers are listed please ping both on issues and requests for visibility.

What to include

A security release, should only include the changes necessary to resolve the security vulnerabilities. Including fixes for regressions in a security patch increases the chances of breaking something, both for users and for our packaging and release process.

Security merge requests will be included in a Security Release if:

  • The security implementation issue is associated to the Security Release Tracking Issue.
  • Security merge request targeting the default branch (e.g master or main) has been approved by an AppSec team member.
    • Note: This only applies for GitLab security fixes. It doesn’t apply for security fixes on Omnibus, GitLab Pages or Gitaly.

The only exception to this policy is release candidates. If the monthly release is in progress as we’re preparing for a security release, it’s acceptable for a new RC to include both security fixes and regression fixes. Care should be taken to coordinate the publishing of an RC package with the other security patches so as to not disclose the security vulnerabilities publicly before we’re ready to disclose them.

Regular Security Releases

From a release manager standpoint, a non-critical security release is fairly similar to a patch release, but with a few extra considerations in order to avoid disclosing vulnerabilities before proper fixes are released. Additional collaboration is needed to coordinate with the AppSec and Marketing teams.

For this type of security releases, the 28th of the month is used as default due date, see security release deadlines for more info.

Overview

  1. An AppSec engineer opens a Security Release Tracking Issue, this is a confidential issue in gitlab-org/gitlab that serves as a high-level overview of an entire security release. This issue is normally created after a Security Release has been completed.

  2. After the 22nd, Release Managers will create the Security Release task issue in the release/tasks repository with:

    # On Slack
/chatops run release prepare --security

    This task contains all the required steps to complete a Security Release, including the ones described below.

  3. Between the start date and one day up to the Security Release due date, Release Managers will start merging security merge requests targeting the default branch with:

    # On Slack
/chatops run release merge --security --default-branch

    Security merge requests targeting the default branch will only be merged if:

    • They belong to GitLab Security
    • They are associated to security issues linked to the current Security Release Tracking Issue.
    • They are associated to security issues that are ready to be processed.

    Other tasks including dealing with other components which may have security picks for the target release as well, such as Gitaly and Omnibus. Each of these are handled manually.

    Once the security merge requests have been merged, Canonical and Security default branches will diverge, security merge-train pipeline schedule will deal with this divergence by updating Security default branch based on the Canonical default branch content, see the troubleshoot guide for more info.

    Security issues that are linked to the Security Release Tracking issue less than 24 hours before the due date will be automatically unlinked by the Release Tools Bot. For example, if the due date of the Security Release Tracking issue is 28th August, security issues added after 27th August 00:00 UTC will be automatically unlinked.

    However, security issues that are linked as blockers (‘is blocked by’ vs ‘relates to’) to the Security Release Tracking issue will not be unlinked. This allows high priority security issues to be linked to the tracking issue close to the due date.

    The automatic unlinking behavior is behind a feature flag, which can be disabled if required: unlink_late_security_issues.

    Release-tools will generate and post a table to the security release issue (in gitlab-org/gitlab). The table shows the status of every security issue linked to the security release issue. Example table: https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1589#note_1055523334.

    Do not edit the table, except to add comments under the “Release manager comments” column. Changes to any other column will be overwritten the next time the /chatops run release merge --security command is executed.

  4. One day before the Security Release due date, Release Managers will merge GitLab security backports and security merge requests associated with other satellite GitLab projects.

    # On Slack
/chatops run release merge --security

  5. On the due date of the Security Release, Release Managers will tag, publish the packages and complete the Security Release.

Critical Security Releases

Critical Security Releases are triggered on demand by an AppSec engineer. From a release manager standpoint, a critical security release requires drafting a release plan with a timeline in addition to tasks defined for a regular security release.

It is important to note that “critical” does not mean that the release needs to happen the very moment it is requested. There might be multiple release tasks that are currently executed, so it is important not to panic.

A few questions that you can discuss with your fellow release manager:

  • Are the fixes ready for issue(s) that prompted this security release?
    • If no, find out when the fixes will be ready for the issue and all backport versions.
    • If yes, are all backports ready? If yes, this means that release managers have everything they need to work on the release.
  • Where are we currently in the release cycle?
    • If you are working on patch releases:
      • If the release is not tagged, consider postponing the patch release.
      • If the release is tagged, complete the patch release before proceeding further.
    • If you are working on the latest monthly release:
      • Will working on security release endanger the monthly release? Is it possible to work on RCs and all backports for the security release without breaching the deadline?
      • If you prepare a security release and it gets postponed for some reason, will creating all other RCs as security releases cause more work?

When you get answers for the questions above, start working on the timeline for the critical security release.

An example of how a release timeline could look like, when a security release is called for during the latest monthly release:

To meet the deadline of 22nd and not block the regular release, proposing a schedule:

* All fixes and backports need to be ready for merge by the end of Monday.
* On Tuesday, @release-manager-1 will merge all security fixes into their respective branches,
  and deploy the security fixes to our different environments.
* On Tuesday, @release-manager-2 will ensure the deployment makes it to production and tag the release.
* On Wednesday, @release-manager-1 will publish the packages and wrap up the rest of the security release tasks.

The plan can be more detailed to include more people with specific tasks. Aim to @-mention people responsible for specific tasks, to avoid the bystander effect.

Once the plan is finalized it is important to stick with the plan. You should refuse any changes with the items that need to be included in the release if they put the release deadlines at risk. You can consider being flexible if there is enough time to recover from failure (CI failing, deploy goes wrong, etc.), but in most cases you will need to stick strictly to the plan to make sure that the deadlines given to the Security team are respected.

Once the plan is defined, Release Managers can create the release tasks issue with the following command and follow the steps listed there.

# In Slack
/chatops run release prepare --security --critical

Troubleshoot guide

When diverged, how is Security default branch updated with the content of Canonical default branch?

Security default branch is automatically mirrored based on the content of Canonical default branch, which means that every commit pushed into Canonical default branch is automatically pushed to Security default branch.

Once security merge requests targeting the default branch start being merged in Security, the content of Canonical and Security default branch will diverge (Security default branch will contain the security fixes), which will halt the automatic mirroring between these two.

A scheduled task in release-tools will check for this divergence periodically and automatically enable the security merge-train pipeline schedule to regularly update the Security default branch until the divergence is resolved.

If you need to force an execution of the merge-train outside of this schedule, you can:

This will trigger a merge-train that will merge Canonical default branch into Security default branch. Full execution of merge-train takes approximately 6 minutes.

What to do if merge-train fails to update Security default branch?

There’s a chance of a file being modified both on Canonical and on Security, if this happens merge-train will fail when performing the merge between the two repositories.

The failure will generate a notification on the #g_delivery Slack channel. Solving this will require coordination by the Release Managers:

  • Analyze how difficult it’s to recover from breakage.
    • If the conflict is easy to solved, fix it and push the changes to a branch with a merge request, ensure pipelines are green before merging.
    • If the conflict is more involved, ping the developers related to the changes, if they’re not available, use the dev-on-call process.
  • If the conflict blocks deployments, follow the deployment blockers process and raise an incident.

Runbooks

Return to Security Guide