Security Releases (Critical / Non-critical) as a Developer

The release deadlines for a critical or non-critical security release are different. Check the Security Release deadlines first to know when the security fixes have to be merged by.

Overview

Delivery team recorded a video explaining the security release process from an engineering perspective. Make sure to see it to get an overview of the necessary work.

The process can be overwhelming and, to ensure all steps are done correctly and consistently, it’s recommended that you:

After reading and watching the aforementioned material, you may consider reaching out to a teammate to double-check that all the procedures are followed correctly. The process is continuously improving, so be sure to check back frequently for updates.

DO NOT PUSH TO gitlab-org/gitlab

As a developer working on a fix for a security vulnerability, your main concern is not disclosing the vulnerability or the fix before we’re ready to publicly disclose it. For the same reason all security fixes are deployed and released as part of the security release process and timing. We cannot apply a security fix to GitLab.com in another way.

To that end, you’ll need to be sure that security vulnerabilities:

This is fundamental to our security release process because Security repositories are not publicly-accessible.

Preparation

Making sure the issue needs to follow the security release workflow

  • Verify if the issue you’re working on gitlab-org/gitlab is confidential, if it’s public fix should be placed on GitLab canonical and no backports are required.
  • If the issue you’re fixing doesn’t appear to be something that can be exploited by a malicious person and is instead simply a security enhancement do not hesitate to ping @gitlab-com/gl-security/appsec to discuss if the issue can be fixed in the canonical repository.
  • If you’re updating a dependency that has a known vulnerability that isn’t exploitable in GitLab or has very low severity feel free to engage @gitlab-com/gl-security/appsec in the related issue to see if the dependency can be updated in the canonical repository.

Preparing the repository

Before starting, add the new security remote on your local GitLab repository:

git remote add security git@gitlab.com:gitlab-org/security/gitlab.git

Then run the scripts/security-harness script. This script will install a Git pre-push hook that will prevent pushing to any remote besides gitlab.com/gitlab-org/security or dev.gitlab.org, in order to prevent accidental disclosure.

Please make sure the output of running scripts/security-harness is:

Security harness installed -- you will only be able to push to gitlab.com/gitlab-org/security!

Process

As with most GitLab development, a security fix starts with an issue identifying the vulnerability. For GitLab.com, it should be a confidential issue in the gitlab-org/gitlab project or other appropriate canonical repository such as gitlab-org/gitaly, gitlab-org/gitlab-workhorse, gitlab-org/gitlab-runner, etc.

Typically, only confidential issues that are security vulnerabilities or dependency updates are to follow the process detailed below. If you are unsure if this process needs to be followed, please ask the AppSec team. Security vulnerability issues are typically marked with the security label, have a priority and severity label/score tied to them and do not have the feature label. For more information, please see this section of the handbook. Please also take note of the upstream security patches process if the vulnerability you’re working on affects a third-party dependency.

Once an eligible confidential security issue is assigned to a developer:

  1. A security implementation issue must be created on the respective Security repository using the security issue template.
    • i.e. If an engineer is working on a security fix for GitLab, they will need to create an issue on GitLab Security repo using the GitLab Security issue template.
    • The security implementation issue describes the required steps that need to be followed to remediate the security vulnerability. The same steps are summarized in this section.
    • IMPORTANT: Link your security implementation issue to the next Security Release Tracking issue or the security release will not pick up your issue
      • If the security fix is required for the release, ensure that the security implementation issue is linked as a blocker to the Security Release Tracking issue. This prevents release-tools from unnecessarily unlinking issues that are added too close to the due date.
  2. If this fix for a security vulnerability has far-reaching impact or is a breaking change for a valid use-case, then additional solution evaluation must be performed. See far-reaching impact or breaking fixes process page
  3. A security merge request targeting the default branch (e.g master or main) must be submitted with the security fix using the security merge request template.
    • i.e. If an engineer is working on a security fix for GitLab, they will need to submit the merge request using the GitLab Security merge request template.
    • Merge requests on GitLab Security follow the same code review and approval process as any other change.
      • Note: The security repos have a setting enabled to remove existing approvals when a new commit is pushed. If you push changes during the approval process, you will need to ping anyone who previously approved the MR and ask for re-review and re-approval.
    • Additionally, the merge request targeting the default branch needs to be approved by an AppSec team member. See the code reviews and approval section for details on who to ping.
  4. Once the merge request targeting the default branch is approved according to our Approval guidelines and by an AppSec team member, the engineer can proceed to prepare the backports
  5. Backports need to be approved by the same maintainer that reviewed and approved the merge request targeting the default branch.
    • It’s not required for the backports to have the AppSec approval.
  6. Once the merge request targeting the default branch and the backports are ready, they must be assigned to @gitlab-release-tools-bot.
  • NOTE: When updating dependencies that could possibly be exploited in the wild (i.e. publicly acknowledged security issue by the vendor like this Mermaid release mentioning the XSS), we can open the MR targeting the default branch in the canonical repository to protect GitLab.com as soon as possible but we must create the backports regardless.

Security implementation issue

Create an issue on the respective repo, GitLab Security Repo or Omnibus Security Repo, using the GitLab Security issue template. The title should be the same as the original created on the Canonical repository, for example: Prevent stored XSS in code blocks.

It’s not required to copy all the labels from the original issue on gitlab-org/gitlab, ~security and ~severity::x are enough.

This issue is now your “Implementation issue” and a single source of truth for all related issues and merge requests. Once the issue is created, assign it to yourself and start working on the tasks listed there.

Versions Affected

Within the description “Details” section, fill out the Versions affected using one of the following formats:

  • All
  • X.Y+
  • X.Y - X.Y
  • X.Y,X.Y

Security merge requests

Security implementation issue will ask you to create merge requests on the Security repository. Ensure they are using the respective Security template.

Your branch name must start with security, such as: security-rs-milestone-xss-12-6.

Branch names that start with security cannot be pushed to the canonical repositories on gitlab.com/gitlab-org. This helps ensure security merge requests don’t get leaked prematurely.

For the first merge request, make sure:

  • Targets the master or main depending on the repository.
  • Has no milestone assigned.
    • This is not needed, and frequently causes confusion because we sometimes start working on a security issue during the current milestone, and sometimes at the beginning of the next one.
  • Has correct labels (normally ~security and ~severity::x are enough).
  • Has green pipelines.

For every backport merge request created, make sure:

  • Targets the X-Y-stable{,-ee} branch that belongs to the target version.
    • For the MR targeting the current release, the stable branch might not yet exist. They are normally created two working days before the 22nd. In the meantime, you can set the MR to target the default branch instead, and change it later once the stable branch has been created. If you’re planning to take time off during the time when the stable branch is created, you can ask somebody from your team for help and reassign the security issue to them.
  • Has a milestone assigned, e.g. a 11-10-stable- backport MR would have 11.10 set as its milestone.
    • Milestones that have already been closed are not displayed in the UI, but can still be set using the quick action command /milestone %X.Y.
  • Has correct labels (normally ~security and ~severity::x are enough).
  • Has green pipelines.

IMPORTANT:

  • In case one of these items is missing or incorrect, Release Managers will re-assign all related merge requests to the original author and remove the issue from the current security release. This is necessary due to the number of merge requests that need to be handled for each security release.
  • Merge requests targeting the default branch are to be merged by Release Managers only.
  • Feature flags are discouraged from security merge requests.
  • Migrations that modify or delete data are discouraged from security merge requests. Best practice would instead be to hide the offending data in the application layer and follow up with a migration using the regular development process.
  • Be careful about information included in documentation updates, as issues are not made public until 30 days after the patch is released. It is OK to link to the issue, even though the link will be broken until the issue is made public (example).

TIP: When implementing a security fix, it’s best to go with the smallest change possible. This is helpful to avoid problems/conflicts when creating backports for older versions. It also helps to reduce the possibility of having unwanted side effects as the fix will be focused on the issue. Improvements can be done publicly after the security release is done.

Backports

Because all security fixes go into at least three monthly releases, three additional branches targeting the last 3 monthly releases, including the current one, will need to be created for your security fix.

Because scheduled security releases happen after a new monthly release on the 22nd, determining which versions need backports may be confusing.

For sake of example, pretend it’s February 15th, we’re about to release GitLab 14.8 on the 22nd, and you’re starting work on a security fix. You should target fixes for the following versions:

  1. A merge request targeting master.
  2. A merge request targeting 14-8-stable[-ee] that will be released as 14.8.1.
    • Note the 14-8-stable[-ee] branch may not exist yet, the stable branch of the current release is created two working days before 22nd.
  3. A merge request targeting 14-7-stable[-ee] that will be released as 14.7.x.
  4. A merge request targeting 14-6-stable[-ee] that will be released as 14.6.x.

In total, there will be four merge requests (one targeting master and three backports). If, for example, a vulnerability was only introduced in 14.7, we’d only require the first three merge requests.

Follow the guidelines in the Security Merge Requests to make sure the backport merge requests are valid.

TIP: When creating your merge requests backports there’s a handy script, secpick, that will allow you to cherry-pick commits across multiple releases. If changes can’t be cleanly picked (e.g. file changed doesn’t exist or file was moved in the previous version), you will need to fix it manually. As an alternative there is a gitlab-dev-cli tool which leverages secpick and creates MRs automatically too.

Missing a security release

If your security fix has to be excluded from a release, you’d “roll forward” the backports. Continuing the example from above, where 14.8.1 has been released and the next upcoming monthly release is 14.9:

  1. Close the 14-6-stable[-ee] merge request, as it’s now outside the backport window.
  2. Update the master merge request to target the 14-9-stable[-ee] branch (when created).
  3. Open a new merge request targeting master

Code reviews and Approvals

Security merge requests follow the same review process stated in our Code Review Guidelines:

  • Merge requests targeting the default branch should be approved:
  • Backport merge requests should be approved by at least one maintainer. The Maintainer must be the same as the one assigned to the merge request targeting the default branch. AppSec approval is not needed for backport merge requests.

Final steps

  • Ensure all items have been completed:
  • Ensure all merge requests associated to the security implementation issue are assigned to @gitlab-release-tools-bot, ping the corresponding maintainer if that’s not the case.
  • Be sure to run scripts/security-harness again to enable pushing to remotes other than GitLab Security.

Contact the AppSec team

To contact the AppSec team, please @-mention the AppSec stable counterpart for the group relevant to the vulnerability being fixed. Alternatively, ping the AppSec engineer associated to the issue in the Canonical repository or ask in the #sec-appsec Slack channel.

Questions?

If you have any doubts or questions, feel free to ask for help in the #releases or #g_delivery channel in Slack.