Security Releases general information #

Security vulnerabilities in GitLab and its dependencies are to be addressed with the highest priority.

Security releases are naturally very similar to patch releases, but on a much shorter timeline. The goal is to make a security release available as soon as possible, while ensuring that the security issue is properly addressed and that the fix does not introduce regressions.

At GitLab, we have two types of security releases:

• critical: immediate patch and mitigation is required for a single issue
• non-critical: monthly release of planned issue

Also see the GitLab Maintenance Policy for the information on determining supported releases and assigning versions.

Guides by role #

• Release Manager
• Security Engineer
• Developer
• Quality Engineer

Terminology #

1. Security vulnerability issue - A confidential issue in a GitLab Canonical project (gitlab-org/some-project), detailing a security vulnerability and steps to reproduce.

   Usually created via HackerOne.

   Example: gitlab-org/gitlab#193100.

2. Security release tracking issue - A confidential issue in gitlab-org/gitlab which is the high-level overview of an entire security release (critical or otherwise).

   Created via issue template.

   Example: gitlab-org/gitlab#209019.

3. Security implementation issue - An issue in a GitLab Security project (gitlab-org/security/some-project), outlining the versions affected, detailing the steps a developer needs to take to remediate the vulnerability, and associating backports.

   Created via issue template.

   Example: gitlab-org/security/gitlab#80.

4. Security merge request (MR) - A merge request in a GitLab Security project (gitlab-org/security/some-project), resolving a vulnerability in a specific branch.

   Created via merge request template.

   Example: gitlab-org/security/gitlab!314.

5. Release task issue - Used by release managers as a checklist of steps to complete the release.

   Created via ChatOps (/chatops run release prepare --security).

   Example: gitlab-org/release/tasks#1272.

Issue association #

graph RL
  classDef canonical fill:#74bd3d
  classDef security fill:#ff7272

  secVuln(Security vulnerability Issue)
  secRelease(Security release Issue)
  secFix(Security implementation issue)
  task{{Release task issue}}

  class secVuln canonical
  class secRelease canonical
  class secFix security

  secFix -->|references| secVuln
  secFix ---|related| secRelease

  task -.->|references| secRelease

Merge request association #

graph RL
  classDef canonical fill:#74bd3d
  classDef security fill:#ff7272

  secFix(Security implementation issue)
  secMR1(["Security MR 1 (default-branch)"])
  secMR2(["Security MR 2 (backport)"])
  secMR3(["Security MR 3 (backport)"])
  secMR4(["Security MR 4 (backport)"])

  class secFix security
  class secMR1,secMR2,secMR3,secMR4 security

  secMR1 & secMR2 & secMR3 & secMR4 -.->|references| secFix

Merging #

graph TD
  classDef security fill:#ff7272

  coMerge{{"/chatops run release merge --security"}}
  coMergeDefaultBranch{{"/chatops run release merge --security --default-branch"}}

  secMR1(["Security MR 1 (default-branch)"])
  secMR2(["Security MR 2 (backport)"])
  secMR3(["Security MR 3 (backport)"])
  secMR4(["Security MR 4 (backport)"])

  autodeploy(auto-deploy)
  defaultbranch(default-branch)

  stable129(12-9-stable-ee)
  stable128(12-8-stable-ee)
  stable127(12-7-stable-ee)

  coMerge --> secMR2
  coMerge --> secMR3
  coMerge --> secMR4

  secMR2 --> |merge| stable129
  secMR3 --> |merge| stable128
  secMR4 --> |merge| stable127

  coMergeDefaultBranch --> secMR1
  secMR1 --> |merge| defaultbranch
  defaultbranch -.-> |cherry-pick| autodeploy

  class secMR1,secMR2,secMR3,secMR4 security
  class stable129,stable128,stable127 security
  class defaultbranch,autodeploy security

Non-critical Security Releases #

Around the 28th of the month, we release patch versions of GitLab containing non-critical security fixes. This date has been chosen as it gives enough time to release a non-security patch release after standard release on the 22nd of the month.

Security Release deadlines #

Just as with patch releases, deadlines for non-critical security releases are at the discretion of release managers, the 28th of the month is used as default due date.

Final deadline will depend on a variety of factors:

• GitLab.com availability.
• Workload on Release Managers.
• Stability of auto-deployments.

In the run-up to the due date, release managers will start processing the security implementation issues associated with the Security Release Tracking issue.

To guarantee the security fix is included on the Security Release, all merge requests associated should be ready to be processed 48h before the due date established in the Security Release Tracking Issue.

Fixes not ready by that date may not be considered for the security release and they’ll have to go into the next one.

Security issues that are linked to the Security Release Tracking issue less than 24 hours before the due date will be automatically unlinked by the Release Tools Bot. For example, if the due date of the Security Release Tracking issue is 28th August, security issues added after 27th August 00:00 UTC will be automatically unlinked. If you have an urgent security issue that cannot wait for the next security release, you can ping the Release Managers for an exception.

Critical Security Releases #

Depending on the severity and impact of the vulnerability, an immediate patch release consisting of just the security fix may be warranted.

The critical release process is a superset of a non-critical release, and for completeness the process is written here:

1. The vulnerability has been reported and discussed between a Security Engineer and a Developer. When the timeline of a possible fix is established, the Security Engineer informs a Release Manager of a need for the critical security release.
2. A release manager proposes a timeline for the critical security release based on the other release tasks, and informs a security engineer and a Quality Engineer in the designated release issue.
3. Depending on the nature of the issue, a security engineer may decide that a temporary mitigation strategy is required. If a patch is required, the security engineer works with the developer and infrastructure engineer following the post-deployment patch process. In other cases, the security engineer will work with infrastructure engineers on remediation.
4. The security engineer creates a security operations issue using the template user_impact_investigation for the security operations team to track further investigations and other post-remediation activities, such as determining and notifying affected users.
5. A security engineer works on a blog post to pre-announce the critical security release. They also work with the marketing team to inform the customers of an upcoming release. They also follow the JiHu Support Security Release Process. The security engineer also prepares the second blog post which will be published at release time.
6. The developer follows the Developer process to create the artifacts necessary for a release.
7. A release manager prepares the release with all backports, and deploys to staging (if applicable). If there are any non security releases pending, such as a release candidate, release managers may decide to release those first.
8. If any post-deployment patches were applied, an on-call Security engineer should verify that the applied patches are still working; unless the newly deployed version removes the need for these patches.
9. A quality engineer prepares environments and executes QA tasks together with the security engineer to verify that all released versions have successfully resolved the vulnerability.
10. When all environments contain the fix, any temporary mitigation strategy is reverted.
11. Following deployment, a release manager coordinates with a security engineer on the exact timing of a release.
12. A release manager will promote the packages at the designated time, and merge the release blog post.
13. A security engineer works with the marketing team to send notification emails to any affected users.
14. A release manager closes all release issues.
15. A security engineer keeps the issues where the vulnerabilities were reported open for the next 30 days.
16. A security engineer prepares a blog post that explains the vulnerability in detail and releases it approximately 30 days after the original release.
17. Once the final blog post is released, a security engineer removes the confidentiality from the issues and closes them.

Each involved role should follow their own guide, and create separate issues linking to the main release issue.

Return to Guides