General process for Security engineer in security releases

The main difference is in who initiates the release process.

  • Critical security releases are initiated on-demand by the security team to resolve the most severe vulnerabilities that are labelled as S1. They are highly urgent and need to be released as soon as the patch is ready.
  • Regular security releases are scheduled monthly and are initiated by the release management team.

Cross-regional AppSec Release Managers

In order to make sure there is always someone available and ready to help get the release finished, the AppSec team assigns two team members to each release.

The ‘Primary’ team member is responsible for the steps described below in the overall process section. They are the main DRI for the security release and the core responsibilities associated with it.

A ‘Secondary’ person located in a different region is also assigned to each release. As with other rotations, the Secondary should find someone to switch with if they are going to be out of office or otherwise unavailable during their assigned rotation.

The Secondary is expected to help with any tasks needed to get the release finalized such as:

  • Copying the reviewed blog post over to the canonical www-gitlab-com repository
  • Working with the Communications team to get the e-mail sent out to customers
  • Assisting the Delivery team as needed in order to get the release finalized

The Primary and Secondary should post handover messages in the #sec-appsec Slack channel at the end of their workday when appropriate leading up to release, and daily during release week. These messages should provide information such as:

  • The current status of the release
  • Links to any relevant Slack threads or merge requests
  • What still needs to be done
  • Any other information that might be helpful to the other team member or the rest of the team

Here is an example template:

Security release appsec handover to <username>
Status: (brief status statement)
Next: (brief description of what should happen next)
Links
- [Security release issue](https://gitlab.com/gitlab-org/gitlab/-/issues?sort=created_date&state=opened&label_name[]=upcoming+security+release) 
- Appsec security release checklist:
- Release draft blog post MR:
- Release blog post MR: 
- Release communications issue:

Overall process

Use the security release tools to create the checklist task issue. That is the single source of truth for security releases.

The following guidance is deprecated, and is kept as a reference if the release tooling is not able to be used.

Expand deprecated manual instructions
  1. Create the AppSec release checklist task issue
  2. Prepare blog post
  3. Follow the appropriate sub-process for a critical or regular security release
  4. Verify fixes
  5. Finalize the release
  6. Post release. Once completed, you have finished the security release.

For all steps in the process, keep in mind that there might be a GitLab Runner security release going on at the same time and that it should be included in the process (blog post, CVE ID requests, notifications on H1 reports, etc.)

Remark: Workhorse, Gitaly, Pages and Omnibus are part of the general security release process. Other products (e.g. gitlab-org/gitlab-runner) follow their own independent security release process.

Create release checklist

Use the security release tools to create the checklist task issue.

Prepare blog post

When all security Issues have an associated CVE request, it is possible to automatically generate the blog post with the make_blogpost.rb script located in the security-release-tools project:

ruby make_blogpost.rb -e <email address> -f "<firstname> <lastname>" -n <gitlab username>

The above command will print out the blog post contents in the terminal for review. To create a draft Merge Request, simply add the --draft flag to the command. When the draft has been approved, it’s possible to create the actual release blog post by adding the --release flag to the command.

See the project’s README for setup instructions and usage.

In case the script does not work, follow the below steps to manually create the blog post:

  • ⚠️ The developer should have provided the range of affected versions: ask on the related security issues to provide it if not already provided.
  • Create a branch for the Draft blog post merge request that will go in the security www-gitlab-com repo.
    1. On the merge request branch in the security www-gitlab-com repo, make a file in the sites/uncategorized/source/releases/posts with this format: YYYY-MM-DD-security-release-gitlab-X-X-X-released.html.md.
    2. Make sure that Draft: is prepended to the blog post merge request title until it has been reviewed, approved, and the packages have been built and ready for release.
    3. Ensure to add the CVE IDs, vulnerability descriptions, affected versions, remediation info, and a thank you to the reporter.
    4. Feel free to use a past security release as a guide, for example this blog post.
  • Please make sure the tags section of the post header includes the security tag, like so: tags: security
  • Please add /images/blogimages/security-cover-new.png to the image_title: field in the front matter.
  • Please include a “Receive Security Release Notifications” section at the very bottom of the blog post with links to our contact us page and RSS feed. See previous security release posts or this issue for examples.
  • Make sure that the blog post only mentions fixes that actually made it into the release. Consider the 48h deadline before the Security Release due date, every issue associated (even after the deadline) will likely not be included and we should ensure it’s not mentioned on the blog post. For anything that was removed, please comment on the related security issue with the blog post text that you have written.
  • Assign the blog post to another member of the appsec team for review.
  • Create a WIP issue using the Security Release Email Alert template in https://gitlab.com/gitlab-com/gl-security/security-communications/communications/issues for the communications team and request an email notification be sent to subscribers of the Security Alert segment on the day of the release. Include a note that this should be sent out after the blog post is live. Also mention that you’ll include the link to the blog post MR once it is prepared. The content of this notification should be similar to the blog post introduction:

“Today we are releasing versions X.X.X, X.X.X, and X.X.X for GitLab Community Edition (CE) and Enterprise Edition (EE).

These versions contain a number of important security fixes, and we strongly recommend that all GitLab installations be upgraded to one of these versions immediately.

Please forward this alert to appropriate people at your organization and have them subscribe to Security Notices. You can also receive security release blog updates by subscribing to our RSS feed.

For more details about this release, please visit our blog.”

Dependencies updates

Fixes that mitigate 3rd party vulnerabilities are typically added at the end of the blog post with a reference to the existing CVE ID that was assigned to that vulnerability by the 3rd party.

Example of previously used wording:

### Update Nokogiri dependency
The Nokogiri dependency has been upgraded to 1.10.8. This upgrade include a security fix for [CVE-2020-7595](https://github.com/advisories/GHSA-7553-jr98-vx47).

Regular security releases

  1. For each provided fix for a security issue:
    1. Make sure that each fix included in the release has all the relevant details like affected versions, bug bounty reporter, etc. in the Issue’s Details table. If this is missing, ping the Issue author and ask them to add this information as it’s needed for the CVE requests in the following step.
    2. Make sure that each fix included in the release has had a pre-assigned CVE requested. This should have been done by the person verifying the fix by following the steps in the Verifying Security Fixes runbook.
    3. Make sure that each fix included in the release has been reviewed and approved by the team’s Security Stable Counterpart. If not, ping the stable counterpart in the MR and ask for them to review and approve.
  2. Set the previous month’s regular security release to public by following the post release steps for it.

Tip:

  1. You should use make_checklist.rb script located in the security-release-tools project.
  2. It is possible to automatically check the status (opened, closed, locked, or merged) of the different merge requests for the security release with the check_merge_request.rb script located in the security-release-tools project.

Critical security releases

The issue must be prepared as soon as we are aware of the critical issue. Please ask @release-managers if the timing is appropriate for a release, if not all efforts should be put on finding a mitigation that doesn’t involve a new release until the agreed date.

Once a fix has been provided for a critical S1 labelled security issue:

  1. Make sure that each fix included in the release has had a pre-assigned CVE requested. This should have been done by the person verifying the fix by following the steps in the Verifying Security Fixes runbook.

  2. Create a security release tracking issue with ~security and ~“upcoming security release” labels (e.g https://gitlab.com/gitlab-org/gitlab/-/issues/297419).

  3. Associate the security issue to the security release tracking issue.

  4. Notify release managers of a need to prepare for the critical security release:

    • Ping them in the security release tracking issue, and
    • Ping them in the #f_upcoming_release channel.

  5. If the fix is not ready yet, provide release managers with an estimate of when the fix is going to be ready so that they can create a release plan.

  6. 30 days after the release is published, follow steps in Post Release.

Verify fixes

Fixes should be already verified at this point, but follow the Verifying Security Fixes runbook if some verifications are still pending.

Finalize release

Once the draft blog post in the security www-gitlab-com repo has been reviewed and all packages are ready for publishing:

  • Release manager starts publishing the packages
  • When the Release Managers notify about the packages being published:
    • Create a merge request in the Canonical www-gitlab-com repo with the path under sites/marketing/sou‎rce/releases/posts/‎ and ask that the Release Managers merge it in.
  • Put the link of the new blog post in the email notification request issue as well as the security release issue
  • Make sure the CVE IDs are documented in the corresponding GitLab.com issues and H1 reports
  • Ping in H1 reports the appsec team member who triaged the issue to notify a fix has been released
  • Close out the issues on gitlab-org/gitlab and the draft blog post merge request that was created in the security www-gitlab-com repo

Post release

30 days after release

  • Open a new security release issue for the next security release. Identify the next release managers from the release managers page and assign the issue to them.
  • Remove confidential from issues unless they have been labelled ~keep confidential.
    • Our bot will automatically remind us to remove confidential from the issues.