Security Releases as a Quality Engineer

General process overview

Quality engineer is contributing to two types of security releases, non-critical and critical.

Quality engineer works closely with Release Managers and Security Engineers to ensure that the provided security fixes are correctly implemented and that they do not cause any regressions in the functionality of the product.

Both types of security releases have a few things in common. Quality engineer:

  1. Gets assigned an issue from Release manager which signals that the QA process can start along with the release package information.
  2. Prepares the environments for testing the security fixes.
  3. Run the automated QA against this dedicated environment.
  4. Notifies the Security Engineer to start the manual QA validations.
  5. Hands-off the release to the Security Engineer and Release Manager.

Non-critical

1. Prepare the environments for testing the security fixes

Confirm if there are any back-ported versions that need to be tested. Generally we limit the back-ports to a total of 3 releases (e.g. if the current release is 10.5, include 10.4 and 10.3).

Repeat the following steps for each back-ported version. Moreover, the security package for the current version is deployed to staging, but if it’s not, follow the steps below for this package too.

  1. In Google Cloud Console (access to this should have been granted during on-boarding), create a new VM instance (in the gitlab-internal project) from the qa-security-1cpu-3-75gb-ram-ubuntu-16-04-lts instance template for each version of GitLab.
  2. Find the .deb package to install:
  3. First find the pipeline for the tag in the pipelines page. For instance, for the 10.5.8+ee.0 Omnibus tag, the pipeline is #81284.
  4. Then on the pipeline page, click the Ubuntu-16.04-staging job in the Staging_upload stage. For instance, for the 10.5.8+ee.0 Omnibus tag, the job is #2379548.
  5. Browse the job’s artifacts, and find the .deb package under pkg/ubuntu-xenial/. For instance, for the 10.5.8+ee.0 Omnibus tag the package page is https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/2379548/artifacts/file/pkg/ubuntu-xenial/gitlab-ee_10.5.8-ee.0_amd64.deb.
  6. Copy the Download link location. For instance, for the 10.5.8+ee.0 Omnibus tag the package link location is https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/2379548/artifacts/raw/pkg/ubuntu-xenial/gitlab-ee_10.5.8-ee.0_amd64.deb.
  7. Install the .deb package from the job artifact:
  8. SSH into the VM via the GCP console.
  9. Create a install-gitlab.sh script in your home folder:
```bash
TEMP_DEB="$(mktemp)" &&
curl -H "PRIVATE-TOKEN: $DEV_TOKEN" "$GITLAB_PACKAGE" -o "$TEMP_DEB" &&
sudo dpkg -i "$TEMP_DEB"
rm -f "$TEMP_DEB"
```
`$DEV_TOKEN` needs to be set with a `dev.gitlab.org` personal access token
so that the script can download the package, `$GITLAB_PACKAGE` needs to be
set to the link location from the job's artifacts.
  1. Change the script’s permission with chmod +x install-gitlab.sh.
  2. Run the script with ./install-gitlab.sh.
  3. Once GitLab installed, set the external_url in /etc/gitlab/gitlab.rb with sudo vim /etc/gitlab/gitlab.rb. You can find the VM’s IP in the GCP console.
  4. Reconfigure and restart GitLab with sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart.
  5. Set the root’s user password:
  6. Visit http://IP_OF_THE_GCP_VM and change root’s password.
  7. Once the environments are ready, capture the information to add to the QA issue.

2. Coordinate the validation of the release

  1. Follow the “Automated QA” steps in the Security QA Issue you’ve been assigned to.
  2. Notify the Security Engineer to verify the security fixes for the release.
  • The manner in which the security fixes are verified can be done in two ways.
    1. By the Quality Engineer executing the validation with close collaboration and guidance from the Security Engineer
    2. By the Security Engineer executing the validation with the Quality Engineer monitoring the steps.
  • Note: When encountered with deadline and resource constraints, the work should be assigned for efficiency. Security Engineer should own verifying complex security validations while Quality Engineer is encouraged to help out with simpler validations. However it is important that the Security team signs off on the result of the validation.
  1. Ensure that all the items for validation are validated and checked off before moving forward
  2. Hand off the release assignment
  3. Once all the validation is completed, Quality Engineer un-assigns themselves from the release issue leaving only the Security Engineer and the Release Manager.

Critical

For now, follow the same process as for Non-critical releases. This will need to be improved.

Return to Security Guide