Release Manager
The release manager drives the monthly release of GitLab, any patch releases and security releases for that version.
To find the current release managers, please consult the release manager schedule.
Responsibilities
You as a release manager have a responsibility to deliver the work of every single person involved in creating the product and running the application at GitLab.com.
This responsibility sometimes requires making difficult decisions. For example, you might need to refuse including a feature or a change within a release. You might need to decide on shipping a feature that is not working as expected to allow for other features/fixes to be available. You might need to revert someone’s work because it was impacting the release schedule.
The decisions you make can have a cascading effect on all other release tasks, so make sure that you collect as much data as you can within the time you have, make an informed decision, and stick with it.
At all times, keep in mind that protecting GitLab users and the stability of GitLab.com are more important than accepting a change in release to please someone from you or another team.
When you start as a release manager, some of your responsibilities are:
- Monitoring the GitLab.com deployments
- Ensuring GitLab’s monthly, security, and patch releases are released in a timely fashion.
- Escalating to responsible parties in case a release task is slowed down or blocked by their area of responsibility
- Handling deployment failures
- Acting as DRI for “Near Miss” incidents relating to deployments
- Ensuring that the automated QA is successfully completed
- Understanding the impact of the deployment on environment
The monthly self-managed releases are a company-wide effort, and should not fall entirely on the release manager’s shoulders. More about that in the sections that follow.
Just one final thing before you get started: keep calm and move fast!
Getting Started
If you are assigned to the release management duty, the first course of action should be:
- To onboard and,
- To create a monthly release issue.
With access to tools required to operate the releases, you can start with your first task.
Release Manager Cheatsheet
The below lists concentrate the usual commands used by release managers in their day-to-day, release, and deployment activities.
Frequently used commands
| Command | Description |
|---|---|
/chatops run auto_deploy status | Returns the current package for each environment (gstg, gprd, gprd-cny, gstg-cng and gstg-ref) |
/chatops run post_deploy_migrations execute | Executes the post-deploy migration pipeline |
/chatops run auto_deploy blockers | Returns any blocker preventing deployments |
Monthly, patch and security releases commands
| Command | Description |
|---|---|
| Patch release specific | |
/chatops run release prepare 15.x.x | Creates a release task issue and preparation branches for the 15.3.3 patch release |
/chatops run release merge 15.x.x | Merges merge requests with ~Pick into 15.x into the prep branch |
/chatops run release status 15.x.x | Checks the progress of the 15.x.x packages |
| Tags and publishing - Monthly and patch release specific | |
/chatops run release tag <VERSION> | Used to tag a release candidate, a monthly version or a patch version |
/chatops run release tag 15.x.x-rc42 --gitlab-sha=XXX | Tags a release candidate for 15.x using a specific SHA from the gitlab-org/gitlab project |
/chatops run publish <VERSION> | Publishes the packages associated with the version. Used in monthly and patch releases |
| Security specific | |
/chatops run release prepare --security | Creates a release task issue for the upcoming security release |
/chatops run release merge --security --default-branch | Merges GitLab security merge requests targeting the default branch |
/chatops run release merge --security | Merges GitLab backports and any pending security merge request |
/chatops run auto_deploy security_status | Verifies if all security merge requests have been deployed to GitLab.com |
/chatops run release status --security | Ensure tests are green on CE and EE for security stable branches |
/chatops run release tag --security 15.x.x | Tags 15.3.3 as a security release |
/chatops run publish --security | Publishes the security release |
| Utilities | |
/chatops run mirror status | Returns the mirror status of all GitLab repositories. Used in all kind of releases. |
Manual deployments and rollbacks commands
| Command | Description |
|---|---|
/chatops run deploy <VERSION> <ENVIRONMENT> | Deploys the auto-deploy package (version) to a specific environment (gstg, gprd, gstg-cny, gprd-cny, pre or release) |
/chatops run deploy <VERSION> gprd --ignore-production-checks 'deployment approved by on call SRE' | Forces a deploy to production (gprd) and overrides production checks |
/chatops run canary --<ENVIRONMENT> | Verifies the status of canary on the target environment production or staging |
/chatops run canary --disable --<ENVIRONMENT> | Disables canary on the target environment, either production or staging |
/chatops run rollback check gstg | Checks if a rollback is feasible in gstg |
/chatops run deploy <VERSION> <ENVIRONMENT> --rollback | Rollback the auto-deploy package (version) to a specific environment (gstg, gprd) |
Utilities
| Command | Description |
|---|---|
/chatops run auto_deploy pause | Pauses the auto-deploy processes. Used on Family and Friends day and PCLs |
/chatops run auto_deploy unpause | Re-enables the auto-deploy processes |
/chatops run auto_deploy lock <ENVIRONMENT> | Prevents deployments to a specific environment (gstg, gprd, gstg-cny, gprd-cny, pre or release) |
/chatops run auto_deploy unlock <ENVIRONMENT> | Allows deployments to a specific environment (gstg, gprd, gstg-cny, gprd-cny, pre or release) |
Onboarding
Create a new issue in the release/tasks project and select the
RM-Onboarding template. Use the title Onboarding Release Manager YOUR_NAME_HERE.
Assign the issue to the previous Release Manager in your timezone.
The tasks are ordered by priority and should be completed before starting your appointed release.
Permissions
To fullfil their responsibilities, Release Managers require push and merge access on protected branches in the following repositories:
- GitLab
- GitLab FOSS
- Omnibus-GitLab
- Gitaly
- GitLab Chart
- GitLab Agent
- GitLab Pages
- GitLab Workhorse
- CNG
- GitLab Elasticsearch Indexer
Permissions are granted through the @gitlab-org/release/managers group which Release Managers
are added automatically when their shift starts.
@gitlab-org/release/managers group has the following permissions:
For Canonical and Security repositories
Release Managers:
- Have
maintaineraccess. - Are allowed to
pushandmergeto*-auto-deploy-*and*-stablebranches. - Are allowed to
pushto the default branch (e.gmasterormain).
For Dev repositories
Dev is a CE instance, therefore the ‘Protected branch’ feature is limited to role. In this case, ~“team::Delivery” members have maintainer access to all the projects.
Bot permissions
@gitlab-bot and @gitlab-release-tools-bot have maintainer role in all the projects
mentioned above. This is necessary since they perform release operations such as mirroring,
Gitaly updates, merging security merge requests, among others. They are specifically allowed to push
and merge on all protected branches.
Training
Now is a good time to talk with the previous Release Managers. They should be able to answer any question you have. If they don’t know the answer, they should direct you to a person who might know. Feel free to go as far as necessary to get your answer. Do remember to document when you find that answer!
Release Candidates
Release Candidate (RC) is a point in time snapshot of what will become a release. Any RC that gets created can be considered for final release.
Deployment
The release manager is also responsible for promoting the latest versions through GitLab.com environments.
QA task
Quality assurance (QA) is how we reduce the possibility of shipping a broken feature. We have two types of QA, automated and manual.
Automated QA is ran with deployment pipelines. Manual QA is in a form of an issue with a list of MRs that engineers need to review through.
Your responsibility is to monitor the automated QA for successful runs and the issue in case something is reported there. As engineers check off items they may raise a concern in this issue and together with them it is necessary to assign a correct priority.
Issues that could cause an incident if deployed should be raised as Deployment blockers and owned by the Release Manager.
Find out more about QA task in a separate qa documentation.
Post-deployment Patch
Post-deployment patches are something we manage the lifecycle of. It would be wise to read up and get to know the process as outlined in the documentation
Patch Release
As the final self-managed release gets published to users on the 22nd, it is expected to have a number of patch releases to resolve any outstanding bugs reported after the release.
The amount and scheduling of patch releases is entirely at your discretion as the release manager.
Exception is reserved for critical security releases, which should be addressed immediately. Security releases diverges from this process.
If a bug affects a large number of users and/or a critical piece of functionality, it’s fine to release a patch with only one fix. Sometimes a patch will include five or more minor fixes. You should use your best judgment to determine when a patch release is warranted. If you are not sure, you can always ask for help on deciding at #releases. We strive to continue releasing patches until all (or most) known regressions for that release are addressed.
To help you understand what is expected from you when doing patch releases, check out Patch release documentation.
Guidelines for Release Issue Assignment
The Delivery team holds weekly status meetings where we decide which members of the team will become Release Managers for upcoming releases. We assign Release Managers to our monthly releases and an MR is created to update the release manager schedule appropriately. Monthly Release Issues, when created, will automatically be assigned to the correct Release Manager based on the data stored in the release manager schedule.
Auto-Deploy Transition
In between monthly releases, auto-deploy continues to operate and this creates a transition window where we shift between the existing RM’s to the new RM’s. This window is loosely defined and should be agreed upon by the two sets of RM’s. The transition can be considered complete when the next monthly release issue is created.
Patch Release Assignment
Patches are unique as there are many varieties of them. A guideline to follow is this:
- RM’s assigned to X.1.0 will perform non critical patch releases until X.2.0
prep work begins
- This style of patch release is normally unrelated to the upcoming X.2.0 release.
- This patch is usually not impacting auto-deploy which is the upcoming RM’s primary responsibility.
- If a critical security issue is impacting auto-deploy leading up to the X.2.0
release, the RM’s associated with X.2.0 should work on that patch.
- This is due to the nature of what is required to maintain the auto-deployment procedure
- Since backports of the security patch will most likely be very similar, logistically, it’s easier for only one set of RM’s to be working on a release.
- Avoid situations where a group of patch releases that are to be released at
the same time, are worked on by differing sets of RM’s
- This can create problems with communications for all Engineers involved
Release Issue Creation
Because release tooling looks at the release manager schedule for issue assignment, issues may be automatically assigned to previous release managers. It is standard practice to assign created release issues to the desired Release Manager if it differs.
Deployment blockers
For any issues that block our ability to deploy, such as high severity bugs, merge conflicts, or failing migrations we should follow the steps in Deployment blockers. Being unable to deploy limits our ability to respond to a S1 incident/security issue and so we must move fast to unblock deployments.
Release Managers will often be the Owner of the incident.
Offboarding
Once there’s no immediate action to create a patch release, Release Managers duties have finished. To recover from the pressure of release management it is strongly encouraged that you take a week of PTO following your rotation.
Temporary permissions
Members of the delivery teams, can temporarily join the release managers. Joining is temporary and the regular permissions are restored daily at 4 UTC by a scheduled task.
Running the following CI jobs will allow the current user to join/leave all the release managers’ groups and the slack handle.