Background Migrations

Background Migrations

The purpose of this document is to help determine if there are background migrations in a select deployment and to determine whether or not migrations are actively running.

For more information on Background Migrations, consult the development documentation.

Determine if latest deploy contains Background Migrations

All Background Migrations are located in lib/gitlab/background_migration. When looking at the diff between two versions of GitLab, this is easiest via the command line using git diff --name-only <commit>..<commit> -- lib/gitlab/background_migration

Example:

% git diff --name-only v12.0.0-ee..v12.1.0-ee -- lib/gitlab/background_migration
lib/gitlab/background_migration/add_merge_request_diff_commits_count.rb
lib/gitlab/background_migration/archive_legacy_traces.rb
lib/gitlab/background_migration/calculate_wiki_sizes.rb
lib/gitlab/background_migration/create_fork_network_memberships_range.rb
lib/gitlab/background_migration/delete_conflicting_redirect_routes_range.rb
lib/gitlab/background_migration/fill_valid_time_for_pages_domain_certificate.rb
lib/gitlab/background_migration/fix_cross_project_label_links.rb
lib/gitlab/background_migration/fix_pages_access_level.rb
lib/gitlab/background_migration/fix_user_namespace_names.rb
lib/gitlab/background_migration/fix_user_project_route_names.rb
lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb
lib/gitlab/background_migration/migrate_null_private_profile_to_false.rb
lib/gitlab/background_migration/migrate_system_uploads_to_new_folder.rb
lib/gitlab/background_migration/move_personal_snippet_files.rb
lib/gitlab/background_migration/normalize_ldap_extern_uids_range.rb
lib/gitlab/background_migration/populate_fork_networks_range.rb
lib/gitlab/background_migration/populate_merge_request_assignees_table.rb
lib/gitlab/background_migration/populate_merge_requests_latest_merge_request_diff_id.rb
lib/gitlab/background_migration/populate_untracked_uploads.rb
lib/gitlab/background_migration/prepare_untracked_uploads.rb

Determine if a Background Migration is actively running

There are at least 4 ways to discover this information.

Metrics (Preferred)

All background migrations run inside of a queue called background_migration. We can then utilize the metric sidekiq_queue_size to determine if there are Background Migrations queued ready to be pulled:

sidekiq_queue_size{name="background_migration"}

And we can also utilize the class name of the worker to determine if there are migrations actively running:

sidekiq_running_jobs_count{name="BackgroundMigrationWorker"}

Here’s a Thanos Query of the above

Here’s a Dashboard that may contain the above

Logs

We can view logs from sidekiq. Utilize this saved query on Kibana

Rails Console

⚠️

This is not desired as not all members that manage the responsibility of said information can actually get this data. Or in the case of an incident, the ability to get this information may be hindered.

⚠️

  1. SSH into the console node of the target environment, example:
ssh -l <username>-rails console-01-sv-gstg.c.gitlab-staging-1.internal
  1. Execute the GitLab::BackgroundMigration.remaining method:
[ gstg ] production> Gitlab::BackgroundMigration.remaining
=> 0

The above steps are documented in GitLab’s Upgrade Documentation

Admin Panel of GitLab

⚠️

This is not desired as not all members that manage the responsibility of said information can actually get this data. Or in the case of an incident, the ability to get to this page may be hindered.

⚠️

  1. Log into GitLab using administrative credentials
  2. Navigate to the Admin Area
    • Monitoring
    • Background Jobs
    • Queues
  3. Find the queue background_migration to find the required data

Background Migration is taking a very long time

Consult What do I do if my background migrations are stuck section in GitLab’s Upgrade Documentation.