Kubernetes

Deploy to Kubernetes using Helm. Supports DigitalOcean and Google Cloud clusters.

Import

include:
  - project: hosstio/gitlab-pipelines
    file: pipelines/kubernetes-helm.yaml

Setup

The deploy jobs need three settings that .gitlab-ci.yml can't hold. Setting them on the project's top-level group covers every project in it.

1. Give the jobs access to the chart. The default HELM_REPOSITORY is the private hosstio/helm-charts, cloned with $CI_JOB_TOKEN. Add the group to its job token allowlist (Settings > CI/CD > Job token permissions), or set HELM_REPOSITORY to a chart the project can read. Without either, helm:<environment> fails with Authentication by CI/CD job token not allowed.
2. Give the jobs access to the cluster. Set DIGITALOCEAN_ACCESS_TOKEN as a masked variable, or set the Google Cloud variables. Without them, helm:<environment> fails with kubernetes cluster unreachable.
3. For a private project, give the cluster access to the image. Create a group deploy token with the read_registry scope. Set DOCKER_REGISTRY_PASSWORD to the token as a masked variable, and KUBERNETES_DOCKER_REGISTRY_USER to the token's username. Without them, the deploy skips the registry-credentials-gitlab pull secret, and the pods can't pull the image.

Jobs

Job	Environment	When
helm:preview	preview/<branch> (prepare)	Always
helm:staging	staging (prepare)	Always
helm:production	production (prepare)	Always
preview	preview/<branch>	Feature branch: automatic in a merge request pipeline and its child pipelines, manual on a branch pipeline; auto-stops in 7 days
uninstall:preview	preview/<branch> (stop)	Manual, on merge request merge or close, or at auto-stop; runs helm uninstall
staging	staging	Default branch, automatic; hotfix branch, manual
production	production	Default branch, automatic once test and staging pass
production:hotfix	production	Hotfix branch, manual
rollback:test	test (access)	Default branch, manual once test passes; runs helm rollback
rollback:staging	staging (access)	Default branch, manual once staging passes; runs helm rollback
rollback:production	production (access)	Default branch, manual once production passes; runs helm rollback

To deploy production by hand instead, set when: manual on .deploy:production. To skip waiting for test and staging, override needs on the production job without them; a job's needs list replaces the included one instead of merging with it.

Preview never runs for the default branch or the long-lived develop, main, and master branches. See When pipelines run for how the pipeline type is chosen.

Hotfix branches

A branch with hotfix in its name, in any case, is a regular feature branch with its preview. It can also deploy test, staging and production with a click, from its own image, in branch and merge request pipelines. The dry runs those deploys need run on their own, and each deploy only appears when its environment's values file exists. The gitflow variant does the same.

Each deploy works on its own. Production deploys through the production:hotfix job, which doesn't wait for test and staging the way production does. A project that overrides needs on production sets them on production:hotfix too.

Each environment's rule is a hidden job, .hotfix:test, .hotfix:staging or .hotfix:production. To turn one off, override it:

.hotfix:production:
  rules:
    - when: never

.hotfix holds the condition alone, for a project's own job that a hotfix deploy needs, such as a production image build:

docker:production:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - !reference [.hotfix, rules]

Diff jobs run in the plan stage with needs: [] so they execute immediately at pipeline start. They expose a Helm Diff artifact in the MR and do not register as deployments or send Slack notifications.

Each deployment job also runs helm diff inline before the actual helm upgrade, so the diff is visible in the job log.

Rollback

Each rollback:<environment> job extends its deploy job, so it rolls back the release and namespace that the job deploys and shows up in the same pipelines. A deploy job you add yourself, such as a second production job, gets no rollback job.

The job prints helm history, then runs helm rollback to the last successful revision before the live one. Failed revisions are skipped, and a revision created by a rollback counts as the revision it restored, so running the job again steps back one more revision. It always steps back from the live revision, so the job in an older pipeline does the same as the one in the newest pipeline.

The job records no GitLab deployment (environment:action: access), so the environment page keeps listing the rolled-back deployment as the latest one.

When the job finishes, it posts one line to the deploy channel, or says that the rollback failed:

Rolled back production for my-app from revision 14 to 12 by Jane Smith ⏪

The job refuses to run unless it was created as a manual job. If you override a deploy job's rules with an explicit when, the rollback job inherits that when and fails without touching the release or posting to Slack; give it its own rules with when: manual.

Concurrent deploys

Deploys to one environment run one at a time. test, staging and production each use a resource group of that name, and preview and uninstall:preview share preview/<branch>. A rollback job extends its deploy job, so it waits in the same queue. A deploy that finds the environment busy waits for the running job instead of failing on Helm's release lock.

GitLab picks the next waiting job in no fixed order (the unordered process mode). The newest commit still ends up deployed as long as the project keeps Prevent outdated deployment jobs on (Settings > CI/CD > General pipelines): an older pipeline's deploy job that starts after a newer deployment fails as outdated.

To deploy only the newest pipeline after several quick merges, switch the environment's resource group to newest_ready_first. Neither the project settings nor .gitlab-ci.yml can set this. Change it through the API, once per project and environment, after the group's first deploy has created it:

glab api --method PUT "projects/<group>%2F<project>/resource_groups/production" --field process_mode=newest_ready_first

Version commits

Set HELM_VALUES_COMMIT to any non-empty value, and every successful test, staging and production deploy commits the image values it passed to Helm, the ones HELM_IMAGE_ARGS sets, to HELM_VALUES_COMMIT_BRANCH, the default branch unless you set another. The commit goes to the last file in HELM_VALUES, by default kubernetes/helm/<environment>.yaml, so the values files show which version runs where:

version: "abc12345"

replicaCount: 2

image:
  repository: registry.example.com/my-app/main
  tag: "abc12345"

A key the file doesn't have yet goes where the chart's values.yaml lists it, and keys already in the file stay where they are. Helm applies the last values file over the others, so a helm upgrade with the same values files from that branch installs the running image. The rest of the file can hold config merged after the last deploy.

The standard chart also takes the image as one string. To commit it that way, set HELM_IMAGE_ARGS as below. The commit won't replace an existing image map with a string, so delete the map by hand when you switch:

variables:
  HELM_IMAGE_ARGS: "--set-string image=$$DOCKER_IMAGE_BRANCH --set-string version=$$DOCKER_IMAGE_VERSION"

The job pushes with its CI/CD job token. GitLab allows that from version 18.4, once you turn on Allow Git push requests to the repository under Settings > CI/CD > Job token permissions, or make this API call:

glab api --method PUT "projects/<group>%2F<project>" --field ci_push_repository_for_job_token_allowed=true

The token has the permissions of the user who started the pipeline. That user needs push rights on that branch and becomes the commit's author. A job token push starts no pipeline.

The job token pushes over HTTPS, so a group that allows Git over SSH only refuses it with Git access over HTTP is not allowed. Allow both protocols under the group's Settings > General > Permissions and group features.

A rollback:<environment> job commits the values of the revision it restored. Previews don't commit; every other deploy does, also from a hotfix branch, a merge request pipeline or a tag. In the gitflow variant, where the default branch is develop, set HELM_VALUES_COMMIT_BRANCH to the branch your manual Helm runs read. To record production only, set HELM_VALUES_COMMIT on .deploy:production; rollback:production follows it. Any non-empty value turns the commit on, "false" included.

A failed commit doesn't fail the deploy. The job log says why, and the Slack message ends with Version not committed to git. The commit refuses files with Windows line endings, and HELM_IMAGE_ARGS keys with brackets or values with commas, which Helm reads differently. It also refuses a file where yq would change other lines, such as a list without indentation, and lists those lines in the job log. Push rules that require signed commits or code owner approval refuse it too.

The deploy keeps setting the image with --set: the tag is the pipeline's own commit SHA, and no commit can contain its own SHA.

Each deploy adds a commit to that branch, up to three per merge. With semi-linear history or fast-forward merges, open merge requests need a rebase after each one.

Gitflow variant

include:
  - project: hosstio/gitlab-pipelines
    file: pipelines/kubernetes-helm-gitflow.yaml

Job	Environment	When
helm:preview	preview/<branch> (prepare)	Always
helm:test	test (prepare)	Always
helm:staging	staging (prepare)	Always
helm:production	production (prepare)	Always
preview	preview/<branch>	Feature branch: automatic in a merge request pipeline and its child pipelines, manual on a branch pipeline; auto-stops in 7 days
uninstall:preview	preview/<branch> (stop)	Manual, on merge request merge or close, or at auto-stop; runs helm uninstall
test	test	Default branch (develop), automatic; hotfix branch, manual
staging	staging	Tag push, automatic; hotfix branch, manual
production	production	Tag push, manual; the button unlocks once staging passes
production:hotfix	production	Hotfix branch, manual
rollback:test	test (access)	Default branch (develop), manual once test passes; runs helm rollback
rollback:staging	staging (access)	Tag push, manual once staging passes; runs helm rollback
rollback:production	production (access)	Tag push, manual once production passes; runs helm rollback

Variables

Helm

Variable	Default	Description
HELM_CHART	./helm-charts/charts/standard	Helm chart path
HELM_CHART_VERSION	""	Version to pin: semver for OCI/helm repo, branch/tag for git; omit for latest/default branch
HELM_RELEASE	$CI_PROJECT_NAME	Helm release name
HELM_NAMESPACE	$KUBERNETES_NAMESPACE	Kubernetes namespace
HELM_REPOSITORY	gitlab.com/hosstio/helm-charts.git	Chart source; see the helm helper for auto-detection rules
HELM_ARGS	--install --wait --server-side=true --force-conflicts	Core helm arguments; see manual changes
HELM_EXTRA_ARGS	""	Additional arguments
HELM_ARGS_SECRET	""	Secret arguments (masked in logs)
HELM_EXTRA_ARGS_SECRET	""	Additional secret arguments (masked in logs)
HELM_IMAGE_ARGS	--set image.repository=$$DOCKER_IMAGE_REPOSITORY_BRANCH --set-string image.tag=$$DOCKER_IMAGE_VERSION --set-string version=$$DOCKER_IMAGE_VERSION	Image values passed to Helm; see Docker image
HELM_VALUES_COMMIT	""	Any non-empty value commits the HELM_IMAGE_ARGS values to HELM_VALUES_COMMIT_BRANCH after each deploy; see version commits
HELM_VALUES_COMMIT_BRANCH	$CI_DEFAULT_BRANCH	Branch the version commits go to
HELM_VALUES_DIR	kubernetes/helm	Directory with the values files; see Helm values

Hostnames

Variable	Default	Description
DOMAIN	example.com	Base domain used as default for all hostname variables
HOSTNAME_PREVIEW	$CI_COMMIT_REF_SLUG.$DOMAIN	Hostname for preview environments; typically overridden per project
PREVIEW_AUTO_STOP	7 days	How long before a preview environment is automatically stopped; accepts any GitLab duration string
HOSTNAME_TEST	test.$DOMAIN	Hostname for the test environment
HOSTNAME_STAGING	staging.$DOMAIN	Hostname for the staging environment
HOSTNAME_PRODUCTION	$DOMAIN	Hostname for the production environment

Kubernetes

Variable	Default	Description
KUBERNETES_NAMESPACE	$CI_PROJECT_NAME	Namespace for deployments
KUBERNETES_CLUSTER	kubernetes	Cluster name
KUBERNETES_DOCKER_REGISTRY_SECRET	registry-credentials-gitlab	Image pull secret name
PREVIEW_KUBERNETES_NAMESPACE	preview-$GITLAB_PROJECT_SLUG	Namespace for preview deploys and their uninstall
PREVIEW_HELM_RELEASE	preview-$CI_COMMIT_REF_SLUG	Release name for preview deploys and their uninstall

To relocate previews, override PREVIEW_KUBERNETES_NAMESPACE/PREVIEW_HELM_RELEASE rather than setting KUBERNETES_NAMESPACE/HELM_RELEASE on .deploy:preview. The uninstall:preview stop job is standalone (it cannot extend .deploy:preview), so an override on the deploy job alone never reaches it, and the uninstall then targets the default namespace while the relocated release stays behind.

Helm values

Values are merged from two files per environment, in HELM_VALUES_DIR (kubernetes/helm by default):

• values.yaml: shared values
• <environment>.yaml: environment-specific values (e.g. staging.yaml, production.yaml)

An environment's deploy and dry run only run when its <environment>.yaml exists in that directory.

Apps in a subdirectory

An app whose values files aren't in kubernetes/helm sets HELM_VALUES_DIR to their directory, such as deploy/helm. The deploy rules, the values files and the version commits all follow it.

A monorepo with several apps sets these variables per app:

Variable	Example	Why per app
HELM_VALUES_DIR	services/backend/kubernetes/helm	Each app has its own values files
HELM_RELEASE	backend	The default, $CI_PROJECT_NAME, is the same for every app
GITLAB_PROJECT_NAME	Backend	Names the app in the deploy messages
GITLAB_PROJECT_SLUG	backend	Names the app's resources in a preview
DOCKER_IMAGE_REPOSITORY	$CI_PROJECT_PATH/backend	Gives each app its own image
DOCKER_BUILD_DIR, DOCKER_FILE	services/backend, services/backend/Dockerfile	Builds the app from its own directory

Either the root pipeline triggers a child pipeline per app from the app's own pipeline file, which sets them as global variables, or one pipeline defines its own jobs per app, which extend .deploy:<environment> and .deploy:preview and set them on each job. In one pipeline, the shared deploy jobs and their dry runs, such as staging and helm:staging, don't run while the repository root has no kubernetes/helm/<environment>.yaml.

GitLab keeps one list of environments and resource groups per project, so each app needs its own names. A child pipeline overrides the environment names where the jobs get them:

.deploy:
  environment:
    name: backend/$ENVIRONMENT

.deploy:preview:
  environment:
    name: backend/preview/$CI_COMMIT_REF_SLUG

uninstall:preview:
  environment:
    name: backend/preview/$CI_COMMIT_REF_SLUG

Resource groups are set on each deploy and dry run job, such as staging and preview/<branch>. Without an override per job, one app's deploy waits for another app's deploy to the same environment.

Docker image

The pipeline passes HELM_IMAGE_ARGS to Helm, by default --set image.repository, --set-string image.tag and --set-string version from $DOCKER_IMAGE_BRANCH. The tag and version go in as strings, because --set turns an all-digit short SHA into a number. If $DOCKER_IMAGE_BRANCH is not set (e.g. for infrastructure-only charts), image args are skipped entirely.

Set HELM_IMAGE_ARGS when the chart reads the image from other keys. Put $$ before the image variables, which the job fills in while it runs: $$DOCKER_IMAGE_BRANCH is the whole reference, $$DOCKER_IMAGE_REPOSITORY_BRANCH the reference without its tag, and $$DOCKER_IMAGE_VERSION the tag.

Running a deploy locally

Every deploy job resolves to a single helm command. To reproduce a staging/production deploy, or a helm:<environment> plan that failed in CI, run the same command from your own machine against a local chart checkout.

1. Authenticate to the target cluster and select its namespace. See Cloud providers for the credentials each provider needs.
2. Check out the chart the pipeline uses. For a git-hosted chart (the default) that's the HELM_REPOSITORY repository, with HELM_CHART as the path within it.
3. From the application repository (so the relative kubernetes/helm/*.yaml paths resolve), run helm with the values files and image args the job builds.

The job variables map to command arguments like this:

Variable	Argument
HELM_ACTION (upgrade)	subcommand
HELM_RELEASE	release name
HELM_CHART	chart path
HELM_NAMESPACE	--namespace
HELM_VALUES	one --values per file
HELM_IMAGE_ARGS	as set, by default --set image.repository=… --set-string image.tag=… --set-string version=…
HELM_ARGS	--install --wait --server-side=true --force-conflicts

The staging job runs the deploy:

helm upgrade my-app /path/to/helm-charts/charts/standard \
  --namespace my-namespace \
  --set image.repository=registry.example.com/my-app/main \
  --set-string image.tag=abc12345 \
  --set-string version=abc12345 \
  --values kubernetes/helm/values.yaml \
  --values kubernetes/helm/staging.yaml \
  --install --wait --server-side=true --force-conflicts

A helm:<environment> job runs the plan: it only diffs, applies nothing, and needs the helm-diff plugin:

helm diff upgrade my-app /path/to/helm-charts/charts/standard \
  --namespace my-namespace \
  --set image.repository=registry.example.com/my-app/main \
  --set-string image.tag=abc12345 \
  --set-string version=abc12345 \
  --values kubernetes/helm/values.yaml \
  --values kubernetes/helm/staging.yaml \
  --allow-unreleased --suppress-secrets --three-way-merge

To catch a bad-values or template error without touching a cluster, render the manifests instead:

helm template my-app /path/to/helm-charts/charts/standard \
  --values kubernetes/helm/values.yaml \
  --values kubernetes/helm/staging.yaml

The three --set values come from $DOCKER_IMAGE_BRANCH; the tag is the short commit SHA the pipeline built ($CI_COMMIT_SHORT_SHA). For an infrastructure chart with no application image, drop them. With version commits on, the values files on HELM_VALUES_COMMIT_BRANCH already hold the deployed image, so a deploy from there can drop them too.

Slack notifications

Each deployment job sends a Slack message on completion. The message format and attachments depend on the environment.

Preview

Preview uses a compact single-line message with the commits listed below it:

my-app (preview) deployment successful for version my-feature (abc12345) by Jane Smith :rocket:
Changes
- Fix login redirect (*abc12345*) - Jane Smith
- Update dependencies (*def67890*) - Jane Smith

Configuration and Kubernetes diff attachments are suppressed when the only change is the image tag or chart version; the commits list already shows what changed. A green preview carries no pod list or container logs either, unless they need attention on the same terms as staging and production below. Preview messages never carry the Release Notes attachment.

When uninstall:preview removes the preview, it posts to the same channel:

Preview uninstalled for my-app version my-feature by Jane Smith 🗑️

The message always posts on its own, never in the branch's preview thread. A failed uninstall says so. The next preview deploy of the branch starts a new thread.

Staging and production

Staging and production send the full message: the app URL beside a View app button, the commits with author and link per commit under Changes, and attachments that depend on the outcome. SLACK_MESSAGE_TEXT_SUCCESS and SLACK_MESSAGE_TEXT_FAILURE replace the URL with your own text.

On success

Attachment	Content	Condition
Configuration	Values diff including chart version, first 50 lines	Only when values changed beyond tag/version
Kubernetes	kubectl get output for the release, first 50 lines	Only when the deployment needs attention (see below)
Kubernetes Containers (logs)	Last 50 lines of pod logs	Only when the deployment needs attention (see below)
Release Notes	Release description, tag message, or CHANGELOG.md changes	Only when present

A successful deployment needs attention when a pod is in a failed state (CrashLoopBackOff, OOMKilled, ImagePullBackOff, ErrImagePull, Error, Failed, Unknown), a pod has restarted more than five times, or the container logs contain a warning or an error. Both the Kubernetes and Kubernetes Containers attachments are sent together on that signal, so the pod list can be read against the logs; neither is sent when the deployment is healthy, where the message title already says everything. Staging and production titles keep the success emoji either way; only preview thread replies swap it for :warning: on restarts or :danger: on failed pods or dirty logs.

The Configuration diff includes a chart: line on both sides so chart upgrades are visible alongside value changes.

On failure

Attachment	Content	Condition
Job logs	Last 10 lines of stderr	Always
Kubernetes	kubectl get output for the release, first 50 lines	Only when state is available
Kubernetes Containers (logs)	Last 50 lines of pod logs	Only when logs are available
Rollback	helm rollback output	Only when rollback ran

Changes and release notes

The Changes list holds the commits since the previous push to the branch. A new branch and a merge request pipeline list every commit since the merge base with the default branch (GIT_COMMITS_BASE_BRANCH); in merge request pipelines GitLab sets CI_COMMIT_BEFORE_SHA to the pipeline's own commit. Merge commits are excluded (git log --no-merges).

Release notes are taken from the first of these sources that has content:

• Tag pipelines: release description → tag message → matching CHANGELOG.md section (up to 100 lines)
• Branch pipelines: lines added to CHANGELOG.md in this push only (only when CHANGELOG.md changed)

Examples

Public Helm chart (infrastructure)

For charts that have no application Docker image (cert-manager, ingress controllers, monitoring stacks), set HELM_REPOSITORY, HELM_REPOSITORY_NAME, and HELM_CHART to point at the public repository and disable preview environments.

A complete example is in examples/cert-manager/.

# .gitlab-ci.yml
include:
  - project: hosstio/gitlab-pipelines
    file: pipelines/default.yaml
  - project: hosstio/gitlab-pipelines
    file: pipelines/kubernetes-helm.yaml

variables:
  HELM_REPOSITORY: https://charts.jetstack.io
  HELM_REPOSITORY_NAME: jetstack
  HELM_CHART: jetstack/cert-manager
  HELM_RELEASE: cert-manager
  HELM_CHART_VERSION: "v1.17.0"
  KUBERNETES_NAMESPACE: cert-manager
  KUBERNETES_CLUSTER: my-cluster
  DOMAIN: example.com

.deploy:
  variables:
    HELM_ARGS: "--install --wait --create-namespace --server-side=true --force-conflicts"

.deploy_preview:
  rules:
    - when: never

Values are loaded from kubernetes/helm/values.yaml (shared) and kubernetes/helm/<environment>.yaml (per-environment) as usual; the chart source is the only difference from a custom chart deployment.

OCI chart (infrastructure)

For charts published to an OCI registry, set HELM_REPOSITORY to the oci:// URL. HELM_CHART is set automatically from the last path component after pull, and HELM_REPOSITORY_NAME is not required.

A complete example is in examples/f5-nginx-ingress/.

# .gitlab-ci.yml
include:
  - project: hosstio/gitlab-pipelines
    file: pipelines/default.yaml
  - project: hosstio/gitlab-pipelines
    file: pipelines/kubernetes-helm.yaml

variables:
  HELM_REPOSITORY: oci://ghcr.io/nginxinc/charts/nginx-ingress
  HELM_RELEASE: nginx-ingress
  HELM_CHART_VERSION: "3.7.0"
  KUBERNETES_NAMESPACE: nginx-ingress
  KUBERNETES_CLUSTER: my-cluster
  DOMAIN: example.com

.deploy:
  variables:
    HELM_ARGS: "--install --wait --create-namespace --server-side=true --force-conflicts"

.deploy_preview:
  rules:
    - when: never

.deploy_staging:
  rules:
    - if: $CI_PIPELINE_SOURCE != "merge_request_event" && $CI_COMMIT_BRANCH

Cloud providers

DigitalOcean

Set these CI/CD variables to connect to a DigitalOcean Kubernetes cluster:

Variable	Description
DIGITALOCEAN_ACCESS_TOKEN	DigitalOcean personal access token
DIGITALOCEAN_KUBERNETES_CLUSTER	Cluster name (defaults to $KUBERNETES_CLUSTER)

Google Cloud

Set these CI/CD variables to connect to a GKE cluster:

Variable	Description
GOOGLE_SERVICE_ACCOUNT	Path to service account key JSON file
GOOGLE_KUBERNETES_CLUSTER	GKE cluster name
GOOGLE_PROJECT	Google Cloud project ID
GOOGLE_REGION	Region (default: europe-west1-b)