Blog 14.10.2019

Very secret development operations part III: CI-pipelines

Good to see you here! We have no doubt this post has good information, but please keep in mind that it is over 5 years old.

This blog series is split into four parts:

  1. General information of secrets management
  2. Example how to store secrets into a version control
  3. Example how to use encrypted secrets in CI-pipelines
  4. Security issue and threat analysis based on previous examples

After the secrets are stored securely on a version control, it is time to look how a continuous integration pipeline should handle the secrets. In this guide, I’ll use Gitlab’s pipelines.

Configuring GCP

First of all, you have to create a new service account and a new KMS key which is used only by the service account.

  1. Head to https://console.cloud.google.com/iam-admin/serviceaccounts, select the project used and create a service account
  2. Create the private key in JSON format and temporarily store it on your computer. It will be used later when configuring Gitlab CI-pipeline.
  3. Then open https://console.cloud.google.com/security/kms, select your preferred keyring (or create a new one) and create a new key
    1. Purpose: Symmetric encrypt/decrypt
    2. Rotation period: 7 days (I’ll tell about this on the last blog post)
  4. Select the key created and add the service account with decryption access (‘Cloud KMS CryptoKey Decrypter’) to it


Modifying access to the cryptographic key

Encrypting with a new key

If you have already encrypted secrets, you must add the key to the secrets after setting up the new cryptographic key.

# Adding GCP key
sops -r -i --add-gcp-kms projects/<gcp project>/locations/global/keyRings/sops/cryptoKeys/gitlab-test api-key.json
# Removing GCP key
# sops -r -i --rm-gcp-kms projects/<gcp project>/locations/global/keyRings/sops/cryptoKeys/gitlab-test api-key.json
# -r option: 'generate a new data encryption key and reencrypt all values with the new key'

For .sops.yaml configuration, you have to add the new cryptographic key and re-encrypt the secrets. The previous blog post contained the Git pre-commit script which will re-encrypt all secrets if the configuration file is changed.

.sops.yaml
creation_rules:
  # Global enc-files (typically for testing and dev-environment)
  - path_regex: .*.enc(.yaml|.json)?$
    gcp_kms: projects/<gcp dev project>/locations/global/keyRings/sops/cryptoKeys/dev-sops-key,projects/<gcp dev project>/locations/global/keyRings/sops/cryptoKeys/gitlab-test

Gitlab configuration

Variables

You can add variables for the CI-pipeline in the project or group settings. I’ll add the following variables to the project settings (Gitlab > Project > Settings > CI/CD > Variables).

Key
Value
GCLOUD_PROJECT_NAME <project name>
GCLOUD_SERVICE_KEY <service account key file content in JSON-format)
SOPS_VERSION 3.4.0

CI Pipeline

Build contains following steps:

  1. Authenticate and setup gcloud CLI
  2. Install SOPS
  3. Decrypt secrets and cache them for the next build steps
    1. decrypt_secrets.sh -script is the same that was used in the previous blog post.
.gitlab-ci.yml
decrypt secrets:
  image: google/cloud-sdk:alpine
  cache:
    # Caches decrypted secrets to next build step (among with other untracked files)
    untracked: true
  script:
    # Authentication for google cloud
    - echo "$GCLOUD_SERVICE_KEY" > ${HOME}/gcloud-service-key.json
    - export GOOGLE_APPLICATION_CREDENTIALS=${HOME}/gcloud-service-key.json
    - gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
    - gcloud config set project $GCLOUD_PROJECT_NAME
     
    # Install SOPS
    - apk update && apk add --no-cache ca-certificates
    - wget https://github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux -O /usr/local/bin/sops
    - chmod 0755 /usr/local/bin/sops
     
    # Install GNU findutils for proper regex support
    - apk add findutils
    - sh decrypt_secrets.sh

Gitlab missing features and bugs

In the last part of blog series: what vulnerabilities my examples could contain and how to manage them.

Systems Development & Integration

Jarkko Koistinaho

Jarkko works as a technical project manager at Gofore and he is a quality and testing oriented professional in the software industry. Depending on the situation, he could be a software engineer or a Scrum Master in addition to being a software tester. Jarkko can also do some DevOps-tasks. Model-based testing and performance testing are his special skills.

Back to top