Very secret development operations part II: Storing skeletons into a version control

This blog series is split into four parts:

General information of secrets management
Example how to store secrets into a version control
Example how to use encrypted secrets in CI-pipelines
Security issue and threat analysis based on previous examples

This time I’ll give a straight forward example how to use Mozilla SOPS with GCP KMS and Git.

Prerequisites

This guide is based on following technologies:

Debian-based OS
Mozilla SOPS 3.4.0
GCP’s Key Management Service (KMS)
Git

Actions:

Download and install SOPS on local machine

On Debian-based OS

wget https://github.com/mozilla/sops/releases/download/3.4.0/sops_3.4.0_amd64.deb
sudo dpkg -i sops_3.4.0_amd64.deb
rm -f sops_3.4.0_amd64.deb

Releases for other operating systems

Download proper installation file of Mozilla SOPS from releases.

Setup Google KMS and key

Make Google Cloud SDK authentication

gcloud auth application-default login

Create a new keyring

# Create a new keyring
gcloud kms keyrings create sops --location global

If the following error occurs: ”ERROR: (gcloud.kms.keyrings.create) FAILED_PRECONDITION: Google Cloud KMS API has not been used in this project before, or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=<id> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.”. Follow the link and enable Google KMS API.

Create a new key

# Create a new key to the keyring
gcloud kms keys create dev-sops-key --location global --keyring sops --purpose encryption
# List all keys in the keyring
gcloud kms keys list --location global --keyring sops
NAME                                                                           PURPOSE          ALGORITHM                    PROTECTION_LEVEL  LABELS  PRIMARY_ID  PRIMARY_STATE
projects/<gcp project>/locations/global/keyRings/sops/cryptoKeys/dev-sops-key  ENCRYPT_DECRYPT  GOOGLE_SYMMETRIC_ENCRYPTION  SOFTWARE                  1           ENABLED

SOPS usage

Basic usage – encrypt and decrypt existing file

# Encrypt
sops --encrypt --gcp-kms projects/<gcp project>/locations/global/keyRings/sops/cryptoKeys/dev-sops-key api-key.json > api-key.enc.json
# Decrypt
sops --decrypt api-key.enc.json > api-key.json

If the following error occurs on encryption: ”Could not generate data key: [failed to encrypt new data key with master key ”projects/<gcp project>/locations/global/keyRings/sops/cryptoKeys/dev-sops-key”: Cannot create GCP KMS service: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.]”. You have to make Google Cloud SDK authentication (a few steps earlier in this post)

Basic usage – create or modify encrypted file by SOPS

With the default text editor, you can create or modify encrypted file by SOPS by running following command:

sops api-key.enc.json

Creating a new encrypted file needs key parameters like those used in the previous step. But if you have configured creation rules for SOPS, you don’t have to add any key parameter – so head to the next step.

Advanced usage – Creation rules configuration

Create a file named .sops.yaml to root directory where you can set specific encrypted file creation rules (example content below).

.sops.yaml

creation_rules:
  # Staging
  - path_regex: staging/.*.enc(.yaml|.json)?$
    gcp_kms: projects/<gcp staging project>/locations/global/keyRings/sops/cryptoKeys/dev-sops-key
    pgp: 43EBC42D5F6BE0B4617A2C78E2855047997055EC
  # Global enc-files (typically for testing and dev-environment)
  - path_regex: .*.enc(.yaml|.json)?$
    gcp_kms: projects/<gcp dev project>/locations/global/keyRings/sops/cryptoKeys/dev-sops-key,projects/<gcp dev project>/locations/global/keyRings/sops/cryptoKeys/gitlab-sops-key

You can specify multiple path_regex -variables where you set specific regex-named path. You can add one or more keys for the specific path_regex -variable.

Whenever you create an encrypted file to root or sub directories of the configuration file, the specific rules are affected.

Adding to version control

When you have created an encrypted file, you can store it to version control and ignore the decrypted file.

# Ignore original file
echo "api-key.json" >> .gitignore
# Add crypted file
git add api-key.enc.json

Automating encrypt and decrypt

Because everybody loves automation I’ll show you my tricks by using Git hooks. The basic flows are:

Encrypt secrets when committing changes.
Decrypt secrets when pulling, merging or doing checkout on branches.

Encrypt

Add the following script to <repository>/.git/hooks/pre-commit -hook. This script will automatically encrypt secrets when you try to commit changes. Also you can add this script to a version control and call it via hook.

encrypt_secrets.sh

#!/bin/sh
# If SOPS configuration is changed, force update
GIT_SOPS_CHANGED=`git status -s | grep ".sops.yaml"`
FORCE=0
if [ ! -z "$GIT_SOPS_CHANGED" ] || [ "$#" -eq 1 -a "$1" = "-f" -o "$1" = "--forced" ]; then
  FORCE=1
fi
# Find all encrypted files
ENCRYPTED_FILES=`find . -type f -regex ".*.enc(.yaml|.json)?$"`
for FILE in ${ENCRYPTED_FILES}; do
  DECRYPTED_FILE=`echo "$FILE" | sed 's/.enc././g'`
  if [ ! -f $DECRYPTED_FILE ]; then
    # Decrypt file if none exists
    echo "Decrypted file does not exist. Decrypt and re-encrypt: $FILE"
    sops --decrypt $FILE > $DECRYPTED_FILE
  fi
  # Check if secret is changed
  SECRET_CHANGED=`sops -d $FILE | diff $DECRYPTED_FILE - -q -Z`
  if [ $FORCE -eq 1 ] || [ ! -z "$SECRET_CHANGED" ]; then
    echo "Secret has changed or update is forced. Update: $FILE"
    # Replace old encrypted file with a new one
    cp $DECRYPTED_FILE $FILE
    sops --encrypt --in-place $FILE
    if [ ! -z "`git status -s $FILE`" ]; then
      # Add encrypted file to commit.
      git add $FILE
    fi
  fi
done

Decrypt

Add the following script to a version control and call it from <repository>/.git/hooks/post-checkout and post-merge. It will decrypt secrets when you are changing, pulling or merging a branch. Also this script will be used in a CI-pipeline in the next blog.

decrypt_secrets.sh

#!/bin/sh
ENCRYPTED_FILES=`find . -type f -regex ".*.enc(.yaml|.json)?$"`
for FILE in ${ENCRYPTED_FILES}; do
  DECRYPTED_FILE=`echo "$FILE" | sed 's/.enc././g'`
  echo "Decrypting $FILE"
  sops --decrypt $FILE > $DECRYPTED_FILE
done

Using Git diff

Add the following configuration to ~/.gitconfig

[diff "sopsdiffer"]
  textconv = "sops -d"

Then create and add <repository>/.gitattributes file with following lines

*.enc diff=sopsdiffer
*.enc.json diff=sopsdiffer
*.enc.yaml diff=sopsdiffer

Now you can use `git diff` so it will show diff between decrypted files!
Next up in the blog series: ”How to use secrets in CI-pipelines”-guide.