Very secret development operations part I: Secrets management

 
Every developer will occasionally run into a similar problem: where and how to store secrets? And this time I don’t mean hiding skeletons in the closet, but storing credentials, passwords, IP addresses, API keys and something like that. There are many ways to store secrets but this time I will focus on how to store them securely in a version control.
This blog series is split into four parts:

1. General information about secrets management
2. Example how to store secrets in a version control
3. Example how to use encrypted secrets in CI-pipelines
4. Security issue and threat analysis based on previous examples

So, why should you be serious about security and secrets management? Too many times I’ve seen that some secrets are stored in a version control as a clear text and I admit that I’ve made the same mistake many times – it is just too easy to store a secret and forget that it exists. Anyone who has access to the repository can expose and use the secret. Not to mention that hackers or possible vulnerabilities in systems which have access to data can also expose the secret. Private repositories aren’t enough.
On one day, I was thinking about what would be the best way to store secrets so that all the following requirements are fulfilled:

1. Multiple encryption types support.
2. Globally working solution with centralised key management.
3. Easy to install and use. Also easy to fix any compromised information.
4. Encrypted information is stored to a version control.

I read up further about secrets management in a version control and came up with five solutions: git-crypt, git-secret, BlackBox, Mozilla SOPS and Transcrypt. There is also Hashicorp Vault but as far as I know, it is not quick to setup from scratch and is not easier to use than other solutions.
I took a look to what encryption types are supported in each solution:

I didn’t want to setup any GPG-based keyservers by myself, so a cloud based key management service (KMS) is the way-to-go. In a current client project we use Google Cloud Platform (GCP), Git and Gitlab, so my biggest motivation was to use an encryption solution which supports those technologies – especially GCP. After all, reading detailed information of encryption solutions, I ended up to SOPS.
Why? For me, the most important feature was that I can encrypt and decrypt secrets with multiple keys. If one key is exposed, the key can be rotated or removed without any bigger hassle – just rotate or remove and then encrypt the secrets again. No need to change all the keys.
Another thing is that if the keys are managed on a cloud platform, all authentications are based on a cloud platform’s user and service management with trace logging.
 

 
Mozilla SOPS (Secrets OPerationS) is an open-source editor of encrypted files that supports YAML, JSON, ENV, INI and binary formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.
SOPS will encrypt and decrypt secrets in the text form so running any diff tools for tracking changes is possible. It is built in the Go programming language and SOPS is compatible with the Windows, Mac and Linux platforms.
In the second part of this blog series, I’ll dive into a technical ”How to add secrets to version control”-guide. It will give a tour of using Mozilla SOPS with GCP and Git.

 

Further reading:

• https://opensource.com/article/19/2/secrets-management-tools-git
• https://medium.com/@snaveen/managing-secrets-2bc57adbb4b
• https://oteemo.com/2019/06/20/hashicorp-vault-is-overhyped-and-mozilla-sops-with-kms-and-git-is-massively-underrated/
• https://embeddedartistry.com/blog/2018/3/15/safely-storing-secrets-in-git
• https://jve.linuxwall.info/blog/index.php?post/2015/10/01/Introducing-Sops%3A-an-editor-of-encrypted-file-that-uses-AWS-KMS-and-PGP