Untitled :: Validated Patterns Workshop

Secrets Management in Validated Patterns

ESO and Vault

The Validated Patterns team settled on using references to secrets

We currently implement GitOps + Secrets via External Secrets Operator and HashiCorp Vault

The main reasons for this choice at the time were the following:

Sealed Secret approach carries additional risks
ESO allows us a certain level of independence from the Secrets Management System
HashiCorp Vault is currently the most popular Secret Management Systems
Advised by our security team
CyberArk leaders (masters) cannot be hosted on OpenShift
Secret injection is incompatible with Pods owned by Operators

Approaches

There are two fundamental approaches to manage secret material within a GitOps context:

Encrypted Secrets stored inside Git repositories
References to Secrets stored inside Git repositories and the actual secret is stored somewhere else

Encrypted Secrets

Secrets are stored in an encrypted form inside the Git repository
Automation and helper tools do the decryption to create Kubernetes secrets from them
A number of projects exist that implement this approach:
- Sealed Secrets
- Mozilla’s Secret OPerationS (SOPS)
- Other smaller ones (Tesoro, Kamus)

These projects provide an easy way to encrypt and decrypt the secrets, and make sure that only authorized users can access them. The encryption method and key management should be considered carefully as they are important to ensure the security of the secrets.

You should evaluate and choose the right project that fits your specific use case and requirements.

References to Secrets

This approach requires two main parts: * A Secret Management System (HashiCorp Vault, Conjur, Confidant) * A controller that retrieves the secret from the Secret Management System and translates it to a Kubernetes Secret or injects it into the Pod directly (External Secrets Operator, Kubernetes Secrets Store CSI Driver, Vault Agent Injector) * Secrets are uploaded/created directly into the Secret Management System * References to secrets are stored in Git * A controller reads the references to secrets and translates them into Kubernetes secrets

HashiCorp Vault runs only on the HUB

The Vault has multiple kubernetes backends configured by the unsealVault Cron Job (one for each cluster)
There is one ESO instance for each cluster (ESO is currently unsupported)
Each ESO instance logs into the HashiCorp Vault using a specific login path that was configured with the credentials of that instance
Vault will authenticate the requests by talking to the API endpoints of the clusters where the login request originated from

At a high-level the External Secrets Operator reads secrets from the Vault and converts them into Kubernetes Secrets
To populate the Vault with our secret material we use make load-secrets
Normally a user would create ~/values-secret-<patternname>.yaml following the contents of values-secret.yaml.template in the pattern’s git repository
Running ./pattern.sh make load-secrets will populate the Vault with our secrets

Provisioning the vault can be complex, so it is taken care of by the framework

Vault unsealing and configuration happens in the imperative namespace with a CronJob

The Unseal vault cron job is idempotent and runs every five minutes on the hub only

The vaultkeys secret containing the keys to unseal the vault and the vault root token that can be used to login to the vault’s UI is created

It is strongly recommended that the vaultkeys secret in the imperative namespace is exported to a safe space and removed. It contains the keys needed to unseal the vault whenever it gets restarted

Vault Concepts

Secrets are stored in Vault in specific paths
secret/global/config-demo is the path
Each path can have multiple attributes
secret is one attribute at this path
In this case secret is autogenerated by default inside the vault directly

By default there are three paths that can be used

secret/global/* accessible by the hub and all managed clusters
secret/hub/* accessible only by the hub cluster
secret/<fqdn of managed cluster>/* accessible only by the fqdn-defined cluster

You can think of a path as being a single dictionary with multiple keys. An external secret looks for a path and then has a templating system to extract keys and put them in kubernetes secrets.

secrets:
  - name: config-demo
	vaultPrefixes:
	- global
	fields:
	- name: secret
  	onMissingValue: generate
  	vaultPolicy: validatedPatternDefaultPolicy

From the vault interface perspective:

ESO Concepts

ESO will read secrets from vault paths (e.g. “secret/data/global/config-demo” is a path)

This ExternalSecret object will read all the attributes of the vault key secret/global/config-demo and will create a secret called config-demo-secret with one entry per attribute.

apiVersion: "external-secrets.io/v1beta1"
kind: ExternalSecret
metadata:
  name: config-demo-secret
  namespace: config-demo
spec:
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: config-demo-secret
    template:
      type: Opaque
  dataFrom:
  - extract:
      key: secret/data/global/config-demo

How secret material ends up in the vault at a specific path is explained in detail in an upcoming section