Vault as a PKI service for Kubernetes authentication

Kubernetes can be configured to use SSL certificates to authenticate users allowing kubernetes internals (RBAC and events) to be used for authorization and accounting. To do this efficiently it’s best to use a pki server to manage the issuance, revocation and maintenance of certificates.

Vault is not only a great tool for storing secrets in Kubernetes, it can also be used as a pki server, managing user certificates on a per-group basis, allowing a hierarchical authorization model as well as hierarchical management of user certificates.

A practical example Vault Kubernetes Authentication

Prerequisites

Start Vault

For the purposes of this example, we’re going to use Vault in debug mode and authenticate to it using a token. Vault has several authentication plugins and the use of those is outside the scope of this document.

Do not run vault like this in production.

# Set a token to be used for vault authentication
export VAULT_ROOT_TOKEN=$(uuidgen)

# Run vault in dev mode
vault server -dev -dev-root-token-id="${VAULT_ROOT_TOKEN}" 2>&1 > /tmp/vault.log &

# connect and authenticate
export VAULT_ADDR='http://127.0.0.1:8200'
vault auth ${VAULT_ROOT_TOKEN}

Start Kubernetes

Again, for example purposes only, we’ll use something that should not be used in production, minikube. This will run kubernetes locally on a VM and produce a root certificate authority that is used by kubernetes services to authenticate servers. We’re going to adopt this CA into Vault.

minikube start

Adopt the Root CA into Vault

export ROOT_PATH="root_ca"

# minikube in default mode generates a CA for the cluster and client certificate in "$HOME/.minikube" directory. 
cat ~/.minikube/ca.{crt,key} > clientCA.bundle

# Register the client CA generaged by minikube.
vault mount -path="${ROOT_PATH}" pki

# Set the max TTL for the root CA to 10 years
vault mount-tune -max-lease-ttl="87600h" "${ROOT_PATH}"

# Write the CA
vault write ${ROOT_PATH}/config/ca pem_bundle=@clientCA.bundle

Create group specific items

Kubernetes groups are used for assigning permissions to uses within specific classes. For instance, you might have a developer group, an accounting group, a sales group, etc. These users can be assigned rights within kubernetes according to their needs using RBAC. To determine the group for which a user is to be a member of, the organization attribute of the SSL certificate is used.

The best practice in issuing user certificates is to use an intermediate certificate that only has permissions to issue certs for each specific subset.

Create the Intermediate Certificate Authority for the group

Intermediate CAs should be created for each organization (kubernetes group). This example will issue certs for the “developers” group.

COMPANY="Kubernetes"
DOMAIN="developers"
USER="jdoe"
INTR_PATH="intermediate_${DOMAIN}"

# Mount the intermediate CA for the zone
vault mount -path=${INTR_PATH} pki

# Set the max TTL for ${DOMAIN} certs to 1 year
vault mount-tune -max-lease-ttl=8760h ${INTR_PATH}

# Generate CSR for ${DOMAIN} to be signed by the root CA, the key is stored
# internally to vault
http POST ${VAULT_ADDR}/v1/${INTR_PATH}/intermediate/generate/internal X-Vault-Token:$VAULT_TOKEN common_name=${DOMAIN} | jq -r .data.csr > ${DOMAIN}.csr

# Generate and sign the ${DOMAIN} certificate as an intermediate CA
http POST ${VAULT_ADDR}/v1/${ROOT_PATH}/root/sign-intermediate X-Vault-Token:$VAULT_TOKEN ttl="8760h" csr=@${DOMAIN}.csr | jq -r .data.certificate > ${DOMAIN}.crt

# Add signed ${DOMAIN} certificate to intermediate CA backend
vault write ${INTR_PATH}/intermediate/set-signed certificate=@${DOMAIN}.crt

# Create role for issuing ${DOMAIN} certificates
# Max lease time is 14 days
vault write ${INTR_PATH}/roles/${DOMAIN} allow_any_name=true organization=${DOMAIN} generate_lease=true server_flag=false lease_max="336h"

Issue a client certificate for a user in this group

# issue certificate for CN:${USER} O:${DOMAIN}
http POST ${VAULT_ADDR}/v1/${INTR_PATH}/issue/${DOMAIN} X-Vault-Token:$VAULT_TOKEN common_name="${USER}" ttl="168h" | jq -r .data.private_key,.data.certificate,.data.issuing_ca > ${USER}_${DOMAIN}.pem

Use this certificate for authenticating to Kubernetes

# Separate out the certificate and key for import into kubeconfig
cat ${USER}_${UNDER_DOM}.pem | sed -n '/certificate/,/END CERTI/p' | sed 's/certificate[ \t]*//g' > ${USER}_${UNDER_DOM}.crt
cat ${USER}_${UNDER_DOM}.pem | sed -n '/private_key /,/END RSA/p' | sed 's/private_key[ \t]*//g' > ${USER}_${UNDER_DOM}.key

# Add user in kubeconfig
kubectl config set-credentials ${USER} --client-certificate=${USER}_${UNDER_DOM}.crt --client-key=${USER}_${UNDER_DOM}.key

Verify the certificate works

kubectl config set-context admin-context --cluster=minikube --user=${USER}
kubectl config use-context admin-context

kubectl get nodes

NAME       STATUS    ROLES     AGE         VERSION
minikube   Ready     <none>    <invalid>   v1.9.0