Creating Users for your Kubernetes Cluster

Posted on July 9, 2020

When it comes to giving people from your organization access to your Kubernetes cluster, things can get a little tricky. Kubernetes does not have an authentication mechanism by default. By doing this, you get stuck with an admin certificate you must share with the developers. In consequence, this gives them access to all the resources in the cluster, which can create holes in your security policy.

In our day to day activity, we faced this challenge and we had to find a solution for it. For this reason, this tutorial is for Operations Engineers and Kubernetes Admins which have deployed a cluster using Kops or using Kubeadm/ Kubespray on on-premise clusters. Additionally, services such as AKS have built-in authentication provided by the cloud provider. On AWS, for example, IAM ensures authentication. Also, other solutions include using a 3rd party application that can handle authentication, like Active Directory.

How to manage users without 3rd party apps

After deploying numerous clusters using Kops and Kubeadm/ Kubespray, we have found the easiest way to create users for Kubernetes clusters without installing and managing 3rd party software.

Even if Kubernetes does not support user authentication by default, it has the possibility of reading the Common Name and the Organization from the certificate—which Kubectl uses to connect to it. In this case, when Kops and Kubeadm create admin certifications, they set the Organization to system:masters. Below is a snippet from a certificate created by Kops:

Issuer: CN=kubernetes
Validity
  Not Before: <redacted>
  Not After : <redacted>
Subject: O=system:masters, CN=kubecfg

Here, we obtained the output by running the following command on the decoded client certificate from the Kubectl config:

$ openssl x509 -in <cert_file> -noout -text

As you can see, the Organization (Group) is set to system:masters and the Common Name (User) is set to kubecfg.

Mapping users & groups with Kubernetes

But, how do we map a group or user to any specific permissions? The answer lies within the Kubernetes RBAC system. This is how the cluster-admin ClusterRole looks like:

$ kubectl get clusterrole cluster-admin -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

Below we have the resource which binds this ClusterRole to our admin certificate:

$ kubectl get clusterrolebinding cluster-admin -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

Certainly, you can see the binding in the last part of our resources, where we declare our subjects. It maps the cluster-admin ClusterRole to all the certificates which have the system:masters Organization. This also maps to a Group in Kubernetes.

Creating individual certificates for cluster users

Now it’s time to create our own certificates designed specifically for other users.

First, we need to get the root certificate and key for our Kubernetes clusters. For kubeadm/kubespray, you can copy it from any master node, as it’s located in the /etc/kubernetes/ssl directory. For Kops, it’s in the S3 bucket configured at install time. The S3 paths are:

ca.crt: s3://state-store/<cluster-name>/pki/issued/ca/<id>.crt

ca.key: s3://state-store/<cluster-name>/pki/private/ca/<id>.key

Put them in a folder and name them ca.crt and ca.key.

Next, we need to do some OpenSSL magic.

First, we need to generate a private key for our new user:

$ openssl genrsa -out <username>.key 2048

Next, create a CSR using the key above, you can use any username and group you want. You can group your users (for example, devs, sysadmins, security, etc.):

$ openssl req -new -key <username>.key -out <username>.csr -subj "/CN=<username>/O=<group>"

Lastly, we need to create a certificate using our private key, our CSR, and our CA for signing. You can choose the number of days the certificate is valid for, in the example below it’s set to 10 years:

$ openssl x509 -req -in <username>.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out <usernamme>.crt -days 3540

Then remove the junk:

$ rm <username>.csr ca.srl

And now convert your ca.crt, <username>.crt and <username>.key to base64:

$ CA_CRT_BASE64=$(base64 ca.crt)
$ CLIENT_CRT_BASE64=$(base64 <username>.crt)
$ CLIENT_KEY_BASE64=$(base64 <username>.key)

Automate individual certification for users with Kubectl config

Here is a Kubectl config template you can use— replace the uppercase variables with useful stuff:

apiVersion: v1
current-context: USER_NAME-ctx
preferences: {}
clusters:
- cluster:
    certificate-authority-data: CA_CRT
    server: K8S_API_URL
  name: CLUSTER_NAME
contexts:
- context:
    cluster: CLUSTER_NAME
    user: USER_NAME
  name: USER_NAME-ctx
kind: Config
users:
- name: USER_NAME
  user:
    client-certificate-data: CLIENT_CRT
    client-key-data: CLIENT_KEY

So now that we have our Kubectl config, we will place it in ~/.kube/config and try to access the cluster with it. Yet, came across the following error:

$ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "<username>" cannot list resource "nodes" in API group "" at the cluster scope

In fact, the user from the certificate does not match any Roles or ClusterRoles.

As a practical and fun example, let’s suppose you want to give Ed the Dev access only to the edsns namespace. After creating his certificate and setting the Common Name to edward and the group to developer, you may want to create a Role and a RoleBinding in the edsns namespace. To give him full control to his namespace, his Role would look like this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: edsns
  name: edsns-rw-role
rules:
- apiGroups: ["", "batch", "extensions", "apps", "networking.istio.io"]
  resources: ["*"]
  verbs: ["*"]

and the RoleBinding:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: edsrolebinding
  namespace: edsns
subjects:
- kind: User
  name: edward
  apiGroup: ""
roleRef:
  kind: Role
  name: edsns-rw-role
  apiGroup: rbac.authorization.k8s.io

$ kubectl apply -f edsns-rw-role.yaml
$ kubectl apply -f edsrolebinding.yaml

Extra options: privileges and removing Kubernetes cluster users

If he now runs any kubectl command with the -n edsns flag, he will have privileges to create, update and delete everything in his namespace.

However, privileges aren’t limited to a single namespace, or all namespaces. You can create multiple Roles and RoleBindings in different namespaces.

In fact, if a user leaves your organisation, just delete his Role and RoleBinding, or ClusterRole and ClusterRoleBinding.

I hope you got some useful information and you can take your Kubernetes Security practices one step further. All in all, create users for Kubernetes clusters without giving everyone the admin certificate.