In part 1 we installed an identity management service; Keycloak. Part 2 showed how to configure Keycloak against AD (or LDAP) with a quickstart option of simply adding a local user.
In this final part we will configure the kube-apiserver to use our identity management (IDM) service – OIDC Kubernetes.
Setting up Kubernetes
The easiest way to configure the kube-apiserver for any auth is to alter the command line arguments it is started with. The method used to do this differs depending on how you have installed kubernetes, and how the service is being started.
It is likely that there is a systemd unit file responsible for starting the kube-apiserver, probably in /usr/lib/system/ or possibly under /etc/systemd/system. Note that if your Kubernetes environment is containerized, as is the case with gravity, then you will first need to enter that environment before hunting for the unit file. In the case of gravity, run sudo gravity enter.
In the case of minikube, Kubernetes runs under Kubernetes defined as static pods. This self-referential startup is something we’ll cover in more detail in a future post, but for now all we really need to understand is how to edit the startup flags passed to kube-apiserver.
Because we cannot make persistent changes to the configuration files minikube copies into the virtual machine, or add certificates, id is easiest to run the oidckube.sh start script. I noticed a small bug in that script, so you will need to make the following changes for this to work:
diff --git a/oidckube.sh b/oidckube.sh
index 19b487c..ba09d58 100755
--- a/oidckube.sh
+++ b/oidckube.sh
@@ -64,7 +64,7 @@ init_minikube() {
inject_keycloak_certs() {
tar -c -C "$PKI_DIR" keycloak-ca.pem | ssh -t -q -o StrictHostKeyChecking=no \
- -i "$(minikube ssh-key)" "docker@$(minikube ip)" 'sudo tar -x --no-same-owner -C /var/lib/localkube/certs'
+ -i "$(minikube ssh-key)" "docker@$(minikube ip)" 'sudo tar -x --no-same-owner -C /var/lib/minikube/certs'
}
@@ -85,7 +85,7 @@ start_minikube() {
--extra-config=apiserver.oidc-username-prefix="oidc:" \
--extra-config=apiserver.oidc-groups-claim=groups \
--extra-config=apiserver.oidc-groups-prefix="oidc:" \
- --extra-config=apiserver.oidc-ca-file=/var/lib/localkube/certs/keycloak-ca.pem
+ --extra-config=apiserver.oidc-ca-file=/var/lib/minikube/certs/keycloak-ca.pem
}
main() {
You can see in the inject_keycloak_certs function the script adds the keycloak-ca.pem file. In a real production setup, this would probably me a real CA, or the root CA of the organization.
The oidc script also runs minikube with a bunch of arguments to ensure that the Keycloak server is used for authentication:
$ minikube start \ --extra-config=apiserver.oidc-client-id=kubernetes \ --extra-config=apiserver.oidc-username-claim=email \ --extra-config=apiserver.oidc-username-prefix=oidc: \ --extra-config=apiserver.oidc-groups-claim=groups \ --extra-config=apiserver.oidc-groups-prefix=oidc: \ --extra-config=apiserver.oidc-issuer-url=https://keycloak.devlocal/auth/realms/master \ --extra-config=apiserver.oidc-ca-file=/var/lib/localkube/certs/keycloak-ca.pem
The options to the –extraconfig flags mimic the command line flags that should be added to the kube-apiserver binary. In order to mimic this minikube setup in a larger cluster, you need to edit the systemd unit file, or the static pod difinition – depending on how the your cluster starts the apiserver.
--oidc-client-id=kubernetes --oidc-username-claim=email --oidc-username-prefix=oidc: --oidc-groups-claim=groups --oidc-groups-prefix=oidc: --oidc-issuer-url=https://keycloak.devlocal/auth/realms/master --oidc-ca-file=/path/to/your/rootca.pem # optional if the CA is public
Once you are satisfied that the new arguments are being used by your kube-apiserver process, it’s time to test out the new auth.
Testing it out
With everything configured on the server side, it’s time to make sure things are working on the client end. There are a few different assets that need to be in place in the configuration for kubectl.
The first thing to consider is how to manage updating the kubeconfig.
Some people want to manage one single kubectl configuration with different contexts driving the parameters needed for different clusters. Others like to use the environment variable KUBECONFIG to maintain many different files in different locations. In this post, we’re going to be using the more self-contained approach of KUBECONFIG, but I recommend using contexts in general and will cover that in a future post.
In addition to the new file, I would recommend the use of a tool like kubelogin (go get github.com/int128/kubelogin) to assist with the handoff between keycloak and kubectl. That tool starts local server and opens a browser against it. The user authenticates to keycloak and, if successful, the kubelogin server receives the callback and updates the KUBECONFIG file with the JWT tokens.
You probably have a working configuration for the cluster already – either minikube or some other Kubernetes installation. Either way, you will probably already have a kubeconifg in .kube/config. In order to get a quick start, just copy .kube/conifg to .kube/oidcconfig and remove the client-certificate: and client-key: fields completely.
Once built, kubelogin is available in $GOROOT/bin/kubelogin, but you can move it wherever you want.
Now we have the kubelogin tool available, just setup kubectl to point to the netw conifg and run kubelogin. It will guide you through the process from there:
$ cp ~/.kube/config ~/.kube/oidcconfig $ kubectl config set-credentials minikube \ --auth-provider oidc \ --auth-provider-arg client-id=kubernetes \ --auth-provider-arg idp-issuer-url=https://keycloak.devlocal/auth/realms/master $ ~/go/bin/kubelogin
When you have logged into keycloak via the web browser, kubelogin will exit and update your kubectl configuration.
---
apiVersion: v1
clusters:
- cluster:
certificate-authority: ca.crt
server: https://192.168.99.102:8443
name: minikube
contexts:
- context:
cluster: minikube
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
auth-provider:
config:
client-id: kubernetes
id-token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJCbGtiVXFVSkdRV09fWEFNeXhDUm84Q01iVy1GQ1FzeXVSLTBhUUZiaUJ3In0.eyJqdGkiOiI1NzY0NGNiOC1jOWM3LTQ0MTMtYThiZi0wYTRjM2EyZGIxNTQiLCJleHAiOjE1NDk4NDAxMjAsIm5iZiI6MCwiaWF0IjoxNTQ5ODQwMDYwLCJpc3MiOiJodHRwczovL2tleWNsb2FrLmRldmxvY2FsL2F1dGgvcmVhbG1zL21hc3RlciIsImF1ZCI6Imt1YmVybmV0ZXMiLCJzdWIiOiIyY2JkNmIzYi1hMWE4LTQxZGYtYTE3ZC03NTYzYzhiZDE1MGEiLCJ0eXAiOiJJRCIsImF6cCI6Imt1YmVybmV0ZXMiLCJhdXRoX3RpbWUiOjE1NDk4NDAwNTYsInNlc3Npb25fc3RhdGUiOiI2ZjJiM2UwZS1hNzY0LTQxODktYjliNi1kMzBhZTRhMWYzYTQiLCJhY3IiOiIxIiwibmFtZSI6InRlc3QgdXNlciIsInByZWZlcnJlZF91c2VybmFtZSI6InRlc3R1c2VyIiwiZ2l2ZW5fbmFtZSI6InRlc3QiLCJmYW1pbHlfbmFtZSI6InVzZXIiLCJlbWFpbCI6InRlc3R1c2VyQGJleW9uZHRoZWt1YmUuY29tIn0.Y3y0GEPsPHaibRNzp0AQ-pAV-b8K5m9rwGe512QKCHINrMu5jrfe1XCnl5qry6coUPSG_nNwDOB8WxFNqW-lTp_rbZ_auz2xy93L2bs1Pb_3QGwHFRXfAP_AZJagCSp8JC3mpopHRvsnuxi4yR4hqNln85a62jBshK-9QEpgR9mRUcSs2PdOicrPvqP0hMMHzOTYsEcsk-YaGxhPqDQJTHzuCa_8fgx6OG2vycG392Vrr1p5RhUg3lUmTv7nYOHnqkhZQLWCDkcndWt8sBiGVOkyJeDsuy0d8QNLP-9Z-BGip-RiZdmpaA4E2LmKO6CK54eo1i2zRSgN1Odm0316cg
idp-issuer-url: https://keycloak.devlocal/auth/realms/master
refresh-token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJCbGtiVXFVSkdRV09fWEFNeXhDUm84Q01iVy1GQ1FzeXVSLTBhUUZiaUJ3In0.eyJqdGkiOiJmMzE5YWFmZC03ZWY5LTQ0OWQtYTljNS1jYjc5YjRhMmRlMzIiLCJleHAiOjE1NDk4NDE4NjAsIm5iZiI6MCwiaWF0IjoxNTQ5ODQwMDYwLCJpc3MiOiJodHRwczovL2tleWNsb2FrLmRldmxvY2FsL2F1dGgvcmVhbG1zL21hc3RlciIsImF1ZCI6Imt1YmVybmV0ZXMiLCJzdWIiOiIyY2JkNmIzYi1hMWE4LTQxZGYtYTE3ZC03NTYzYzhiZDE1MGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoia3ViZXJuZXRlcyIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjZmMmIzZTBlLWE3NjQtNDE4OS1iOWI2LWQzMGFlNGExZjNhNCIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJ1bWFfYXV0aG9yaXphdGlvbiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX19.Fb8_P_UVGDsR9hPYm6pbHbU3AWavS9DLjOxDUdTsEPn2gcLZi0e42GbJCFZMz3sLliTBizxFXGAbaR4c6jS3wO5mZgIIa-ek9OgX1Qo7jsI3w0NegFXdFJHG2HgXr_T8gYcHFkG3FP7j7qi8z52GKfA1T1M_Ki97ovUfLJGu0CxfnCFNpXz3xfIj8tmuV_QZc9_s9jVgXyaQfyq0QYyNbMCtgFG1AZkv70ycUoJ6EtB3R7HOGUPd5MQjtih7GIal8E3U9PS5yp_DWSBig10T-wYKiViEiILxoXO90CF2n-4v8Q3P6YZ6HL-CtjHK4XkHi2jHEJGFqxeeXXBTgVUAUA
name: oidc
If you take the content of the id_token: field and paste it into http://jwt.io you can decode the token into human-readable form:
{
"jti": "57644cb8-c9c7-4413-a8bf-0a4c3a2db154",
"exp": 1549840120,
"nbf": 0,
"iat": 1549840060,
"iss": "https://keycloak.devlocal/auth/realms/master",
"aud": "kubernetes",
"sub": "2cbd6b3b-a1a8-41df-a17d-7563c8bd150a",
"typ": "ID",
"azp": "kubernetes",
"auth_time": 1549840056,
"session_state": "6f2b3e0e-a764-4189-b9b6-d30ae4a1f3a4",
"acr": "1",
"name": "test user",
"preferred_username": "testuser",
"given_name": "test",
"family_name": "user",
"email": "testuser@beyondthekube.com"
}
Finally, now that we have inspected our id_token, we can offer it up to the kube-apiserver to see if it knows who we are:
$ kubectl get pods Error from server (Forbidden): pods is forbidden: User "oidc:testuser@beyondthekube.com" cannot list resource "pods" in API group "" in the namespace "default" $
Although the outcome here is unexciting – we’re not actually allowed to interact with the cluster – this does show the user prefixing (oidc:) working, in addition to the kube-apiserver recognizing the username.
Stay tuned for more on how to authorize users to do stuff with the cluster – I’m planning to cove a few different options for that.