Category: on-prem

Evaluation :: kubespray

Published on by admin

Introduction

Kubespray is different to the recently evaluated Gravity deploy. It uses ansible to drive deployment on the base system, rather than attempting to package all of the kubernetes components into a container

If it is not obvious, Kubespray makes extensive use of connectivity to the internet in order to download and configure a lot of the required assets, and command line tools. This is not always workable in an on-premis environment.

Installation

In order to get started, clone the official kubespray repo:

$ git clone https://github.com/kubernetes-sigs/kubespray.git
Cloning into 'kubespray'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 33233 (delta 1), reused 2 (delta 1), pack-reused 33228
Receiving objects: 100% (33233/33233), 9.67 MiB | 14.10 MiB/s, done.
Resolving deltas: 100% (18561/18561), done.
$

You will need to edit the inventory file to reflect the hosts that you wish to install the Kubernetes cluster on. If this is a large production installation, you may want to take a look at dynamic inventories for Ansible.

In my case, I’m planning to install the cluster on three nodes named playground[1-3] so I’ve edited the inventory file to reflect that:

Creating the inventory

$ cat inventory/local/hosts.ini 
playground1 ansible_connection=local local_release_dir={{ansible_env.HOME}}/releases

[kube-master]
playground1

[etcd]
playground1

[kube-node]
playground1
playground2
playground3

[k8s-cluster:children]
kube-node
kube-master
$

Installing the kubespray requirements

Next, make sure that you have all of the requirements required:

$ sudo pip install -r requirements.txt

Customization

You may want to review and possibly change some of the variables in the group_vars directories.

$ find inventory/local/group_vars/
inventory/local/group_vars/
inventory/local/group_vars/all
inventory/local/group_vars/all/all.yml
inventory/local/group_vars/all/azure.yml
inventory/local/group_vars/all/coreos.yml
inventory/local/group_vars/all/docker.yml
inventory/local/group_vars/all/oci.yml
inventory/local/group_vars/all/openstack.yml
inventory/local/group_vars/etcd.yml
inventory/local/group_vars/k8s-cluster
inventory/local/group_vars/k8s-cluster/addons.yml
inventory/local/group_vars/k8s-cluster/k8s-cluster.yml
inventory/local/group_vars/k8s-cluster/k8s-net-calico.yml
inventory/local/group_vars/k8s-cluster/k8s-net-canal.yml
inventory/local/group_vars/k8s-cluster/k8s-net-cilium.yml
inventory/local/group_vars/k8s-cluster/k8s-net-contiv.yml
inventory/local/group_vars/k8s-cluster/k8s-net-flannel.yml
inventory/local/group_vars/k8s-cluster/k8s-net-kube-router.yml
inventory/local/group_vars/k8s-cluster/k8s-net-weave.yml
$

If you have not used Ansible before, it works over ssh and requires the user running it to have passwordless ssh access to all of the nodes. In the case of kubespray, this user is root, so you need to have a strategy to make that work — at least for installation time.

Running the install

$ sudo bash
# ansible-playbook -i inventory/local/hosts.ini cluster.yml

After running the installation, a slow 10 minutes on my test setup, we get to the end of the Ansible run:


PLAY RECAP *********************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   
playground1                : ok=404  changed=126  unreachable=0    failed=0   
playground2                : ok=253  changed=76   unreachable=0    failed=0   
playground3                : ok=254  changed=76   unreachable=0    failed=0   

Monday 25 February 2019  18:36:02 -0600 (0:00:00.027)       0:10:49.135 ******* 
=============================================================================== 
download : container_download | download images for kubeadm config images ---------------------------------------------------------------------- 39.74s
download : file_download | Download item ------------------------------------------------------------------------------------------------------- 38.43s
kubernetes/master : kubeadm | Initialize first master ------------------------------------------------------------------------------------------ 28.25s
kubernetes/node : install | Copy hyperkube binary from download dir ---------------------------------------------------------------------------- 27.08s
container-engine/docker : ensure docker packages are installed --------------------------------------------------------------------------------- 25.38s
download : file_download | Download item ------------------------------------------------------------------------------------------------------- 19.69s
kubernetes/kubeadm : Join to cluster ----------------------------------------------------------------------------------------------------------- 16.65s
kubernetes/preinstall : Update package management cache (YUM) ---------------------------------------------------------------------------------- 16.56s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------- 11.67s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------- 11.57s
container-engine/docker : Docker | pause while Docker restarts --------------------------------------------------------------------------------- 10.08s
kubernetes/kubeadm : Restart all kube-proxy pods to ensure that they load the new configmap ----------------------------------------------------- 9.44s
kubernetes/node : install | Copy kubelet binary from download dir ------------------------------------------------------------------------------- 9.20s
etcd : Configure | Check if etcd cluster is healthy --------------------------------------------------------------------------------------------- 7.34s
container-engine/docker : Ensure old versions of Docker are not installed. | RedHat ------------------------------------------------------------- 7.20s
download : container_download | Download containers if pull is required or told to always pull (all nodes) -------------------------------------- 7.18s
download : container_download | Download containers if pull is required or told to always pull (all nodes) -------------------------------------- 6.75s
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------------- 6.58s
download : container_download | Download containers if pull is required or told to always pull (all nodes) -------------------------------------- 6.47s
kubernetes/node : install | Write kubelet systemd init file ------------------------------------------------------------------------------------- 6.44s

Playing with the cluster

# kubectl get nodes
NAME          STATUS   ROLES         AGE     VERSION
playground1   Ready    master,node   4m9s    v1.13.3
playground2   Ready    node          3m43s   v1.13.3
playground3   Ready    node          3m43s   v1.13.3
# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-8594b5df7b-26gn4   1/1     Running   0          3m18s
calico-node-424g9                          1/1     Running   0          3m25s
calico-node-8wpfc                          1/1     Running   0          3m25s
calico-node-f4jbh                          1/1     Running   0          3m25s
coredns-6fd7dbf94c-9wtjj                   1/1     Running   0          2m40s
coredns-6fd7dbf94c-ldvcj                   1/1     Running   0          2m44s
dns-autoscaler-5b4847c446-m2df7            1/1     Running   0          2m41s
kube-apiserver-playground1                 1/1     Running   0          4m40s
kube-controller-manager-playground1        1/1     Running   0          4m40s
kube-proxy-ldntn                           1/1     Running   0          3m10s
kube-proxy-skvcq                           1/1     Running   0          3m1s
kube-proxy-tj2bk                           1/1     Running   0          3m21s
kube-scheduler-playground1                 1/1     Running   0          4m40s
kubernetes-dashboard-8457c55f89-pjn9g      1/1     Running   0          2m39s
nginx-proxy-playground2                    1/1     Running   0          4m27s
nginx-proxy-playground3                    1/1     Running   0          4m27s
#

Kubespray helpfully creates and populates a Kubeconfig file with embedded certs for client identity. This is useful, but should probably be thought of as ‘the root user’ of the Kubernetes cluster.

In order to grant additional access to the cluster, now would be a good time to look into identity management and role based access control

Adding additional nodes

Adding a node is done by adding to the inventory and running ansible-playbook with the -l option to limit the install to the new nodes. All of the certificate handling is performed under the hood by kubespray.

Conclusion

If you are familiar with Ansible, then Kubespray is a great way to go; it encapsulates all of the complexity of orchestrating a cluster into a single command to run.

There are downsides: Kubespray is hiding a lot of complexity here, and ultimately it is useful to understand in detail what is going on. As stated earlier in the post, it’s also an absolute necessity to have relatively wide-open access to the internet for Kubespray to work. Of course, this is plain old Ansible, so it is possible to alter Kubespray to work with in-house repos if needed.

I found Kubespray surprisingly easy to get along with, and would recommend it for those wanting to get up and running on bare metal if internet access is not an issue for you.

Identity management Part 1 :: Installing Keycloak

Published on by admin

One of the first challenges associated with owning the control plane of Kubernetes is that you are responsible for authz and authn.

While there are some great resources for getting set up with some form of identity management, ultimately the correct solution will depend on your existing on-premise setup. If you already have something like Active Directory or LDAP configured, then it probably makes sense to integrate with those.

Thankfully, Kubernetes provides a rich set of possibilities for authentication; from standards like OIDC right through to just sharing a token to figure out who the current user is.

Objective

In this post, I’ll demonstrate how to setup OIDC authentication with the kube-apiserver, which will allow user identity to be established. If you’re lucky, your enterprise will already have an OIDC provider such as Okta., or the Google IAM services. Even if you’re starting from scratch, it’s relatively simple to set up a bridge to LDAP or AD.

In order to follow along, you’ll need a Kubernetes cluster on which you are able to run kubectl.

The procedure for getting keycloak up and running on top of kubernetes is relatively well documented elsewhere, so I’ll assume you’re able to get that up and running. If you’re using minikube for this, then I would recommend this project.

Setting up access to the service

Once you have a Keycloak pod running, you’ll navigate to the Keycloak end. Given that we have no solution for ingress yet, our best option is to expose the service on a nodeport.

This is achieved by adding some definitions to the existing service definition. The default keycloak service definition will be something like this:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: keycloak
    component: keycloak
  name: keycloak
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: keycloak
    component: keycloak
  type: ClusterIP

Nodeports cannot be added to services of type ClusterIP, but we can take advantage of Kubernetes service abstraction to simply create a new service for our nodeport. We will start by grabbing the existing one:

$ kubectl get svc keycloak -o yaml > keycloak-nodeport.yaml

Now we can simply edit it to look like this (notice how similar these services look)

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: keycloak
    component: keycloak
  name: keycloak-nodeport
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    nodePort: 32080
    protocol: TCP
    targetPort: http
  selector:
    app: keycloak
    component: keycloak
  sessionAffinity: None
  type: NodePort

In fact, the diff is tiny:

7c7
<   name: keycloak
---
>   name: keycloak-nodeport
10d9
<   clusterIP: None
13a13
>     nodePort: 32080
20c20
<   type: ClusterIP
---
>   type: NodePort

Accessing the Keycloak service

If you used the minikube and oidckube script, then you should have keycloak available at https://keycloak.devlocal. You should see this page, and be able to click through to the Administration Console.

Keycloak landing page
Keycload admin console login page

The username is ‘keycloak’ and the password is ‘keycloak.’ It is hashed and available in a secret resource:

$  kubectl get secret  keycloak-admin-user -o yaml
 apiVersion: v1
 data:
   password: THIS_IS_THE_PASSWORD_HASH
 kind: Secret
 metadata:
   labels:
     app: keycloak
     component: keycloak
   name: keycloak-admin-user
   namespace: default
 type: Opaque

So we now have Keycloak installed and running. Stick around for Part 2.

Installing Kubernetes on-prem

Published on by admin

There are several methods of getting Kubernetes installed on physical tin, or just on top of your favorite OS. Many developers will probably have some familiarity with minikube. That is a great self-contained quick start configuration for hacking on your laptop.

That’s not really what we want here. We want a fully featured multi-node cluster installed on physical hosts. There are a number of ways to achieve that. They range in terms of flexibility vs convenience, vendor tie-in and so on.

Over the coming weeks, I’ll be providing more in-depth evaluations of some of these implementations. For now here are the resources:

Manual and semi-manual

Kubernetes the hard way is a really complete introduction to getting Kubernetes up and running entirely manually. It’s a great starting point if you want to understand the dirty details of how the components hang together.

kubespray is an ansible repo that does the heavy lifting for you. You simply need to provide your ansible inventory and away you go.

kops is a neat little cli for setting up clusters and managing them throughout the lifecycle.

Packaged

Gravity is an installer and containerized kubernetes offering from Gravitational. The stack includes helpful built-in components like tiller. In addition it provides a full cluster management service (as part of the paid-for enterprise version)

Rancher is also a pre-packaged Kubernetes offering from the company of the same name.

Openshift is the RedHat offering of Kubernetes.

On prem Kubernetes is a fast-growing area. I’m sure I have not captured all of the available offerings. If you feel like I’ve missed an important one, and you’d like to see it evaluated, leave a commend and I’ll add it to the mix.