Category: packaging

Evaluation :: kubespray

Published on by admin

Introduction

Kubespray is different to the recently evaluated Gravity deploy. It uses ansible to drive deployment on the base system, rather than attempting to package all of the kubernetes components into a container

If it is not obvious, Kubespray makes extensive use of connectivity to the internet in order to download and configure a lot of the required assets, and command line tools. This is not always workable in an on-premis environment.

Installation

In order to get started, clone the official kubespray repo:

$ git clone https://github.com/kubernetes-sigs/kubespray.git
Cloning into 'kubespray'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 33233 (delta 1), reused 2 (delta 1), pack-reused 33228
Receiving objects: 100% (33233/33233), 9.67 MiB | 14.10 MiB/s, done.
Resolving deltas: 100% (18561/18561), done.
$

You will need to edit the inventory file to reflect the hosts that you wish to install the Kubernetes cluster on. If this is a large production installation, you may want to take a look at dynamic inventories for Ansible.

In my case, I’m planning to install the cluster on three nodes named playground[1-3] so I’ve edited the inventory file to reflect that:

Creating the inventory

$ cat inventory/local/hosts.ini 
playground1 ansible_connection=local local_release_dir={{ansible_env.HOME}}/releases

[kube-master]
playground1

[etcd]
playground1

[kube-node]
playground1
playground2
playground3

[k8s-cluster:children]
kube-node
kube-master
$

Installing the kubespray requirements

Next, make sure that you have all of the requirements required:

$ sudo pip install -r requirements.txt

Customization

You may want to review and possibly change some of the variables in the group_vars directories.

$ find inventory/local/group_vars/
inventory/local/group_vars/
inventory/local/group_vars/all
inventory/local/group_vars/all/all.yml
inventory/local/group_vars/all/azure.yml
inventory/local/group_vars/all/coreos.yml
inventory/local/group_vars/all/docker.yml
inventory/local/group_vars/all/oci.yml
inventory/local/group_vars/all/openstack.yml
inventory/local/group_vars/etcd.yml
inventory/local/group_vars/k8s-cluster
inventory/local/group_vars/k8s-cluster/addons.yml
inventory/local/group_vars/k8s-cluster/k8s-cluster.yml
inventory/local/group_vars/k8s-cluster/k8s-net-calico.yml
inventory/local/group_vars/k8s-cluster/k8s-net-canal.yml
inventory/local/group_vars/k8s-cluster/k8s-net-cilium.yml
inventory/local/group_vars/k8s-cluster/k8s-net-contiv.yml
inventory/local/group_vars/k8s-cluster/k8s-net-flannel.yml
inventory/local/group_vars/k8s-cluster/k8s-net-kube-router.yml
inventory/local/group_vars/k8s-cluster/k8s-net-weave.yml
$

If you have not used Ansible before, it works over ssh and requires the user running it to have passwordless ssh access to all of the nodes. In the case of kubespray, this user is root, so you need to have a strategy to make that work — at least for installation time.

Running the install

$ sudo bash
# ansible-playbook -i inventory/local/hosts.ini cluster.yml

After running the installation, a slow 10 minutes on my test setup, we get to the end of the Ansible run:


PLAY RECAP *********************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   
playground1                : ok=404  changed=126  unreachable=0    failed=0   
playground2                : ok=253  changed=76   unreachable=0    failed=0   
playground3                : ok=254  changed=76   unreachable=0    failed=0   

Monday 25 February 2019  18:36:02 -0600 (0:00:00.027)       0:10:49.135 ******* 
=============================================================================== 
download : container_download | download images for kubeadm config images ---------------------------------------------------------------------- 39.74s
download : file_download | Download item ------------------------------------------------------------------------------------------------------- 38.43s
kubernetes/master : kubeadm | Initialize first master ------------------------------------------------------------------------------------------ 28.25s
kubernetes/node : install | Copy hyperkube binary from download dir ---------------------------------------------------------------------------- 27.08s
container-engine/docker : ensure docker packages are installed --------------------------------------------------------------------------------- 25.38s
download : file_download | Download item ------------------------------------------------------------------------------------------------------- 19.69s
kubernetes/kubeadm : Join to cluster ----------------------------------------------------------------------------------------------------------- 16.65s
kubernetes/preinstall : Update package management cache (YUM) ---------------------------------------------------------------------------------- 16.56s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------- 11.67s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------- 11.57s
container-engine/docker : Docker | pause while Docker restarts --------------------------------------------------------------------------------- 10.08s
kubernetes/kubeadm : Restart all kube-proxy pods to ensure that they load the new configmap ----------------------------------------------------- 9.44s
kubernetes/node : install | Copy kubelet binary from download dir ------------------------------------------------------------------------------- 9.20s
etcd : Configure | Check if etcd cluster is healthy --------------------------------------------------------------------------------------------- 7.34s
container-engine/docker : Ensure old versions of Docker are not installed. | RedHat ------------------------------------------------------------- 7.20s
download : container_download | Download containers if pull is required or told to always pull (all nodes) -------------------------------------- 7.18s
download : container_download | Download containers if pull is required or told to always pull (all nodes) -------------------------------------- 6.75s
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------------- 6.58s
download : container_download | Download containers if pull is required or told to always pull (all nodes) -------------------------------------- 6.47s
kubernetes/node : install | Write kubelet systemd init file ------------------------------------------------------------------------------------- 6.44s

Playing with the cluster

# kubectl get nodes
NAME          STATUS   ROLES         AGE     VERSION
playground1   Ready    master,node   4m9s    v1.13.3
playground2   Ready    node          3m43s   v1.13.3
playground3   Ready    node          3m43s   v1.13.3
# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-8594b5df7b-26gn4   1/1     Running   0          3m18s
calico-node-424g9                          1/1     Running   0          3m25s
calico-node-8wpfc                          1/1     Running   0          3m25s
calico-node-f4jbh                          1/1     Running   0          3m25s
coredns-6fd7dbf94c-9wtjj                   1/1     Running   0          2m40s
coredns-6fd7dbf94c-ldvcj                   1/1     Running   0          2m44s
dns-autoscaler-5b4847c446-m2df7            1/1     Running   0          2m41s
kube-apiserver-playground1                 1/1     Running   0          4m40s
kube-controller-manager-playground1        1/1     Running   0          4m40s
kube-proxy-ldntn                           1/1     Running   0          3m10s
kube-proxy-skvcq                           1/1     Running   0          3m1s
kube-proxy-tj2bk                           1/1     Running   0          3m21s
kube-scheduler-playground1                 1/1     Running   0          4m40s
kubernetes-dashboard-8457c55f89-pjn9g      1/1     Running   0          2m39s
nginx-proxy-playground2                    1/1     Running   0          4m27s
nginx-proxy-playground3                    1/1     Running   0          4m27s
#

Kubespray helpfully creates and populates a Kubeconfig file with embedded certs for client identity. This is useful, but should probably be thought of as ‘the root user’ of the Kubernetes cluster.

In order to grant additional access to the cluster, now would be a good time to look into identity management and role based access control

Adding additional nodes

Adding a node is done by adding to the inventory and running ansible-playbook with the -l option to limit the install to the new nodes. All of the certificate handling is performed under the hood by kubespray.

Conclusion

If you are familiar with Ansible, then Kubespray is a great way to go; it encapsulates all of the complexity of orchestrating a cluster into a single command to run.

There are downsides: Kubespray is hiding a lot of complexity here, and ultimately it is useful to understand in detail what is going on. As stated earlier in the post, it’s also an absolute necessity to have relatively wide-open access to the internet for Kubespray to work. Of course, this is plain old Ansible, so it is possible to alter Kubespray to work with in-house repos if needed.

I found Kubespray surprisingly easy to get along with, and would recommend it for those wanting to get up and running on bare metal if internet access is not an issue for you.

Evaluation :: Gravity

Published on by admin

Gravity is a product from gravitational that is designed to solve the problem of packaging and deploying a multi-node Kubernetes application. It has native support for the big cloud providers.

One of the advantages of Gravity is the opscenter, which is a registry on which you can publish your applications. In theory, if hardware or cloud resources are available, you can single-click instantiate a whole cluster running the intended application.

One of the huge advantages of Gravity is that it bundles everything together in one tarball, so you do not need any internet connectivity in order to deploy your cluster and application. This makes it well suited for environments like DMZ that have little to no internet access.

In this post, I’m going to slightly abuse the use-case for Gravity, and use it as a pre-packaged system to deploy a base Kubernetes cluster

Creating the ‘application’

There is some basic documentation on how to build Gravity applications on the Gravitational site. The basic idea is that you define a gravity application ‘bundle’ which will produce a tarball of kubernetes in addition to embedding any docker container images and your installation resources.

The first step is to create a set of manifests that define our application. When done, our directory will look like this:

 $ find .
.
./app.yaml
./install-hook.yaml
./update-hook.yaml
./uninst-hook.yaml

Lets take a look at each file one-by-one

app.yaml

---
apiVersion: bundle.gravitational.io/v2
kind: Bundle
metadata:
  name: beyondthekube
  resourceVersion: 0.0.1
  description: This is a test app for gravity
  author: beyondthekube.com

installer:
  flavors:
    default: one
    items:
    - name: one
      description: Single node installation
      nodes:
      - profile: node
        count: 1
    - name: three
      description: Three node cluster
      nodes:
      - profile: node
        count: 3

nodeProfiles:
- name: node
  description: worker node

hooks:
  install:
    job: file://install-hook.yaml
  update:
    job: file://update-hook.yaml
  uninstall:
    job: file://uninst-hook.yaml

systemOptions:
  docker:
    storageDriver: overlay2
    args:
    - --log-level=DEBUG

install-hook.yaml

apiversion: batch/v1
kind: Job
metadata:
  name: install-hook
spec:
  template:
    metadata:
      name: install-hook
    spec:
      restartPolicy: OnFailure
      containers:
      - name: debian-tall
        image: quay.io/gravitational/debian-tall:0.0.1
        command:
        - /usr/local/bin/kubectl
        - apply
        - -f
        - /var/lib/gravity/resources/myapp.yaml

Performing the build

 $ ../tele build app.yaml 
* [1/6] Selecting application runtime
	Will use latest runtime version 5.4.6
* [2/6] Downloading dependencies from s3://hub.gravitational.io
	Still downloading dependencies from s3://hub.gravitational.io (10 seconds elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (20 seconds elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (30 seconds elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (40 seconds elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (50 seconds elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (2 minutes elapsed)
	Still downloading dependencies from s3://hub.gravitational.io (2 minutes elapsed)
* [3/6] Embedding application container images
	Detected application manifest app.yaml
	Detected resource file install-hook.yaml
	Detected resource file uninst-hook.yaml
	Detected resource file update-hook.yaml
	Using local image quay.io/gravitational/debian-tall:0.0.1
	Using local image quay.io/gravitational/debian-tall:0.0.1
	Vendored image gravitational/debian-tall:0.0.1
* [4/6] Using runtime version 5.4.6
* [5/6] Generating the cluster snapshot
	Still generating the cluster snapshot (10 seconds elapsed)
	Still generating the cluster snapshot (20 seconds elapsed)
* [6/6] Saving the snapshot as beyondthekube-0.0.1.tar
	Still saving the snapshot as beyondthekube-0.0.1.tar (10 seconds elapsed)
	Still saving the snapshot as beyondthekube-0.0.1.tar (20 seconds elapsed)
* [6/6] Build completed in 2 minutes 
 $

The build produces a tarball as indicated in the output. If we look inside, we can see that the tar includes bundled scripts, binaries and packaged blobs – these are the container images.

 $ tar tvf beyondthekube-0.0.1.tar 
-rwxr-xr-x 1000/1000  64053744 2019-02-14 21:36 gravity
-rw-r--r-- 1000/1000      5364 2019-02-14 21:36 app.yaml
-rwxr-xr-x 1000/1000       907 2019-02-14 21:36 install
-rwxr-xr-x 1000/1000       411 2019-02-14 21:36 upload
-rwxr-xr-x 1000/1000       344 2019-02-14 21:36 upgrade
-rw-r--r-- 1000/1000      1086 2019-02-14 21:36 README
-rw------- beyond/beyond 262144 2019-02-14 21:36 gravity.db
drwxr-xr-x beyond/beyond      0 2019-02-14 21:35 packages
drwxr-xr-x beyond/beyond      0 2019-02-14 21:36 packages/blobs
drwxr-xr-x beyond/beyond      0 2019-02-14 21:35 packages/blobs/128
-rw------- beyond/beyond 443247654 2019-02-14 21:35 packages/blobs/128/128cb957bf304b8ac62f7404dd80b2d9353b7a8b8b94c3d1aefce54d0b749752
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/19d
-rw------- beyond/beyond 151150973 2019-02-14 21:36 packages/blobs/19d/19df5d94b336fd5d59d5957122ad3b75f6bb550281593dd30a9ffc2cd4a51984
drwxr-xr-x beyond/beyond         0 2019-02-14 21:35 packages/blobs/1eb
-rw------- beyond/beyond  64053744 2019-02-14 21:35 packages/blobs/1eb/1eb29eaf77d0cf883b9636e7a92c4466bb476ade645821eb7df6a6aff7f62dac
drwxr-xr-x beyond/beyond         0 2019-02-14 21:35 packages/blobs/21a
-rw------- beyond/beyond   5131031 2019-02-14 21:35 packages/blobs/21a/21a2a700c454ed032ddca4c581480e384d172b6c3ad8e592be2a7abb6a07ba69
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/2f8
-rw------- beyond/beyond  67647165 2019-02-14 21:36 packages/blobs/2f8/2f8cb1d2724d9d68f684b9a3d552119005e745e343f12b5528996557da9af8a9
drwxr-xr-x beyond/beyond         0 2019-02-14 21:35 packages/blobs/6ad
-rw------- beyond/beyond  56612218 2019-02-14 21:35 packages/blobs/6ad/6adb28689c4c87153c09526123991aa7a241b3aa4ee0677b14c95ff1f5853d9b
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/7fa
-rw------- beyond/beyond  23333959 2019-02-14 21:36 packages/blobs/7fa/7fa881d2c9d847e44638354ce3bed1585ac4cc14da0f0a831a8160f35b40e98a
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/874
-rw------- beyond/beyond 305392980 2019-02-14 21:36 packages/blobs/874/8740c910a5040fe9f018c88d6c773e5e3eaf7041be495a52fc2aaa19eeb2ba79
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/9bc
-rw------- beyond/beyond   5131049 2019-02-14 21:36 packages/blobs/9bc/9bcd1569b28fab6e7e4faa3e1add67e10ed152583de537de3ca6c0e397e380fe
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/a6a
-rw------- beyond/beyond   5130375 2019-02-14 21:36 packages/blobs/a6a/a6a499fec0dcee59da5e50595605087b37ab39ced9888d91344da63ea77927cc
drwxr-xr-x beyond/beyond         0 2019-02-14 21:35 packages/blobs/beb
-rw------- beyond/beyond  16082373 2019-02-14 21:35 packages/blobs/beb/beb9f38e50c05cfb45db81f898b927554a3f3aa46df22c5d0134c3bbef414bf7
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/blobs/d40
-rw------- beyond/beyond  74265843 2019-02-14 21:36 packages/blobs/d40/d401e2bd53fc7d08bd22745d45ba8004199150c1af0ed38c5b53cd0cd0cb3289
drwxr-xr-x beyond/beyond         0 2019-02-14 21:35 packages/blobs/f42
-rw------- beyond/beyond   1258660 2019-02-14 21:35 packages/blobs/f42/f4262bdd8f893444ce43321e753dd8c198ba25974a82e4ed3b722cc2ce08a666
drwxr-xr-x beyond/beyond         0 2019-02-14 21:36 packages/tmp
drwxr-xr-x beyond/beyond         0 2019-02-14 21:35 packages/unpacked
 $

Installation

Potential Errors

If you get errors running the install, your underlying OS installation is probably failing the preflight checks. The installer is pretty good about letting you know what is required in the error message:

$ sudo ./gravity install --flavor=one
Sat Feb 16 23:17:23 UTC	Starting installer
Sat Feb 16 23:17:23 UTC	Preparing for installation...
Sat Feb 16 23:18:08 UTC	Installing application beyondthekube:0.0.1
Sat Feb 16 23:18:08 UTC	Starting non-interactive install
Sat Feb 16 17:18:08 UTC	Auto-loaded kernel module: overlay
Sat Feb 16 17:18:08 UTC	Auto-set kernel parameter: net.ipv4.ip_forward=1
[ERROR]: The following pre-flight checks failed:
	[×] open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory (br_netfilter module is either not loaded, or sysctl net.bridge.bridge-nf-call-iptables is not set, see https://www.gravitational.com/docs/faq/#bridge-driver)

One issue with the gravity installer is in an environment where a directory service is the authoritative source of user logins. By default the gravity installer wants to perform a useradd if the ‘planet’ user and group are not found in their respective files. The easiest way to work around this (if you have pam_extrausers) is to add the users to the respective extrausers files:

$ sudo mkdir -p /var/lib/extrausers
$ sudo getent passwd planet > /var/lib/extrausers/passwd
$ sudo getent group planet > /var/lib/extrausers/group

This is the case even if you change the user and group used to run gravity by passing the --service-uid and --service-gid flags.

Types of installation

When we created the application using the app.yaml manifest above, we defined two flavors; “one” and “three” which specify single node and 3 node installations respectively. Let’s start with the simple, single node installation:

Single node

 $ mkdir ../inst
 $ cd ../inst
 $ tar xf ../app/beyondthekube-0.0.1.tar 
 $ sudo ./gravity install --flavor=one
Mon Feb 18 04:14:16 UTC	Starting installer
Mon Feb 18 04:14:16 UTC	Preparing for installation...
Mon Feb 18 04:14:46 UTC	Installing application beyondthekube:0.0.1
Mon Feb 18 04:14:46 UTC	Starting non-interactive install
Mon Feb 18 04:14:46 UTC	Still waiting for 1 nodes of role "node"
Mon Feb 18 04:14:47 UTC	All agents have connected!
Mon Feb 18 04:14:48 UTC	Starting the installation
Mon Feb 18 04:14:48 UTC	Operation has been created
Mon Feb 18 04:14:49 UTC	Execute preflight checks
Mon Feb 18 04:14:56 UTC	Configure packages for all nodes
Mon Feb 18 04:15:03 UTC	Bootstrap all nodes
Mon Feb 18 04:15:04 UTC	Bootstrap master node playground1
Mon Feb 18 04:15:11 UTC	Pull packages on master node playground1
Mon Feb 18 04:16:15 UTC	Install system software on master node playground1
Mon Feb 18 04:16:16 UTC	Install system package teleport:2.4.7 on master node playground1
Mon Feb 18 04:16:17 UTC	Install system package planet:5.4.7-11302 on master node playground1
Mon Feb 18 04:16:51 UTC	Wait for system services to start on all nodes
Mon Feb 18 04:17:32 UTC	Bootstrap Kubernetes roles and PSPs
Mon Feb 18 04:17:34 UTC	Populate Docker registry on master node playground1
Mon Feb 18 04:18:19 UTC	Install system application dns-app:0.2.0
Mon Feb 18 04:18:20 UTC	Install system application logging-app:5.0.2
Mon Feb 18 04:18:29 UTC	Install system application monitoring-app:5.2.2
Mon Feb 18 04:18:53 UTC	Install system application tiller-app:5.2.1
Mon Feb 18 04:19:18 UTC	Install system application site:5.4.6
Mon Feb 18 04:20:55 UTC	Install system application kubernetes:5.4.6
Mon Feb 18 04:20:56 UTC	Install application beyondthekube:0.0.1
Mon Feb 18 04:20:59 UTC	Enable elections
Mon Feb 18 04:21:01 UTC	Operation has completed
Mon Feb 18 04:21:01 UTC	Installation succeeded in 6m15.298851759s
 $

Multi node

The installation is slightly more complicated to orchestrate with 3 nodes. In this case, a node called ‘playground1’ will run the installation and the others (playground2 and playground3) will join that installer. A token is used to permit the nodes to join 

[playground1] $ mkdir ../inst
[playground1] $ cd ../inst
[playground1] $ scp ../app/beyoundthekube-0.0.1.tar playground2:[playground1] $ scp ../app/beyoundthekube-0.0.1.tar playground3:
[playground1] $ tar xf ../app/beyondthekube-0.0.1.tar 
[playground1] $ sudo ./gravity install --flavor=three --token=multinode
Mon Feb 18 05:09:47 UTC	Starting installer
Mon Feb 18 05:09:47 UTC	Preparing for installation...
Mon Feb 18 05:10:21 UTC	Installing application beyondthekube:0.0.1
Mon Feb 18 05:10:21 UTC	Starting non-interactive install
Sun Feb 17 23:10:21 UTC	Auto-loaded kernel module: br_netfilter
Sun Feb 17 23:10:21 UTC	Auto-loaded kernel module: iptable_nat
Sun Feb 17 23:10:21 UTC	Auto-loaded kernel module: iptable_filter
Sun Feb 17 23:10:21 UTC	Auto-loaded kernel module: ebtables
Sun Feb 17 23:10:21 UTC	Auto-loaded kernel module: overlay
Sun Feb 17 23:10:21 UTC	Auto-set kernel parameter: net.ipv4.ip_forward=1
Sun Feb 17 23:10:21 UTC	Auto-set kernel parameter: net.bridge.bridge-nf-call-iptables=1
Mon Feb 18 05:10:21 UTC	Still waiting for 3 nodes of role "node"
Mon Feb 18 05:10:22 UTC	Still waiting for 1 nodes of role "node"
Mon Feb 18 05:10:23 UTC	Still waiting for 1 nodes of role "node"
Mon Feb 18 05:10:24 UTC	All agents have connected!
Mon Feb 18 05:10:25 UTC	Starting the installation
Mon Feb 18 05:10:25 UTC	Operation has been created
Mon Feb 18 05:10:27 UTC	Execute preflight checks
Mon Feb 18 05:10:57 UTC	Configure packages for all nodes
Mon Feb 18 05:11:15 UTC	Bootstrap all nodes
Mon Feb 18 05:11:17 UTC	Bootstrap master node playground1
Mon Feb 18 05:11:25 UTC	Pull configured packages
Mon Feb 18 05:11:26 UTC	Pull packages on master node playground1
Mon Feb 18 05:15:50 UTC	Install system software on master nodes
Mon Feb 18 05:15:51 UTC	Install system package teleport:2.4.7 on master node playground2
Mon Feb 18 05:15:52 UTC	Install system package teleport:2.4.7 on master node playground3
Mon Feb 18 05:16:00 UTC	Install system package planet:5.4.7-11302 on master node playground3
Mon Feb 18 05:17:58 UTC	Wait for system services to start on all nodes
Mon Feb 18 05:19:25 UTC	Bootstrap Kubernetes roles and PSPs
Mon Feb 18 05:19:28 UTC	Export applications layers to Docker registries
Mon Feb 18 05:19:29 UTC	Populate Docker registry on master node playground1
Mon Feb 18 05:22:20 UTC	Install system applications
Mon Feb 18 05:22:21 UTC	Install system application dns-app:0.2.0
Mon Feb 18 05:22:22 UTC	Install system application logging-app:5.0.2
Mon Feb 18 05:22:43 UTC	Install system application monitoring-app:5.2.2
Mon Feb 18 05:23:31 UTC	Install system application tiller-app:5.2.1
Mon Feb 18 05:24:43 UTC	Install system application site:5.4.6
Mon Feb 18 05:29:56 UTC	Install system application kubernetes:5.4.6
Mon Feb 18 05:29:58 UTC	Install user application
Mon Feb 18 05:29:59 UTC	Install application beyondthekube:0.0.1
Mon Feb 18 05:30:39 UTC	Enable elections
Mon Feb 18 05:30:43 UTC	Operation has completed
Mon Feb 18 05:30:44 UTC	Installation succeeded in 20m22.974575584s
[playground1]$

Once the main installer is running, you can join the installation with the other two nodes (I have omitted palyground3, as the output is almost identical to playground2)

[playground2] $ mkdir ~/inst
[playground2] $ cd ~/inst
[playground2] $ tar xf ../app/beyondthekube-0.0.1.tar 
[playground2] $ sudo ./gravity join playground1 --token=multinode
Mon Feb 18 05:10:11 UTC	Connecting to cluster
Mon Feb 18 05:10:12 UTC	Connecting to cluster
Mon Feb 18 05:10:12 UTC	Connecting to cluster
Mon Feb 18 05:10:13 UTC	Connecting to cluster
Mon Feb 18 05:10:16 UTC	Connecting to cluster
Mon Feb 18 05:10:18 UTC	Connecting to cluster
Mon Feb 18 05:10:22 UTC	Connecting to cluster
Sun Feb 17 23:10:23 UTC	Auto-loaded kernel module: br_netfilter
Sun Feb 17 23:10:23 UTC	Auto-loaded kernel module: iptable_nat
Sun Feb 17 23:10:23 UTC	Auto-loaded kernel module: iptable_filter
Sun Feb 17 23:10:23 UTC	Auto-loaded kernel module: ebtables
Sun Feb 17 23:10:23 UTC	Auto-loaded kernel module: overlay
Sun Feb 17 23:10:23 UTC	Auto-set kernel parameter: net.ipv4.ip_forward=1
Sun Feb 17 23:10:23 UTC	Auto-set kernel parameter: net.bridge.bridge-nf-call-iptables=1
Mon Feb 18 05:10:23 UTC	Connected to installer at playground1
Mon Feb 18 05:10:24 UTC	Operation has been created
Mon Feb 18 05:10:27 UTC	Execute preflight checks
Mon Feb 18 05:10:57 UTC	Configure packages for all nodes
Mon Feb 18 05:11:15 UTC	Bootstrap all nodes
Mon Feb 18 05:11:17 UTC	Bootstrap master node playground1
Mon Feb 18 05:11:26 UTC	Pull packages on master node playground1
Mon Feb 18 05:15:50 UTC	Install system software on master nodes
Mon Feb 18 05:15:51 UTC	Install system package teleport:2.4.7 on master node playground2
Mon Feb 18 05:15:52 UTC	Install system package teleport:2.4.7 on master node playground3
Mon Feb 18 05:16:01 UTC	Install system package planet:5.4.7-11302 on master node playground3
Mon Feb 18 05:18:03 UTC	Wait for system services to start on all nodes
Mon Feb 18 05:19:25 UTC	Bootstrap Kubernetes roles and PSPs
Mon Feb 18 05:19:28 UTC	Export applications layers to Docker registries
Mon Feb 18 05:19:29 UTC	Populate Docker registry on master node playground1
Mon Feb 18 05:22:20 UTC	Install system applications
Mon Feb 18 05:22:21 UTC	Install system application dns-app:0.2.0
Mon Feb 18 05:22:22 UTC	Install system application logging-app:5.0.2
Mon Feb 18 05:22:43 UTC	Install system application monitoring-app:5.2.2
Mon Feb 18 05:23:31 UTC	Install system application tiller-app:5.2.1
Mon Feb 18 05:24:43 UTC	Install system application site:5.4.6
Mon Feb 18 05:29:57 UTC	Install system application kubernetes:5.4.6
Mon Feb 18 05:29:58 UTC	Install user application
Mon Feb 18 05:29:59 UTC	Install application beyondthekube:0.0.1
Mon Feb 18 05:30:38 UTC	Enable elections
Mon Feb 18 05:30:43 UTC	Operation has completed
Mon Feb 18 05:30:44 UTC	Joined cluster in 20m31.733348076s
[playground2]$

What is actually happening above is that playground1 is launching an installer and listening for other nodes to join. When we issue the gravity join command on playground2 and playground3 those nodes connect to playground1 using the token we passed in.

Post installation goodness

When the installation is done, you can inspect the gravity environment by running sudo gravity status. Enter the planet environment and inspect the kubernetes environment.

 $ sudo gravity status
Cluster status:	active
Application:	beyondthekube, version 0.0.1
Join token:	c47d9d1329eab6f624116591315b0841aca60f3ed2c6f2fd1a7a02d65111e2b3
Last completed operation:
    * operation_install (33072172-7852-47cc-80cf-e6d2dd10916c)
      started:		Mon Feb 18 04:14 UTC (10 minutes ago)
      completed:	Mon Feb 18 04:14 UTC (10 minutes ago)
Cluster:		awesomedarwin4747
    Masters:
        * playground1 (192.168.1.177, node)
            Status:	healthy
 $ sudo gravity enter
                                                     ___
                                                  ,o88888
                                               ,o8888888'
                         ,:o:o:oooo.        ,8O88Pd8888"
                     ,.::.::o:ooooOoOoO. ,oO8O8Pd888'"
                   ,.:.::o:ooOoOoOO8O8OOo.8OOPd8O8O"
                  , ..:.::o:ooOoOOOO8OOOOo.FdO8O8"
                 , ..:.::o:ooOoOO8O888O8O,COCOO"
                , . ..:.::o:ooOoOOOO8OOOOCOCO"
                 . ..:.::o:ooOoOoOO8O8OCCCC"o
                    . ..:.::o:ooooOoCoCCC"o:o
                    . ..:.::o:o:,cooooCo"oo:o:
                 `   . . ..:.:cocoooo"'o:o:::'
                 .`   . ..::ccccoc"'o:o:o:::'
                :.:.    ,c:cccc"':.:.:.:.:.'
              ..:.:"'`::::c:"'..:.:.:.:.:.'
            ...:.'.:.::::"'    . . . . .'
           .. . ....:."' `   .  . . ''
         . . . ...."'
         .. . ."'     -hrr-
        .
playground1:/$ kubectl get nodes
NAME            STATUS   ROLES    AGE     VERSION
192.168.1.177   Ready    <none>   9m18s   v1.13.2
playground1:/$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   5m41s

In addition to looking at the default namespace, gravity provides an alias kctl which is a quick way to run commands against the kube-system namespace. With that, we can see all of the services that gravity installs by default. This provides us with local docker registries for the cluster, monitoring, tiller – the helm server process.

playground1:/$ type kctl
kctl is aliased to `kubectl -nkube-system'
playground1:/$ kctl get all
NAME                                      READY   STATUS      RESTARTS   AGE
pod/gravity-site-79wc6                    1/1     Running     0          3m
pod/install-hook-8fef86-2w9lq             0/1     Completed   0          2m9s
pod/install-telekube-55c2a4-sp4sz         0/1     Completed   0          3m47s
pod/log-collector-697d94486-7glht         1/1     Running     0          4m38s
pod/log-forwarder-kc222                   1/1     Running     0          4m38s
pod/logging-app-bootstrap-c07491-wdsb2    0/1     Completed   0          4m45s
pod/monitoring-app-install-1d8e1d-7q6wk   0/1     Completed   0          4m37s
pod/site-app-post-install-4359d0-vjn2d    0/1     Completed   2          2m57s
pod/tiller-app-bootstrap-b443e8-7dx9s     0/1     Completed   0          4m12s
pod/tiller-deploy-69c5787759-g9td4        1/1     Running     0          3m49s

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                    AGE
service/gravity-site    LoadBalancer   10.100.197.83   <pending>     3009:32009/TCP             3m
service/log-collector   ClusterIP      10.100.186.31   <none>        514/UDP,514/TCP,8083/TCP   4m38s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                      AGE
daemonset.apps/gravity-site    1         1         1       1            1           gravitational.io/k8s-role=master   3m
daemonset.apps/log-forwarder   1         1         1       1            1           <none>                             4m38s

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/log-collector   1/1     1            1           4m38s
deployment.apps/tiller-deploy   1/1     1            1           3m49s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/log-collector-697d94486    1         1         1       4m38s
replicaset.apps/tiller-deploy-69c5787759   1         1         1       3m49s

NAME                                      COMPLETIONS   DURATION   AGE
job.batch/install-hook-8fef86             1/1           2s         2m9s
job.batch/install-telekube-55c2a4         1/1           50s        3m47s
job.batch/logging-app-bootstrap-c07491    1/1           7s         4m46s
job.batch/monitoring-app-install-1d8e1d   1/1           23s        4m37s
job.batch/site-app-post-install-4359d0    1/1           45s        2m57s
job.batch/tiller-app-bootstrap-b443e8     1/1           24s        4m13s
playground1:/$

Note that in multi-node clusters, there will be 3 instances of each system service:

playground1:/$ kctl get pods
NAME                                  READY   STATUS      RESTARTS   AGE
gravity-site-c25rk                    1/1     Running     2          17m
gravity-site-gjfvp                    0/1     Running     1          17m
gravity-site-vd8b7                    0/1     Running     2          17m
log-collector-697d94486-l9cx7         1/1     Running     0          20m
log-forwarder-8nqj9                   1/1     Running     0          20m
log-forwarder-cggjs                   1/1     Running     0          20m
log-forwarder-vpd4d                   1/1     Running     0          20m
tiller-deploy-69c5787759-v9vz5        1/1     Running     0          18m

Things to watch out for

Verbose logging

During the installation, the installer writes /var/log/telekube-install.log and /var/log/telekube-system.log. If you chose to watch those logs during installation, it is easy to get the impression that the installation is not working:

019-02-17T17:52:10-06:00 DEBU             Unsuccessful attempt 1/100: failed to query cluster status from agent, retry in 5s. install/hook.go:56
2019-02-17T17:52:15-06:00 DEBU             Unsuccessful attempt 2/100: failed to query cluster status from agent, retry in 5s. install/hook.go:56
2019-02-17T17:52:20-06:00 DEBU             Unsuccessful attempt 3/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:25-06:00 DEBU             Unsuccessful attempt 4/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:31-06:00 DEBU             Unsuccessful attempt 5/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:36-06:00 DEBU             Unsuccessful attempt 6/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU [KEYGEN]    generated user key for [root] with expiry on (1550483561) 2019-02-18 03:52:41.2294291 -0600 CST m=+36251.083685033 install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU [AUDITLOG]  EmitAuditEvent(user.login: map[user:opscenter@gravitational.io method:local]) install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU             [TELEPORT] generated certificate for opscenter@gravitational.io install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU             Unsuccessful attempt 7/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:56-06:00 DEBU             Unsuccessful attempt 8/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:53:02-06:00 DEBU             Unsuccessful attempt 9/100: not all planets have come up yet: &amp;{unknown []}, retry in 5s. install/hook.go:56

The gravity installation logging is very verbose, and these messages are simply the installation waiting for kubernetes services to come up.

Firewall issues

If the installation is not working for you, attempt it without the firewall enabled. I’m not recommending running live without a firewall running in production, but at least rule it out before considering other issues. In a cloud environment, be sure to check rules built into the cloud provider. On premis, check your network firewalls in addition to the host firewall.

When you have achieved a working installation of gravity without the firewall, add it back in and make sure the recommended ports have been opened.

Conclusion

The Gravity system is a great option for admins that want to entrust the configuration and setup to a third party.

It is a very good option if your objective is to package up your application for a DMZ or to distribute to customers who may not have a good understanding of Kubernetes.

On the less positive side, the installer makes a lot of assumptions about your infrastructure that may be difficult to work around. Further to this, gravity works most seamlessly when you buy into the complete gravitational ecosystem. If you want to run with more esoteric settings or plugins, or you want to run enormous clusters, then gravity may not be the best fit for you.

In the coming weeks, I will be evaluating other methods of on-premis Kubernetes installation, and will eventually post a comparison of all of them.

Stay tuned for that. If you just got your gravity cluster up and running,