Gravity is a product from gravitational that is designed to solve the problem of packaging and deploying a multi-node Kubernetes application. It has native support for the big cloud providers.
One of the advantages of Gravity is the opscenter, which is a registry on which you can publish your applications. In theory, if hardware or cloud resources are available, you can single-click instantiate a whole cluster running the intended application.
One of the huge advantages of Gravity is that it bundles everything together in one tarball, so you do not need any internet connectivity in order to deploy your cluster and application. This makes it well suited for environments like DMZ that have little to no internet access.
In this post, I’m going to slightly abuse the use-case for Gravity, and use it as a pre-packaged system to deploy a base Kubernetes cluster
Creating the ‘application’
There is some basic documentation on how to build Gravity applications on the Gravitational site. The basic idea is that you define a gravity application ‘bundle’ which will produce a tarball of kubernetes in addition to embedding any docker container images and your installation resources.
The first step is to create a set of manifests that define our application. When done, our directory will look like this:
$ find . . ./app.yaml ./install-hook.yaml ./update-hook.yaml ./uninst-hook.yaml
Lets take a look at each file one-by-one
app.yaml
---
apiVersion: bundle.gravitational.io/v2
kind: Bundle
metadata:
name: beyondthekube
resourceVersion: 0.0.1
description: This is a test app for gravity
author: beyondthekube.com
installer:
flavors:
default: one
items:
- name: one
description: Single node installation
nodes:
- profile: node
count: 1
- name: three
description: Three node cluster
nodes:
- profile: node
count: 3
nodeProfiles:
- name: node
description: worker node
hooks:
install:
job: file://install-hook.yaml
update:
job: file://update-hook.yaml
uninstall:
job: file://uninst-hook.yaml
systemOptions:
docker:
storageDriver: overlay2
args:
- --log-level=DEBUG
install-hook.yaml
apiversion: batch/v1
kind: Job
metadata:
name: install-hook
spec:
template:
metadata:
name: install-hook
spec:
restartPolicy: OnFailure
containers:
- name: debian-tall
image: quay.io/gravitational/debian-tall:0.0.1
command:
- /usr/local/bin/kubectl
- apply
- -f
- /var/lib/gravity/resources/myapp.yaml
Performing the build
$ ../tele build app.yaml * [1/6] Selecting application runtime Will use latest runtime version 5.4.6 * [2/6] Downloading dependencies from s3://hub.gravitational.io Still downloading dependencies from s3://hub.gravitational.io (10 seconds elapsed) Still downloading dependencies from s3://hub.gravitational.io (20 seconds elapsed) Still downloading dependencies from s3://hub.gravitational.io (30 seconds elapsed) Still downloading dependencies from s3://hub.gravitational.io (40 seconds elapsed) Still downloading dependencies from s3://hub.gravitational.io (50 seconds elapsed) Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed) Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed) Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed) Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed) Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed) Still downloading dependencies from s3://hub.gravitational.io (1 minute elapsed) Still downloading dependencies from s3://hub.gravitational.io (2 minutes elapsed) Still downloading dependencies from s3://hub.gravitational.io (2 minutes elapsed) * [3/6] Embedding application container images Detected application manifest app.yaml Detected resource file install-hook.yaml Detected resource file uninst-hook.yaml Detected resource file update-hook.yaml Using local image quay.io/gravitational/debian-tall:0.0.1 Using local image quay.io/gravitational/debian-tall:0.0.1 Vendored image gravitational/debian-tall:0.0.1 * [4/6] Using runtime version 5.4.6 * [5/6] Generating the cluster snapshot Still generating the cluster snapshot (10 seconds elapsed) Still generating the cluster snapshot (20 seconds elapsed) * [6/6] Saving the snapshot as beyondthekube-0.0.1.tar Still saving the snapshot as beyondthekube-0.0.1.tar (10 seconds elapsed) Still saving the snapshot as beyondthekube-0.0.1.tar (20 seconds elapsed) * [6/6] Build completed in 2 minutes $
The build produces a tarball as indicated in the output. If we look inside, we can see that the tar includes bundled scripts, binaries and packaged blobs – these are the container images.
$ tar tvf beyondthekube-0.0.1.tar -rwxr-xr-x 1000/1000 64053744 2019-02-14 21:36 gravity -rw-r--r-- 1000/1000 5364 2019-02-14 21:36 app.yaml -rwxr-xr-x 1000/1000 907 2019-02-14 21:36 install -rwxr-xr-x 1000/1000 411 2019-02-14 21:36 upload -rwxr-xr-x 1000/1000 344 2019-02-14 21:36 upgrade -rw-r--r-- 1000/1000 1086 2019-02-14 21:36 README -rw------- beyond/beyond 262144 2019-02-14 21:36 gravity.db drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/blobs/128 -rw------- beyond/beyond 443247654 2019-02-14 21:35 packages/blobs/128/128cb957bf304b8ac62f7404dd80b2d9353b7a8b8b94c3d1aefce54d0b749752 drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/19d -rw------- beyond/beyond 151150973 2019-02-14 21:36 packages/blobs/19d/19df5d94b336fd5d59d5957122ad3b75f6bb550281593dd30a9ffc2cd4a51984 drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/blobs/1eb -rw------- beyond/beyond 64053744 2019-02-14 21:35 packages/blobs/1eb/1eb29eaf77d0cf883b9636e7a92c4466bb476ade645821eb7df6a6aff7f62dac drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/blobs/21a -rw------- beyond/beyond 5131031 2019-02-14 21:35 packages/blobs/21a/21a2a700c454ed032ddca4c581480e384d172b6c3ad8e592be2a7abb6a07ba69 drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/2f8 -rw------- beyond/beyond 67647165 2019-02-14 21:36 packages/blobs/2f8/2f8cb1d2724d9d68f684b9a3d552119005e745e343f12b5528996557da9af8a9 drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/blobs/6ad -rw------- beyond/beyond 56612218 2019-02-14 21:35 packages/blobs/6ad/6adb28689c4c87153c09526123991aa7a241b3aa4ee0677b14c95ff1f5853d9b drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/7fa -rw------- beyond/beyond 23333959 2019-02-14 21:36 packages/blobs/7fa/7fa881d2c9d847e44638354ce3bed1585ac4cc14da0f0a831a8160f35b40e98a drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/874 -rw------- beyond/beyond 305392980 2019-02-14 21:36 packages/blobs/874/8740c910a5040fe9f018c88d6c773e5e3eaf7041be495a52fc2aaa19eeb2ba79 drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/9bc -rw------- beyond/beyond 5131049 2019-02-14 21:36 packages/blobs/9bc/9bcd1569b28fab6e7e4faa3e1add67e10ed152583de537de3ca6c0e397e380fe drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/a6a -rw------- beyond/beyond 5130375 2019-02-14 21:36 packages/blobs/a6a/a6a499fec0dcee59da5e50595605087b37ab39ced9888d91344da63ea77927cc drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/blobs/beb -rw------- beyond/beyond 16082373 2019-02-14 21:35 packages/blobs/beb/beb9f38e50c05cfb45db81f898b927554a3f3aa46df22c5d0134c3bbef414bf7 drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/blobs/d40 -rw------- beyond/beyond 74265843 2019-02-14 21:36 packages/blobs/d40/d401e2bd53fc7d08bd22745d45ba8004199150c1af0ed38c5b53cd0cd0cb3289 drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/blobs/f42 -rw------- beyond/beyond 1258660 2019-02-14 21:35 packages/blobs/f42/f4262bdd8f893444ce43321e753dd8c198ba25974a82e4ed3b722cc2ce08a666 drwxr-xr-x beyond/beyond 0 2019-02-14 21:36 packages/tmp drwxr-xr-x beyond/beyond 0 2019-02-14 21:35 packages/unpacked $
Installation
Potential Errors
If you get errors running the install, your underlying OS installation is probably failing the preflight checks. The installer is pretty good about letting you know what is required in the error message:
$ sudo ./gravity install --flavor=one Sat Feb 16 23:17:23 UTC Starting installer Sat Feb 16 23:17:23 UTC Preparing for installation... Sat Feb 16 23:18:08 UTC Installing application beyondthekube:0.0.1 Sat Feb 16 23:18:08 UTC Starting non-interactive install Sat Feb 16 17:18:08 UTC Auto-loaded kernel module: overlay Sat Feb 16 17:18:08 UTC Auto-set kernel parameter: net.ipv4.ip_forward=1 [ERROR]: The following pre-flight checks failed: [×] open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory (br_netfilter module is either not loaded, or sysctl net.bridge.bridge-nf-call-iptables is not set, see https://www.gravitational.com/docs/faq/#bridge-driver)
One issue with the gravity installer is in an environment where a directory service is the authoritative source of user logins. By default the gravity installer wants to perform a useradd if the ‘planet’ user and group are not found in their respective files. The easiest way to work around this (if you have pam_extrausers) is to add the users to the respective extrausers files:
$ sudo mkdir -p /var/lib/extrausers $ sudo getent passwd planet > /var/lib/extrausers/passwd $ sudo getent group planet > /var/lib/extrausers/group
This is the case even if you change the user and group used to run gravity by passing the --service-uid and --service-gid flags.
Types of installation
When we created the application using the app.yaml manifest above, we defined two flavors; “one” and “three” which specify single node and 3 node installations respectively. Let’s start with the simple, single node installation:
Single node
$ mkdir ../inst $ cd ../inst $ tar xf ../app/beyondthekube-0.0.1.tar $ sudo ./gravity install --flavor=one Mon Feb 18 04:14:16 UTC Starting installer Mon Feb 18 04:14:16 UTC Preparing for installation... Mon Feb 18 04:14:46 UTC Installing application beyondthekube:0.0.1 Mon Feb 18 04:14:46 UTC Starting non-interactive install Mon Feb 18 04:14:46 UTC Still waiting for 1 nodes of role "node" Mon Feb 18 04:14:47 UTC All agents have connected! Mon Feb 18 04:14:48 UTC Starting the installation Mon Feb 18 04:14:48 UTC Operation has been created Mon Feb 18 04:14:49 UTC Execute preflight checks Mon Feb 18 04:14:56 UTC Configure packages for all nodes Mon Feb 18 04:15:03 UTC Bootstrap all nodes Mon Feb 18 04:15:04 UTC Bootstrap master node playground1 Mon Feb 18 04:15:11 UTC Pull packages on master node playground1 Mon Feb 18 04:16:15 UTC Install system software on master node playground1 Mon Feb 18 04:16:16 UTC Install system package teleport:2.4.7 on master node playground1 Mon Feb 18 04:16:17 UTC Install system package planet:5.4.7-11302 on master node playground1 Mon Feb 18 04:16:51 UTC Wait for system services to start on all nodes Mon Feb 18 04:17:32 UTC Bootstrap Kubernetes roles and PSPs Mon Feb 18 04:17:34 UTC Populate Docker registry on master node playground1 Mon Feb 18 04:18:19 UTC Install system application dns-app:0.2.0 Mon Feb 18 04:18:20 UTC Install system application logging-app:5.0.2 Mon Feb 18 04:18:29 UTC Install system application monitoring-app:5.2.2 Mon Feb 18 04:18:53 UTC Install system application tiller-app:5.2.1 Mon Feb 18 04:19:18 UTC Install system application site:5.4.6 Mon Feb 18 04:20:55 UTC Install system application kubernetes:5.4.6 Mon Feb 18 04:20:56 UTC Install application beyondthekube:0.0.1 Mon Feb 18 04:20:59 UTC Enable elections Mon Feb 18 04:21:01 UTC Operation has completed Mon Feb 18 04:21:01 UTC Installation succeeded in 6m15.298851759s $
Multi node
The installation is slightly more complicated to orchestrate with 3 nodes. In this case, a node called ‘playground1’ will run the installation and the others (playground2 and playground3) will join that installer. A token is used to permit the nodes to join
[playground1] $ mkdir ../inst [playground1] $ cd ../inst [playground1] $ scp ../app/beyoundthekube-0.0.1.tar playground2:[playground1] $ scp ../app/beyoundthekube-0.0.1.tar playground3: [playground1] $ tar xf ../app/beyondthekube-0.0.1.tar [playground1] $ sudo ./gravity install --flavor=three --token=multinode Mon Feb 18 05:09:47 UTC Starting installer Mon Feb 18 05:09:47 UTC Preparing for installation... Mon Feb 18 05:10:21 UTC Installing application beyondthekube:0.0.1 Mon Feb 18 05:10:21 UTC Starting non-interactive install Sun Feb 17 23:10:21 UTC Auto-loaded kernel module: br_netfilter Sun Feb 17 23:10:21 UTC Auto-loaded kernel module: iptable_nat Sun Feb 17 23:10:21 UTC Auto-loaded kernel module: iptable_filter Sun Feb 17 23:10:21 UTC Auto-loaded kernel module: ebtables Sun Feb 17 23:10:21 UTC Auto-loaded kernel module: overlay Sun Feb 17 23:10:21 UTC Auto-set kernel parameter: net.ipv4.ip_forward=1 Sun Feb 17 23:10:21 UTC Auto-set kernel parameter: net.bridge.bridge-nf-call-iptables=1 Mon Feb 18 05:10:21 UTC Still waiting for 3 nodes of role "node" Mon Feb 18 05:10:22 UTC Still waiting for 1 nodes of role "node" Mon Feb 18 05:10:23 UTC Still waiting for 1 nodes of role "node" Mon Feb 18 05:10:24 UTC All agents have connected! Mon Feb 18 05:10:25 UTC Starting the installation Mon Feb 18 05:10:25 UTC Operation has been created Mon Feb 18 05:10:27 UTC Execute preflight checks Mon Feb 18 05:10:57 UTC Configure packages for all nodes Mon Feb 18 05:11:15 UTC Bootstrap all nodes Mon Feb 18 05:11:17 UTC Bootstrap master node playground1 Mon Feb 18 05:11:25 UTC Pull configured packages Mon Feb 18 05:11:26 UTC Pull packages on master node playground1 Mon Feb 18 05:15:50 UTC Install system software on master nodes Mon Feb 18 05:15:51 UTC Install system package teleport:2.4.7 on master node playground2 Mon Feb 18 05:15:52 UTC Install system package teleport:2.4.7 on master node playground3 Mon Feb 18 05:16:00 UTC Install system package planet:5.4.7-11302 on master node playground3 Mon Feb 18 05:17:58 UTC Wait for system services to start on all nodes Mon Feb 18 05:19:25 UTC Bootstrap Kubernetes roles and PSPs Mon Feb 18 05:19:28 UTC Export applications layers to Docker registries Mon Feb 18 05:19:29 UTC Populate Docker registry on master node playground1 Mon Feb 18 05:22:20 UTC Install system applications Mon Feb 18 05:22:21 UTC Install system application dns-app:0.2.0 Mon Feb 18 05:22:22 UTC Install system application logging-app:5.0.2 Mon Feb 18 05:22:43 UTC Install system application monitoring-app:5.2.2 Mon Feb 18 05:23:31 UTC Install system application tiller-app:5.2.1 Mon Feb 18 05:24:43 UTC Install system application site:5.4.6 Mon Feb 18 05:29:56 UTC Install system application kubernetes:5.4.6 Mon Feb 18 05:29:58 UTC Install user application Mon Feb 18 05:29:59 UTC Install application beyondthekube:0.0.1 Mon Feb 18 05:30:39 UTC Enable elections Mon Feb 18 05:30:43 UTC Operation has completed Mon Feb 18 05:30:44 UTC Installation succeeded in 20m22.974575584s [playground1]$
Once the main installer is running, you can join the installation with the other two nodes (I have omitted palyground3, as the output is almost identical to playground2)
[playground2] $ mkdir ~/inst [playground2] $ cd ~/inst [playground2] $ tar xf ../app/beyondthekube-0.0.1.tar [playground2] $ sudo ./gravity join playground1 --token=multinode Mon Feb 18 05:10:11 UTC Connecting to cluster Mon Feb 18 05:10:12 UTC Connecting to cluster Mon Feb 18 05:10:12 UTC Connecting to cluster Mon Feb 18 05:10:13 UTC Connecting to cluster Mon Feb 18 05:10:16 UTC Connecting to cluster Mon Feb 18 05:10:18 UTC Connecting to cluster Mon Feb 18 05:10:22 UTC Connecting to cluster Sun Feb 17 23:10:23 UTC Auto-loaded kernel module: br_netfilter Sun Feb 17 23:10:23 UTC Auto-loaded kernel module: iptable_nat Sun Feb 17 23:10:23 UTC Auto-loaded kernel module: iptable_filter Sun Feb 17 23:10:23 UTC Auto-loaded kernel module: ebtables Sun Feb 17 23:10:23 UTC Auto-loaded kernel module: overlay Sun Feb 17 23:10:23 UTC Auto-set kernel parameter: net.ipv4.ip_forward=1 Sun Feb 17 23:10:23 UTC Auto-set kernel parameter: net.bridge.bridge-nf-call-iptables=1 Mon Feb 18 05:10:23 UTC Connected to installer at playground1 Mon Feb 18 05:10:24 UTC Operation has been created Mon Feb 18 05:10:27 UTC Execute preflight checks Mon Feb 18 05:10:57 UTC Configure packages for all nodes Mon Feb 18 05:11:15 UTC Bootstrap all nodes Mon Feb 18 05:11:17 UTC Bootstrap master node playground1 Mon Feb 18 05:11:26 UTC Pull packages on master node playground1 Mon Feb 18 05:15:50 UTC Install system software on master nodes Mon Feb 18 05:15:51 UTC Install system package teleport:2.4.7 on master node playground2 Mon Feb 18 05:15:52 UTC Install system package teleport:2.4.7 on master node playground3 Mon Feb 18 05:16:01 UTC Install system package planet:5.4.7-11302 on master node playground3 Mon Feb 18 05:18:03 UTC Wait for system services to start on all nodes Mon Feb 18 05:19:25 UTC Bootstrap Kubernetes roles and PSPs Mon Feb 18 05:19:28 UTC Export applications layers to Docker registries Mon Feb 18 05:19:29 UTC Populate Docker registry on master node playground1 Mon Feb 18 05:22:20 UTC Install system applications Mon Feb 18 05:22:21 UTC Install system application dns-app:0.2.0 Mon Feb 18 05:22:22 UTC Install system application logging-app:5.0.2 Mon Feb 18 05:22:43 UTC Install system application monitoring-app:5.2.2 Mon Feb 18 05:23:31 UTC Install system application tiller-app:5.2.1 Mon Feb 18 05:24:43 UTC Install system application site:5.4.6 Mon Feb 18 05:29:57 UTC Install system application kubernetes:5.4.6 Mon Feb 18 05:29:58 UTC Install user application Mon Feb 18 05:29:59 UTC Install application beyondthekube:0.0.1 Mon Feb 18 05:30:38 UTC Enable elections Mon Feb 18 05:30:43 UTC Operation has completed Mon Feb 18 05:30:44 UTC Joined cluster in 20m31.733348076s [playground2]$
What is actually happening above is that playground1 is launching an installer and listening for other nodes to join. When we issue the gravity join command on playground2 and playground3 those nodes connect to playground1 using the token we passed in.
Post installation goodness
When the installation is done, you can inspect the gravity environment by running sudo gravity status. Enter the planet environment and inspect the kubernetes environment.
$ sudo gravity status
Cluster status: active
Application: beyondthekube, version 0.0.1
Join token: c47d9d1329eab6f624116591315b0841aca60f3ed2c6f2fd1a7a02d65111e2b3
Last completed operation:
* operation_install (33072172-7852-47cc-80cf-e6d2dd10916c)
started: Mon Feb 18 04:14 UTC (10 minutes ago)
completed: Mon Feb 18 04:14 UTC (10 minutes ago)
Cluster: awesomedarwin4747
Masters:
* playground1 (192.168.1.177, node)
Status: healthy
$ sudo gravity enter
___
,o88888
,o8888888'
,:o:o:oooo. ,8O88Pd8888"
,.::.::o:ooooOoOoO. ,oO8O8Pd888'"
,.:.::o:ooOoOoOO8O8OOo.8OOPd8O8O"
, ..:.::o:ooOoOOOO8OOOOo.FdO8O8"
, ..:.::o:ooOoOO8O888O8O,COCOO"
, . ..:.::o:ooOoOOOO8OOOOCOCO"
. ..:.::o:ooOoOoOO8O8OCCCC"o
. ..:.::o:ooooOoCoCCC"o:o
. ..:.::o:o:,cooooCo"oo:o:
` . . ..:.:cocoooo"'o:o:::'
.` . ..::ccccoc"'o:o:o:::'
:.:. ,c:cccc"':.:.:.:.:.'
..:.:"'`::::c:"'..:.:.:.:.:.'
...:.'.:.::::"' . . . . .'
.. . ....:."' ` . . . ''
. . . ...."'
.. . ."' -hrr-
.
playground1:/$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.1.177 Ready <none> 9m18s v1.13.2
playground1:/$ kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 5m41s
In addition to looking at the default namespace, gravity provides an alias kctl which is a quick way to run commands against the kube-system namespace. With that, we can see all of the services that gravity installs by default. This provides us with local docker registries for the cluster, monitoring, tiller – the helm server process.
playground1:/$ type kctl kctl is aliased to `kubectl -nkube-system' playground1:/$ kctl get all NAME READY STATUS RESTARTS AGE pod/gravity-site-79wc6 1/1 Running 0 3m pod/install-hook-8fef86-2w9lq 0/1 Completed 0 2m9s pod/install-telekube-55c2a4-sp4sz 0/1 Completed 0 3m47s pod/log-collector-697d94486-7glht 1/1 Running 0 4m38s pod/log-forwarder-kc222 1/1 Running 0 4m38s pod/logging-app-bootstrap-c07491-wdsb2 0/1 Completed 0 4m45s pod/monitoring-app-install-1d8e1d-7q6wk 0/1 Completed 0 4m37s pod/site-app-post-install-4359d0-vjn2d 0/1 Completed 2 2m57s pod/tiller-app-bootstrap-b443e8-7dx9s 0/1 Completed 0 4m12s pod/tiller-deploy-69c5787759-g9td4 1/1 Running 0 3m49s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/gravity-site LoadBalancer 10.100.197.83 <pending> 3009:32009/TCP 3m service/log-collector ClusterIP 10.100.186.31 <none> 514/UDP,514/TCP,8083/TCP 4m38s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/gravity-site 1 1 1 1 1 gravitational.io/k8s-role=master 3m daemonset.apps/log-forwarder 1 1 1 1 1 <none> 4m38s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/log-collector 1/1 1 1 4m38s deployment.apps/tiller-deploy 1/1 1 1 3m49s NAME DESIRED CURRENT READY AGE replicaset.apps/log-collector-697d94486 1 1 1 4m38s replicaset.apps/tiller-deploy-69c5787759 1 1 1 3m49s NAME COMPLETIONS DURATION AGE job.batch/install-hook-8fef86 1/1 2s 2m9s job.batch/install-telekube-55c2a4 1/1 50s 3m47s job.batch/logging-app-bootstrap-c07491 1/1 7s 4m46s job.batch/monitoring-app-install-1d8e1d 1/1 23s 4m37s job.batch/site-app-post-install-4359d0 1/1 45s 2m57s job.batch/tiller-app-bootstrap-b443e8 1/1 24s 4m13s playground1:/$
Note that in multi-node clusters, there will be 3 instances of each system service:
playground1:/$ kctl get pods NAME READY STATUS RESTARTS AGE gravity-site-c25rk 1/1 Running 2 17m gravity-site-gjfvp 0/1 Running 1 17m gravity-site-vd8b7 0/1 Running 2 17m log-collector-697d94486-l9cx7 1/1 Running 0 20m log-forwarder-8nqj9 1/1 Running 0 20m log-forwarder-cggjs 1/1 Running 0 20m log-forwarder-vpd4d 1/1 Running 0 20m tiller-deploy-69c5787759-v9vz5 1/1 Running 0 18m
Things to watch out for
Verbose logging
During the installation, the installer writes /var/log/telekube-install.log and /var/log/telekube-system.log. If you chose to watch those logs during installation, it is easy to get the impression that the installation is not working:
019-02-17T17:52:10-06:00 DEBU Unsuccessful attempt 1/100: failed to query cluster status from agent, retry in 5s. install/hook.go:56
2019-02-17T17:52:15-06:00 DEBU Unsuccessful attempt 2/100: failed to query cluster status from agent, retry in 5s. install/hook.go:56
2019-02-17T17:52:20-06:00 DEBU Unsuccessful attempt 3/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:25-06:00 DEBU Unsuccessful attempt 4/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:31-06:00 DEBU Unsuccessful attempt 5/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:36-06:00 DEBU Unsuccessful attempt 6/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU [KEYGEN] generated user key for [root] with expiry on (1550483561) 2019-02-18 03:52:41.2294291 -0600 CST m=+36251.083685033 install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU [AUDITLOG] EmitAuditEvent(user.login: map[user:opscenter@gravitational.io method:local]) install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU [TELEPORT] generated certificate for opscenter@gravitational.io install/hook.go:56
2019-02-17T17:52:41-06:00 DEBU Unsuccessful attempt 7/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:52:56-06:00 DEBU Unsuccessful attempt 8/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
2019-02-17T17:53:02-06:00 DEBU Unsuccessful attempt 9/100: not all planets have come up yet: &{unknown []}, retry in 5s. install/hook.go:56
The gravity installation logging is very verbose, and these messages are simply the installation waiting for kubernetes services to come up.
Firewall issues
If the installation is not working for you, attempt it without the firewall enabled. I’m not recommending running live without a firewall running in production, but at least rule it out before considering other issues. In a cloud environment, be sure to check rules built into the cloud provider. On premis, check your network firewalls in addition to the host firewall.
When you have achieved a working installation of gravity without the firewall, add it back in and make sure the recommended ports have been opened.
Conclusion
The Gravity system is a great option for admins that want to entrust the configuration and setup to a third party.
It is a very good option if your objective is to package up your application for a DMZ or to distribute to customers who may not have a good understanding of Kubernetes.
On the less positive side, the installer makes a lot of assumptions about your infrastructure that may be difficult to work around. Further to this, gravity works most seamlessly when you buy into the complete gravitational ecosystem. If you want to run with more esoteric settings or plugins, or you want to run enormous clusters, then gravity may not be the best fit for you.
In the coming weeks, I will be evaluating other methods of on-premis Kubernetes installation, and will eventually post a comparison of all of them.
Stay tuned for that. If you just got your gravity cluster up and running,