Table of Content

minikube setup

How to setup minikube on portabledevops, please refer previous post.

Assume minikube and kubectl install and run properly,

$ which minikube
/C/oldhorse/portableapps/dockertoolbox/minikube
$ which kubectl
/C/oldhorse/portableapps/dockertoolbox/kubectl

cluster

check version

minikube version
kubectl version

$ minikube version
minikube version: v0.24.1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

start cluster

$ minikube start
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready 19h v1.8.0

deploy app

  • self-healing mechanism to address machine failure or maintenance, if node down or deleted, controller will replace it
  • need to specify the container image for your app and number of replicas running

deploy app "kubernetes-bootcamp" using docker image,

$ kubectl run kubernetes-bootcamp --image=docker.io/jocatalin/kubernetes-bootcamp:v1 --port=8080
deployment "kubernetes-bootcamp" created

check pod status,

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 14s

$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubernetes-bootcamp 1 1 1 1 22s

get more details about pod and container,

$ kubectl describe pods
Name: kubernetes-bootcamp-6db74b9f76-jvtkp
Namespace: default
Node: minikube/192.168.99.100
Start Time: Sun, 10 Dec 2017 08:27:50 -0500
Labels: pod-template-hash=2863065932
run=kubernetes-bootcamp
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"kubernetes-bootcamp-6db74b9f76","uid":"eb4a5d41-ddad-11e7-8d2b-0...
Status: Running
IP: 172.17.0.2
Created By: ReplicaSet/kubernetes-bootcamp-6db74b9f76
Controlled By: ReplicaSet/kubernetes-bootcamp-6db74b9f76
Containers:
kubernetes-bootcamp:
Container ID: docker://76d8fb9084d70d27fccd1dd2e43a311ab6c86f75fa91de36e8c1c1ab20e07b5d
Image: docker.io/jocatalin/kubernetes-bootcamp:v1
Image ID: docker-pullable://jocatalin/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
Port: 8080/TCP
State: Running
Started: Sun, 10 Dec 2017 08:28:02 -0500
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8fqgm (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-8fqgm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8fqgm
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18m default-scheduler Successfully assigned kubernetes-bootcamp-6db74b9f76-jvtkp to minikube
Normal SuccessfulMountVolume 18m kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-8fqgm"
Normal Pulling 18m kubelet, minikube pulling image "docker.io/jocatalin/kubernetes-bootcamp:v1"
Normal Pulled 18m kubelet, minikube Successfully pulled image "docker.io/jocatalin/kubernetes-bootcamp:v1"
Normal Created 18m kubelet, minikube Created container
Normal Started 18m kubelet, minikube Started container

explore app

Pods that are running inside Kubernetes are running on a private, isolated network. By default they are visible from other pods and services within the same kubernetes cluster, but not outside that network.

You can open 2nd shell terminal, running a proxy to forward communications into cluster-wide, private network. It can be stopped only by ctrl-C, no printout when it is running.

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

then you can access to APIs through proxy endpoint at http://localhost:8001.

query version from 1st shell terminal,

$ curl http://localhost:8001/version
{
"major": "1",
"minor": "8",
"gitVersion": "v1.8.0",
"gitCommit": "0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4",
"gitTreeState": "clean",
"buildDate": "2017-11-29T22:43:34Z",
"goVersion": "go1.9.1",
"compiler": "gc",
"platform": "linux/amd64"
}

export POD_NAME to env,

$ export POD_NAME=$(kubectl get pods -o go-template --template '{{range.items}}{{.metadata.name}}{{"\n"}}{{end}}')

$ echo Name of the Pod: $POD_NAME
Name of the Pod: kubernetes-bootcamp-6db74b9f76-jvtkp

access to api of pod,

$ curl http://localhost:8001/api/v1/proxy/namespaces/default/pods/$POD_NAME/
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6db74b9f76-jvtkp | v=1

check logs of pod

$ kubectl logs $POD_NAME
Kubernetes Bootcamp App Started At: 2017-12-10T13:28:02.757Z | Running On: kubernetes-bootcamp-6db74b9f76-jvtkp

Running On: kubernetes-bootcamp-6db74b9f76-jvtkp | Total Requests: 1 | App Uptime: 583.754 seconds | Log Time: 2017-12-10T13:37:46.511Z
Running On: kubernetes-bootcamp-6db74b9f76-jvtkp | Total Requests: 2 | App Uptime: 587.405 seconds | Log Time: 2017-12-10T13:37:50.162Z

when pod is running, can directly run command on containers,

$ kubectl exec $POD_NAME env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=kubernetes-bootcamp-6db74b9f76-jvtkp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
NPM_CONFIG_LOGLEVEL=info
NODE_VERSION=6.3.1
HOME=/root

we can go to container,

$ kubectl exec -ti $POD_NAME bash
root@kubernetes-bootcamp-6db74b9f76-jvtkp:/# cat server.js
root@kubernetes-bootcamp-6db74b9f76-jvtkp:/# curl localhost:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6db74b9f76-jvtkp | v=1

expose app

kubernetes uses service to expose app, there are few type of ways:

  • ClusterIp(default), service only reachable from within cluster
  • NodePort, expose service using NAT, make a service accessible from outside the cluster using NodeIp:NodePort
  • Loadbalancer, assign external ip to service
  • ExternalName, exposes the service using an arbitrary name, need v1.7 + or kube-dns

we expose new service called kubernetes-bootcamp on $NodePort,

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 38m

$ kubectl expose deployment/kubernetes-bootcamp --type="NodePort" --port 8080
service "kubernetes-bootcamp" exposed

$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 42m
kubernetes-bootcamp NodePort 10.105.210.176 8080:31251/TCP 4s

check which port opened externally,

$ kubectl describe services/kubernetes-bootcamp
Name: kubernetes-bootcamp
Namespace: default
Labels: run=kubernetes-bootcamp
Annotations:
Selector: run=kubernetes-bootcamp
Type: NodePort
IP: 10.105.210.176
Port: 8080/TCP
TargetPort: 8080/TCP
NodePort: 31251/TCP
Endpoints: 172.17.0.2:8080
Session Affinity: None
External Traffic Policy: Cluster
Events:

export $NODE_PORT to env,

$ export NODE_PORT=$(kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}')
$ echo NODE_PORT=$NODE_PORT
NODE_PORT=31251

test app is exposed outside of cluster, in our case we run minikube on win msys and minikube running in virtualbox, so node ip is vm host ip.

$ minikube ip
192.168.99.100

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6db74b9f76-jvtkp | v=1

label

check the label:

Labels: run=kubernetes-bootcamp

$ kubectl describe deployment
Name: kubernetes-bootcamp
Namespace: default
CreationTimestamp: Sun, 10 Dec 2017 08:27:50 -0500
Labels: run=kubernetes-bootcamp
Annotations: deployment.kubernetes.io/revision=1
Selector: run=kubernetes-bootcamp
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: run=kubernetes-bootcamp
Containers:
kubernetes-bootcamp:
Image: docker.io/jocatalin/kubernetes-bootcamp:v1
Port: 8080/TCP
Environment:
Mounts:
Volumes:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets:
NewReplicaSet: kubernetes-bootcamp-6db74b9f76 (1/1 replicas created)
Events:

use label to list pod,

$ kubectl get pods -l run=kubernetes-bootcamp
NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 1h

same for service,

$ kubectl get services -l run=kubernetes-bootcamp
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-bootcamp NodePort 10.105.210.176 8080:31251/TCP 35m

export POD_NAME to env,

$ export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

$ echo Name of the Pod: $POD_NAME
Name of the Pod: kubernetes-bootcamp-6db74b9f76-jvtkp

change label,

$ kubectl label pod $POD_NAME app=v1
pod "kubernetes-bootcamp-6db74b9f76-jvtkp" labeled

we can see label changed,

$ kubectl describe pods $POD_NAME
Name: kubernetes-bootcamp-6db74b9f76-jvtkp
Namespace: default
Node: minikube/192.168.99.100
Start Time: Sun, 10 Dec 2017 08:27:50 -0500
Labels: app=v1
pod-template-hash=2863065932
run=kubernetes-bootcamp

we can use new label,

$ kubectl get pods -l app=v1
NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 1h

scale app

  • new Pods are created and scheduled to Nodes with available resource
  • will distribute the traffic to multi instances
  • can do rolling updates without downtime
    $ kubectl scale deployments/kubernetes-bootcamp --replicas=4
    deployment "kubernetes-bootcamp" scaled

$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubernetes-bootcamp 4 4 4 4 1h

we have 4 pods running now,

$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kubernetes-bootcamp-6db74b9f76-cv8cj 1/1 Running 0 1m 172.17.0.7 minikube
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 1h 172.17.0.2 minikube
kubernetes-bootcamp-6db74b9f76-lfqmk 1/1 Running 0 1m 172.17.0.4 minikube
kubernetes-bootcamp-6db74b9f76-w4269 1/1 Running 0 1m 172.17.0.5 minikube

let’s see more details about scaled deployments,

$ kubectl describe deployments/kubernetes-bootcamp
Name: kubernetes-bootcamp
Namespace: default
CreationTimestamp: Sun, 10 Dec 2017 08:27:50 -0500
Labels: run=kubernetes-bootcamp
Annotations: deployment.kubernetes.io/revision=1
Selector: run=kubernetes-bootcamp
Replicas: 4 desired | 4 updated | 4 total | 4 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: run=kubernetes-bootcamp
Containers:
kubernetes-bootcamp:
Image: docker.io/jocatalin/kubernetes-bootcamp:v1
Port: 8080/TCP
Environment:
Mounts:
Volumes:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets:
NewReplicaSet: kubernetes-bootcamp-6db74b9f76 (4/4 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3m deployment-controller Scaled up replica set kubernetes-bootcamp-6db74b9f76 to 4

load balance

check exposed ip and port,

$ kubectl describe services/kubernetes-bootcamp
Name: kubernetes-bootcamp
Namespace: default
Labels: run=kubernetes-bootcamp
Annotations:
Selector: run=kubernetes-bootcamp
Type: NodePort
IP: 10.105.210.176
Port: 8080/TCP
TargetPort: 8080/TCP
NodePort: 31251/TCP
Endpoints: 172.17.0.2:8080,172.17.0.4:8080,172.17.0.5:8080 + 1 more...
Session Affinity: None
External Traffic Policy: Cluster
Events:

we need to export NODE_PORT to env,

$ export NODE_PORT=$(kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}')
$ echo NODE_PORT=$NODE_PORT
NODE_PORT=31251

then we can query app from exposed ip and node port.

In my case I run minikube from win7 msys64 so exposed ip is vm ip here,

$ minikube ip
192.168.99.100

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6db74b9f76-jvtkp | v=1
$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6db74b9f76-lfqmk | v=1

you will see pod name changed means load balancing working.

scale down

$ kubectl scale deployments/kubernetes-bootcamp --replicas=2
deployment "kubernetes-bootcamp" scaled

$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubernetes-bootcamp 2 2 2 2 1h

$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kubernetes-bootcamp-6db74b9f76-cv8cj 0/1 Terminating 0 7m minikube
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 1h 172.17.0.2 minikube
kubernetes-bootcamp-6db74b9f76-lfqmk 1/1 Running 0 7m 172.17.0.4 minikube
kubernetes-bootcamp-6db74b9f76-w4269 0/1 Terminating 0 7m minikube

$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kubernetes-bootcamp-6db74b9f76-jvtkp 1/1 Running 0 1h 172.17.0.2 minikube
kubernetes-bootcamp-6db74b9f76-lfqmk 1/1 Running 0 8m 172.17.0.4 minikube

rolling updates

  • allow Deployments’ update to take place with zero downtime by incrementally updating Pods instances with new ones
  • promote an application from one environment to another
  • rollback to previous versions
  • CI(Continuous Integration) and CD(Continuous Delivery) of applications with zero downtime

as test, we change image version to v2,

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v2
deployment "kubernetes-bootcamp" image updated

check NodePort,

$ kubectl describe services/kubernetes-bootcamp | grep -i nodeport
Type: NodePort
NodePort: 31251/TCP

verify the exposed ip and port, check pod name and image version,

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6566f8f9db-j4rvx | v=2

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6566f8f9db-lbl22 | v=2

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6566f8f9db-lbl22 | v=2

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6566f8f9db-mhj6m | v=2

$ curl 192.168.99.100:$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6566f8f9db-xqcnf | v=2

check rollout status,

$ kubectl rollout status deployments/kubernetes-bootcamp
deployment "kubernetes-bootcamp" successfully rolled out

rollback

let’s change to image v10 which is not exist in repository,

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v10
deployment "kubernetes-bootcamp" image updated

from below check will find v10 not exist and pods get to wrong status,

kubectl get deployments
kubectl get pods
kubectl describe pods

Failed to pull image "jocatalin/kubernetes-bootcamp:v10": rpc error: code = Unknown desc = Error response from daemon: manifest for jocatalin/kubernetes-bootcamp:v10 not found

we can quickly rollback the update by,

$ kubectl rollout undo deployments/kubernetes-bootcamp
deployment "kubernetes-bootcamp" rolled back

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-5466c44897-x5fgp 0/1 Terminating 0 8m
kubernetes-bootcamp-6566f8f9db-j4rvx 1/1 Running 0 18m
kubernetes-bootcamp-6566f8f9db-lbl22 1/1 Running 0 16m
kubernetes-bootcamp-6566f8f9db-mhj6m 1/1 Running 0 18m
kubernetes-bootcamp-6566f8f9db-nj44l 1/1 Running 0 4s

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubernetes-bootcamp-6566f8f9db-j4rvx 1/1 Running 0 18m
kubernetes-bootcamp-6566f8f9db-lbl22 1/1 Running 0 16m
kubernetes-bootcamp-6566f8f9db-mhj6m 1/1 Running 0 18m
kubernetes-bootcamp-6566f8f9db-nj44l 1/1 Running 0 10s

further reading