K8s - Networking Service Discovery

21 Apr 2023 • kubernetes

How Network Container Work

Let’s assume we have an app which stores data in MySql database. It is not a good idea to incorporate the app and database (MySql) in a single container. Docker container is designed to handle a single service. Eventhough it is possible a single container handling multiple service (app and database), but it is not a good design and not recommended.

In Kubernetes it is also possible to put multiple container in a single Pod. If we do that with the case of app and database, the app access the database as localhost with a specific port exposed by database container.

Eventhough designing multiple container in a single pod is acceptable and make the network simple, but it is not a good design and it’s not recommended.

Having an app and a database in a single Pod would make the Pod much more complicated to manage. If the Pod fails, then we are going to have to find out if it’s failed because of the database container or the app container. Generally, it just makes things more complicated.

The recommended thing is the app and database should have seperated containers and also seperated Pods.

In previous section we know that the functionality of Pods are exposed as Services. Each of the Services has it’s own private IP address. And the IP is just visible inside the Kubernetes cluster. If we want to communicate from the app Pod to the database Pod, we just refer to the IP address of the database Service. The IP address of the Services are allocated dynamically by Kubernetes.

The next time we run the Kubernetes cluster, we may get different IP addresses allocated to the services. The solution is the Kubernetes maintains its own private DNS service. The DNS Service basically is a database containing a set of key value pairs. The keys are just label name of Kubernetes services and the values are the IP addresses of those Services. And Kubernetes takes full responsibility in maintaining this DNS system. The Kubernetes service which does the job is called kube-dns

Namespace

Namespace basically is partioning resources in Kubernetes.

In Kubernetes, if we don’t specify namespace, it will be default namespace.

To get list of namespace we check with command

# long command
kubectl get namespaces

# short command
kubectl get ns

We also can get resources by adding flag --all-namespaces for all namespaces and -n for specific namespace

# get pods in kube-system namespace
kubectl get po -n kube-system

# get all resources in kube-system namespace
kubectl get all -n kube-system

If we are in different namespace, we should specify the namespace

kubectl describe svc kube-dns -n kube-system

Working with Custom Namespace

# creating a namespace
kubectl create ns demo

# run image inside a specific namespace 
kubectl run busybox -t -i --image busybox --namespace demo

Service Discovery

To demonstrate service dicovery, let’s create a Pod and Service (mysql.yaml) for a MySQL database. Then we will try to connect to the database from the previous web app Pod.

apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  containers:
   - name: mysql
     image: mysql:5
     env:
      # Use secret in real implementation
      - name: MYSQL_ROOT_PASSWORD
        value: password
      - name: MYSQL_DATABASE
        value: fleetman
---
kind: Service
apiVersion: v1
metadata:
  name: database
spec:
  selector:
    app: mysql
  ports:
  - port: 3306
  type: ClusterIP


kubectl apply -f mysql.yaml

How does the web app Pod know how to find the dns Service? The answer is Kubernetes will automatically do some management of the containers and it automatically configures the DNS system. We can verify it by viewing the file in the /etc/resolve.conf

# get into the Pod shell
kubectl exec -it webapp-666c69fc7f-hk94d sh

From the above screenshot we can see that the corresponding DNS entry for the database is database.default.svc.cluster.local. In short the general format of fully qualified domain name of service is \<service-name\>.\<namespace-name\>.svc.cluster.local

Let try to connect the MySql Pod from the web app Pod. Before doing that, we will install mysql client in the web app Pod.

The important thing we have established here is that we can find the IP address of a service just by its name. That is called service discovery.

Reference

K8s - Deployment

20 Apr 2023 • kubernetes

About Deployment

In most cases we don’t want to work with Pods directly. As well as ReplicaSet, we might want to work with Deployment instead. A Deployment provides declarative updates for Pods and ReplicaSets.

Basically it’s a more sophisticated form of ReplicaSet. It is a ReplicaSet with an additional feature. With a Deployment, we get automatic rolling updates with zero downtime.

In the previous section of Service, we did a rolling deployment by changing the labels of selector in the service. The rolling updates was zero downtime. But with Deployment we have more elegant way without have to manual change the selector label. We could also do rollback if something goes wrong.

Note:
Do not manage ReplicaSets owned by a Deployment.

Creating Deployment

The yaml structure of Deployment is similar with ReplicaSet. To demonstrate the Deployment creation, let’s delete the previous ReplicaSet and Service.

kubectl delete rs webapp
kubectl delete svc fleetman-webapp

In creating a Deployment we can copy the previous ReplicaSet yaml and modify the kind element. To make it more closer to real condition, we will remove the release in the Pod label and Service selector. Since we should not tightly to any specific version.

The first deployment, we will use release0 tag of the docker image. We also make slightly increase the number of replica to 2.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template: # template for the pod
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: richardchesterwood/k8s-fleetman-webapp-angular:release0

---
apiVersion: v1
kind: Service
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  # This defines which pods are going to be represented by this service
  # The service becomes a network endpoint for either other services
  # or maybe external users to connect to (browser)
  selector:
    app: webapp # can be any selector according to the label defined in pod ex: myapp:webapp
  ports:
  - name: http
    port: 80
    # the nodeport should be greater than 30000
    nodePort: 30080 
  type: NodePort

The next steps are saving the above yaml as deployment.yaml and executing it to create deployment.

kubectl -f apply deployment.yaml

If we notice, a Deployment will create a corresponding a ReplicaSet. Meanwhile, the ReplicaSet will create corresponding Pods.

We can think that a Deployment is an entity in Kubernetes that manages the ReplicaSet for us. So from now we don’t need to work with the ReplicaSet directly, but we just need to work with Deployment instead.

Now, lets check and open in browser http://192.168.59.100:30080.

Rolling Deployment

In this section we will simulate how we do a rolling deployment without needing to fiddle around with lables like we did before.

To demonstrate it, let’s we update the tag of docker image to a new version release0-5.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template: # template for the pod
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: richardchesterwood/k8s-fleetman-webapp-angular:release0-5

---
apiVersion: v1
kind: Service
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  # This defines which pods are going to be represented by this service
  # The service becomes a network endpoint for either other services
  # or maybe external users to connect to (browser)
  selector:
    app: webapp # can be any selector according to the label defined in pod ex: myapp:webapp
  ports:
  - name: http
    port: 80
    # the nodeport should be greater than 30000
    nodePort: 30080 
  type: NodePort

Then apply the updated deployment.yaml and compare the previous entities to the updated ones.

Now, lets check the updated app by opening in browser http://192.168.59.100:30080. The app should be updated.

How the Deployment does the elegant way ilustrated in the following picture.

Managing Rollout

To check the rollout deployment status and history, run the commands.

# model 1
kubectl rollout status deployment webapp

# model 2
kubectl rollout status deploy webapp

kubectl rollout history deployment webapp

Rolling Back

To rollback deployment we can do with the following command

kubectl rollout undo deploy webapp --to-revision=1

When we open the browser, we will get previous version. The revision number will keep increment as history tracking of Kubernates.

Tips

Applying annotation in deployment will fill out the column CHANGE-CAUSE which can inform us the reason on each deployment. Please see here

References:

K8s - ReplicaSet

19 Apr 2023 • kubernetes

About ReplicaSet

A ReplicaSet’s purpose is to maintain number of Pod available at any given time. It is used to guarantee the availability of a specified number of identical Pods.

From the above picture, if the Pod die for any reason, Kubernetes will spring up a new one Pod to ensure the Pod availability.

In short, if want to deploy a Pod in Kubernates cluster, we could wrap it in a ReplicaSet to ensure its availability in the cluster in case the Pod dies for any reason.

Creating ReplicaSet

Let’s wrap the previous Pod in ReplicaSet in the following replica-set.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
      release: "0-5"
  template: # template for the pod
    metadata:
      labels:
        app: webapp
        release: "0-5"
    spec:
      containers:
      - name: webapp
        image: richardchesterwood/k8s-fleetman-webapp-angular:release0-5

---
apiVersion: v1
kind: Service
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  # This defines which pods are going to be represented by this service
  # The service becomes a network endpoint for either other services
  # or maybe external users to connect to (browser)
  selector:
    app: webapp # can be any selector according to the label defined in pod ex: myapp:webapp
    release: "0-5"
  ports:
  - name: http
    port: 80
    # the nodeport should be greater than 30000
    nodePort: 30080 
  type: NodePort

Before applying the above yaml, let’s delete the previous pods to make it clean.

# deleting all pods
kubectl delete pods --all

# creating replica-set
kubectl apply -f replicaset.yaml

Before open the app through browser, we could evaluate the ReplicaSet creation and the Service.

If the ReplicaSet creation as expected, then we can open the app through browser (http://192.168.59.100:30080) to ensure the service links to Pods.

Simulating Pod Crash

To simulate Pod crash, let’s delete the pod. The Kubernetes should recreated the a new similar pod for us.

On the detail checking of replica set shown that the pod recreated when it’s deleted.

kubectl describe replicaset {NAME_REPLICASET}
kubectl describe rs {NAME_REPLICASET}

Deleting Replica Set

If delete ReplicaSet, the corresponding pods will be deleted as well.

kubectl delete rs {NAME_REPLICA_SET}

K8s - Service

18 Apr 2023 • kubernetes

Creating a Service

a Service is a method for exposing a network application that is running as one or more Pods in your cluster. Different from Pod, Service is a long running object. A service has an IP address and has stable fixed port. We can attach a Service to a Pod.

The Pod has a label which is a key value pair. The Service has a selector which is a key value pair to. The Service pointing to the Pod based on the match the key value pair on the selector and lable.

If a Service we create is designed to be access from internal cluster (such as microservice internal communication), we set the type to ClusterIP. On the other hand if we design it to be accessible from outside cluster, set the type to NodePort/.

The port that is allowed in Kubernetes cluster is greater than 30000

Let’s create a service.yaml

apiVersion: v1
kind: Service
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  # This defines which pods are going to be represented by this service
  # The service becomes a network endpoint for either other services
  # or maybe external users to connect to (browser)
  selector:
    app: webapp # # can be any selector according to the label defined in pod ex: myapp:webapp
  ports:
  - name: http
    port: 80
    # the nodeport should be greater than 30000
    nodePort: 30080 
  type: NodePort

kubectl apply -f service.yaml

Once the service created, check the status of serice and get ip of node (in this case we use minikube).

Then open browser: http://192.168.59.100:30080

Updating Version of App

Let’s assume we want to update app to a new release, from richardchesterwood/k8s-fleetman-webapp-angular:release0 to richardchesterwood/k8s-fleetman-webapp-angular:release0-5

In this example we will add additional label called release and a new Pod named webapp-release-0-5 which represent a new release app.

apiVersion: v1
kind: Pod
metadata:
  name: webapp
  labels:
    app: webapp # can be any label ex, myapp: webapp
    release: "0" # should be a string
spec:
  containers:
  - name: webapp
    image: richardchesterwood/k8s-fleetman-webapp-angular:release0

---
apiVersion: v1
kind: Pod
metadata:
  name: webapp-release-0-5
  labels:
    app: webapp # can be any label ex, myapp: webapp
    release: "0-5" # should be a string
spec:
  containers:
  - name: webapp
    image: richardchesterwood/k8s-fleetman-webapp-angular:release0-5

For service, we also add additional selector

apiVersion: v1
kind: Service
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  # This defines which pods are going to be represented by this service
  # The service becomes a network endpoint for either other services
  # or maybe external users to connect to (browser)
  selector:
    app: webapp # # can be any selector according to the label defined in pod ex: myapp:webapp
    release: "0" # if we change this to "0-5" the service will point to app version 0-5. It is very fast without downtime.
  ports:
  - name: http
    port: 80
    # the nodeport should be greater than 30000
    nodePort: 30080 
  type: NodePort

Let’s apply the pod.yaml and service.yaml

kubectl apply -f pod.yaml 
kubectl apply -f service.yaml

Once the second Pod webapp-release-0-5 up, we can change the release value from "0" to "0-5" in the service. The changes in release in Service will pointing it to app version 0-5. It is very fast without downtime.

We can check further in the detail service information

Tips

We can search/list Pod by filtering on the command

kubectl get pod --show-labels -l release=0-5

If we refresh/force reload browser (http://192.168.59.100:30080), it will load the new release of app (version 0-5).

Service Port, TargetPort, and NodePort

There are several different port configurations for Kubernetes services:

  • Port
    The port of this service

  • TargetPort
    The target port on the pod(s) to forward traffic to

  • NodePort
    The port on the node where external traffic will come in on.

Beside may have the 3 kind of ports, Kubernates service may have multiple port as well. Here is an example of service which has multiple port.


kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
      # From inside the cluster, if you hit the my-service:8089 the traffic is routed to 8080 of the container(targetPort)
      # From outside the cluster, if you hit host_ip:30475 the traffic is routed to 8080 of the container(targetPort)
    - name: http
      protocol: TCP
      targetPort: 8080
      port: 8089
      nodePort: 30475

    - name: metrics
      protocol: TCP
      targetPort: 5555
      port: 5555
      nodePort: 31261
     
     # From inside the cluster, if you hit my-service:8443 then it is redirected to 8085 of the container(targetPort)
     # From outside the cluster, if you hit host_ip:30013 then it is redirected to 8085 of the container(targetPort)
    - name: health
      protocol: TCP
      targetPort: 8085
      port: 8443 
      nodePort: 30013

K8s - Pod

17 Apr 2023 • kubernetes

Creating a Pod

Let’s create a simple pod.yaml using a docker image richardchesterwood/k8s-fleetman-position-tracker

apiVersion: v1
kind: Pod
metadata:
  name: webapp
  labels:
    app: webapp # can be any label ex, myapp: webapp
spec:
  containers:
  - name: webapp
    image: richardchesterwood/k8s-fleetman-webapp-angular:release0

To create a Pod, run the following command


# apply will create if it does not exist, and update if it exists
kubectl apply -f pod.yaml

Operating to Pods can be done with the following command


# get all pods
kubectl get pods

# get the namespaces
kubectl get namespaces

# get pods with namespace
kubectl get pods --all-namespaces
kubectl get pods -A

# We can also check all resources at once
kubectl get all

# Deleting Pod. The actual pod.yaml on your local working directory will not be deleted
kubectl delete -f pod.yaml

# delete using pod name
kubectl delete pod {POD_NAME}
kubectl delete po {POD_NAME}

# delete all pods
kubectl delete pods --all

# delete all pods in specfific namespace
kubectl delete pods --all --namespace {NAMESPACE_NAME}

If we got status ContainerCreating for long time, we can consider doing a full reboot of VMs that are running the Kubernetes master node and Kubernetes worker nodes.

If we get an error while creating Pod, we can check the detail with describe command.

kubectl describe pod {POD_NAME}

we can also check the log

kubectl logs --tail=[number] -p {POD_NAME}

Interacting with Container

Pod is not visible from outside the cluster.

But we still able to connect and execute command inside the pod.

kubectl exec {POD_NAME} {COMMAND}
kubectl exec webapp ls

We could also get a shell interraction inside the container in the pod.


# execute shell command
kubectl -it exec {POD_NAME} sh
kubectl -it exec webapp sh

Inside the container we can do

# for the example
wget http://localhost:80

The other port characteristics are having shot lifetime, regularly die, and regularly recreated.

References

  1. https://kubernetes.io/docs/concepts/workloads/pods/
  2. https://signoz.io/blog/kubectl-logs-tail/