One of the most important and basic concept in Kubernetes is the Service exposition. How about exposing a Kubernetes Service deployed inside a cluster to outside traffic?
Let’s have a look on the different approaches to expose a Kubernetes Service outside the cluster and particularly the role of the Ingress Controller.
NodePort, Load Balancers, and Ingress Controllers
In the Kubernetes world, there are three general approaches to exposing your application.
But we need to consider the following points when using NodePort:
- Only possible to have 1 service per port
- Limited Ports Range 30000 to 32767
- In case of node IP address change, you need to deal with that
Using a LoadBalancer service type automatically deploys an external load balancer. This external load balancer is associated with a specific IP address and routes external traffic to a Kubernetes service in your cluster.
The advantage of using LoadBalancer service type is the direct exposition of your services, without routing, filtering, etc. This means you can send almost any kind of traffic to it, like HTTP, TCP, UDP, etc. The downside is that each service you expose with a LoadBalancer will get its own IP address, and you have to pay for a LoadBalancer per exposed service.
Kubernetes supports a high level abstraction called Ingress, which allows simple host or URL based HTTP(S) routing. An ingress controller is responsible for reading the Ingress Resource information and processing that data accordingly. Unlike above service, Ingress is not a type of service. It’s a smart-router acting as the entry point to the cluster for external communication.
Ingress is the most powerful way to expose your services and became a standard over time. Ingress is an external component and multiple Ingress Controller exists in the market. In the rest of this blog we will see how Ingress controllers work, how to deploy them, but before we have to take into account a few points.
Cloud Environment vs Bare-Metal
In traditional cloud environments, network load balancers are available on-demand to become the single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster.
Basically this is a lack for bare-metal clusters. Therefore few recommended approaches exists to bypass this lack (see here: https://kubernetes.github.io/ingress-nginx/images/baremetal)
Nginx Ingress Controller combined with MetalLB
In this section we will describe how to use Nginx as an Ingress Controller for our cluster combined with MetalLB which will act as a network load-balancer for all incoming communications.
The official documentation can be found here: https://metallb.universe.tf/
➜ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml namespace/metallb-system created ➜ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml podsecuritypolicy.policy/controller created podsecuritypolicy.policy/speaker created serviceaccount/controller created serviceaccount/speaker created clusterrole.rbac.authorization.k8s.io/metallb-system:controller created clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created role.rbac.authorization.k8s.io/config-watcher created role.rbac.authorization.k8s.io/pod-lister created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created rolebinding.rbac.authorization.k8s.io/config-watcher created rolebinding.rbac.authorization.k8s.io/pod-lister created daemonset.apps/speaker created deployment.apps/controller created ➜ kubectl get all -n metal
NGINX Ingress Installation
All installation details can be found here: https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
Let’s start by cloning the GitHub repository:
➜ git clone https://github.com/nginxinc/kubernetes-ingress/ ➜ cd kubernetes-ingress/deployments ➜ git checkout v1.7.2
Create the namespace and the service account
➜ kubectl create -f common/ns-and-sa.yaml namespace/nginx-ingress created serviceaccount/nginx-ingress created
Create the cluster role and the cluster role binding:
➜ kubectl create -f rbac/rbac.yaml clusterrole.rbac.authorization.k8s.io/nginx-ingress created clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress created
Create the secret with a TLS certificate for the default server (only for testing purpose):
➜ kubectl create -f common/default-server-secret.yaml
Create the default config map:
➜ kubectl create -f common/nginx-config.yaml configmap/nginx-config created
Create the CRDs for VirtualServer and VirtualServerRoute and TransportServer:
➜ kubectl create -f common/vs-definition.yaml ➜ kubectl create -f common/vsr-definition.yaml ➜ kubectl create -f common/ts-definition.yaml customresourcedefinition.apiextensions.k8s.io/virtualservers.k8s.nginx.org created customresourcedefinition.apiextensions.k8s.io/virtualserverroutes.k8s.nginx.org created customresourcedefinition.apiextensions.k8s.io/transportservers.k8s.nginx.org created
For TCP and UDP loadbalancing, create the CRD for GlobalConfiguration:
➜ kubectl create -f common/gc-definition.yaml customresourcedefinition.apiextensions.k8s.io/globalconfigurations.k8s.nginx.org created ➜ kubectl create -f common/global-configuration.yaml globalconfiguration.k8s.nginx.org/nginx-configuration created
Deploy the Ingress Controller as a Deployment to only deploy it on worker nodes
➜ kubectl create -f deployment/nginx-ingress.yaml
Check the deployment status:
➜ kubectl get deployment -o wide -n nginx-ingress NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-ingress 1/1 1 1 3m51s nginx-ingress nginx/nginx-ingress:1.7.2 app=nginx-ingress
Increase the number of replicas:
➜ kubectl scale --replicas=3 deployment/nginx-ingress -n nginx-ingress deployment.apps/nginx-ingress scaled
Check the pods deployment status now:
➜ kubectl get pods -o wide -n nginx-ingress NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-577f8dff74-69j7w 1/1 Running 0 21m 10.60.1.3 gke-dbi-test-default-pool-2f971eed-0b39 nginx-ingress-577f8dff74-gdgf9 1/1 Running 0 12s 10.60.2.5 gke-dbi-test-default-pool-2f971eed-7clq nginx-ingress-577f8dff74-nfpbd 1/1 Running 0 34m 10.60.2.4 gke-dbi-test-default-pool-2f971eed-7clq
We can observe that 2 ingress controller pods have been deployed on each worker nodes. The deployment is now completed.
External Access to Ingress Controller
To get access from outside the cluster we will create a service in LoadBalancer mode:
➜ kubectl create -f service/loadbalancer.yaml service/nginx-ingress created
Then we can track the status of the service until we get the external IP address:
➜ kubectl get svc -w -n nginx-ingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-ingress LoadBalancer 10.3.247.17 pending 80:32420/TCP,443:30400/TCP 33s nginx-ingress LoadBalancer 10.3.247.17 220.127.116.11 80:32420/TCP,443:30400/TCP 39s
Nginx Web Server deployment
Once the ingress controller is deployed and configured, we can deploy our application and test the external access.
➜vi nginx-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: run: nginx name: nginx-deploy spec: replicas: 1 selector: matchLabels: run: nginx-deploy template: metadata: labels: run: nginx-deploy spec: containers: - image: nginx name: nginx ➜ kubectl create -f nginx-deploy.yaml -n default
Then create a service ClusterIP for our deployment:
➜ kubectl expose deploy/nginx-deploy --port 80 -n default service/nginx-deploy exposed ➜ kubectl get svc -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-deploy ClusterIP 10.3.240.25 80/TCP 4s
The ultime step is to create an ingress rule to route requests coming from nginx.example.com to the service nginx-deploy:
➜ vi ingress-nginx-deploy.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-nginx-deploy spec: rules: - host: nginx.example.com http: paths: - backend: serviceName: nginx-deploy servicePort: 80 ➜ kubectl create -f ingress-nginx-deploy.yaml ingress.networking.k8s.io/ingress-nginx-deploy created ➜ kubectl describe ing ingress-nginx-deploy Name: ingress-nginx-deploy Namespace: default Address: Default backend: default-http-backend:80 (10.0.0.10:8080) Rules: Host Path Backends ---- ---- -------- nginx.example.com nginx-deploy:80 (10.0.1.5:80) Annotations: Events: