Infrastructure at your Service

Daniel Westermann

Rancher, up and running, on EC2 – 2 – Three nodes

In the last post we’ve brought up a RKE Kubernetes cluster on a single node. While that is cool for demonstration purposes or testing, this is nothing for a real life setup. Running the control pane, the etcd nodes and the worker nodes all on one node, is nothing you want to do usually, as you can not guarantee fault tolerance with such a setup. To make the RKE cluster highly available we’ll be adding two additional nodes to the configuration in this post. We’ll end up with three nodes, all running etcd, control pane and workers.

Before you can add the additional nodes, they need to be prepared in very much the same way as the first node: Bring the system to the latest release, install a supported version Docker, create the group and the user, and use the same SSH configuration as on the first node.

$ sudo apt update && sudo apt dist-upgrade -y && sudo systemctl reboot
$ sudo hostnamectl set-hostname rancher2
$ # sudo hostnamectl set-hostname rancher3 # on the third node
$ echo "10.0.1.168 rancher2 rancher2.it.dbi-services.com" >> /etc/hosts
$ # echo "10.0.1.168 rancher3 rancher3.it.dbi-services.com" >> /etc/hosts # on the third node
$ exit
$ sudo curl https://releases.rancher.com/install-docker/19.03.sh | sh
$ sudo bash
$ echo "rancher ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
$ exit
$ sudo systemctl reboot
$ sudo groupadd rancher
$ sudo useradd -g rancher -G docker -m -s /bin/bash rancher
$ sudo passwd rancher
$ sudo su - rancher

Before proceeding make sure that you use the same ssk key for the rancher user on the additional nodes, and that you can login from the first node without being prompted for a password:

$ mkdir .ssh
$ chmod 700 .ssh/
$ echo "-----BEGIN OPENSSH PRIVATE KEY-----
> b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn
> NhAAAAAwEAAQAAAQEAx+iJ2W/nGWytnVxyEeRuUDf8UyX3XOxEv7w+TeNGm3o6votXzsEY
> CclNxZ0KBt72OnPlpCjNgMOZhKC7XIDwEkhldLyMUVV8jdh/03qfJDyVBp4zqpQ2s1yf/b
> SU8cqOrj0gSYmozQdbGybZHmzgj+q9HS5iCAJZ7DUeM43E6kUvHpBJ6a1uP2fIr6+BRd25
> sejcT7kgu50Dv/cVxQ1s0hVydX29kAe0S9IFZUWIlsPCNzPxUGNJxigoC2tAcsXttyeguQ
> dtCzTYPgm3wBOoIOR9pAns8kHfiaajZK36vdF6/nEuaI2pw0IpkAct6aFqWq54utgdG9zv
> a8mqci/94QAAA8i6pMTbuqTE2wAAAAdzc2gtcnNhAAABAQDH6InZb+cZbK2dXHIR5G5QN/
> xTJfdc7ES/vD5N40abejq+i1fOwRgJyU3FnQoG3vY6c+WkKM2Aw5mEoLtcgPASSGV0vIxR
> VXyN2H/Tep8kPJUGnjOqlDazXJ/9tJTxyo6uPSBJiajNB1sbJtkebOCP6r0dLmIIAlnsNR
> 4zjcTqRS8ekEnprW4/Z8ivr4FF3bmx6NxPuSC7nQO/9xXFDWzSFXJ1fb2QB7RL0gVlRYiW
> w8I3M/FQY0nGKCgLa0Byxe23J6C5B20LNNg+CbfAE6gg5H2kCezyQd+JpqNkrfq90Xr+cS
> 5ojanDQimQBy3poWparni62B0b3O9ryapyL/3hAAAAAwEAAQAAAQAueLVK8cOUWnpFoY72
> 79ZhGZKztZi6ZkZZGCaXrqTkUdbEItpnuuWeqMhGjwocrMoqrnSM49tZ+p5+gWrsxyCH74
> J+T7KC2c+ZneGhRNkn8Flob3BtUAUjTv32WXtidgcTJCyUS8cM2o/oUPCaLQ9LBXOvC/BI
> ElvbGEIMFAHZv4+eVcZt1NJG3qlu8CXfxRAe6UPLAJOATRyFoNBycPyYu9Hhpr2vXvzksc
> QJUT177q2nu5U+UbCAatekQSGVqv18RWnECKJP4ntSbUMhg/PoPQALnWC09epD+397Yqwp
> uevR76u7S78q0SnycCvT9EMwpGRjl1e/FTZFejEs9rY9AAAAgQDlMVjYrJ4l5jIrT6GBPE
> 7cBBlMW7P0sr1qFxjQQ05JC4CpgCkvqQDqL4alErQ5KTwk9ZsgJY1N49tQk6Rtxv98BK8K
> x3d0dth/2q690iDG6LzExTFI26fjPK0a22FLouXSexoQtsHqnpefR9HuJWHPAIhBlgjX98
> Ce/A9McrIfOAAAAIEA/jhYGQaiqhZJIo7ggXVT3yj2ysXjPQ9TR+WRb+Ze3esi/bAUfKfK
> 2XtZTALNTFw6+KlorHK5ZgvMdpPLSeAg0htO5g6dLhmVv8VuAItVFQMm/R6AGFc/+EJw9k
> iWaGakJzmzCBRwfyZFh3MeMM9sxq60HyV1VHx/SzQvwKNVOJsAAACBAMlO2QU4r1H8kyzu
> jn5/NgX0lO6iHDhQWKQywrQ3NjYmtYRBhwpT62MpnpHpev6OpkR2xPOJ+9fDG2K1Q3raSP
> jfKaurZlMqmvVeziIhQEXrB3L3vnyq5Jx85oqHv7sh7PYCBD4J6zgL5o66fZOoqdc57GLC
> K+XnWjDZpULuQxUzAAAAD3JhbmNoZXJAcmFuZ2VyMQECAw==
> -----END OPENSSH PRIVATE KEY-----" > .ssh/id_rsa
$ chmod 600 .ssh/id_rsa 
$ echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH6InZb+cZbK2dXHIR5G5QN/xTJfdc7ES/vD5N40abejq+i1fOwRgJyU3FnQoG3vY6c+WkKM2Aw5mEoLtcgPASSGV0vIxRVXyN2H/Tep8kPJUGnjOqlDazXJ/9tJTxyo6uPSBJiajNB1sbJtkebOCP6r0dLmIIAlnsNR4zjcTqRS8ekEnprW4/Z8ivr4FF3bmx6NxPuSC7nQO/9xXFDWzSFXJ1fb2QB7RL0gVlRYiWw8I3M/FQY0nGKCgLa0Byxe23J6C5B20LNNg+CbfAE6gg5H2kCezyQd+JpqNkrfq90Xr+cS5ojanDQimQBy3poWparni62B0b3O9ryapyL/3h [email protected]" >> .ssh/authorized_keys
[email protected]:~$ ssh 10.0.1.253
The authenticity of host '10.0.1.253 (10.0.1.253)' can't be established.
ECDSA key fingerprint is SHA256:/JzK5lFQv6qsM5zi4A+1JYwS5u0Iup3uUUV8927MF50.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.253' (ECDSA) to the list of known hosts.
Linux rancher2 4.19.0-14-cloud-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
[email protected]:~$ logout
Connection to 10.0.1.253 closed.
[email protected]:~$ ssh 10.0.1.73
The authenticity of host '10.0.1.73 (10.0.1.73)' can't be established.
ECDSA key fingerprint is SHA256:oVfRCbqh5PIdTx16+wNmMS8CNnHTnQXsjlpybHmPVlY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.73' (ECDSA) to the list of known hosts.
Linux rancher3 4.19.0-14-cloud-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Once that is confirmed, we need to adjust the RKE cluster configuration file, to include the new nodes. Currently the node section looks like this:

# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 10.0.1.168
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []

We need to add the two additional nodes. As this setup is on EC2, you need to specify the public and the internal IP addresses:

The node section in the yaml file looks like this (I am assuming that you are familiar with security groups and traffic is allowed between the node):

nodes:
- address: 18.195.249.125
  port: "22"
  internal_address: "10.0.1.168"
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: "rancher1"
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
- address: 3.64.193.173
  port: "22"
  internal_address: "10.0.1.253"
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: "rancher2"
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
- address: 18.185.105.131
  port: "22"
  internal_address: "10.0.1.73"
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: "rancher3"
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []

That’s all you need to do. Use “rke up” to apply the changed configuration:

[email protected]:~$ rke up
INFO[0000] Running RKE version: v1.1.15                 
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [dialer] Setup tunnel for host [3.64.193.173] 
INFO[0000] [dialer] Setup tunnel for host [18.185.105.131] 
INFO[0000] [dialer] Setup tunnel for host [18.195.249.125] 
INFO[0000] Checking if container [cluster-state-deployer] is running on host [3.64.193.173], try #1 
INFO[0000] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0000] Starting container [cluster-state-deployer] on host [3.64.193.173], try #1 
INFO[0000] [state] Successfully started [cluster-state-deployer] container on host [3.64.193.173] 
INFO[0000] Checking if container [cluster-state-deployer] is running on host [18.185.105.131], try #1 
INFO[0000] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0000] Starting container [cluster-state-deployer] on host [18.185.105.131], try #1 
INFO[0001] [state] Successfully started [cluster-state-deployer] container on host [18.185.105.131] 
INFO[0001] Checking if container [cluster-state-deployer] is running on host [18.195.249.125], try #1 
INFO[0001] [certificates] Generating CA kubernetes certificates 
INFO[0001] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0002] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates 
INFO[0002] [certificates] Generating Kubernetes API server certificates 
INFO[0003] [certificates] Generating Service account token key 
INFO[0003] [certificates] Generating Kube Controller certificates 
INFO[0003] [certificates] Generating Kube Scheduler certificates 
INFO[0003] [certificates] Generating Kube Proxy certificates 
INFO[0003] [certificates] Generating Node certificate   
INFO[0003] [certificates] Generating admin certificates and kubeconfig 
INFO[0003] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0004] [certificates] Generating kube-etcd-10-0-1-168 certificate and key 
INFO[0004] [certificates] Generating kube-etcd-10-0-1-253 certificate and key 
INFO[0004] [certificates] Generating kube-etcd-10-0-1-73 certificate and key 
INFO[0005] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0005] Building Kubernetes cluster                  
INFO[0005] [dialer] Setup tunnel for host [18.185.105.131] 
INFO[0005] [dialer] Setup tunnel for host [18.195.249.125] 
INFO[0005] [dialer] Setup tunnel for host [3.64.193.173] 
INFO[0005] [network] Deploying port listener containers 
INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0005] Starting container [rke-etcd-port-listener] on host [18.185.105.131], try #1 
INFO[0005] Starting container [rke-etcd-port-listener] on host [18.195.249.125], try #1 
INFO[0005] Starting container [rke-etcd-port-listener] on host [3.64.193.173], try #1 
INFO[0005] [network] Successfully started [rke-etcd-port-listener] container on host [18.185.105.131] 
INFO[0005] [network] Successfully started [rke-etcd-port-listener] container on host [18.195.249.125] 
INFO[0005] [network] Successfully started [rke-etcd-port-listener] container on host [3.64.193.173] 
INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0006] Starting container [rke-cp-port-listener] on host [18.195.249.125], try #1 
INFO[0006] Starting container [rke-cp-port-listener] on host [18.185.105.131], try #1 
INFO[0006] Starting container [rke-cp-port-listener] on host [3.64.193.173], try #1 
INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [3.64.193.173] 
INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [18.185.105.131] 
INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [18.195.249.125] 
INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0006] Starting container [rke-worker-port-listener] on host [18.185.105.131], try #1 
INFO[0006] Starting container [rke-worker-port-listener] on host [18.195.249.125], try #1 
INFO[0006] Starting container [rke-worker-port-listener] on host [3.64.193.173], try #1 
INFO[0006] [network] Successfully started [rke-worker-port-listener] container on host [3.64.193.173] 
INFO[0006] [network] Successfully started [rke-worker-port-listener] container on host [18.185.105.131] 
INFO[0006] [network] Successfully started [rke-worker-port-listener] container on host [18.195.249.125] 
INFO[0006] [network] Port listener containers deployed successfully 
INFO[0006] [network] Running etcd  etcd port checks  
INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0007] Starting container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0007] Starting container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0007] Starting container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0007] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] 
INFO[0007] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] 
INFO[0007] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] 
INFO[0007] Removing container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0007] Removing container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0008] Removing container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0008] [network] Running control plane -> etcd port checks 
INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0008] Starting container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0008] Starting container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0008] Starting container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0008] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] 
INFO[0008] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] 
INFO[0008] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] 
INFO[0008] Removing container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0008] Removing container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0008] Removing container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0008] [network] Running control plane -> worker port checks 
INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0009] Starting container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0009] Starting container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0009] Starting container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] 
INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] 
INFO[0009] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] 
INFO[0009] Removing container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0009] Removing container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0009] Removing container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0009] [network] Running workers -> control plane port checks 
INFO[0009] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0009] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0009] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0009] Starting container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0009] Starting container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0009] Starting container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0009] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] 
INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] 
INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] 
INFO[0009] Removing container [rke-port-checker] on host [3.64.193.173], try #1 
INFO[0009] Removing container [rke-port-checker] on host [18.185.105.131], try #1 
INFO[0009] Removing container [rke-port-checker] on host [18.195.249.125], try #1 
INFO[0009] [network] Checking KubeAPI port Control Plane hosts 
INFO[0009] [network] Removing port listener containers  
INFO[0009] Removing container [rke-etcd-port-listener] on host [18.195.249.125], try #1 
INFO[0009] Removing container [rke-etcd-port-listener] on host [18.185.105.131], try #1 
INFO[0009] Removing container [rke-etcd-port-listener] on host [3.64.193.173], try #1 
INFO[0010] [remove/rke-etcd-port-listener] Successfully removed container on host [3.64.193.173] 
INFO[0010] [remove/rke-etcd-port-listener] Successfully removed container on host [18.185.105.131] 
INFO[0010] [remove/rke-etcd-port-listener] Successfully removed container on host [18.195.249.125] 
INFO[0010] Removing container [rke-cp-port-listener] on host [18.195.249.125], try #1 
INFO[0010] Removing container [rke-cp-port-listener] on host [3.64.193.173], try #1 
INFO[0010] Removing container [rke-cp-port-listener] on host [18.185.105.131], try #1 
INFO[0010] [remove/rke-cp-port-listener] Successfully removed container on host [18.185.105.131] 
INFO[0010] [remove/rke-cp-port-listener] Successfully removed container on host [18.195.249.125] 
INFO[0010] [remove/rke-cp-port-listener] Successfully removed container on host [3.64.193.173] 
INFO[0010] Removing container [rke-worker-port-listener] on host [18.195.249.125], try #1 
INFO[0010] Removing container [rke-worker-port-listener] on host [18.185.105.131], try #1 
INFO[0010] Removing container [rke-worker-port-listener] on host [3.64.193.173], try #1 
INFO[0010] [remove/rke-worker-port-listener] Successfully removed container on host [3.64.193.173] 
INFO[0010] [remove/rke-worker-port-listener] Successfully removed container on host [18.185.105.131] 
INFO[0010] [remove/rke-worker-port-listener] Successfully removed container on host [18.195.249.125] 
INFO[0010] [network] Port listener containers removed successfully 
INFO[0010] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0010] Checking if container [cert-deployer] is running on host [18.195.249.125], try #1 
INFO[0010] Checking if container [cert-deployer] is running on host [3.64.193.173], try #1 
INFO[0010] Checking if container [cert-deployer] is running on host [18.185.105.131], try #1 
INFO[0010] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0010] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0010] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0011] Starting container [cert-deployer] on host [3.64.193.173], try #1 
INFO[0011] Starting container [cert-deployer] on host [18.185.105.131], try #1 
INFO[0011] Starting container [cert-deployer] on host [18.195.249.125], try #1 
INFO[0011] Checking if container [cert-deployer] is running on host [3.64.193.173], try #1 
INFO[0011] Checking if container [cert-deployer] is running on host [18.185.105.131], try #1 
INFO[0011] Checking if container [cert-deployer] is running on host [18.195.249.125], try #1 
INFO[0016] Checking if container [cert-deployer] is running on host [3.64.193.173], try #1 
INFO[0016] Removing container [cert-deployer] on host [3.64.193.173], try #1 
INFO[0016] Checking if container [cert-deployer] is running on host [18.185.105.131], try #1 
INFO[0016] Removing container [cert-deployer] on host [18.185.105.131], try #1 
INFO[0016] Checking if container [cert-deployer] is running on host [18.195.249.125], try #1 
INFO[0016] Removing container [cert-deployer] on host [18.195.249.125], try #1 
INFO[0016] [reconcile] Rebuilding and updating local kube config 
INFO[0016] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0016] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0016] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0016] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0016] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [18.195.249.125] 
INFO[0016] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0016] Starting container [file-deployer] on host [18.195.249.125], try #1 
INFO[0017] Successfully started [file-deployer] container on host [18.195.249.125] 
INFO[0017] Waiting for [file-deployer] container to exit on host [18.195.249.125] 
INFO[0017] Waiting for [file-deployer] container to exit on host [18.195.249.125] 
INFO[0017] Container [file-deployer] is still running on host [18.195.249.125]: stderr: [], stdout: [] 
INFO[0018] Waiting for [file-deployer] container to exit on host [18.195.249.125] 
INFO[0018] Removing container [file-deployer] on host [18.195.249.125], try #1 
INFO[0018] [remove/file-deployer] Successfully removed container on host [18.195.249.125] 
INFO[0018] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [3.64.193.173] 
INFO[0018] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0018] Starting container [file-deployer] on host [3.64.193.173], try #1 
INFO[0018] Successfully started [file-deployer] container on host [3.64.193.173] 
INFO[0018] Waiting for [file-deployer] container to exit on host [3.64.193.173] 
INFO[0018] Waiting for [file-deployer] container to exit on host [3.64.193.173] 
INFO[0018] Container [file-deployer] is still running on host [3.64.193.173]: stderr: [], stdout: [] 
INFO[0019] Waiting for [file-deployer] container to exit on host [3.64.193.173] 
INFO[0019] Removing container [file-deployer] on host [3.64.193.173], try #1 
INFO[0019] [remove/file-deployer] Successfully removed container on host [3.64.193.173] 
INFO[0019] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [18.185.105.131] 
INFO[0019] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0019] Starting container [file-deployer] on host [18.185.105.131], try #1 
INFO[0020] Successfully started [file-deployer] container on host [18.185.105.131] 
INFO[0020] Waiting for [file-deployer] container to exit on host [18.185.105.131] 
INFO[0020] Waiting for [file-deployer] container to exit on host [18.185.105.131] 
INFO[0020] Container [file-deployer] is still running on host [18.185.105.131]: stderr: [], stdout: [] 
INFO[0021] Waiting for [file-deployer] container to exit on host [18.185.105.131] 
INFO[0021] Removing container [file-deployer] on host [18.185.105.131], try #1 
INFO[0021] [remove/file-deployer] Successfully removed container on host [18.185.105.131] 
INFO[0021] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes 
INFO[0021] [reconcile] Reconciling cluster state        
INFO[0021] [reconcile] This is newly generated cluster  
INFO[0021] Pre-pulling kubernetes images                
INFO[0021] Pulling image [rancher/hyperkube:v1.18.16-rancher1] on host [18.185.105.131], try #1 
INFO[0021] Pulling image [rancher/hyperkube:v1.18.16-rancher1] on host [3.64.193.173], try #1 
INFO[0021] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] 
INFO[0047] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] 
INFO[0047] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] 
INFO[0047] Kubernetes images pulled successfully        
INFO[0047] [etcd] Building up etcd plane..              
INFO[0047] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0047] Starting container [etcd-fix-perm] on host [18.195.249.125], try #1 
INFO[0047] Successfully started [etcd-fix-perm] container on host [18.195.249.125] 
INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [18.195.249.125] 
INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [18.195.249.125] 
INFO[0047] Container [etcd-fix-perm] is still running on host [18.195.249.125]: stderr: [], stdout: [] 
INFO[0048] Waiting for [etcd-fix-perm] container to exit on host [18.195.249.125] 
INFO[0048] Removing container [etcd-fix-perm] on host [18.195.249.125], try #1 
INFO[0048] [remove/etcd-fix-perm] Successfully removed container on host [18.195.249.125] 
INFO[0048] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [18.195.249.125] 
INFO[0048] Starting container [etcd] on host [18.195.249.125], try #1 
INFO[0049] [etcd] Successfully started [etcd] container on host [18.195.249.125] 
INFO[0049] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [18.195.249.125] 
INFO[0049] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0049] Starting container [etcd-rolling-snapshots] on host [18.195.249.125], try #1 
INFO[0049] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.195.249.125] 
INFO[0054] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0054] Starting container [rke-bundle-cert] on host [18.195.249.125], try #1 
INFO[0054] [certificates] Successfully started [rke-bundle-cert] container on host [18.195.249.125] 
INFO[0054] Waiting for [rke-bundle-cert] container to exit on host [18.195.249.125] 
INFO[0054] Container [rke-bundle-cert] is still running on host [18.195.249.125]: stderr: [], stdout: [] 
INFO[0055] Waiting for [rke-bundle-cert] container to exit on host [18.195.249.125] 
INFO[0055] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.195.249.125] 
INFO[0055] Removing container [rke-bundle-cert] on host [18.195.249.125], try #1 
INFO[0056] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0056] Starting container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0056] [etcd] Successfully started [rke-log-linker] container on host [18.195.249.125] 
INFO[0056] Removing container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0056] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] 
INFO[0056] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0062] Starting container [etcd-fix-perm] on host [3.64.193.173], try #1 
INFO[0062] Successfully started [etcd-fix-perm] container on host [3.64.193.173] 
INFO[0062] Waiting for [etcd-fix-perm] container to exit on host [3.64.193.173] 
INFO[0062] Waiting for [etcd-fix-perm] container to exit on host [3.64.193.173] 
INFO[0062] Container [etcd-fix-perm] is still running on host [3.64.193.173]: stderr: [], stdout: [] 
INFO[0063] Waiting for [etcd-fix-perm] container to exit on host [3.64.193.173] 
INFO[0063] Removing container [etcd-fix-perm] on host [3.64.193.173], try #1 
INFO[0063] [remove/etcd-fix-perm] Successfully removed container on host [3.64.193.173] 
INFO[0063] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [3.64.193.173], try #1 
INFO[0067] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [3.64.193.173] 
INFO[0067] Starting container [etcd] on host [3.64.193.173], try #1 
INFO[0067] [etcd] Successfully started [etcd] container on host [3.64.193.173] 
INFO[0067] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [3.64.193.173] 
INFO[0067] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0067] Starting container [etcd-rolling-snapshots] on host [3.64.193.173], try #1 
INFO[0067] [etcd] Successfully started [etcd-rolling-snapshots] container on host [3.64.193.173] 
INFO[0072] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0073] Starting container [rke-bundle-cert] on host [3.64.193.173], try #1 
INFO[0073] [certificates] Successfully started [rke-bundle-cert] container on host [3.64.193.173] 
INFO[0073] Waiting for [rke-bundle-cert] container to exit on host [3.64.193.173] 
INFO[0073] Container [rke-bundle-cert] is still running on host [3.64.193.173]: stderr: [], stdout: [] 
INFO[0074] Waiting for [rke-bundle-cert] container to exit on host [3.64.193.173] 
INFO[0074] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [3.64.193.173] 
INFO[0074] Removing container [rke-bundle-cert] on host [3.64.193.173], try #1 
INFO[0074] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0074] Starting container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0074] [etcd] Successfully started [rke-log-linker] container on host [3.64.193.173] 
INFO[0074] Removing container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0075] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] 
INFO[0075] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0078] Starting container [etcd-fix-perm] on host [18.185.105.131], try #1 
INFO[0079] Successfully started [etcd-fix-perm] container on host [18.185.105.131] 
INFO[0079] Waiting for [etcd-fix-perm] container to exit on host [18.185.105.131] 
INFO[0079] Waiting for [etcd-fix-perm] container to exit on host [18.185.105.131] 
INFO[0079] Container [etcd-fix-perm] is still running on host [18.185.105.131]: stderr: [], stdout: [] 
INFO[0080] Waiting for [etcd-fix-perm] container to exit on host [18.185.105.131] 
INFO[0080] Removing container [etcd-fix-perm] on host [18.185.105.131], try #1 
INFO[0080] [remove/etcd-fix-perm] Successfully removed container on host [18.185.105.131] 
INFO[0080] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [18.185.105.131], try #1 
INFO[0084] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [18.185.105.131] 
INFO[0084] Starting container [etcd] on host [18.185.105.131], try #1 
INFO[0084] [etcd] Successfully started [etcd] container on host [18.185.105.131] 
INFO[0084] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [18.185.105.131] 
INFO[0084] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0084] Starting container [etcd-rolling-snapshots] on host [18.185.105.131], try #1 
INFO[0084] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.185.105.131] 
INFO[0089] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0089] Starting container [rke-bundle-cert] on host [18.185.105.131], try #1 
INFO[0090] [certificates] Successfully started [rke-bundle-cert] container on host [18.185.105.131] 
INFO[0090] Waiting for [rke-bundle-cert] container to exit on host [18.185.105.131] 
INFO[0090] Container [rke-bundle-cert] is still running on host [18.185.105.131]: stderr: [], stdout: [] 
INFO[0091] Waiting for [rke-bundle-cert] container to exit on host [18.185.105.131] 
INFO[0091] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.185.105.131] 
INFO[0091] Removing container [rke-bundle-cert] on host [18.185.105.131], try #1 
INFO[0091] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0091] Starting container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0091] [etcd] Successfully started [rke-log-linker] container on host [18.185.105.131] 
INFO[0091] Removing container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0092] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] 
INFO[0092] [etcd] Successfully started etcd plane.. Checking etcd cluster health 
INFO[0092] [etcd] etcd host [18.195.249.125] reported healthy=true 
INFO[0092] [controlplane] Building up Controller Plane.. 
INFO[0092] Checking if container [service-sidekick] is running on host [18.195.249.125], try #1 
INFO[0092] Checking if container [service-sidekick] is running on host [18.185.105.131], try #1 
INFO[0092] Checking if container [service-sidekick] is running on host [3.64.193.173], try #1 
INFO[0092] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0092] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0092] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0092] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] 
INFO[0092] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] 
INFO[0092] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] 
INFO[0092] Starting container [kube-apiserver] on host [18.185.105.131], try #1 
INFO[0092] Starting container [kube-apiserver] on host [3.64.193.173], try #1 
INFO[0092] Starting container [kube-apiserver] on host [18.195.249.125], try #1 
INFO[0092] [controlplane] Successfully started [kube-apiserver] container on host [18.185.105.131] 
INFO[0092] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.185.105.131] 
INFO[0092] [controlplane] Successfully started [kube-apiserver] container on host [18.195.249.125] 
INFO[0092] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.195.249.125] 
INFO[0092] [controlplane] Successfully started [kube-apiserver] container on host [3.64.193.173] 
INFO[0092] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [3.64.193.173] 
INFO[0102] [healthcheck] service [kube-apiserver] on host [18.185.105.131] is healthy 
INFO[0102] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0102] [healthcheck] service [kube-apiserver] on host [3.64.193.173] is healthy 
INFO[0102] [healthcheck] service [kube-apiserver] on host [18.195.249.125] is healthy 
INFO[0102] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0102] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0102] Starting container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0102] Starting container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0102] Starting container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.185.105.131] 
INFO[0103] Removing container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [3.64.193.173] 
INFO[0103] Removing container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.195.249.125] 
INFO[0103] Removing container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] 
INFO[0103] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] 
INFO[0103] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] 
INFO[0103] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] 
INFO[0103] Starting container [kube-controller-manager] on host [18.185.105.131], try #1 
INFO[0103] Starting container [kube-controller-manager] on host [3.64.193.173], try #1 
INFO[0103] Starting container [kube-controller-manager] on host [18.195.249.125], try #1 
INFO[0103] [controlplane] Successfully started [kube-controller-manager] container on host [18.185.105.131] 
INFO[0103] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.185.105.131] 
INFO[0103] [controlplane] Successfully started [kube-controller-manager] container on host [3.64.193.173] 
INFO[0103] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [3.64.193.173] 
INFO[0103] [controlplane] Successfully started [kube-controller-manager] container on host [18.195.249.125] 
INFO[0103] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.195.249.125] 
INFO[0108] [healthcheck] service [kube-controller-manager] on host [18.185.105.131] is healthy 
INFO[0108] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0108] [healthcheck] service [kube-controller-manager] on host [3.64.193.173] is healthy 
INFO[0108] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0108] [healthcheck] service [kube-controller-manager] on host [18.195.249.125] is healthy 
INFO[0108] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0109] Starting container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0109] Starting container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0109] Starting container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0109] [controlplane] Successfully started [rke-log-linker] container on host [18.185.105.131] 
INFO[0109] [controlplane] Successfully started [rke-log-linker] container on host [3.64.193.173] 
INFO[0109] Removing container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0109] Removing container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0109] [controlplane] Successfully started [rke-log-linker] container on host [18.195.249.125] 
INFO[0109] Removing container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0109] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] 
INFO[0109] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] 
INFO[0109] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] 
INFO[0109] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] 
INFO[0109] Starting container [kube-scheduler] on host [3.64.193.173], try #1 
INFO[0109] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] 
INFO[0109] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] 
INFO[0109] Starting container [kube-scheduler] on host [18.185.105.131], try #1 
INFO[0109] Starting container [kube-scheduler] on host [18.195.249.125], try #1 
INFO[0109] [controlplane] Successfully started [kube-scheduler] container on host [3.64.193.173] 
INFO[0109] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [3.64.193.173] 
INFO[0109] [controlplane] Successfully started [kube-scheduler] container on host [18.185.105.131] 
INFO[0109] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.185.105.131] 
INFO[0109] [controlplane] Successfully started [kube-scheduler] container on host [18.195.249.125] 
INFO[0109] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.195.249.125] 
INFO[0115] [healthcheck] service [kube-scheduler] on host [3.64.193.173] is healthy 
INFO[0115] [healthcheck] service [kube-scheduler] on host [18.185.105.131] is healthy 
INFO[0115] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0115] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0115] [healthcheck] service [kube-scheduler] on host [18.195.249.125] is healthy 
INFO[0115] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0115] Starting container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0115] Starting container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0115] Starting container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0115] [controlplane] Successfully started [rke-log-linker] container on host [3.64.193.173] 
INFO[0115] Removing container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0115] [controlplane] Successfully started [rke-log-linker] container on host [18.185.105.131] 
INFO[0115] Removing container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0115] [controlplane] Successfully started [rke-log-linker] container on host [18.195.249.125] 
INFO[0115] Removing container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] 
INFO[0116] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] 
INFO[0116] [controlplane] Successfully started Controller Plane.. 
INFO[0116] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0116] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0116] [authz] Creating system:node ClusterRoleBinding 
INFO[0116] [authz] system:node ClusterRoleBinding created successfully 
INFO[0116] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding 
INFO[0116] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully 
INFO[0116] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0116] [state] Saving full cluster state to Kubernetes 
INFO[0116] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state 
INFO[0116] [worker] Building up Worker Plane..          
INFO[0116] Checking if container [service-sidekick] is running on host [18.185.105.131], try #1 
INFO[0116] Checking if container [service-sidekick] is running on host [18.195.249.125], try #1 
INFO[0116] Checking if container [service-sidekick] is running on host [3.64.193.173], try #1 
INFO[0116] [sidekick] Sidekick container already created on host [3.64.193.173] 
INFO[0116] [sidekick] Sidekick container already created on host [18.185.105.131] 
INFO[0116] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] 
INFO[0116] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] 
INFO[0116] [sidekick] Sidekick container already created on host [18.195.249.125] 
INFO[0116] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] 
INFO[0116] Starting container [kubelet] on host [3.64.193.173], try #1 
INFO[0116] Starting container [kubelet] on host [18.185.105.131], try #1 
INFO[0116] Starting container [kubelet] on host [18.195.249.125], try #1 
INFO[0116] [worker] Successfully started [kubelet] container on host [3.64.193.173] 
INFO[0116] [healthcheck] Start Healthcheck on service [kubelet] on host [3.64.193.173] 
INFO[0116] [worker] Successfully started [kubelet] container on host [18.185.105.131] 
INFO[0116] [healthcheck] Start Healthcheck on service [kubelet] on host [18.185.105.131] 
INFO[0116] [worker] Successfully started [kubelet] container on host [18.195.249.125] 
INFO[0116] [healthcheck] Start Healthcheck on service [kubelet] on host [18.195.249.125] 
INFO[0121] [healthcheck] service [kubelet] on host [3.64.193.173] is healthy 
INFO[0121] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0121] [healthcheck] service [kubelet] on host [18.185.105.131] is healthy 
INFO[0121] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0121] [healthcheck] service [kubelet] on host [18.195.249.125] is healthy 
INFO[0121] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0121] Starting container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0121] Starting container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0121] Starting container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0121] [worker] Successfully started [rke-log-linker] container on host [3.64.193.173] 
INFO[0121] Removing container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0121] [worker] Successfully started [rke-log-linker] container on host [18.185.105.131] 
INFO[0121] Removing container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0122] [worker] Successfully started [rke-log-linker] container on host [18.195.249.125] 
INFO[0122] Removing container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0122] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] 
INFO[0122] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] 
INFO[0122] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] 
INFO[0122] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] 
INFO[0122] Starting container [kube-proxy] on host [3.64.193.173], try #1 
INFO[0122] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] 
INFO[0122] Starting container [kube-proxy] on host [18.185.105.131], try #1 
INFO[0122] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] 
INFO[0122] [worker] Successfully started [kube-proxy] container on host [3.64.193.173] 
INFO[0122] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.64.193.173] 
INFO[0122] Starting container [kube-proxy] on host [18.195.249.125], try #1 
INFO[0122] [worker] Successfully started [kube-proxy] container on host [18.185.105.131] 
INFO[0122] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.185.105.131] 
INFO[0122] [worker] Successfully started [kube-proxy] container on host [18.195.249.125] 
INFO[0122] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.195.249.125] 
INFO[0127] [healthcheck] service [kube-proxy] on host [3.64.193.173] is healthy 
INFO[0127] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0127] [healthcheck] service [kube-proxy] on host [18.185.105.131] is healthy 
INFO[0127] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0127] Starting container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0127] [healthcheck] service [kube-proxy] on host [18.195.249.125] is healthy 
INFO[0127] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0127] Starting container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0127] Starting container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0128] [worker] Successfully started [rke-log-linker] container on host [3.64.193.173] 
INFO[0128] Removing container [rke-log-linker] on host [3.64.193.173], try #1 
INFO[0128] [worker] Successfully started [rke-log-linker] container on host [18.185.105.131] 
INFO[0128] Removing container [rke-log-linker] on host [18.185.105.131], try #1 
INFO[0128] [worker] Successfully started [rke-log-linker] container on host [18.195.249.125] 
INFO[0128] Removing container [rke-log-linker] on host [18.195.249.125], try #1 
INFO[0128] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] 
INFO[0128] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] 
INFO[0128] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] 
INFO[0128] [worker] Successfully started Worker Plane.. 
INFO[0128] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] 
INFO[0128] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] 
INFO[0128] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] 
INFO[0128] Starting container [rke-log-cleaner] on host [18.185.105.131], try #1 
INFO[0128] Starting container [rke-log-cleaner] on host [18.195.249.125], try #1 
INFO[0128] Starting container [rke-log-cleaner] on host [3.64.193.173], try #1 
INFO[0129] [cleanup] Successfully started [rke-log-cleaner] container on host [18.185.105.131] 
INFO[0129] Removing container [rke-log-cleaner] on host [18.185.105.131], try #1 
INFO[0129] [cleanup] Successfully started [rke-log-cleaner] container on host [18.195.249.125] 
INFO[0129] Removing container [rke-log-cleaner] on host [18.195.249.125], try #1 
INFO[0129] [cleanup] Successfully started [rke-log-cleaner] container on host [3.64.193.173] 
INFO[0129] Removing container [rke-log-cleaner] on host [3.64.193.173], try #1 
INFO[0129] [remove/rke-log-cleaner] Successfully removed container on host [18.185.105.131] 
INFO[0129] [remove/rke-log-cleaner] Successfully removed container on host [18.195.249.125] 
INFO[0129] [remove/rke-log-cleaner] Successfully removed container on host [3.64.193.173] 
INFO[0129] [sync] Syncing nodes Labels and Taints       
INFO[0129] [sync] Successfully synced nodes Labels and Taints 
INFO[0129] [network] Setting up network plugin: canal   
INFO[0129] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0129] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0129] [addons] Executing deploy job rke-network-plugin 
INFO[0134] [addons] Setting up coredns                  
INFO[0134] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes 
INFO[0134] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes 
INFO[0134] [addons] Executing deploy job rke-coredns-addon 
INFO[0139] [addons] CoreDNS deployed successfully       
INFO[0139] [dns] DNS provider coredns deployed successfully 
INFO[0139] [addons] Setting up Metrics Server           
INFO[0139] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0139] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0139] [addons] Executing deploy job rke-metrics-addon 
INFO[0144] [addons] Metrics Server deployed successfully 
INFO[0144] [ingress] Setting up nginx ingress controller 
INFO[0144] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0144] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0144] [addons] Executing deploy job rke-ingress-controller 
INFO[0149] [ingress] ingress controller nginx deployed successfully 
INFO[0149] [addons] Setting up user addons              
INFO[0149] [addons] no user addons defined              
INFO[0149] Finished building Kubernetes cluster successfully

If all went fine you should have a three node cluster after a few minutes:

[email protected]:~$ export KUBECONFIG=kube_config_cluster.yml 
[email protected]:~$ kubectl get nodes -o wide
NAME       STATUS   ROLES                      AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION          CONTAINER-RUNTIME
rancher2    Ready    controlplane,etcd,worker   4m51s   v1.18.16   10.0.1.253            Debian GNU/Linux 10 (buster)   4.19.0-14-cloud-amd64   docker://19.3.15
rancher1    Ready    controlplane,etcd,worker   4m51s   v1.18.16   10.0.1.168            Debian GNU/Linux 10 (buster)   4.19.0-14-cloud-amd64   docker://19.3.15
rancher3    Ready    controlplane,etcd,worker   4m51s   v1.18.16   10.0.1.73             Debian GNU/Linux 10 (buster)   4.19.0-14-cloud-amd64   docker://19.3.15

Again, very easy to setup. We still do not have Rancher running, just RKE in a three node configuration. The installation of Rancher itself will be the topic for the next post.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Daniel Westermann
Daniel Westermann

Principal Consultant & Technology Leader Open Infrastructure