Infrastructure at your Service

Before we begin: This is in no way production ready, as the title states. In a production setup you would put the internal registry on a persistent storage, you would probably have more than one master node and you would probably have more than on compute node. Security is not covered at all here. This post is intended to quickly bring up something you can play with, that’s it. In future posts will explore more details of OpenShift. So, lets start.

What I used as a starting point are three t2.xlarge instances:

One of them will be the master, there will be one infrastructure and one compute node. All of them are based on the Red Hat Enterprise Linux 7.5 (HVM) AMI:

Once these three instances are running the most important thing is that you set persistent hostnames (if you do not do this the OpenShift installation will fail):

[[email protected] ec2-user]$ hostnamectl set-hostname --static master.it.dbi-services.com
[[email protected] ec2-user]$ echo "preserve_hostname: true" >> /etc/cloud/cloud.cfg

Of course you need to do that on all three hosts. Once that is done, because I have no DNS in my setup, /etc/hosts should be adjusted on all the machines, in my case:

[[email protected] ec2-user]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.1.167  master master.it.dbi-services.com
10.0.1.110  node1 node1.it.dbi-services.com
10.0.1.13   node2 node2.it.dbi-services.com

As everything is based on RedHat you need to register all the machines:

[[email protected] ec2-user]$ subscription-manager register
Registering to: subscription.rhsm.redhat.com:443/subscription
Username: xxxxxx
Password: 
The system has been registered with ID: xxxxxxx
The registered system name is: master

Once done, refresh and then list the available subscriptions. There should be at least one which is named like “Red Hat OpenShift”. Having identified the “Pool ID” for that one attach it (on all machines):

[[email protected] ec2-user]$ subscription-manager refresh
[[email protected] ec2-user]$ subscription-manager list --available
[[email protected] ec2-user]$ subscription-manager attach --pool=xxxxxxxxxxxxxxxxxxxxxxxxx

Now you are ready to enable the required repositories (on all machines):

[[email protected] ec2-user]$ subscription-manager repos --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
     --enable="rhel-7-server-ose-3.11-rpms" \
     --enable="rhel-7-server-ansible-2.6-rpms"

Repository 'rhel-7-server-rpms' is enabled for this system.
Repository 'rhel-7-server-extras-rpms' is enabled for this system.
Repository 'rhel-7-server-ansible-2.6-rpms' is enabled for this system.
Repository 'rhel-7-server-ose-3.11-rpms' is enabled for this system.

Having the repos enabled the required packages can be installed (on all machines):

[[email protected] ec2-user]$ yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct

Updating all packages to the latest release and rebooting to the potentially new kernel is recommended. As we will be using Docker for this deployment we will install that as well (on all machines):

[[email protected] ec2-user]$ yum install -y docker
[[email protected] ec2-user]$ yum update -y
[[email protected] ec2-user]$ systemctl reboot

Now, that we are up to date and the prerequisites are met we create a new group and a new user. Why that? The complete OpenShift installation is driven by Ansible. You could run all of the installation directly as root, but a better way is to use a dedicated user that has sudo permissions to perform the tasks (on all machines):

[[email protected] ec2-user]$ useradd -g dbi dbi
[[email protected] ec2-user]$ useradd -g dbi dbi

As Ansible needs to login to all the machines you will need to setup password-less ssh connections for the user. I am assuming that you know how to do that. If not, please check here.

Several tasks of the OpenShift Ansible playbooks need to be executed as root so the “dbi” user needs permissions to do that (on all machines):

[[email protected] ec2-user]$ cat /etc/sudoers | grep dbi
dbi	ALL=(ALL)	NOPASSWD: ALL

There is one last preparation step to be executed on the master only: Installing the Ansible playbooks required to bring up OpenShift:

[[email protected] ec2-user]$ yum -y install openshift-ansible

That’s all the preparation required for this playground setup. As all the installation is Ansible based we need an inventory file on the master:

[[email protected] ~]$ id -a
uid=1001(dbi) gid=1001(dbi) groups=1001(dbi),994(dockerroot) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[[email protected] ~]$ pwd
/home/dbi
[[email protected] ~]$ cat inventory 
# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=dbi
# If ansible_ssh_user is not root, ansible_become must be set to true
ansible_become=true
become_method = sudo
openshift_deployment_type=openshift-enterprise
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_users={'admin': '$apr1$4ZbKL26l$3eKL/6AQM8O94lRwTAu611', 'developer': '$apr1$4ZbKL26l$3eKL/6AQM8O94lRwTAu611'}
# Registry settings
oreg_url=registry.redhat.io/openshift3/ose-${component}:${version}
oreg_auth_user=dbiservices2800
oreg_auth_password=eIJAy7LsyA
# disable checks
openshift_disable_check=disk_availability,docker_storage,memory_availability

openshift_master_default_subdomain=apps.it.dbi-services.com

# host group for masters
[masters]
master.it.dbi-services.com

# host group for etcd
[etcd]
master.it.dbi-services.com

# host group for nodes, includes region info
[nodes]
master.it.dbi-services.com openshift_node_group_name='node-config-master'
node1.it.dbi-services.com openshift_node_group_name='node-config-compute'
node2.it.dbi-services.com openshift_node_group_name='node-config-infra'

If you need more details about all the variables and host groups used here, please check the OpenShift documentation.

In any case pleas execute the prerequisites playbook before starting with the installation. When that does not run until the end or does show any “failed” tasks then you need to fix something before proceeding:

[[email protected] ~]$ ansible-playbook -i inventory /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml 

PLAY [Fail openshift_kubelet_name_override for new hosts] **********************************************

TASK [Gathering Facts] *********************************************************************************
ok: [master.it.dbi-services.com]
ok: [node1.it.dbi-services.com]

...

PLAY RECAP *********************************************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0   
master.it.dbi-services.com : ok=80   changed=17   unreachable=0    failed=0   
node1.it.dbi-services.com  : ok=56   changed=16   unreachable=0    failed=0   


INSTALLER STATUS ***************************************************************************************
Initialization  : Complete (0:01:40)

When it is fine, install OpenShift:

[[email protected] ~]$ ansible-playbook -i inventory /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml 

That will take some time but at the end your OpenShift cluster should be up and running:

[[email protected] ~]$ oc login -u system:admin
Logged into "https://master:8443" as "system:admin" using existing credentials.

You have access to the following projects and can switch between them with 'oc project ':

  * default
    kube-public
    kube-service-catalog
    kube-system
    management-infra
    openshift
    openshift-ansible-service-broker
    openshift-console
    openshift-infra
    openshift-logging
    openshift-monitoring
    openshift-node
    openshift-sdn
    openshift-template-service-broker
    openshift-web-console

Using project "default".

[[email protected] ~]$ oc get nodes 
NAME                         STATUS    ROLES     AGE       VERSION
master.it.dbi-services.com   Ready     master    1h        v1.11.0+d4cacc0
node1.it.dbi-services.com    Ready     compute   1h        v1.11.0+d4cacc0
node2.it.dbi-services.com    Ready     infra     1h        v1.11.0+d4cacc0

As expected there is one master, one infratructure and one compute node. All the pods in the default namespace should be running fine:

[[email protected] ~]$ oc get pods -n default
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-lmjzs    1/1       Running   0          1h
registry-console-1-n4z5j   1/1       Running   0          1h
router-1-5wl27             1/1       Running   0          1h

All the default Image Streams are there as well:

[[email protected] ~]$ oc get is -n openshift
NAME                                           DOCKER REPO                                                                               TAGS                          UPDATED
apicurito-ui                                   docker-registry.default.svc:5000/openshift/apicurito-ui                                   1.2                           2 hours ago
dotnet                                         docker-registry.default.svc:5000/openshift/dotnet                                         latest,1.0,1.1 + 3 more...    2 hours ago
dotnet-runtime                                 docker-registry.default.svc:5000/openshift/dotnet-runtime                                 2.2,latest,2.0 + 1 more...    2 hours ago
eap-cd-openshift                               docker-registry.default.svc:5000/openshift/eap-cd-openshift                               14.0,15.0,13 + 6 more...      2 hours ago
fis-java-openshift                             docker-registry.default.svc:5000/openshift/fis-java-openshift                             1.0,2.0                       2 hours ago
fis-karaf-openshift                            docker-registry.default.svc:5000/openshift/fis-karaf-openshift                            1.0,2.0                       2 hours ago
fuse-apicurito-generator                       docker-registry.default.svc:5000/openshift/fuse-apicurito-generator                       1.2                           2 hours ago
fuse7-console                                  docker-registry.default.svc:5000/openshift/fuse7-console                                  1.0,1.1,1.2                   2 hours ago
fuse7-eap-openshift                            docker-registry.default.svc:5000/openshift/fuse7-eap-openshift                            1.0,1.1,1.2                   2 hours ago
fuse7-java-openshift                           docker-registry.default.svc:5000/openshift/fuse7-java-openshift                           1.0,1.1,1.2                   2 hours ago
fuse7-karaf-openshift                          docker-registry.default.svc:5000/openshift/fuse7-karaf-openshift                          1.0,1.1,1.2                   2 hours ago
httpd                                          docker-registry.default.svc:5000/openshift/httpd                                          2.4,latest                    2 hours ago
java                                           docker-registry.default.svc:5000/openshift/java                                           8,latest                      2 hours ago
jboss-amq-62                                   docker-registry.default.svc:5000/openshift/jboss-amq-62                                   1.3,1.4,1.5 + 4 more...       2 hours ago
jboss-amq-63                                   docker-registry.default.svc:5000/openshift/jboss-amq-63                                   1.0,1.1,1.2 + 1 more...       2 hours ago
jboss-datagrid73-openshift                     docker-registry.default.svc:5000/openshift/jboss-datagrid73-openshift                     1.0                           
jboss-datavirt63-driver-openshift              docker-registry.default.svc:5000/openshift/jboss-datavirt63-driver-openshift              1.0,1.1                       2 hours ago
jboss-datavirt63-openshift                     docker-registry.default.svc:5000/openshift/jboss-datavirt63-openshift                     1.0,1.1,1.2 + 2 more...       2 hours ago
jboss-decisionserver62-openshift               docker-registry.default.svc:5000/openshift/jboss-decisionserver62-openshift               1.2                           2 hours ago
jboss-decisionserver63-openshift               docker-registry.default.svc:5000/openshift/jboss-decisionserver63-openshift               1.3,1.4                       2 hours ago
jboss-decisionserver64-openshift               docker-registry.default.svc:5000/openshift/jboss-decisionserver64-openshift               1.0,1.1,1.2 + 1 more...       2 hours ago
jboss-eap64-openshift                          docker-registry.default.svc:5000/openshift/jboss-eap64-openshift                          1.7,1.3,1.4 + 6 more...       2 hours ago
jboss-eap70-openshift                          docker-registry.default.svc:5000/openshift/jboss-eap70-openshift                          1.5,1.6,1.7 + 2 more...       2 hours ago
jboss-eap71-openshift                          docker-registry.default.svc:5000/openshift/jboss-eap71-openshift                          1.1,1.2,1.3 + 1 more...       2 hours ago
jboss-eap72-openshift                          docker-registry.default.svc:5000/openshift/jboss-eap72-openshift                          1.0,latest                    2 hours ago
jboss-fuse70-console                           docker-registry.default.svc:5000/openshift/jboss-fuse70-console                           1.0                           2 hours ago
jboss-fuse70-eap-openshift                     docker-registry.default.svc:5000/openshift/jboss-fuse70-eap-openshift                     1.0                           
jboss-fuse70-java-openshift                    docker-registry.default.svc:5000/openshift/jboss-fuse70-java-openshift                    1.0                           2 hours ago
jboss-fuse70-karaf-openshift                   docker-registry.default.svc:5000/openshift/jboss-fuse70-karaf-openshift                   1.0                           2 hours ago
jboss-processserver63-openshift                docker-registry.default.svc:5000/openshift/jboss-processserver63-openshift                1.3,1.4                       2 hours ago
jboss-processserver64-openshift                docker-registry.default.svc:5000/openshift/jboss-processserver64-openshift                1.2,1.3,1.0 + 1 more...       2 hours ago
jboss-webserver30-tomcat7-openshift            docker-registry.default.svc:5000/openshift/jboss-webserver30-tomcat7-openshift            1.1,1.2,1.3                   2 hours ago
jboss-webserver30-tomcat8-openshift            docker-registry.default.svc:5000/openshift/jboss-webserver30-tomcat8-openshift            1.2,1.3,1.1                   2 hours ago
jboss-webserver31-tomcat7-openshift            docker-registry.default.svc:5000/openshift/jboss-webserver31-tomcat7-openshift            1.0,1.1,1.2                   2 hours ago
jboss-webserver31-tomcat8-openshift            docker-registry.default.svc:5000/openshift/jboss-webserver31-tomcat8-openshift            1.0,1.1,1.2                   2 hours ago
jenkins                                        docker-registry.default.svc:5000/openshift/jenkins                                        2,latest,1                    2 hours ago
mariadb                                        docker-registry.default.svc:5000/openshift/mariadb                                        10.1,10.2,latest              2 hours ago
mongodb                                        docker-registry.default.svc:5000/openshift/mongodb                                        2.4,3.2,3.6 + 3 more...       2 hours ago
mysql                                          docker-registry.default.svc:5000/openshift/mysql                                          5.7,latest,5.6 + 1 more...    2 hours ago
nginx                                          docker-registry.default.svc:5000/openshift/nginx                                          1.8,latest,1.10 + 1 more...   2 hours ago
nodejs                                         docker-registry.default.svc:5000/openshift/nodejs                                         8-RHOAR,0.10,6 + 3 more...    2 hours ago
perl                                           docker-registry.default.svc:5000/openshift/perl                                           5.20,5.24,5.16 + 1 more...    2 hours ago
php                                            docker-registry.default.svc:5000/openshift/php                                            5.6,5.5,7.0 + 1 more...       2 hours ago
postgresql                                     docker-registry.default.svc:5000/openshift/postgresql                                     latest,10,9.2 + 3 more...     2 hours ago
python                                         docker-registry.default.svc:5000/openshift/python                                         2.7,3.3,3.4 + 3 more...       2 hours ago
redhat-openjdk18-openshift                     docker-registry.default.svc:5000/openshift/redhat-openjdk18-openshift                     1.0,1.1,1.2 + 2 more...       2 hours ago
redhat-sso70-openshift                         docker-registry.default.svc:5000/openshift/redhat-sso70-openshift                         1.3,1.4                       2 hours ago
redhat-sso71-openshift                         docker-registry.default.svc:5000/openshift/redhat-sso71-openshift                         1.1,1.2,1.3 + 1 more...       2 hours ago
redhat-sso72-openshift                         docker-registry.default.svc:5000/openshift/redhat-sso72-openshift                         1.0,1.1,1.2                   2 hours ago
redis                                          docker-registry.default.svc:5000/openshift/redis                                          3.2,latest                    2 hours ago
rhdm70-decisioncentral-openshift               docker-registry.default.svc:5000/openshift/rhdm70-decisioncentral-openshift               1.0,1.1                       2 hours ago
rhdm70-kieserver-openshift                     docker-registry.default.svc:5000/openshift/rhdm70-kieserver-openshift                     1.0,1.1                       2 hours ago
rhdm71-controller-openshift                    docker-registry.default.svc:5000/openshift/rhdm71-controller-openshift                    1.0,1.1                       2 hours ago
rhdm71-decisioncentral-indexing-openshift      docker-registry.default.svc:5000/openshift/rhdm71-decisioncentral-indexing-openshift      1.0,1.1                       2 hours ago
rhdm71-decisioncentral-openshift               docker-registry.default.svc:5000/openshift/rhdm71-decisioncentral-openshift               1.1,1.0                       2 hours ago
rhdm71-kieserver-openshift                     docker-registry.default.svc:5000/openshift/rhdm71-kieserver-openshift                     1.0,1.1                       2 hours ago
rhdm71-optaweb-employee-rostering-openshift    docker-registry.default.svc:5000/openshift/rhdm71-optaweb-employee-rostering-openshift    1.0,1.1                       2 hours ago
rhdm72-controller-openshift                    docker-registry.default.svc:5000/openshift/rhdm72-controller-openshift                    1.0,1.1                       2 hours ago
rhdm72-decisioncentral-indexing-openshift      docker-registry.default.svc:5000/openshift/rhdm72-decisioncentral-indexing-openshift      1.0,1.1                       2 hours ago
rhdm72-decisioncentral-openshift               docker-registry.default.svc:5000/openshift/rhdm72-decisioncentral-openshift               1.1,1.0                       2 hours ago
rhdm72-kieserver-openshift                     docker-registry.default.svc:5000/openshift/rhdm72-kieserver-openshift                     1.0,1.1                       2 hours ago
rhdm72-optaweb-employee-rostering-openshift    docker-registry.default.svc:5000/openshift/rhdm72-optaweb-employee-rostering-openshift    1.0,1.1                       2 hours ago
rhpam70-businesscentral-indexing-openshift     docker-registry.default.svc:5000/openshift/rhpam70-businesscentral-indexing-openshift     1.0,1.1,1.2                   2 hours ago
rhpam70-businesscentral-monitoring-openshift   docker-registry.default.svc:5000/openshift/rhpam70-businesscentral-monitoring-openshift   1.1,1.2,1.0                   2 hours ago
rhpam70-businesscentral-openshift              docker-registry.default.svc:5000/openshift/rhpam70-businesscentral-openshift              1.0,1.1,1.2                   2 hours ago
rhpam70-controller-openshift                   docker-registry.default.svc:5000/openshift/rhpam70-controller-openshift                   1.0,1.1,1.2                   2 hours ago
rhpam70-kieserver-openshift                    docker-registry.default.svc:5000/openshift/rhpam70-kieserver-openshift                    1.0,1.1,1.2                   2 hours ago
rhpam70-smartrouter-openshift                  docker-registry.default.svc:5000/openshift/rhpam70-smartrouter-openshift                  1.0,1.1,1.2                   2 hours ago
rhpam71-businesscentral-indexing-openshift     docker-registry.default.svc:5000/openshift/rhpam71-businesscentral-indexing-openshift     1.0,1.1                       2 hours ago
rhpam71-businesscentral-monitoring-openshift   docker-registry.default.svc:5000/openshift/rhpam71-businesscentral-monitoring-openshift   1.0,1.1                       2 hours ago
rhpam71-businesscentral-openshift              docker-registry.default.svc:5000/openshift/rhpam71-businesscentral-openshift              1.0,1.1                       2 hours ago
rhpam71-controller-openshift                   docker-registry.default.svc:5000/openshift/rhpam71-controller-openshift                   1.0,1.1                       2 hours ago
rhpam71-kieserver-openshift                    docker-registry.default.svc:5000/openshift/rhpam71-kieserver-openshift                    1.0,1.1                       2 hours ago
rhpam71-smartrouter-openshift                  docker-registry.default.svc:5000/openshift/rhpam71-smartrouter-openshift                  1.0,1.1                       2 hours ago
rhpam72-businesscentral-indexing-openshift     docker-registry.default.svc:5000/openshift/rhpam72-businesscentral-indexing-openshift     1.1,1.0                       2 hours ago
rhpam72-businesscentral-monitoring-openshift   docker-registry.default.svc:5000/openshift/rhpam72-businesscentral-monitoring-openshift   1.0,1.1                       2 hours ago
rhpam72-businesscentral-openshift              docker-registry.default.svc:5000/openshift/rhpam72-businesscentral-openshift              1.0,1.1                       2 hours ago
rhpam72-controller-openshift                   docker-registry.default.svc:5000/openshift/rhpam72-controller-openshift                   1.0,1.1                       2 hours ago
rhpam72-kieserver-openshift                    docker-registry.default.svc:5000/openshift/rhpam72-kieserver-openshift                    1.0,1.1                       2 hours ago
rhpam72-smartrouter-openshift                  docker-registry.default.svc:5000/openshift/rhpam72-smartrouter-openshift                  1.0,1.1                       2 hours ago
ruby                                           docker-registry.default.svc:5000/openshift/ruby                                           2.2,2.3,2.4 + 3 more...       2 hours ago

Happy playing …

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Daniel Westermann
Daniel Westermann

Principal Consultant & Technology Leader Open Infrastructure