How can you connect an external DFC Client to docbrokers and Repositories hosted on Kubernetes Pods? That seems to be a very simple question yet it might prove difficult… Let’s talk about this challenge in this blog and possible solutions/workarounds.
As you all know, Kubernetes is using containers so just like for a basic Docker container, you won’t be able to access it from the outside by default. On Docker, you will need to expose some ports and then you can interact with whatever is running on that port. For Kubernetes, it’s the same principle but it obviously add other layers in addition which makes it even more complicated. Therefore, if you want to be able to connect to a docbroker inside a K8s Pod from the outside of K8s, then you will need to do a few things:
- at the container level, to open the ports 1489/1490 (default ones, you can change them obviously)
- a K8s Service to expose these ports inside K8s
- an Nginx Ingres Controller for which the TCP ports 1489/1490 have been configured for external accesses (or other ports if these are already used for another namespace for example)
- a “Load Balancer” K8s Service (still at the Nginx Ingres Controller level) which exposes these ports using an external IP
Once you have that, you should be able to communicate with a docbroker that is inside a K8s pod. If you want to have a chance to talk to a Repository, then you will also need to do the same thing but for the Repository ports. When you install a repo, you will specify in the /etc/services the ports it should use (just like for the docbroker).
For this example, let’s start simple with the same ports internally and externally:
- DFC Client host: vm
- K8s pod short name (hostname): cs-0
- K8s pod full name (headless service / full hostname): cs-0.cs.dbi-ns01.svc.cluster.local
- K8s pod IP: 1.1.1.100
- K8s pod docbroker port: 1489/1490
- K8s pod Repositories port: gr_repo=49400/49401 // REPO1=49402/49403
- K8s external hostname/lb: k8s-cs-dbi-ns01.domain.com
- K8s external IP: 2.2.2.200
- K8s external docbroker port: 1489/1490
- K8s external Repositories port: gr_repo=49400/49401 // REPO1=49402/49403
Considering the above setup (both the docbroker and Repositories ports configured on K8s), you can already talk to the docbroker properly:
[dmadmin@vm ~]$ grep "dfc.docbroker" dfc.properties dfc.docbroker.host[0]=k8s-cs-dbi-ns01.domain.com dfc.docbroker.port[0]=1489 [dmadmin@vm ~]$ [dmadmin@vm ~]$ nc -v k8s-cs-dbi-ns01.domain.com 1489 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 2.2.2.200:1489. ^C [dmadmin@vm ~]$ [dmadmin@vm ~]$ nc -v k8s-cs-dbi-ns01.domain.com 49402 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 2.2.2.200:49402. ^C [dmadmin@vm ~]$ [dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c ping dmqdocbroker: A DocBroker Query Tool dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185 Using specified port: 1489 Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0170.0234 Linux64). [dmadmin@vm ~]$ [dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c getdocbasemap dmqdocbroker: A DocBroker Query Tool dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185 Using specified port: 1489 ************************************************** ** D O C B R O K E R I N F O ** ************************************************** Docbroker host : cs-0 Docbroker port : 1490 Docbroker network address : INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100 Docbroker version : 16.4.0170.0234 Linux64 ************************************************** ** D O C B A S E I N F O ** ************************************************** -------------------------------------------- Docbase name : gr_repo Docbase id : 1234567 Docbase description : dbi-ns01 dev k8s gr Govern docbase : Federation name : Server version : 16.4.0170.0234 Linux64.Oracle Docbase Roles : Global Registry Docbase Dormancy Status : -------------------------------------------- Docbase name : REPO1 Docbase id : 1234568 Docbase description : dbi-ns01 dev k8s repo1 Govern docbase : Federation name : Server version : 16.4.0170.0234 Linux64.Oracle Docbase Roles : Docbase Dormancy Status : -------------------------------------------- [dmadmin@vm ~]$
So as you can see above, the docbroker does respond properly with the list of Repositories that it is aware of (Repo name, ID, hostname, …) and for that purpose, there is no need for the Repositories’ ports to be opened, only the docbroker is enough. However, as soon as you want to go further and start talking to the Repositories, you will obviously need to open these additional ports as well. Above, I used 49402/49403 for the REPO1 Repository (both internal and external). When trying to login to a target Repository, it will fail:
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO1
^C[dmadmin@vm ~]$
[dmadmin@vm ~]$
Why is that? Well the reason is that to connect to a docbroker, a DFC Client will use the value from the well-known “dfc.properties” file. By reading it, it will know where the docbroker can be found: in our case, it’s “k8s-cs-dbi-ns01.domain.com:1489“. When that is done, the docbroker replies with the list of Repositories known and it will also reply with the “host” that should be used to communicate with the Repositories. That’s because the Repositories might not be on the same host as the docbroker and therefore it needs to provides the information to the DFC Client. However, that “host” is actually an IP! When a Repository register itself with a docbroker, the docbroker records the source IP of the request and it will then forward this IP to the DFC Client that wants to talk to this Repository.
The problem here is that the Repositories are installed on K8s Pods and therefore the IP that the docbroker knows is actually the IP of the K8s Pod… Which is, therefore, not reachable from outside of K8s!
1. IP Forwarding, a solution?
If you want to validate a setup or do some testing, it’s pretty simple on Linux, you can quickly setup an IP Forwarding between the IP of the K8s Pod (which points to nothing) and the IP of the K8s LB Service that you configured previously for the docbroker and Repositories ports. Here is an example:
[dmadmin@vm ~]$ nslookup k8s-cs-dbi-ns01.domain.com
Server: 1.1.1.10
Address: 1.1.1.10#53
k8s-cs-dbi-ns01.domain.com canonical name = k8s-cluster-worker2.domain.com.
Name: k8s-cluster-worker2.domain.com
Address: 2.2.2.200
[dmadmin@vm ~]$
[dmadmin@vm ~]$ external_ip=2.2.2.200
[dmadmin@vm ~]$ ping -c 1 ${external_ip}
PING 2.2.2.200 (2.2.2.200) 56(84) bytes of data.
64 bytes from 2.2.2.200: icmp_seq=1 ttl=63 time=0.980 ms
--- 2.2.2.200 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.980/0.980/0.980/0.000 ms
[dmadmin@vm ~]$
[dmadmin@vm ~]$ internal_ip=1.1.1.100
[dmadmin@vm ~]$ ping -c 1 ${internal_ip}
PING 1.1.1.100 (1.1.1.100) 56(84) bytes of data.
--- 1.1.1.100 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO1
^C[dmadmin@vm ~]$
[dmadmin@vm ~]$
[dmadmin@vm ~]$
[dmadmin@vm ~]$ sudo sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
[dmadmin@vm ~]$
[dmadmin@vm ~]$ sudo iptables -t nat -A OUTPUT -d ${internal_ip} -j DNAT --to-destination ${external_ip}
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info: "Session 0112d6888000152a started for user dmadmin."
Connected to OpenText Documentum Server running Release 16.4.0170.0234 Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@vm ~]$
As you can see above, as soon as you configure an IP Forwarding from the Pod IP to the K8s LB IP, then the Repository connection is successful. Here, I just executed a “quit” command to close the iAPI session but it shows that the session creation is working so you are sure that the end-to-end communication is fine.
Obviously, that is just for testing… Indeed, your Pod IP is going to change in the future (after each restart of the pod for example) which means that the IP Forwarding will be broken at that time. This also requires being setup on the client directly because the DFC Client will try to communicate with a specific IP… But this IP most probably doesn’t point to anything and therefore the only way to make it happen correctly is either setting that up on the client or on the network layer, which is super annoying and isn’t really reliable anyway so this isn’t a solution.
2. Docbroker Translation, a solution?
Several years ago, a feature has been introduced in the docbroker that was initially planned to handle blocking rules on a FireWall: IP and Port Translation. I believe it was introduced for Documentum 6.5 but I might be wrong, it was a long time ago… Since the issue here for K8s is pretty similar to what would happen with a FireWall blocking the IP, we can actually use this feature to help us. Contrary to the IP Forwarding, which is done on the client side, the Translation is done on the server side which is therefore global for all clients. This has an obvious advantage that you can just do it once for all clients (or rather you will need to re-do this configuration at each start of your K8s Pod since the IP will be changed). However, it also has a drawback which is that there is no exception: all communications will be translated, even K8s internal communications… So this might be a problem. There is a KB to describe how it works (KB7701941) and you can also look at the documentation as well. However, the documentation might not be really correct. Indeed, if you look at the CS 7.1 documentation, you will find this definition:
[TRANSLATION]
port=inside_firewall_port=outside_firewall_port
{,inside_firewall_port=outside_firewall_port}
host=inside_firewall_IP=outside_firewall_IP
{,inside_firewall_IP=outside_firewall_IP}
If you look at the CS 16.4 documentation, you will find this definition:
[TRANSLATION]
port=inside_firewall_port=outside_firewall_port
{,inside_firewall_port=outside_firewall_port}
host=outside_firewall_IP=inside_firewall_IP
{,outside_firewall_IP=inside_firewall_IP}
Finally, if you look at the CS 16.7 documentation, you will find yet another definition:
[TRANSLATION]port=["]outside_firewall_port=inside_firewall_port
{,outside_firewall_port=inside_firewall_port}["]
host=["]outside_firewall_ip=inside_firewall_ip
{,outside_firewall_ip=inside_firewall_ip}["]
Three documentations on the same feature, three different definitions :D. In addition to that, there is an example in the documentation which is also wrong, on the three documentations. The real definition is the last one, after fixing the formatting errors that is… So in short, this is what you can do with the docbroker translation:
[TRANSLATION]
port=["]ext_port_1=int_port_1{,ext_port_2=int_port_2}{,ext_port_3=int_port_3}{,...}["]
host=["]ext_ip_1=int_ip_1{,ext_ip_2=int_ip_2}{,ext_ip_3=int_ip_3}{,...}["]
From what I could see, the double quotes aren’t mandatory but you can use them if you want to…
Let’s test all that after removing the IP Forwarding, obviously:
[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1489
**************************************************
** D O C B R O K E R I N F O **
**************************************************
Docbroker host : cs-0
Docbroker port : 1490
Docbroker network address : INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100
Docbroker version : 16.4.0170.0234 Linux64
**************************************************
** D O C B A S E I N F O **
**************************************************
--------------------------------------------
Docbase name : gr_repo
Docbase id : 1234567
Docbase description : dbi-ns01 dev k8s gr
Govern docbase :
Federation name :
Server version : 16.4.0170.0234 Linux64.Oracle
Docbase Roles : Global Registry
Docbase Dormancy Status :
--------------------------------------------
Docbase name : REPO1
Docbase id : 1234568
Docbase description : dbi-ns01 dev k8s repo1
Govern docbase :
Federation name :
Server version : 16.4.0170.0234 Linux64.Oracle
Docbase Roles :
Docbase Dormancy Status :
--------------------------------------------
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO1
^C[dmadmin@vm ~]$
[dmadmin@vm ~]$
On the docbroker side (k8s), let’s configure the translation properly and restart for the new configuration to be applied:
[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba
[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ external_ip=2.2.2.200
[dmadmin@cs-0 dba]$ external_port=1489
[dmadmin@cs-0 dba]$ internal_ip=1.1.1.100
[dmadmin@cs-0 dba]$ internal_port=1489
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ echo "[TRANSLATION]" >> Docbroker.ini
[dmadmin@cs-0 dba]$ echo "port=${external_port}=${internal_port}" >> Docbroker.ini
[dmadmin@cs-0 dba]$ echo "host=${external_ip}=${internal_ip}" >> Docbroker.ini
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[TRANSLATION]
port=1489=1489
host=2.2.2.200=1.1.1.100
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ ./dm_stop_Docbroker; sleep 1; ./dm_launch_Docbroker
./dmshutdown 16.4.0000.0248 Linux64 Copyright (c) 2018. OpenText Corporation.
Shutdown request was processed by Docbroker on host cs-0 (INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100)
Reply status indicates a success: OK
starting connection broker on current host: [cs-0.cs.dbi-ns01.svc.cluster.local]
with connection broker log: [$DOCUMENTUM/dba/log/docbroker.cs-0.cs.dbi-ns01.svc.cluster.local.1489.log]
connection broker pid: 18219
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ head -7 log/docbroker.cs-0.cs.dbi-ns01.svc.cluster.local.1489.log
OpenText Documentum Connection Broker (version 16.4.0170.0234 Linux64)
Copyright (c) 2018. OpenText Corporation
HOST TRANSLATION TABLE:
[1] From(1.1.1.100), to(2.2.2.200)
PORT TRANSLATION TABLE:
[1] From(1489), to(1489)
2019-12-15T10:25:22.307379 [DM_DOCBROKER_I_START]info: "Docbroker has started. Process id: 18219"
[dmadmin@cs-0 dba]$
Once that is done, back on the DFC Client side, trying to connect to the Repository:
[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c getdocbasemap dmqdocbroker: A DocBroker Query Tool dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185 Using specified port: 1489 ************************************************** ** D O C B R O K E R I N F O ** ************************************************** Docbroker host : cs-0 Docbroker port : 1490 Docbroker network address : INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100 Docbroker version : 16.4.0170.0234 Linux64 ************************************************** ** D O C B A S E I N F O ** ************************************************** -------------------------------------------- Docbase name : gr_repo Docbase id : 1234567 Docbase description : dbi-ns01 dev k8s gr Govern docbase : Federation name : Server version : 16.4.0170.0234 Linux64.Oracle Docbase Roles : Global Registry Docbase Dormancy Status : -------------------------------------------- Docbase name : REPO1 Docbase id : 1234568 Docbase description : dbi-ns01 dev k8s repo1 Govern docbase : Federation name : Server version : 16.4.0170.0234 Linux64.Oracle Docbase Roles : Docbase Dormancy Status : -------------------------------------------- [dmadmin@vm ~]$
As you can see above, the dmqdocbroker will still print the Internal IP (1.1.1.100), that’s fine/normal. However the Repository connection should now work:
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info: "Session 0112d6888000175b started for user dmadmin."
Connected to OpenText Documentum Server running Release 16.4.0170.0234 Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@vm ~]$
So as you can see above, using the docbroker translation mechanisms is indeed a solution to be able to connect to a Repository that is inside a K8s pod. There are drawbacks as mentioned above but at least, that’s a valid workaround.
3. Using different ports externally
Above, I have always been using the same ports internally and externally. However, in a real case, you will probably have, in the end, hundreds or even thousands of CS pods. So how do you manage that? Well you saw above that the docbroker translation can be used to translate an external port into an internal port but it’s not just for the docbroker port! You can actually use that for the Repository ports as well.
Let’s say for this example that I have a second namespace (dbi-ns02) with the following:
- DFC Client Host: vm
- K8s pod short name (hostname): cs-0
- K8s pod full name (headless service / full hostname): cs-0.cs.dbi-ns02.svc.cluster.local
- K8s pod IP: 1.1.1.200
- K8s pod docbroker port: 1489/1490
- K8s pod Repositories port: gr_repo=49400/49401 // REPO2=49402/49403
- K8s external hostname/lb: k8s-cs-dbi-ns02.domain.com
- K8s external IP: 2.2.2.200
- K8s external docbroker port: 1491/1492
- K8s external Repositories port: gr_repo=49404/49405 // REPO2=49406/49407
The external IP is still the same because it’s the same K8s Cluster but the external ports are now different. The internal IP is also different because it’s another namespace. So with the default docbroker configuration (no translation), then we have the same issue, obviously, where the iAPI session will hang and never respond because of the IP that doesn’t exist.
So if we try to setup the basic docbroker translation just like what we did above, then on the K8s pod, we will have the following:
[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ ifconfig | grep inet | grep -v 127.0.0.1
inet 1.1.1.200 netmask 255.255.255.255 broadcast 0.0.0.0
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[TRANSLATION]
port=1491=1489
host=2.2.2.200=1.1.1.200
[dmadmin@cs-0 dba]$
With this configuration, if you are trying to connect from an external DFC Client, then it will be able to talk to the docbroker (assuming you have all the K8s stuff in place for redirecting the ports properly) but won’t be able to talk to the Repository:
[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns02.domain.com -p 1491 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1491
**************************************************
** D O C B R O K E R I N F O **
**************************************************
Docbroker host : cs-0
Docbroker port : 1490
Docbroker network address : INET_ADDR: 02 3d4 02020286 cs-0 1.1.1.200
Docbroker version : 16.4.0170.0234 Linux64
**************************************************
** D O C B A S E I N F O **
**************************************************
--------------------------------------------
Docbase name : gr_repo
Docbase id : 1234569
Docbase description : dbi-ns02 dev k8s gr
Govern docbase :
Federation name :
Server version : 16.4.0170.0234 Linux64.Oracle
Docbase Roles : Global Registry
Docbase Dormancy Status :
--------------------------------------------
Docbase name : REPO2
Docbase id : 1234570
Docbase description : dbi-ns02 dev k8s repo2
Govern docbase :
Federation name :
Server version : 16.4.0170.0234 Linux64.Oracle
Docbase Roles :
Docbase Dormancy Status :
--------------------------------------------
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO2 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO2
:Wrong docbase id: (1234570) expecting: (1234568)
Could not connect
[dmadmin@vm ~]$
The reason for that is that I have been talking to the docbroker on the external port 1491, which is therefore the docbroker 1489 of the second namespace (“dbi-ns02“). This docbroker replied to the DFC Client that the Repository is using the port 49402/49403, which is true but only internally… Therefore, my DFC Client has been trying to connect to the Repository REPO2 (from the second namespace) using the port which is actually the one used by the REPO1 (from the first namespace) and therefore there is a mismatch in the Repository ID.
For that purpose, you can update the docbroker translation to include the Repositories ports as well:
[dmadmin@cs-0 dba]$ cat Docbroker.ini [DOCBROKER_CONFIGURATION] secure_connect_mode=dual [TRANSLATION] port=1491=1489,49404=49400,49405=49401,49406=49402,49407=49403 host=2.2.2.200=1.1.1.200 [dmadmin@cs-0 dba]$
With this new docbroker translation configuration, the external DFC Client should be able to communicate properly with the repository:
[dmadmin@vm ~]$ echo "quit" | iapi REPO2 -Udmadmin -P${dm_pw}
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0000.0185
Connecting to Server using docbase REPO2
[DM_SESSION_I_SESSION_START]info: "Session 0112d68a80001403 started for user dmadmin."
Connected to OpenText Documentum Server running Release 16.4.0170.0234 Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@vm ~]$
Alternatively to all that, you might want to take a look at Traefik or Istio which might also help you to configure the correct communications from the outside of K8s to the inside. I had a case opened with the OpenText Support so that they could correct the documentation for all versions.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2023/03/KKE_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ENB_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/CHT_web-min-scaled.jpg)