Documentum – dm_DMFilescan fails with invalid method_verb?

Recently in a project, one of the Documentum environments we were working on started showing failure on the execution of the dm_DMFilescan job and only this one. After a couple minutes of investigation, I quickly found the root cause of the issue which wasn’t very important, but I thought I would share it because of the implications it has. In addition to that, it’s also not the first time I’m seeing this kind of issue so let’s dig into it.

The error shown in the job log was the following one:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba/log/000f1234/sysadmin
[dmadmin@cs-0 sysadmin]$
[dmadmin@cs-0 sysadmin]$ cat ../agentexec/job_080f12348000035c
Sun Mar 15 08:50:32 2020 [INFORMATION] [LAUNCHER 25934] Detected while preparing job dm_DMFilescan for execution: Agent Exec connected to server Repo01:  [DM_SESSION_I_SESSION_START]info:  "Session 010f123480032063 started for user dmadmin."

[dmadmin@cs-0 sysadmin]$
[dmadmin@cs-0 sysadmin]$ cat DMFilescanDoc.txt
DMFilescan Report For DocBase Repo01 As Of 3/15/2020 09:50:37

Remove orphan content older than 168 hours.
  Generated DMFilescan script will be executed...
  The trace level is set to 0...

DMFilescan utility syntax: apply,c,NULL,DO_METHOD,METHOD,S,dmfilescan,ARGUMENTS,S,'-grace_period 168 '
Executing DMFilescan...
Unable to execute DMFilescan method...
[DM_METHOD_E_INVALID_MTHD_VERB]error:  "The dm_method named (dmfilescan) of type dmbasic has invalid method_verb (./dmfilescan)."


Exiting...
Report End  3/15/2020 09:50:37
[dmadmin@cs-0 sysadmin]$

Before explaining what happened, I think it’s necessary to take some time to explain how the execution of the dm_DMFilescan is done. The job “dm_DMFilescan” is a content type job which is used to scan storage areas in order to find files that do not have any relation to database objects. You can use it in report mode only (nothing done) or not for example. There are more details on the OpenText documentation but the thing to note here is that this job will execute the method of the same name: “dm_DMFilescan”. This method is a dmbasic one and it’s actually just an encapsulation. What I mean by that is it will actually execute a common dmbasic for several jobs, the only thing that changes are the parameters of the method. There are four jobs that will use the exact same dmbasic script and you can find the list easily:

[dmadmin@cs-0 sysadmin]$ iapi ${repo} -Udmadmin -Pxxx << EOC
> ?,c,select object_name, method_verb from dm_method where method_verb like '%mthd1.ebs%'
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f123480032086 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> object_name      method_verb
--------------------  ---------------------------------------------------------
dm_ContentWarning     ./dmbasic -f../install/admin/mthd1.ebs -eContentWarning
dm_DMClean            ./dmbasic -f../install/admin/mthd1.ebs -eDMClean
dm_DMFilescan         ./dmbasic -f../install/admin/mthd1.ebs -eDMFilescan
dm_DMArchive          ./dmbasic -f../install/admin/mthd1.ebs -eDMArchive
(4 rows affected)

API> Bye
[dmadmin@cs-0 sysadmin]$

Take a look at these “mthdx.ebs” files if you aren’t familiar with them, it’s always good to know how it actually works. This dmbasic method will “only” prepare and execute another method which is then specific to each of the jobs. For the “dm_DMFilescan” dmbasic method for example, it will execute the “dmfilescan” method. This second method isn’t a dmbasic one, it’s a binary. So why am I saying all that? Well it’s simply to explain where the issue is coming from… When reading the error message from the log file, one might thing at first sight that the issue is with the “dm_DMFilescan” method but actually it’s not. The issue was with the second method and if you read it carefully, you actually have all key items to solve it. As said, the “dmfilescan” method is executing a binary but on the error message above, it is saying that this method is of type dmbasic currently. Comparing the final method for the “dm_DMFilescan” and “dm_DMClean” jobs, you can see the error:

[dmadmin@cs-0 sysadmin]$ iapi ${repo} -Udmadmin -Pxxx << EOC
> ?,c,select object_name, method_type, method_verb from dm_method where object_name in ('dmfilescan','dmclean')
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f12348003209b started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> object_name      method_type    method_verb
--------------------  -------------  --------------
dmclean                              ./dmclean
dmfilescan            dmbasic        ./dmfilescan
(2 rows affected)

API> Bye
[dmadmin@cs-0 sysadmin]$

What happened, in my opinion, is that someone opened the properties of the “dmfilescan” method in Documentum Administrator but instead of using “Cancel” to close it, he clicked on “OK” which then saved the current configuration. Unfortunately, with DA, there are often cases where opening a property page can load values that aren’t currently configured and therefore when you click on “OK”, it will save back a value that wasn’t the initial one… I have seen that quite often on job’s target_server for example when working on HA environment: you have a job configured to run on ANY (‘ ‘) and while opening it on DA, it can show the Primary CS target (‘Repo.Repo@cs-0’ for example). Therefore, while clicking on “OK”, it will change the target_server of this job. I believe this is what happened here for this specific method because by default it’s a binary and therefore the method_type should be ‘ ‘ but in this case, it was changed recently to dmbasic. Changing the method_type back to the correct value and the job is working again:

[dmadmin@cs-0 sysadmin]$ iapi ${repo} -Udmadmin -Pxxx << EOC
> ?,c,update dm_method object set method_type=' ' where object_name='dmfilescan'
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f1234800320a7 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> objects_updated
---------------
              1
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info:  "1 objects were affected by your UPDATE statement."

API> Bye
[dmadmin@cs-0 sysadmin]$
[dmadmin@cs-0 sysadmin]$ cat DMFilescanDoc.txt | grep -v ^#
DMFilescan Report For DocBase Repo01 As Of 3/15/2020 10:03:21

Remove orphan content older than 168 hours.
  Generated DMFilescan script will be executed...
  The trace level is set to 0...

DMFilescan utility syntax: apply,c,NULL,DO_METHOD,METHOD,S,dmfilescan,ARGUMENTS,S,'-grace_period 168 '
Executing DMFilescan...
Shell was successful!

Generated script from the DMFilescan method:
----- Start $DOCUMENTUM/dba/log/000f1234/sysadmin/080f12348000035c.bat output ------------------------

------- End $DOCUMENTUM/dba/log/000f1234/sysadmin/080f12348000035c.bat output ------------------------
Destroying DMFilescan script with ID 090f12348003d932...
Report End  3/15/2020 10:05:27
[dmadmin@cs-0 sysadmin]$

So, take care while using Documentum Administrator because it might surprise you.