Infrastructure at your Service

Cesare Cervini

Enhancing idql/iapi with rlwrap (part I)

The rant

I have a confession to make: I’m obsessed with idql/iapi. These tools so central to Documentum administrative tasks are so lame for interactive work that they have probably disgusted more than their share of administrators. The problem is that I have to use them almost daily, and everyday I grumble: there must be a better way. Sure, there were third-party utilities such as Samson and Repoint in the past, and now dqman and DQLBuddy, when they are allowed on the desktop, but my work occurs mostly server-side, on Unix or Linux machines. How come those tools received no update during all these years ? It’s been almost 30 years if senility has not confused my memory. What was the company doing all these years ? They sure deserve some enhancements as the current ones go through the roof on the Suckness scale.

If you ever have been working as an administrator of Documentum installations and repositories, you inevitably have used the idql & iapi command-line utilities. While there are several GUI-based third-party alternatives, e.g. the already mentioned dqman for Windows, idql and iapi are still very useful to automate administrative tasks in repositories. As they don’t need a graphical interface, they are are much lighter weight and can work server-side from a GUI-less terminal. However, the least that can be said about them is that they are extremely crude. They are effective for batch execution but they are so spartan that they even lack the very basic functionalities for a comfortable interactive work. Under Unix for example, idql doesn’t provide command-line history (although they do under Windows). Some functionalities, e.g. spooling its output into a file, are only available at launch time and no longer once it is running (this is true for idql but not for iapi which has the little known $ command). Also, no facilities for formatting the output are available (there is the -W|w command-line argument and command but they just silently truncate the columns), resulting in a totally messed up display when long rows are output. In addition, some frequently used functionalities, e.g. the dump, get and set commands, are only accessible from within the iapi tool (although there is the describe command in idql) and it would be more effective to have these two tools merged into a unique one. Other functionalities quite common in Unix tools, e.g. shelling out an external command without quitting the tool, are missing.

Despite all this, there is hope. One could still develop an enhanced alternative tool using the DFCs from within a JVM-based language (e.g. java, jython, groovy, etc..) or use the dmcl dmapp.h interface from within C or any language that can link to the libdmcl.so library such C, gawk, perl or python. See my previous blog posts here, here and here.
If you are fluent in C/C++, the source code of an ancient version of idql is available in any Documentum installation’s folder $DOCUMENTUM/share/sdk/example/code and could be the basis for a much better enhanced version.

But is there a way to enhance idql/iapi without coding ? As surprising as it sound, there are several tools that can help here, one of these is rlwrap . This tool is really amazing and can address several of the shortcomings of the Documentum command-line tools. Let’s list the peskiest ones.

The shortcomings

Here is what idql and iapi lack the most, in no particular order:

  1. command-line history: recall and execute past commands;
  2. save the history so it is available the next time the tools are launched;
  3. command-line editing using vi or emacs commands, like in the shells;
  4. start vi from the input line;
  5. multi-lines commands or statements;
  6. easy piping so filters can be used to further customize the output, e.g. for grepping and colorization;
  7. really nice to have: macros;
  8. better visualization of output;
  9. execute external programs
  10. name completion

Well, believe it or not, all of them can be alleviated by rlwrap without writing a single line of code !
As its author Hans Lub says it in the readme file:
rlwrap is a ‘readline wrapper’, a small utility that uses the GNU
readline library to allow the editing of keyboard input for any
command. (…) You should consider rlwrap
especially when you need user-defined completion (by way of completion word
lists) and persistent history, or if you want to program ‘special
effects’ using the filter mechanism.
So what rlwrap basically does is use the GNU readline library to read input from the keyboard (and therefore opening up all the features of the library) and pass along the input to the specified program that it invoked. Its syntax is:

$ rlwrap [-options] <command> <args>

Example:

$ rlwrap idql dmtest72 -Udmadmin -Pdmadmin

Now, idql behaves as if it had been linked with the readline library, and the result is utterly impressive.
To install it, just use your distribution’s usual package manager. After that, a man page is available for reference, but it can also be read from here if your prefer a web page.

The miracle

rlwrap has been much publicized to enhance Oracle sql*plus, even though sql*plus is already light-years ahead of idql/iapi in terms of flexibility and power. For example, sql*plus’ built-in small line editor allows to simply and easily edit multi-line statements; column formatting is also easy, it just lacks command-line history under Unix.
All that follows in this article also applies to any rlwrapped executable, not just to sql*plus, idql or iapi. Thus, what you learn here can be leveraged anywhere. Furthermore, as it is possible to specify distinct configuration files separately for each executable, and even define a $RLWRAP_HOME directory, conflicts and namespace pollution are prevented.
rlwrap works beautifully with idql/iapi and enhances them tremendously. Here is the simplest usage:

$ rlwrap idql dmtest72 -Udmadmin -Pdmadmin
...
EMC Documentum idql - Interactive document query interface
...
Connected to Documentum Server running Release 7.2.0000.0155 Linux64.Oracle
1>

Now type a few DQL statements followed by <enter> and use the arrow keys to navigate the history. A dream comes true !
vi or emacs editing mode can be switched to forth and back as follows:
vi —> emacs: <esc> (back to command mode) <esc>-ctrl-e
emacs —> vi: <esc> ctrl-j
Generally, there is little need to switch editing mode once the initial one has been selected from the configuration file.
As rlwrap uses readline, it takes its configuration from the same default file as readline, i.e. ~/.inputrc and adds a few settings of its own. Here is an example of this file:

set convert-meta Off

# editing mode, either emacs or vi;
# I prefer vi mode by default;
#set editing-mode emacs
set editing-mode vi

set history-size -1
set horizontal-scroll-mode Off
set input-meta On
set keymap vi

# this is for rlwrap's rlwrap-call-editor:
"\C-x\C-e": rlwrap-call-editor

If you exit idql now and re-start it via rlwrap, all the commands typed so far are still available; by default, they were saved in ~/.<wrapped_program>_history, here ~/.idql_history.
Now, while on the rlwrap’s command-line, type <esc> ctrl-x ctrl-e if in vi editing mode, ctrl-x ctrl-e if in emacs editing mode; this will start the editor defined in $RLWRAP_EDITOR, $VISUAL or $EDITOR environment variables, typically vi. rlwrap’s internal command rlwrap-call-editor is bound to the key sequence ^X^E, as shown above; it is possible to remap it as you fancy.

So far, through the simplest syntax, rlwrap already covered items 1 to 4, quite remarkable.

In order to use multi-line edit, it is necessary first to define a line separator; we cannot use the <enter> key as it causes the line input to be terminated and sent to the rlwrapped executable so we must choose a rarely used character string; it is a choice that can be done on a per-execution basis so it doesn’t need to be the ultimate, definitive, unique string in the history of mankind. Let’s choose the string “__” for instance and launch rlwrap with the proper option named, you guesses it, ––multi-line:

rlwrap --multi-line='__' idql dmtest72 -Udmadmin -Pdmadmin
...
EMC Documentum idql - Interactive document query interface
...
Connected to Documentum Server running Release 7.2.0000.0155 Linux64.Oracle
1> select __ count(*) __ from __dm_user
-- While still on this line, invoke the editor, <esc> ^x^e:
select
count(*)
from
dm_user
:q
1> select
2> count(*)
3> from
4> dm_user
<up arrow>
5> select __ count(*) __ from __dm_user

Thus, the original line with the line separator strings is still in the command history, unchanged, but when editing it, the editor splits it on multiple lines as specified, and sends those multiple physical lines to idql on exiting. This makes it pleasing to edit a complex statement with an external editor, and also easy to reach for the statement in the history as one line.
And that was item 5 in the list. Before tackling the big issue of output formatting, let’s first see a few nice goodies, a kind of intermission if you will.

Prompts, colors and macros

Prompts

rlwrap also allows to change the prompt of the launched executable (idql here) through the ––substitute-prompt, e.g.:

rlwrap --substitute-prompt="howdy, mate ?>" iapi dmtest72 -Udmadmin -Pdmadmin
...
	EMC Documentum iapi - Interactive API interface
...
Connecting to Server using docbase dmtest72
...
Connected to Documentum Server running Release 7.2.0000.0155  Linux64.Oracle
Session id is s0
howdy, mate ?> quit 
Bye

However, sometimes it does not work flawlessly because idql already displays its own prompt and even completes it with a line counter if a statement spans multiple lines; if rlwrap’s prompt takes over, it overwrites the numbering and the output can be confusing. It is said to nonetheless be useful for programs with no prompt but when I tried with iapi and the -V- command-line option, the substitution prompt was not displayed.

Another oddity of the substitution prompt is that sometimes rlwrap starts displaying the prompt too early while iapi is not ready yet, e.g. it still connecting or displaying the banner, which results in the prompt being pushed up by the incoming text and printed several times, as shown below:

$ rlwrap --substitute-prompt="howdy, mate ?>" iapi dmtest72 -Udmadmin -Pdmadmin
howdy, mate ?>

EMC Documentum iapi - Interactive API interface
(c) Copyright EMC Corp., 1992 - 2016
All rights reserved.
Client Library Release 7.3.0000.0205
...
Connecting to Server using docbase dmtest72
howdy, mate ?>[DM_SESSION_I_SESSION_START]info: "Session 0100c39880002253 started for user dmadmin."
...
howdy, mate ?>Connected to Documentum Server running Release 7.2.0000.0155 Linux64.Oracle
Session id is s0
howdy, mate ?>

The option ––wait-before-prompt exists to prevent this; it tells rlwrap to wait until no output is displayed, presumably meaning that the rlwrapped executable finished outputting stuff (e.g. its banner), displayed its prompt and is waiting now for an user input.
Example of use with a more useful prompt:

$ user_name=dmadmin repo=dmtest72 rlwrap --substitute-prompt="${user_name}@${repo} >" --wait-before-prompt=-1000 iapi $repo -U${user_name} -Pdmadmin
...
Connected to Documentum Server running Release 7.2.0000.0155  Linux64.Oracle
[email protected]> quit
Bye

Here, ––wait-before-prompt is set to 1 seconds, but this does not always work. Worst, this delay slows down every line of the subsequent output so the cure is worse than the disease. Nevertheless, it could still be useful in e.g. non-interactive demos, preparation of slides or documentation, etc., i.e. anything that doesn’t make one fall asleep behind the keyboard. So this rlwrap’s feature may cause some problem sometimes, and sometimes it works as expected. Use it with caution.

In order to add more visibility to the prompt, either the original or the substituted one, let’s colorize it:

Colors are the usual 8 ones written in lowercase for normal intensity, i.e. black, red, green, yellow, blue, magenta, cyan and white, and in uppercase for bright intensity, i.e. BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN and WHITE, a clever notation I think. E.g.:

Macros

Another powerful feature is macros; they are implemented by the filter simple_macro.
To enable macros, rlwrap must be started with the option -z or ––filter=simple_macro. Then, to define a macro name, use the syntax $name((text)). To use the macro name, type $name. Example:

# define a shortcut c for clear:
1> $c((clear))
1> 
...
# use it;
1>
2>
3>
4>
5> $c
1>

The prompt counter has been reset to 1, which shows that the input buffer has been emptied by the clear command.
Unfortunately, there is a small peculiarity:

1>
2>
3>
4> $c((clear))
1>

Defining a macro executes it too. While this looks harmless, although it can produce spurious error messages, it prevents commands such as quit to be abbreviated because idql is exited right after the definition:

1> $q((quit))
Bye
Instead of:
1> $q((quit))
2>
3>
....
> $q
Bye

But completion (see below) provides an even better solution here. Nevertheless, macros are a nice addition. Their usefulness stands in that they allow to shorten long, regularly used commands, just like bash aliases do.
Other examples:

-- count users;
$cu((select count(*) from dm_user ^Mgo))
go
2> count(*)
----------------------
54
(1 row affected)

1> $cu
go
2> count(*)
----------------------
54
(1 row affected)

-- count documents;
cd((select count(*) from dm_document ^Mgo))
go
2> count(*)
----------------------
814
(1 row affected)
1> $cu
go
2> count(*)
----------------------
54
(1 row affected)
1> $cd
go
2> count(*)
----------------------
814
(1 row affected)

It is quite visible here that the macros were executed right after their definition; I guess, we can live with that.

In iapi:

$ rlwrap --no-warnings --filter=simple_macro --ansi-colour-aware --prompt-colour=RED --multi-line='__' iapi dmtest72 -Udmadmin -Pdmadmin
# show dm_server_config;
$sc((retrieve,c,dm_server_config ^M dump,c,l^M))
3d00c39880000102
API> ...
USER ATTRIBUTES

  object_name                     : dmtest72
  title                           : 
  subject                         : 
  authors                       []: 
...
# show a few config's infos;
$sci((retrieve,c,dm_server_config ^M get,c,l,object_name^Mget,c,l,r_server_version^Mget,c,l,r_host_name^M))
...
3d00c39880000102
API> ...
dmtest72
API> ...
7.2.0000.0155  Linux64.Oracle
API> ...
dmtest.cec

^M is entered by typing ctrl-v ctrl-m as usual in vi editing mode (i.e. like in the vi editor).

Macros are saved in the same history file rlwrap uses for program <wrapped_program>: ~/.<wrapped_program>_history, e.g. ~/.idql_history. Thus, they can be shared, recalled and used between idql sessions.

Filters can be chained using the “pipeline” filter and the syntax:

--filter='pipeline filter{:filter}'

E.g.:

--filter='pipeline pipeto:simple_macro'

A common user error that happens when using macros is to forget to include simple_macro in rlwrap’s pipeto argument. No error message will be output when defining a macro or using it, but no result will be produced either, as illustrated below:

rlwrap --no-warnings -f idql-completions.txt --filter='pipeline pipeto' --ansi-colour-aware --prompt-colour=RED
...
	EMC Documentum idql - Interactive document query interface
...
Connecting to Server using docbase dmtest73
...
Connected to Documentum Server running Release 7.3.0000.0214  Linux64.Oracle
1>$qq((gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {printf("%s%s%s\n", !match($0, /^[0-9a-f]{16}/) ? col[l%2] : col[++l%2], $0, reset)}'))
2> clear 
1> select * from dm_user 
 go| $qq

This silent behavior can cause some frustration while troubleshooting but is easily circumvented by systematically including the macro on rlwrap’s command-line.

Another hard-to-find error occurs when double closing parentheses “))” are present in a macro definition; rlwrap considers them to be the closing macro delimiters and removes them from the definition which subsequently invalidates the script, e.g.:

1> $qq((gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {if (!match($0, /^[0-9a-f]{16}/)) printf("%s", col[l%2]); else printf("%s", col[++l%2]); printf("%s%s\n", $0, reset)}'))
1> gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {if (!match($0, /^[0-9a-f]{16}/ printf("%s", col[l%2]); else printf("%s", col[++l%2]); printf("%s%s\n", , reset)}'))
2> clear 
1> select * from dm_user __ go| $qq
1> select * from dm_user 
 go| gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {if (!match($0, /^[0-9a-f]{16}/

sh: 1: Syntax error: Unterminated quoted string

On line 1, the if (!match(…)) statement has been digested into if (!match($0, /^[0-9a-f]{16}/ and the new line has replaced the line with the macro definition (line 2) which now invalidates the statement; in effect, when invoking the macro in line 4, a syntax error is raised (line 6).
To fix this, just insert a space between both closing parentheses, as follow:

1> $qq((gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {if (!match($0, /^[0-9a-f]{16}/) ) printf("%s", col[l%2]); else printf("%s", col[++l%2]); printf("%s%s\n", $0, reset)}'))

Another quirk is that the order of the filters matters; e.g. if pipeto precedes simple_macro (––filter=’pipeline pipeto:simple_macro’) and an in-line gawk script is used, the macro definition and invocation will be error-free but nothing will be produced as a result and no error will be raised, just as if simple_macro were missing. The obvious solution is to specify simple_macro before pipeto to avoid this situation, e.g. ––filter=’pipeline simple_macro:pipeto’. Use a shell alias to call rlwrap with its filter arguments in the correct order to get rid of this issue once for all.

However, even when the filters are correctly ordered, if the macro contains a pipe character, e.g. aimed at the shell or as part of gawk’s || logical operator, the shell will report a syntax error, e.g.:

rlwrap --no-warnings -f idql-completions.txt --filter='pipeline simple_macro:pipeto' --ansi-colour-aware --prompt-colour=RED --multi-line='__' -q \' idql dmtest73 -Udmadmin -Pdmadmin
...
1> $qq((gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {if (!match($0, /^[0-9a-f]{16}/) ) printf("%s", col[l%2]); else printf("%s", col[++l%2]); printf("%s%s\n", $0, reset)}' | less))
1> $qq((gawk 'BEGIN {col[0]="\033[1m\033[93m"; col[1]="\033[1m\033[94m"; reset="\033[0m"; l = 0} {if (!match($0, /^[0-9a-f]{16}/) ) printf("%s", col[l%2]); else printf("%s", col[++l%2]); printf("%s%s\n", $0, reset)}' 

sh: 1: Syntax error: ")" unexpected

One line 3, the macro is defined with the pipe symbol | to invoke the less command on the output; on line 4, the offending “| less” part is removed and the definition fails with a syntax error from the shell on line 6. Fortunately, this error message can be ignored because the macro invocation works just fine. I guess the error comes from the fact that rlwrap prematurely executes the macro by passing it to the shell as part of its definition.

rlwrap comes with several pre-installed filters; for a complete list, use the command:

rlwrap -z listing
The following filters can be found in  /usr/share/rlwrap/filters
count_in_prompt                replace prompt by simple counter
ftp_filter                     run plain Netkit ftp with completion for commands, local and remote files
history_format                 Append  to every history item, and strip it off again when input is accepted
null                           a filter that does nothing
paint_prompt                   paint the prompt in colour gradient between X11 colours  and 
pipeline                       combines the effects of 2 or more filters
pipeto                         Allow piping of  output through pagers or other shell commands
scrub_prompt                   removes all junk from prompt
simple_macro                   simple on-the-fly macro processing
template                       filter template
unbackspace                    remove backspaces from output

Help for a filter can be requested thusly, e.g.:

$ rlwrap -z pipeline
Usage: rlwrap [-options] -z 'pipeline filter_1:filter_2:..., filter_n' 
combines the effects of 2 or more filters
messages will be passed through filter_1, ..., filter_n.
Use a backslash to pass a ':' that is not meant as a pipe symbol, e.g:
rlwrap -z 'pipeline prompt hello\: : logger out' command

This is the end of Part I. Please, turn to Part II for even more rlwrap’s goodies.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Cesare Cervini
Cesare Cervini

Consultant