Last month I was confronted to a very interesting and strange problem while deploying a Cloud Control 12c infrastructure. One of my monitored server was a SuSE Linux Enterprise Server (SLES) 64 bits on which a 11.2.0.3 Oracle database was installed. The x86_64 cloud control 12c agent has been deployed on this server with the standard Cloud Control 12c agent deployment procedure (not the scope of this post). However despite several trials to configure the target database, this one remained in the status “Pending”.

The situation is presented in the screenhshot below :

This post summarizes the following steps I had to perform to troubleshoot this issue :

  • Find out the appropriate cause of this trouble
  • Identify the correct solution to fix it
  • Document all the steps in this post to allow to solve this issue on your infrastructure

First of all, as usual, My Oracle Support is a good starting point to find out the possible causes. The first interesting note about Targets in pending status I found was the following one :
“12c: Cluster Database Target Remains in Pending Delete Status Indefinitely after Re-adding [ID 1403833.1]”

At a first glance I thought the note was exactly fitting to my situation. Indeed I performed several trials to configure the database target, therefore I configured and removed it several times to get rid of the “Pending” state.

I checked the note and I started the attached script (Fixdelete.sql) to find outs any target remaining in the “deletion” status. Below the SQL statement started by the script :

select target_type,target_name 
from mgmt_targets_delete d 
where delete_complete_time is null 
group by target_type,target_name 
having count(*) > 1 ;

The output of the query was : “no rows selected …”.
The next interesting My Oracle Support note found was the following one :
Troubleshooting the ‘Pending’ status for a Database Target in Enterprise Manager 12c Cloud Control [ID 1418873.1]

In this note a very interesting verification was pointed out (section 7) : “Does the target server/agent have the appropriate plugins  to monitor the several targets, for instance the database plugin to monitoring the databases”.

As a first check, I confirmed, on the OMS side, that the same target was registered several times (through my multiple trials) and that the status was still “Pending” (emcli used at the OMS side) :

oracle@chhs-sora011:/home/oracle/ [oms12c] emcli get_targets | grep oracle_db 
6       Status Pending   oracle_dbsys          ODGMGT1_SITE1.ch_sys01 
6       Status Pending   oracle_dbsys          ODGMGT1_SITE1.ch_sys0 
6       Status Pending   oracle_dbsys          ODGMGT1_SITE1.ch_sys012 
6       Status Pending   oracle_dbsys          ODGMGT1_SITE1.ch_sys

However on the agent side, I couldn’t get the confirmation that the agent really sees the database targets :

oracle@chbas0-sid092:/home/oracle/ [agent12c] emctl config agent listtargets 
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved. 
[oramgt2, host] 
[oramgt2:3872, oracle_emd]

Only the host and the agent itself are visible for the agent.

One possible reason presented in the My Oracle Support note 1418873.1 was that the plugins were not installed correctly on the  agent side. Therefore we tried to deploy the database plugin on the oramgt2 target host (setup -> Extensibility -> Plugins).
Click on database , Oracle Database, right click , deploy on ……) :

 

The plugin deployment procedure fails with the following error :

This issue is documented in the following My Oracle Support note : “EM 12c: Deploying 12.1.0.2 DB Plugin on Suse 10 64-bit   server fails with “java.io.IOException: Broken pipe” [ID 1468543.1]”

Even if the target system has a SLES 64 bits Operating system :
oracle@chbas0-sid092:/u01/app/oracle/product/12.1.0/agent12c/ [agent12c] uname -a
Linux chbas0-sid092 2.6.16.60-0.59.1-smp #1 SMP Thu Jan 14 18:30:10 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

a 32 bits Linux 12c agent must be installed

This is exactly what I did. To perform this operation I :

  • removed the existing agent (using the Oracle Universal Installer, and removing first the plugin, then the agent itself)
  • configured the My Oracle Support connection from the OMS, I configured the proxy and the Software library
  • downloaded the linux 32 bits agent in from the OMS :

  • I deployed it to the 64bits SLES sever (ormgt2) :

  • I configured the target again, finally it worked perfectly :

Oracle states that this point is documented in the “compatibility matrix”, however I really didin’t find any clear statement in this matrix, the only source of information confirming this “incompatibility” was the My Oracle Support notes themselves.