When you use Enterprise Manager Cloud control 12c, the monitor username commonly used is dbsnmp. Depending on the Oracle profile used for this user, the dbsnmp password can expire, and as a consequence multiple targets will be seen in a pending status by Enterprise Manager Cloud control 12c.

An interesting way to solve this problem is to create a metric extension detecting in how many days the password will expire:

You realize the operation as follows:

dbsnmp1

You define the collection frequency to 1 day

dbsnmp2

You define the SQL request

dbsnmp3

You define the metrics columns

dbsnmp4

The metric extension defined, saved as deployable, published and applied to your database targets you will be told when the dbsnmp password will expire.

In case the dbsnmp password is about to expire, such an incident will be created:

dbsnmp7

By the way, you will notice that the incident created by the password expiry will not be evaluated before one day, even if you change the password and the incident will remain in the EM12c incident manager.

Of course you can use the Reevaluate Alert button in the incident manager page:

dbsnmp6

Nevertheless with EM12c, you have the possibility to manually trigger the metric collection by using the emctl control agent runCollection command.

For example, if you need to reevaluate the dbsnmp_expiry metric collection manually and not waiting one day, you will have to use the following command:

emctl control agent runcollection OMSREP:oracle_database ‘ME$dbsnmp_expiry’

The metric will be reevaluated and the incident will disappear.

Manually triggering the metrics can be very helpful when you administer a lot of targets. The usual case is when you receive a tablespace full alert. You increase or add a data file to correct the error, but as previously the metric collection will not be evaluated immediately and the generated incident will always be present.

The syntax of the command described in the Oracle documentation is:

emctl control agent runCollection <targetName>:<targetType> <collectionItemName>

 

Oracle’s documentation tells us to look in xml files located in $AGENT_BASE/plugins in order to find the collectionItemName:

oracle@vm12c:/u01/app/oracle/agent12cR5/plugins/ [agent12c] ls
oracle.sysman.beacon.agent.plugin_12.1.0.5.0
oracle.sysman.emrep.agent.plugin_12.1.0.5.0
oracle.sysman.db.agent.plugin_12.1.0.8.0
oracle.sysman.oh.agent.plugin_12.1.0.5.0
oracle.sysman.db.discovery.plugin_12.1.0.8.0
oracle.sysman.oh.discovery.plugin_12.1.0.5.0
oracle.sysman.emas.agent.plugin_12.1.0.8.0
oracle.sysman.xa.discovery.plugin_12.1.0.6.0
oracle.sysman.emas.discovery.plugin_12.1.0.8.0

 

In our case we have a look in oracle.sysman.db.agent.plugin_12.1.0.8.0/default_collections and have a look at the database.xmlp file, you have to search Tablespaces Full:

======================================================================
== Category: Tablespaces Full - 10i - locally managed - not CDB
======================================================================
-->
<CollectionItem NAME="problemTbsp_10i_Loc">
<ValidIf>
<CategoryProp NAME="VersionCategory" CHOICES="10gR2;10gR203;10gR204;10gR205;11gR1;11gR2;11gR202;12c"/>

 

<CategoryProp NAME=”MetricScope” CHOICES=”DB”/>

<CategoryProp NAME=”DBCategoryDetails” CHOICES=”FullLLFile;none;TRUE;FALSE;1;0″/>

</ValidIf>

<Schedule>

<IntervalSchedule INTERVAL=”30″ TIME_UNIT=”Min”/>

</Schedule>

In order to manually trigger the tablespace full alert, we use the command:

oracle@vm12c:/u01/app/oracle/agent12cR5/ [agent12c] emctl control agent runCollection OMSREP:oracle_database problemTbsp_10i_Loc
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

There is also an easy way to find all the collectionItemName. All those collections are scheduled by the agent, so if you run the command:

oracle@vmtestoraem12c:/u01/app/oracle/ [agent12c] emctl status agent scheduler | grep oracle_database | grep OMSREP
2015-09-24 15:16:53.222 : oracle_database:OMSREP:UserLocksCollection
2015-09-24 15:16:55.432 : oracle_database:OMSREP:Response
2015-09-24 15:16:57.805 : oracle_database:OMSREP:wait_sess_cls_10i
2015-09-24 15:17:12.566 : oracle_database:OMSREP:observer_11g
2015-09-24 15:17:42.703 : oracle_database:OMSREP:haconfig2_collection
2015-09-24 15:18:41.928 : oracle_database:OMSREP:Recovery_Area
2015-09-24 15:18:51.331 : oracle_database:OMSREP:latest_hdm_findings_coll_item
2015-09-24 15:18:54.788 : oracle_database:OMSREP:dgprimarydb_collection
2015-09-24 15:19:09.136 : oracle_database:OMSREP:adr_alert_log_rollup
2015-09-24 15:19:38.388 : oracle_database:OMSREP:incident_meter
2015-09-24 15:19:59.818 : oracle_database:OMSREP:topWaitEvents_col
2015-09-24 15:20:16.315 : oracle_database:OMSREP:db_alertlog_coll_12
2015-09-24 15:20:32.234 : oracle_database:OMSREP:haconfig4_collection
2015-09-24 15:20:48.166 : oracle_database:OMSREP:memory_usage_coll_item
2015-09-24 15:20:48.307 : oracle_database:OMSREP:activity_pending
2015-09-24 15:20:49.534 : oracle_database:OMSREP:archFull
2015-09-24 15:20:59.886 : oracle_database:OMSREP:haconfig3_collection
2015-09-24 15:21:14.840 : oracle_database:OMSREP:db_inst_pga_alloc_11g
2015-09-24 15:21:43.593 : oracle_database:OMSREP:rac_global_cache_10i
2015-09-24 15:21:47.527 : oracle_database:OMSREP:dataguard_11gR2
2015-09-24 15:22:09.095 : oracle_database:OMSREP:memory_usage_sga_pga
2015-09-24 15:22:11.189 : oracle_database:OMSREP:sga_pool_wastage_10i
2015-09-24 15:22:19.278 : oracle_database:OMSREP:UserAudit
2015-09-24 15:22:21.311 : oracle_database:OMSREP:Database_Resource_Usage_10i
2015-09-24 15:22:45.416 : oracle_database:OMSREP:sysTimeModel_col
2015-09-24 15:23:41.486 : oracle_database:OMSREP:db_inst_opt_sga
2015-09-24 15:23:43.516 : oracle_database:OMSREP:instance_efficiency_10i
2015-09-24 15:23:51.902 : oracle_database:OMSREP:system_response_time_per_call_10i
2015-09-24 15:24:32.947 : oracle_database:OMSREP:wait_bottlenecks_10i
2015-09-24 15:24:53.137 : oracle_database:OMSREP:topSqlMonitoringList_col
2015-09-24 15:25:04.134 : oracle_database:OMSREP:instance_throughput_10i
2015-09-24 15:26:19.263 : oracle_database:OMSREP:service_10i
2015-09-24 15:33:34.681 : oracle_database:OMSREP:log_full
2015-09-24 15:33:35.160 : oracle_database:OMSREP:problemTbsp_10i_Loc
2015-09-24 15:33:35.223 : oracle_database:OMSREP:problemTbsp_10i_Dct
2015-09-24 15:33:35.474 : oracle_database:OMSREP:problemTbspTemp_10i_Loc
2015-09-24 15:33:35.504 : oracle_database:OMSREP:problemTbspUndo_10i_Loc
2015-09-24 15:33:35.797 : oracle_database:OMSREP:dbjob_status
2015-09-24 15:33:37.800 : oracle_database:OMSREP:audit_failed_logins
2015-09-24 15:33:38.363 : oracle_database:OMSREP:aq_monitoring_alerts
2015-09-24 15:33:38.455 : oracle_database:OMSREP:streams_processes_count_item
2015-09-24 15:33:38.762 : oracle_database:OMSREP:streams_statistics
2015-09-24 15:33:40.027 : oracle_database:OMSREP:Temporary File Status
2015-09-24 16:03:34.340 : oracle_database:OMSREP:scn_instance_collection
2015-09-24 16:03:34.671 : oracle_database:OMSREP:db_inst_cpu_usage
2015-09-24 16:03:38.055 : oracle_database:OMSREP:DatabaseVaultRealmViolation_collection
2015-09-24 16:03:38.222 : oracle_database:OMSREP:DatabaseVaultRealmConfigurationIssue_collection
2015-09-24 16:03:38.239 : oracle_database:OMSREP:DatabaseVaultCommandRuleViolation_collection
2015-09-24 16:03:38.542 : oracle_database:OMSREP:DatabaseVaultCommandRuleConfigurationIssue_collection
2015-09-24 16:03:39.403 : oracle_database:OMSREP:DatabaseVaultPolicyChanges_collection
2015-09-24 16:03:39.871 : oracle_database:OMSREP:scn_growth_collection
2015-09-24 18:03:25.455 : oracle_database:OMSREP:oracle_dbconfig
2015-09-24 18:03:33.694 : oracle_database:OMSREP:ocm_instrumentation
2015-09-24 18:03:33.712 : oracle_database:OMSREP:mgmt_sql
2015-09-24 18:03:33.714 : oracle_database:OMSREP:has_resources
2015-09-24 18:03:33.905 : oracle_database:OMSREP:sizeOfOSAuditFiles_collection
2015-09-24 18:03:34.589 : oracle_database:OMSREP:problemSegTbsp
2015-09-24 18:03:34.849 : oracle_database:OMSREP:ha_dg_target_summary
2015-09-24 18:03:34.858 : oracle_database:OMSREP:invalid_objects_rollup
2015-09-24 18:03:35.003 : oracle_database:OMSREP:tbspAllocation
2015-09-24 18:03:35.917 : oracle_database:OMSREP:haconfig1_collection
2015-09-24 18:03:36.848 : oracle_database:OMSREP:audit_failed_logins_historical
2015-09-24 18:03:38.968 : oracle_database:OMSREP:mgmt_database_listener_config
2015-09-24 18:03:38.996 : oracle_database:OMSREP:exadataCollection
2015-09-24 18:03:39.449 : oracle_database:OMSREP:scn_max_collection
2015-09-24 18:03:39.546 : oracle_database:OMSREP:db_instance_caging
2015-09-24 18:03:47.855 : oracle_database:OMSREP:oracle_cdbconfig
2015-09-24 22:08:59.848 : oracle_database:OMSREP:ME$dbsnmp_expiry

 

You will find the entire collection item name you can reevaluate with the emctl control agent runCollection command.

For a host target, the behavior is equivalent:

oracle@vmtestoraem12c:/u01/app/oracle/ [agent12c] emctl status agent scheduler | grep host | grep vmtestoraem12c

2015-09-24 15:22:12.565 : host:vm12c:Processes_diagnosticsLinux
2015-09-24 15:23:45.713 : host:vm12c:LoadLinux
2015-09-24 15:23:52.670 : host:vm12c:ProgramResourceUtilizationLinux
2015-09-24 15:24:10.227 : host:vm12c:PagingActivityLinux
2015-09-24 15:24:41.241 : host:vm12c:TotalDiskUsageLinux
2015-09-24 15:25:34.795 : host:vm12c:CPUUsageLinux
2015-09-24 15:26:19.727 : host:vm12c:FileMonitoringLinux
2015-09-24 15:26:31.792 : host:vm12c:NetworkLinux
2015-09-24 15:27:26.153 : host:vm12c:DiskActivityLinux
2015-09-24 15:33:35.080 : host:vm12c:proc_zombieLinux
2015-09-24 15:33:46.489 : host:vm12c:LogFileMonitorLinux
2015-09-24 15:34:22.695 : host:vm12c:FilesystemsLinux
2015-09-24 18:02:53.840 : host:vm12c:host_storage
2015-09-24 18:02:54.169 : host:vm12c:HostStorageSupport
2015-09-24 18:02:54.201 : host:vm12c:DiscoverTargets
2015-09-24 18:02:55.599 : host:vm12c:oracle_security
2015-09-24 18:02:56.532 : host:vm12c:Swap_Area_StatusLinux
2015-09-24 18:26:28.295 : host:vm12c:ll_host_config

 

Conclusion:

Depending of the metric evaluation frequency, it might be very useful to manually trigger the metric collections in order to cleanup false open incidents in the Enterprise Manager Cloud 12c console.