When you use Enterprise Manager Cloud control 12c, the monitor username commonly used is dbsnmp. Depending on the Oracle profile used for this user, the dbsnmp password can expire, and as a consequence multiple targets will be seen in a pending status by Enterprise Manager Cloud control 12c.
An interesting way to solve this problem is to create a metric extension detecting in how many days the password will expire:
You realize the operation as follows:
You define the collection frequency to 1 day
You define the SQL request
You define the metrics columns
The metric extension defined, saved as deployable, published and applied to your database targets you will be told when the dbsnmp password will expire.
In case the dbsnmp password is about to expire, such an incident will be created:
By the way, you will notice that the incident created by the password expiry will not be evaluated before one day, even if you change the password and the incident will remain in the EM12c incident manager.
Of course you can use the Reevaluate Alert button in the incident manager page:
Nevertheless with EM12c, you have the possibility to manually trigger the metric collection by using the emctl control agent runCollection command.
For example, if you need to reevaluate the dbsnmp_expiry metric collection manually and not waiting one day, you will have to use the following command:
emctl control agent runcollection OMSREP:oracle_database ‘ME$dbsnmp_expiry’
The metric will be reevaluated and the incident will disappear.
Manually triggering the metrics can be very helpful when you administer a lot of targets. The usual case is when you receive a tablespace full alert. You increase or add a data file to correct the error, but as previously the metric collection will not be evaluated immediately and the generated incident will always be present.
The syntax of the command described in the Oracle documentation is:
emctl control agent runCollection <targetName>:<targetType> <collectionItemName>
Oracle’s documentation tells us to look in xml files located in $AGENT_BASE/plugins in order to find the collectionItemName:
oracle@vm12c:/u01/app/oracle/agent12cR5/plugins/ [agent12c] ls oracle.sysman.beacon.agent.plugin_12.1.0.5.0 oracle.sysman.emrep.agent.plugin_12.1.0.5.0 oracle.sysman.db.agent.plugin_12.1.0.8.0 oracle.sysman.oh.agent.plugin_12.1.0.5.0 oracle.sysman.db.discovery.plugin_12.1.0.8.0 oracle.sysman.oh.discovery.plugin_12.1.0.5.0 oracle.sysman.emas.agent.plugin_12.1.0.8.0 oracle.sysman.xa.discovery.plugin_12.1.0.6.0 oracle.sysman.emas.discovery.plugin_12.1.0.8.0
In our case we have a look in oracle.sysman.db.agent.plugin_12.1.0.8.0/default_collections and have a look at the database.xmlp file, you have to search Tablespaces Full:
====================================================================== == Category: Tablespaces Full - 10i - locally managed - not CDB ====================================================================== --> <CollectionItem NAME="problemTbsp_10i_Loc"> <ValidIf> <CategoryProp NAME="VersionCategory" CHOICES="10gR2;10gR203;10gR204;10gR205;11gR1;11gR2;11gR202;12c"/>
<CategoryProp NAME=”MetricScope” CHOICES=”DB”/>
<CategoryProp NAME=”DBCategoryDetails” CHOICES=”FullLLFile;none;TRUE;FALSE;1;0″/>
</ValidIf>
<Schedule>
<IntervalSchedule INTERVAL=”30″ TIME_UNIT=”Min”/>
</Schedule>
In order to manually trigger the tablespace full alert, we use the command:
oracle@vm12c:/u01/app/oracle/agent12cR5/ [agent12c] emctl control agent runCollection OMSREP:oracle_database problemTbsp_10i_Loc Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD runCollection completed successfully
There is also an easy way to find all the collectionItemName. All those collections are scheduled by the agent, so if you run the command:
oracle@vmtestoraem12c:/u01/app/oracle/ [agent12c] emctl status agent scheduler | grep oracle_database | grep OMSREP 2015-09-24 15:16:53.222 : oracle_database:OMSREP:UserLocksCollection 2015-09-24 15:16:55.432 : oracle_database:OMSREP:Response 2015-09-24 15:16:57.805 : oracle_database:OMSREP:wait_sess_cls_10i 2015-09-24 15:17:12.566 : oracle_database:OMSREP:observer_11g 2015-09-24 15:17:42.703 : oracle_database:OMSREP:haconfig2_collection 2015-09-24 15:18:41.928 : oracle_database:OMSREP:Recovery_Area 2015-09-24 15:18:51.331 : oracle_database:OMSREP:latest_hdm_findings_coll_item 2015-09-24 15:18:54.788 : oracle_database:OMSREP:dgprimarydb_collection 2015-09-24 15:19:09.136 : oracle_database:OMSREP:adr_alert_log_rollup 2015-09-24 15:19:38.388 : oracle_database:OMSREP:incident_meter 2015-09-24 15:19:59.818 : oracle_database:OMSREP:topWaitEvents_col 2015-09-24 15:20:16.315 : oracle_database:OMSREP:db_alertlog_coll_12 2015-09-24 15:20:32.234 : oracle_database:OMSREP:haconfig4_collection 2015-09-24 15:20:48.166 : oracle_database:OMSREP:memory_usage_coll_item 2015-09-24 15:20:48.307 : oracle_database:OMSREP:activity_pending 2015-09-24 15:20:49.534 : oracle_database:OMSREP:archFull 2015-09-24 15:20:59.886 : oracle_database:OMSREP:haconfig3_collection 2015-09-24 15:21:14.840 : oracle_database:OMSREP:db_inst_pga_alloc_11g 2015-09-24 15:21:43.593 : oracle_database:OMSREP:rac_global_cache_10i 2015-09-24 15:21:47.527 : oracle_database:OMSREP:dataguard_11gR2 2015-09-24 15:22:09.095 : oracle_database:OMSREP:memory_usage_sga_pga 2015-09-24 15:22:11.189 : oracle_database:OMSREP:sga_pool_wastage_10i 2015-09-24 15:22:19.278 : oracle_database:OMSREP:UserAudit 2015-09-24 15:22:21.311 : oracle_database:OMSREP:Database_Resource_Usage_10i 2015-09-24 15:22:45.416 : oracle_database:OMSREP:sysTimeModel_col 2015-09-24 15:23:41.486 : oracle_database:OMSREP:db_inst_opt_sga 2015-09-24 15:23:43.516 : oracle_database:OMSREP:instance_efficiency_10i 2015-09-24 15:23:51.902 : oracle_database:OMSREP:system_response_time_per_call_10i 2015-09-24 15:24:32.947 : oracle_database:OMSREP:wait_bottlenecks_10i 2015-09-24 15:24:53.137 : oracle_database:OMSREP:topSqlMonitoringList_col 2015-09-24 15:25:04.134 : oracle_database:OMSREP:instance_throughput_10i 2015-09-24 15:26:19.263 : oracle_database:OMSREP:service_10i 2015-09-24 15:33:34.681 : oracle_database:OMSREP:log_full 2015-09-24 15:33:35.160 : oracle_database:OMSREP:problemTbsp_10i_Loc 2015-09-24 15:33:35.223 : oracle_database:OMSREP:problemTbsp_10i_Dct 2015-09-24 15:33:35.474 : oracle_database:OMSREP:problemTbspTemp_10i_Loc 2015-09-24 15:33:35.504 : oracle_database:OMSREP:problemTbspUndo_10i_Loc 2015-09-24 15:33:35.797 : oracle_database:OMSREP:dbjob_status 2015-09-24 15:33:37.800 : oracle_database:OMSREP:audit_failed_logins 2015-09-24 15:33:38.363 : oracle_database:OMSREP:aq_monitoring_alerts 2015-09-24 15:33:38.455 : oracle_database:OMSREP:streams_processes_count_item 2015-09-24 15:33:38.762 : oracle_database:OMSREP:streams_statistics 2015-09-24 15:33:40.027 : oracle_database:OMSREP:Temporary File Status 2015-09-24 16:03:34.340 : oracle_database:OMSREP:scn_instance_collection 2015-09-24 16:03:34.671 : oracle_database:OMSREP:db_inst_cpu_usage 2015-09-24 16:03:38.055 : oracle_database:OMSREP:DatabaseVaultRealmViolation_collection 2015-09-24 16:03:38.222 : oracle_database:OMSREP:DatabaseVaultRealmConfigurationIssue_collection 2015-09-24 16:03:38.239 : oracle_database:OMSREP:DatabaseVaultCommandRuleViolation_collection 2015-09-24 16:03:38.542 : oracle_database:OMSREP:DatabaseVaultCommandRuleConfigurationIssue_collection 2015-09-24 16:03:39.403 : oracle_database:OMSREP:DatabaseVaultPolicyChanges_collection 2015-09-24 16:03:39.871 : oracle_database:OMSREP:scn_growth_collection 2015-09-24 18:03:25.455 : oracle_database:OMSREP:oracle_dbconfig 2015-09-24 18:03:33.694 : oracle_database:OMSREP:ocm_instrumentation 2015-09-24 18:03:33.712 : oracle_database:OMSREP:mgmt_sql 2015-09-24 18:03:33.714 : oracle_database:OMSREP:has_resources 2015-09-24 18:03:33.905 : oracle_database:OMSREP:sizeOfOSAuditFiles_collection 2015-09-24 18:03:34.589 : oracle_database:OMSREP:problemSegTbsp 2015-09-24 18:03:34.849 : oracle_database:OMSREP:ha_dg_target_summary 2015-09-24 18:03:34.858 : oracle_database:OMSREP:invalid_objects_rollup 2015-09-24 18:03:35.003 : oracle_database:OMSREP:tbspAllocation 2015-09-24 18:03:35.917 : oracle_database:OMSREP:haconfig1_collection 2015-09-24 18:03:36.848 : oracle_database:OMSREP:audit_failed_logins_historical 2015-09-24 18:03:38.968 : oracle_database:OMSREP:mgmt_database_listener_config 2015-09-24 18:03:38.996 : oracle_database:OMSREP:exadataCollection 2015-09-24 18:03:39.449 : oracle_database:OMSREP:scn_max_collection 2015-09-24 18:03:39.546 : oracle_database:OMSREP:db_instance_caging 2015-09-24 18:03:47.855 : oracle_database:OMSREP:oracle_cdbconfig 2015-09-24 22:08:59.848 : oracle_database:OMSREP:ME$dbsnmp_expiry
You will find the entire collection item name you can reevaluate with the emctl control agent runCollection command.
For a host target, the behavior is equivalent:
oracle@vmtestoraem12c:/u01/app/oracle/ [agent12c] emctl status agent scheduler | grep host | grep vmtestoraem12c
2015-09-24 15:22:12.565 : host:vm12c:Processes_diagnosticsLinux 2015-09-24 15:23:45.713 : host:vm12c:LoadLinux 2015-09-24 15:23:52.670 : host:vm12c:ProgramResourceUtilizationLinux 2015-09-24 15:24:10.227 : host:vm12c:PagingActivityLinux 2015-09-24 15:24:41.241 : host:vm12c:TotalDiskUsageLinux 2015-09-24 15:25:34.795 : host:vm12c:CPUUsageLinux 2015-09-24 15:26:19.727 : host:vm12c:FileMonitoringLinux 2015-09-24 15:26:31.792 : host:vm12c:NetworkLinux 2015-09-24 15:27:26.153 : host:vm12c:DiskActivityLinux 2015-09-24 15:33:35.080 : host:vm12c:proc_zombieLinux 2015-09-24 15:33:46.489 : host:vm12c:LogFileMonitorLinux 2015-09-24 15:34:22.695 : host:vm12c:FilesystemsLinux 2015-09-24 18:02:53.840 : host:vm12c:host_storage 2015-09-24 18:02:54.169 : host:vm12c:HostStorageSupport 2015-09-24 18:02:54.201 : host:vm12c:DiscoverTargets 2015-09-24 18:02:55.599 : host:vm12c:oracle_security 2015-09-24 18:02:56.532 : host:vm12c:Swap_Area_StatusLinux 2015-09-24 18:26:28.295 : host:vm12c:ll_host_config
Conclusion:
Depending of the metric evaluation frequency, it might be very useful to manually trigger the metric collections in order to cleanup false open incidents in the Enterprise Manager Cloud 12c console.