As the Enterprise Manager Cloud 12c is configured for sending warnings and alerts if a target database is down, what’s happening if the Enterprise Manager Repository database crashes? With Enterprise Manager Cloud Control 12c, a quite good solution is available: after an easy configuration, the management agent is able to run script to notify database administrators there is a problem with the repository. But what happens if the OMS itself is down? Enterprise Manager 13c offers a good solution wit ALWAYS-ON Monitoring.
AOM (Always-on Monitoring) allows the Enterprise Manager Administrators to keep informed of the target’s critical alerts even when the OMS is down.
I installed it on the Enterprise Manager 13c repository for testing but it would obviously be better to create a dedicated database.
Two pre-requisites are mandatory:
Optimizer_adaptive_features = FALSE:
SQL> ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE SCOPE=BOTH SID=’*’;
System altered.
Chearcter Set AL32UTF8
SQL> SELECT * from nls_database_parameters where parameter='NLS_CHARACTERSET'; PARAMETER VALUE NLS_CHARACTERSET AL32UTF8
The emsca binaries are located in $MW_HOME/sysman/ems:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ [oms13c] ls ems_13.1.0.0.0.zip
We uncompress the binaries:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ [oms13c] unzip ems_13.1.0.0.0.zip Archive: ems_13.1.0.0.0.zip creating: ems/ creating: ems/lib/ inflating: ems/lib/com.bea.core.apache.commons.lang_1.0.0.0_2-6.jar inflating: ems/lib/com.oracle.http_client.http_client_12.1.3.jar inflating: ems/lib/jackson-core-asl-1.9.2.jar …
We copy the Emkey in the repository:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emctl config emkey -copy_to_repos Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure. After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".
Then we run the emsca configuration script from /opt/oracle/Middleware13c/sysman/ems/ems/scripts/
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsca Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Always-On Monitoring Repository Connection String : cloud13c:1521:EMREP13C Create Always-On Monitoring Repository User [n] : y Always-On Monitoring Repository SYSDBA Username : sys Always-On Monitoring Repository SYSDBA Password : Always-On Monitoring Repository Username [ems] : aom Always-On Monitoring Repository Password [ems] : Enterprise Manager Repository Connection String : cloud13c:1521:EMREP13C Enterprise Manager Repository Username : sysman Enterprise Manager Repository Password : Creating Always-On Monitoring repository user aom Enter Enterprise Manager Middleware Home : /opt/oracle/Middleware13c/ Connecting to Always-On Monitoring repository. Creating Always-On Monitoring repository schema Creating repository storage for Targets data. Creating repository storage for Alerts and Availability data. Creating repository storage for Notification Metadata data. Creating repository storage for Target Metric Metadata data. Registering Always-On Monitoring instance Always-On Monitoring Upload URL: https://cloud13c:8081/upload Oracle PKI Tool : Version 12.1.3.0.0 Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved. Certificate was added to keystore
We have to define downtime contacts in order AOM to be able to send notifications:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emcli set_oms_property -property_name='oracle.sysman.core.events.ems.downtimeContact' -property_value='[email protected]' Property "oracle.sysman.core.events.ems.downtimeContact" has been successfully set to value "[email protected]" on all Management Servers. Server restart is not required for the property change to get reflected
You can also define downtime contact from the target property:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emcli set_target_property_value -property_records="TESTDBA:oracle_database:Downtime Contact:[email protected]" Properties updated successfully
Then we synchronize Always-on Monitoring with EM13c for the first time:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl sync Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Connecting to Always-On Monitoring repository. Starting synchronization with EM. Synchronizing with EM repository: sysman@cloud13c:1521:EMREP13C Synchronizing Targets data. Synchronizing Alerts and Availability data. Synchronizing Notification Metadata data. Synchronizing Target Metric Metadata data. Synchronization complete at : Fri Feb 12 11:56:32 CET 2016
We configure EM to work with Always-on:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value "https://cloud13c:8081/upload" -sysman_pwd manager1 Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Property oracle.sysman.core.events.ems.emsURL has been set to value https://cloud13c:8081/upload for all Management Servers OMS restart is not required to reflect the new property value
emsctl gives us the possibility to check the status:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl status Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Always-On Monitoring Version : 13.1.0.0.0 Always-On Monitoring Home : /opt/oracle/Middleware13c/sysman/ems/ems Started At : February 12, 2016 12:01:40 PM CET Last Repository Sync : February 12, 2016 11:56:21 AM CET Upload URL : https://cloud13c:8081/upload Always-On Monitoring Process ID : 20588 Always-On Monitoring Repository : cloud13c:1521:EMREP13C Enterprise Manager Repository : cloud13c:1521:EMREP13C Notifications Enabled : false Total Downtime Contacts Configured : 1
We can check if the URL is present in the agent:
oracle@cloud13c:/opt/oracle/agent13c/agent_inst/sysman/config/ [agent13c] grep 8081 emd.properties EMS_URL=https://cloud13c:8081/upload
It is also mandatory to enable the notifications (by default it is not …), the enable notification command will automatically run an incremental sync:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl enable_notification Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Notifications have been enabled. There are downtime contacts configured. Connecting to Always-On Monitoring repository. Starting synchronization with EM. Synchronizing with EM repository: sysman@cloud13c:1521:EMREP13C Synchronizing Targets data. Synchronizing Alerts and Availability data. Synchronizing Notification Metadata data. Synchronizing Target Metric Metadata data. Synchronization complete at : Fri Feb 12 13:31:40 CET 2016
We can easily check if AOM is running:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl ping Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. ------------------------------------------------------------------ Always-On Monitoring is running.
The logfiles are located in $OMS_HOME/sysman/ems/ems/log:
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/log/ [oms13c] ls emsca.err emsca.log.0 emsctl.err emsctl.log.0 ems.err ems.log.0 ems.log.0.lck ems.out
Finally we can test:
We shutdown the OMS
oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/log/ [oms13c] emctl stop oms -all Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Stopping Oracle Management Server... WebTier Successfully Stopped Oracle Management Server Successfully Stopped Oracle Management Server is Down JVMD Engine is Down BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command Stopping BI Publisher Server... BI Publisher Server Already Stopped BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command AdminServer Successfully Stopped BI Publisher Server is Down BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
We stop our TESTDBA database, and we receive a mail telling us our database target is down:
Conclusion:
AOM is a nice feature allowing you to continue to receive critical alerts from your database targets when OMS is down or under maintenance.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/12/oracle-square.png)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/DWE_web-min-scaled.jpg)