Infrastructure at your Service

As the Enterprise Manager Cloud 12c is configured for sending warnings and alerts if a target database is down, what’s happening¬†if the Enterprise Manager Repository database crashes? With Enterprise Manager Cloud Control 12c, a quite good solution is available: after an easy configuration, the management agent is able to run script to notify database administrators there is a problem with the repository. But what happens if the OMS itself is down? Enterprise Manager 13c offers a good solution wit ALWAYS-ON Monitoring.

AOM (Always-on Monitoring) allows the Enterprise Manager Administrators to keep informed of the target’s critical alerts even when the OMS is down.

I installed it on the Enterprise Manager 13c repository for testing but it would obviously be better to create a dedicated database.

Two pre-requisites are mandatory:

Optimizer_adaptive_features = FALSE:

SQL> ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE SCOPE=BOTH SID=’*’;

System altered.

Chearcter Set AL32UTF8

SQL> SELECT * from nls_database_parameters where parameter='NLS_CHARACTERSET';
PARAMETER                         VALUE
NLS_CHARACTERSET                  AL32UTF8

The emsca binaries are located in $MW_HOME/sysman/ems:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ [oms13c] ls
ems_13.1.0.0.0.zip

We uncompress the binaries:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ [oms13c] unzip ems_13.1.0.0.0.zip
Archive: ems_13.1.0.0.0.zip
   creating: ems/
   creating: ems/lib/
inflating: ems/lib/com.bea.core.apache.commons.lang_1.0.0.0_2-6.jar
inflating: ems/lib/com.oracle.http_client.http_client_12.1.3.jar
inflating: ems/lib/jackson-core-asl-1.9.2.jar
…

We copy the Emkey in the repository:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. 
This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey 
by running "emctl config emkey -remove_from_repos".

Then we run the emsca configuration script from /opt/oracle/Middleware13c/sysman/ems/ems/scripts/

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsca
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Always-On Monitoring Repository Connection String : cloud13c:1521:EMREP13C
Create Always-On Monitoring Repository User [n] : y
Always-On Monitoring Repository SYSDBA Username : sys
Always-On Monitoring Repository SYSDBA Password :
Always-On Monitoring Repository Username [ems] : aom
Always-On Monitoring Repository Password [ems] :
Enterprise Manager Repository Connection String : cloud13c:1521:EMREP13C
Enterprise Manager Repository Username : sysman
Enterprise Manager Repository Password :
Creating Always-On Monitoring repository user aom
Enter Enterprise Manager Middleware Home : /opt/oracle/Middleware13c/
Connecting to Always-On Monitoring repository.
Creating Always-On Monitoring repository schema
Creating repository storage for Targets data.
Creating repository storage for Alerts and Availability data.
Creating repository storage for Notification Metadata data.
Creating repository storage for Target Metric Metadata data.
Registering Always-On Monitoring instance
Always-On Monitoring Upload URL: https://cloud13c:8081/upload
Oracle PKI Tool : Version 12.1.3.0.0
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.
Certificate was added to keystore

We have to define downtime contacts in order AOM to be able to send notifications:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emcli set_oms_property 
-property_name='oracle.sysman.core.events.ems.downtimeContact'
 -property_value='[email protected]'
Property "oracle.sysman.core.events.ems.downtimeContact" 
has been successfully set to value "[email protected]" on all Management Servers.
Server restart is not required for the property change to get reflected

 

You can also define downtime contact from the target property:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emcli set_target_property_value 
-property_records="TESTDBA:oracle_database:Downtime Contact:[email protected]"
Properties updated successfully

 

Then we synchronize Always-on Monitoring with EM13c for the first time:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl sync
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: [email protected]:1521:EMREP13C
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Fri Feb 12 11:56:32 CET 2016

 

We configure EM to work with Always-on:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emctl set property 
-name "oracle.sysman.core.events.ems.emsURL" -value "https://cloud13c:8081/upload" -sysman_pwd manager1
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Property oracle.sysman.core.events.ems.emsURL has been set to value
https://cloud13c:8081/upload for all Management Servers
OMS restart is not required to reflect the new property value

emsctl gives us the possibility to check the status:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version               : 13.1.0.0.0
Always-On Monitoring Home                   : /opt/oracle/Middleware13c/sysman/ems/ems
Started At                                 : February 12, 2016 12:01:40 PM CET
Last Repository Sync                       : February 12, 2016 11:56:21 AM CET
Upload URL                                 : https://cloud13c:8081/upload
Always-On Monitoring Process ID             : 20588
Always-On Monitoring Repository             : cloud13c:1521:EMREP13C
Enterprise Manager Repository               : cloud13c:1521:EMREP13C
Notifications Enabled                       : false
Total Downtime Contacts Configured         : 1

 

We can check if the URL is present in the agent:

[email protected]:/opt/oracle/agent13c/agent_inst/sysman/config/ [agent13c] grep 8081 emd.properties
EMS_URL=https://cloud13c:8081/upload

 

It is also mandatory to enable the notifications (by default it is not …), the enable notification command will automatically run an incremental sync:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl enable_notification
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notifications have been enabled. There are downtime contacts configured.
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: [email protected]:1521:EMREP13C
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Fri Feb 12 13:31:40 CET 2016

We can easily check if AOM is running:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl ping
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring is running.

 

The logfiles are located in $OMS_HOME/sysman/ems/ems/log:

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/log/ [oms13c] ls
emsca.err emsca.log.0 emsctl.err emsctl.log.0 ems.err ems.log.0 ems.log.0.lck ems.out

 

Finally we can test:

We shutdown the OMS

[email protected]:/opt/oracle/Middleware13c/sysman/ems/ems/log/ [oms13c] emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
Stopping BI Publisher Server...
BI Publisher Server Already Stopped
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
AdminServer Successfully Stopped
BI Publisher Server is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command

 

We stop our TESTDBA database, and we receive a mail telling us our database target is down:

aom1

Conclusion:

AOM is a nice feature allowing you to continue to receive critical alerts from your database targets when OMS is down or under maintenance.

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Oracle Team
Oracle Team