Infrastructure at your Service

Pierre Sicot

Always-on Monitoring EM 13C

As the Enterprise Manager Cloud 12c is configured for sending warnings and alerts if a target database is down, what’s happening if the Enterprise Manager Repository database crashes? With Enterprise Manager Cloud Control 12c, a quite good solution is available: after an easy configuration, the management agent is able to run script to notify database administrators there is a problem with the repository. But what happens if the OMS itself is down? Enterprise Manager 13c offers a good solution wit ALWAYS-ON Monitoring.

AOM (Always-on Monitoring) allows the Enterprise Manager Administrators to keep informed of the target’s critical alerts even when the OMS is down.

I installed it on the Enterprise Manager 13c repository for testing but it would obviously be better to create a dedicated database.

Two pre-requisites are mandatory:

Optimizer_adaptive_features = FALSE:

SQL> ALTER SYSTEM SET OPTIMIZER_ADAPTIVE_FEATURES=FALSE SCOPE=BOTH SID=’*';

System altered.

Chearcter Set AL32UTF8

SQL> SELECT * from nls_database_parameters where parameter='NLS_CHARACTERSET';
PARAMETER                         VALUE
NLS_CHARACTERSET                  AL32UTF8

The emsca binaries are located in $MW_HOME/sysman/ems:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ [oms13c] ls
ems_13.1.0.0.0.zip

We uncompress the binaries:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ [oms13c] unzip ems_13.1.0.0.0.zip
Archive: ems_13.1.0.0.0.zip
   creating: ems/
   creating: ems/lib/
inflating: ems/lib/com.bea.core.apache.commons.lang_1.0.0.0_2-6.jar
inflating: ems/lib/com.oracle.http_client.http_client_12.1.3.jar
inflating: ems/lib/jackson-core-asl-1.9.2.jar
…

We copy the Emkey in the repository:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. 
This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey 
by running "emctl config emkey -remove_from_repos".

Then we run the emsca configuration script from /opt/oracle/Middleware13c/sysman/ems/ems/scripts/

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsca
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Always-On Monitoring Repository Connection String : cloud13c:1521:EMREP13C
Create Always-On Monitoring Repository User [n] : y
Always-On Monitoring Repository SYSDBA Username : sys
Always-On Monitoring Repository SYSDBA Password :
Always-On Monitoring Repository Username [ems] : aom
Always-On Monitoring Repository Password [ems] :
Enterprise Manager Repository Connection String : cloud13c:1521:EMREP13C
Enterprise Manager Repository Username : sysman
Enterprise Manager Repository Password :
Creating Always-On Monitoring repository user aom
Enter Enterprise Manager Middleware Home : /opt/oracle/Middleware13c/
Connecting to Always-On Monitoring repository.
Creating Always-On Monitoring repository schema
Creating repository storage for Targets data.
Creating repository storage for Alerts and Availability data.
Creating repository storage for Notification Metadata data.
Creating repository storage for Target Metric Metadata data.
Registering Always-On Monitoring instance
Always-On Monitoring Upload URL: https://cloud13c:8081/upload
Oracle PKI Tool : Version 12.1.3.0.0
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.
Certificate was added to keystore

We have to define downtime contacts in order AOM to be able to send notifications:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emcli set_oms_property 
-property_name='oracle.sysman.core.events.ems.downtimeContact'
 -property_value='pierre.sicot@dbi-services.com'
Property "oracle.sysman.core.events.ems.downtimeContact" 
has been successfully set to value "pierre.sicot@dbi-services.com" on all Management Servers.
Server restart is not required for the property change to get reflected

 

You can also define downtime contact from the target property:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emcli set_target_property_value 
-property_records="TESTDBA:oracle_database:Downtime Contact:pierre.sicot@dbi-services.com"
Properties updated successfully

 

Then we synchronize Always-on Monitoring with EM13c for the first time:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl sync
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: sysman@cloud13c:1521:EMREP13C
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Fri Feb 12 11:56:32 CET 2016

 

We configure EM to work with Always-on:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] emctl set property 
-name "oracle.sysman.core.events.ems.emsURL" -value "https://cloud13c:8081/upload" -sysman_pwd manager1
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Property oracle.sysman.core.events.ems.emsURL has been set to value
https://cloud13c:8081/upload for all Management Servers
OMS restart is not required to reflect the new property value

emsctl gives us the possibility to check the status:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version               : 13.1.0.0.0
Always-On Monitoring Home                   : /opt/oracle/Middleware13c/sysman/ems/ems
Started At                                 : February 12, 2016 12:01:40 PM CET
Last Repository Sync                       : February 12, 2016 11:56:21 AM CET
Upload URL                                 : https://cloud13c:8081/upload
Always-On Monitoring Process ID             : 20588
Always-On Monitoring Repository             : cloud13c:1521:EMREP13C
Enterprise Manager Repository               : cloud13c:1521:EMREP13C
Notifications Enabled                       : false
Total Downtime Contacts Configured         : 1

 

We can check if the URL is present in the agent:

oracle@cloud13c:/opt/oracle/agent13c/agent_inst/sysman/config/ [agent13c] grep 8081 emd.properties
EMS_URL=https://cloud13c:8081/upload

 

It is also mandatory to enable the notifications (by default it is not …), the enable notification command will automatically run an incremental sync:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl enable_notification
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notifications have been enabled. There are downtime contacts configured.
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: sysman@cloud13c:1521:EMREP13C
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Fri Feb 12 13:31:40 CET 2016

We can easily check if AOM is running:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/scripts/ [oms13c] ./emsctl ping
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring is running.

 

The logfiles are located in $OMS_HOME/sysman/ems/ems/log:

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/log/ [oms13c] ls
emsca.err emsca.log.0 emsctl.err emsctl.log.0 ems.err ems.log.0 ems.log.0.lck ems.out

 

Finally we can test:

We shutdown the OMS

oracle@cloud13c:/opt/oracle/Middleware13c/sysman/ems/ems/log/ [oms13c] emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
Stopping BI Publisher Server...
BI Publisher Server Already Stopped
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
AdminServer Successfully Stopped
BI Publisher Server is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command

 

We stop our TESTDBA database, and we receive a mail telling us our database target is down:

aom1

Conclusion:

AOM is a nice feature allowing you to continue to receive critical alerts from your database targets when OMS is down or under maintenance.

 

 

Leave a Reply

Pierre Sicot
Pierre Sicot

Senior Consultant