Error Deploying Applications in Config Tool

When trying to deploy to application server, config tool crashes with "An unexpected error occurred and the application will have to shut down. Review the error log, correct the error condition and re-launch the application."





The first thing to check is the config tool log. Digging into the log reveals,

[SRC_CLASS: com.hyperion.hit.wizard.Wizard] trace: Unexpected error:[[
java.lang.NullPointerException
at com.hyperion.hit.tool.deploy.utils.UIPortUtil.resetData(UIPortUtil.java:183)
at com.hyperion.hit.tool.deploy.utils.UIPortUtil.<init>(UIPortUtil.java:65)
at com.hyperion.hit.tool.deploy.utils.UIPortUtil.getInstance(UIPortUtil.java:48)
at com.hyperion.config.wizard.DeploymentField.getAppServerDeploymentData(DeploymentField.java:309)
at com.hyperion.config.wizard.DeploymentField.<init>(DeploymentField.java:70)
at com.hyperion.config.wizard.impl.DeploymentForm.<init>(DeploymentForm.java:57)
at com.hyperion.config.wizard.impl.DeploymentState.initializeBeforeShowOnNext(DeploymentState.java:58)
at com.hyperion.hit.wizard.Wizard$NextClass.run(Wizard.java:551)
at java.lang.Thread.run(Thread.java:662)
]]

Upon further investigation - this logic is basically trying to enumerate the logical web applications for each host. While iterating through the logical web applications, the config tool encounters a null value and crashes. The logical web applications are found in the Shared Services Registry. So I ran the registry-cleanup utility,

NOTE: The registry-cleanup utility has a high probability of corrupting your environment. Take a database backup of your Shared Services/Foundation schema before starting and be prepared to take downtime to restore the schema if required.
Essentially, the cleanup utility found some invalid entries to delete - so I removed them one at a time until the config tool worked again.

registry-cleanup.bat
Launching the Hyperion Registry Cleanup Utility, please wait...
Enter the password for Shared Services Database :
Reading rules XML file: \Oracle\Middleware\EPMSystem11R1\common\config\11.1.2.
0\resources\registry\cleanup\default-rules.xml
Found 6 rules

Looking for components matching rule: Remove components without parent HOST node
Processing results for rule: Remove components without parent HOST node

Do you really want to delete WORKSPACE_WEBAPP(WORKSPACE_WEBAPP):808d62cd746781a0
S201b96e113e13916fd3S6ae7 component?
Y - Yes
A - Yes, for all components matching this rule
N - No, skip this component
S - No, skip this and all the next components matching this rule
I - Show full component info
y
Deleting: WORKSPACE_WEBAPP(WORKSPACE_WEBAPP):808d62cd746781a0S201b96e113e13916fd
3S6ae7


This fixed the error in config tool.

Learning EPM - Level Up

I have been spending time lately focused on how to bring seasoned IT folks up to speed in Oracle EPM. I wanted to help identify specific levels of familiarity with EPM and help folks who are in one tier have resources to help go into more depth in their quest for EPM knowledge. The following post is a work in progress. The expectation is that a more polished version will eventually end up as part of a future presentation.

Levels of EPM Skills

Basic Operational Skills -
    No expectation prior EPM knowledge
   This is good for the typical "on call" resource who is not specializing in EPM.

1. Familiar with start and stop operations - specific to company
2. Familiar with the architecture diagrams for each ENV - specific to company
3. What's up and whats down - able to identify products that are not running and use the startup script to start or restart components - familiar with monitoring specific to company

Basic EPM Admin Skills

1. Has logged in to each EPM server
2. Has admin access to each product and has logged into each EPM module
3. Familiar with EPM modules
   Knows the basic end user functionality of each EPM module
   Comfortable explaining end user interactions
   Can explain the importance of EPM regarding the business and why it is critical for good infrastructure support
4. Knowledge of jobs which run in the environment - LCM exports, FDM loads, etc. - specific to company
5. Ability to execute basic validation procedures to determine if the ENV is working as expected after planned maintenance or during issue resolution.
6. Traversing the Oracle documentation - Read up!
   http://docs.oracle.com/cd/E17236_01/index.htm
   
   EPM Module Admin Guides
   example: http://docs.oracle.com/cd/E17236_01/epm.1112/esb_dbag.pdf
   EPM Module User Guides
   example: http://docs.oracle.com/cd/E17236_01/epm.1112/sv_user.pdf
   
   Infrastructure:
   -Support Matrix:   http://docs.oracle.com/cd/E17236_01/epm.1112/epm_install_start_here_1112200.pdf
   -Installation and Configuration
   http://docs.oracle.com/cd/E17236_01/epm.1112/epm_install_1112200.pdf
   -High Availability and Disaster Recovery
   http://docs.oracle.com/cd/E17236_01/epm.1112/epm_high_avail_11121.pdf
   -Security and SSL
   http://docs.oracle.com/cd/E17236_01/epm.1112/epm_security_11121.pdf
7. Experience Installing and Configuring basic EPM environments

Intermediate EPM Skills

1. Understands Opatch
   Can evaluate patch readme's
   Understands patch prerequisites and orders of patches
   Knows difference between patch sets vs one off patches.
   Understands where the patches should be applied - potentially 3 different places
      Product web
      Product app
      WebLogic admin node
2. Understands implications of running config tool
    Resets registry heap settings
    Potentially wipes out custom changes
3. Understands load balancing and SSL setup
   Key files:
      ssl.conf - basic SSL settings - defines location of wallet and SSL port
      mod_wl_ohs.conf - The Web->WebLogic forwarding
      httpd.conf - Web->IIS fowarding
   Files are auto generated when running web configuration in config tool - custom changes are blown away
4. Knows what happens behind the scenes during the proces of deploying an application in config tool
5. Can perform SSL certificate rotation based on predefined procedures
6. Can perform password rotations
   a. database accounts
   b. EPM system accounts (admin, essadmin, jobs)
   c. windows DCOM account
7. Familiar with oracle support - able to follow tickets from inception to resolution based on oracle's instructions.
8. Knowledge of log files locations to troubleshoot issues.

Advanced Skills

1. Can put together instructions for SSL implementation and rotation
2. Can create architecture to implement high availability and the architecture can be tested to achieve high availability
3. Create and refine DR process for EPM
4. Troubleshooting
   Quickly understands interaction between products and can evaluate error messages from logs to quickly resolve issues
5. Able to plan and implement a upgrade from one EPM version to another
6. Begins to train others in the art of EPM infrastructure

EPM Guru (subjective)

1. Ability to reach out and solve others problems
2. Takes responsibility for infrastructure issues
3. Multitasking - understands most EPM modules and is able to help on every front 
4. Begins to see EPM at the "code" level - can start to evaluate why issues are occurring based on how the code is structured.
5. Historical reference of EPM - watching EPM evolve over time and multiple versions can help bring invaluable context to situations.

EPM Config Tool: The next hosts are unreachable.

When opening the config tool in 11.1.2.2 the utility actually tries to connect to every host in the EPM Registry. It show the error message if any hosts are unreachable.

I started down this path,

Checking closer it appears that the config tool uses the Java InetAddress isReachable() method.
Looking into the doc on the class,
http://docs.oracle.com/javase/6/docs/api/java/net/InetAddress.html

public boolean isReachable(int timeout)
                    throws IOException

Test whether that address is reachable. Best effort is made by the implementation to try to reach the host, but firewalls and server configuration may block requests resulting in a unreachable status while some specific ports may be accessible. A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.

I wrote up a simple test which confirmed the issue (Can't connect):

import java.net.InetAddress;

public class test

{

    public static void main(String[] args)

    throws Exception

    {

  

     InetAddress address =  InetAddress.getByName("hostname.fullyqualified.com");

     if (address == null) {

         System.out.println("Is null");

     }

     else if (! address.isReachable(3000)) {

         System.out.println("Cant connect");

     }

     else { 

         System.out.println("Host is reachable");

     }

     }

}

The host is pingable so ICMP traffic is being allowed. It's puzzling why the Java method cannot connect.

Further research on the web brings up some articles regarding why the isReachable method might fail, 

http://stackoverflow.com/questions/5126697/java-networking-issue

So this sort of pinpoints a firewall issue - Next I disabled the Windows Firewall - The method began working! 

About this time I came across John Goodwin's article while researching:

http://john-goodwin.blogspot.com/2013/01/epm-11122-configurator-host-unreachable.html

John had gone down almost the exact same path, written up nicely in his blog.

Final thoughts: 

  Why does Oracle introduce a new check in the config tool using a strange ICMP/TCP method that has nothing to do with the actual requirements of EPM? Crazy... Assuming you are implementing firewalls - you will need to include this as an additional requirement. 

Browser Setting Recommendations for 11.1.2.2

I have been working with users facing strange issues in their browser for 11.1.2.2. These range from stuck "Loading..." prompts in Planning, Missing Arrows in Shared Services and disabled menus in DRM. This post will explain some recommended browser settings to prevent such issues.

The first main setting is to disable compatibility mode in Internet Explorer. This applies for both IE8 and IE9, though the steps are a slight bit different.

To make this harder - IE comes with a not well known default setting that automatically enables compatibility mode for intranet sites - where Hyperion typically resides. Looking carefully at the dialog box below see the option by default "Display intranet sites in Compatibility View". This should be unchecked for Hyperion! Hyperion will not work in Compatibility Mode.

Under Internet Explorer:
  Page -> Compatibility View Settings, Uncheck the box "Display intranet sites in Compatibility View".Here is an additional link to help for IE9 users to get the "Page" menu to display:
Disabling Compatibility Mode in IE9

Also, the following KB article has a ton of other recommended settings such as disabling popups, setting Hyperion as a trusted site, and lowering your security settings to allow specific functions:
      Oracle support Doc "Internet Explorer (IE7, IE8 and IE9) Recommended Settings for Oracle Hyperion Products [ID 820892.1]"

EPM System Configurator Crashes During EPMA Configure Database

I just saw this one on the forums. Since I spent some time scratching my head on the issue and others are starting to see it I thought I would write it up...

In a patched 11.1.2.2 environment while configuring the EPMA database connection in config tool the config tool will crash.

In the configtool.log you would find:
[2013-01-30T11:34:00.420-05:00] [EPMCFG] [TRACE] [EPMCFG-01758] [oracle.EPMCFG] [tid: 16] [ecid: 0000JmCJgspEwGIFmrADS31H2KhK000003,0] [SRC_CLASS: com.hyperion.config.wizard.impl.DBConnectionForm] DB type is Oracle
[2013-01-30T11:34:00.420-05:00] [EPMCFG] [ERROR] [EPMCFG-01020] [oracle.EPMCFG] [tid: 16] [ecid: 0000JmCJgspEwGIFmrADS31H2KhK000003,0] [SRC_CLASS: com.hyperion.cis.config.Configurator] Error: [[
java.lang.NoClassDefFoundError: com/hyperion/cis/ISMPWizardServices
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at com.hyperion.cis.config.Configurator.loadClass(Configurator.java:172)
at com.hyperion.cis.config.Configurator.loadProductDBConfigurators(Configurator.java:201)
at com.hyperion.config.wizard.impl.DBWizardState$1.validate(DBWizardState.java:348)
at com.hyperion.hit.wizard.Wizard$NextClass.run(Wizard.java:531)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.hyperion.cis.ISMPWizardServices
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 7 more

]]



The reason is that the Oracle patch has broken the jar file AWBConfig.jar located
d:\oracle\Middleware\EPMSystem11R1\products\Foundation\BPMA\Common

The broken EPMA patch number is 14031912

There are missing classes in the patched , newer version of this jar

To fix the config tool you can make a copy of this existing jar and replace the older unpatched version. To find the old version search,
\oracle\Middleware\EPMSystem11R1\.patch_storage\ for AwbConfig.jar

To confirm this, look at the size of the current jar and the older jar - the new one is 158k and the old one was 317k - obviously the new one is missing some classes.

Once you are done with the config tool put the "newer" one back.

Error Saving User Preferences in Planning

While performing an upgrade to the planning application, the new application is facing an error saving the user preferences in planning. 

When trying to save the user preferences the following end user error is displayed:

"The SQL operation failed. Check logs for details.

User Preferences have been saved."

Diving into the Planning logs identifies the actual error message:

java.sql.SQLSyntaxErrorException: ORA-00904: "ENABLE_MRU_SCREEN": invalid identifier


This error was found searching the knowledge base, 
Unable to Save Form Preference in Planning After Upgrade [ID 889744.1]


According to Oracle:
This issue has been caused by migrating the application and a column must be added to HSP_USER_PREF table.

One would think that when you upgrade the application, the schema update it performs when logging in for the first time would add this column, nevertheless...

The following SQL was executed against the planning application database to add the appropriate column.

ALTER TABLE HSP_USER_PREFS ADD ENABLE_MRU_SCREEN INTEGER DEFAULT 1 NOT NULL

epmsys_registry Command Line Options

The epmsys_registry command is the key to manipulating the insides of the EPM System. However, it is not well documented. Here are a few ways to further explore the tool...

A command that is useful is the host report. This report can show you what hosts are stored inside the epm registry and what those hosts are being resolved to on each node in the environment. This is useful if you have DNS names or virtual names inside the registry and how those names are being resolved on the host itself. Over time, host resolution may change based on host file entries, DNS changes, failover scenarios, load balancing...etc. The report can identify host related issues buried inside the registry.

epmsys_registry viewhosts

Hosts in EPM Registry

HOST NAME IN EPM REGISTRY           HOST NAME AS RESOLVED ON THIS MACHINE

*************************                            *************************************

1       host1.full                                      host1.full

2       VirtualIP                                       host2.full

Generate a deployment report

  - Creates a report enumerating your web deployments and a history of deployment activities.

epmsys_registry report deployment

Components of the report:

• Logical Web Addresses
• Application Tier Components
• EPM Deployment Topology Report 
• EPM Deployment History Report
• User Directories - SSO Not Enabled
• Database Connections
• Data Directories

Sample Command Line Usage:

epmsys_registry deletecomponent #8dafd4d781a01633a4a713b43baa1b1S6c71
    The component has been removed. However the linked components remain.

epmsys_registry view FOUNDATION_SERVICES_PRODUCT

Change the database password in the EPM Registry:
epmsys_registry updateencryptedproperty #64f85ee85aef2a62eccaf7d13c02cb3445S76a3/@dbPassword testpw
    The new value of the property has been encrypted and updated on the component.

Hierarchical changes:

epmsys_registry createlink #18349408fd0bc34c2226d26dafds88d019S7a87 #834940894494fdcS6fe8c90313b3e28c59dS6d7f

-- first argument is parent component , second is the child link that will be removed. epmsys_registry removelink #18349408fd0bc34dfda33b3e28c59dS6d7f #8349408fd0bc34addd26213aad88d019S7a87

Full List of Command Line Options

 view

 viewencryptedproperty (and yes this does display the unencrypted password)

 viewhierarchy

 addproperty

 updateproperty

 removeproperty

 addencryptedproperty

 updateencryptedproperty

 addfile

 updatefile

 removefile

 createlink

 removelink

 createcomponenthierarchy

 deletecomponent

 updatehost