IBM Installation Manager - Installing across the 'web' - no packages found

Recently my client decided to try and set up Installation Manager for IBM products on a webserver.   I guessed this was possible but just to make sure I spoke to my go-to expert, Dave Hay,  who pointed me at his blog  http://planetlotus.org/profiles/dave-hay_131315

Perfect!

I followed his steps to the letter but when I ran:

./imcl listAvailablePackages -repositories  http://jslwebserver/repos/WebSphere/8.5.5/Base/repository

I received the message "No packages were found"

Hmmm, how odd.  When I used IM to test connections to the repository it worked ok.  A quick look at the webserver logs showed that IM was trying to find repository.xml as well as repository.config and getting a 404 error.

A but more reading and I spotted this http://www-01.ibm.com/support/knowledgecenter/SSDV2W_1.8.2/com.ibm.cic.auth.ui.doc/topics/t_introduction.html    which states

Restriction: Installing from an HTTP server requires by using the Packaging Utility. You must use Packaging Utility to create the repository that is on the HTTP server.

Ok, so I downloaded the packaging utility and then carried out the next few steps:

1  Copied pu.offering.disk.linux.gtk.x86_64_1.8.2000.20150303_1543.zip   to /home/wasadm/PU

2   Unzipped the above file

3.  cd disk_linux.gtk.x86_64/InstallerImage_linux.gtk.x86_64/

4.    ./userinstc -acceptLicense

"Installed com.ibm.cic.packagingUtility_1.2.2000.20150303_1543 to the /home/wasadm/IBM/PackagingUtility directory.

5.    cd   /home/wasadm/IBM/PackagingUtility

6.    ./IBMPU


At which point a GUI starts up.  I selected "Copy repository" which allowed me to point at my exisitng repository.config file and provide a new location for the updated repository.

The utility then ran and in my new location, the repository now included a few new files, one of which was repository.xml

So now when I ran the following I was able to see the package


./imcl listAvailablePackages -repositories  http://jslwebserver/repos/WebSphere/8.5.5/BasePU/repository

com.ibm.websphere.BASE.v85_8.5.5000.20130514_1044


I was then able to run my install using the repository on hosted on the webserver.

Business Monitor and Cognos performance problem

Recently we have been seeing some issues with our Business Monitor servers on start up.   After running IBM Business Monitor 7.5.1 for over a year, every time we restarted the environment there was a spike in CPU consumption that lasted for over half an hour.

On closer inspection it was the cognos process spawned by Monitor that was consuming all this CPU.    A quick call to IBM support pointed us at the NC tables used by cognos.  A few of our tables had over half a million rows, where as the following document suggestes anything over 1000 could impact performance:

http://www-01.ibm.com/support/docview.wss?uid=swg21637944

So we followed the steps in the technote and cleared down the NC tables and all of a sudden there was no issue following the next restart.

We have been keeping an eye on these tables since then and noticed they do continue to grow.   Further questions to IBM support suggest this is normal behavior as long as there is no rapid increase in the size of the tables.  It just means every now and then we may have to clear down these tables again.

BPM portal images not displaying

Another battle with BPM today, this time to do with the images in Process Portal not displaying correctly.

When I tried to access the url for Process Portal  the screen displayed but some of the images/css files were missing so the page didn't look complete.

After finding this page it seemed quite a common problem

http://www-01.ibm.com/support/docview.wss?uid=swg21587668

I followed the instructions to the letter but still no joy.  A colleague mentioned it might be an issue with hostnames.  If BPM is set to a short name e.g.   mybpm.host  but the network then appends the fully qualified host name mybpm.host.mydomain.com this may cause issues.  I made sure the 99local and 100custom xml files were all using the FQDN.

Still no joy then I noticed something odd.  When logging on from my desktop the page worked correctly but when using the browser on the BPM host VM the error was still there.  A look in the http log showed that both requests looked identical and both showed up http respone code 200 for the CSS and images.

At this point I realised it might be the browser.  Good old IE.  Simply adding mybpm.host.mydomain.com  to the trusted sites fixed the problem and css files suddenly displayed correctly.

Well that was a few hours wasted!!!!

Snapshot install error

Wow, has it really been a year since my last blog post!

I have recently been having issues on my BPM 8.0.1.1 environment with some snapshot installs.
I had set up my Process server environment and initially set up a single Process server environment, using http ports and direct (not via a LB).  In this scenario everything worked as expected.

I then extended my Process server environment to a second node, and fronted it all with a webserver that was only enabled for https.

I followed the instructions in this useful infocenter document to ensure I was making the correct changes in the xml files:

http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0mx/index.jsp?topic=%2Fcom.ibm.wbpm.imuc.sbpm.doc%2Ftopics%2Ftconfig_custom_cluster_env.html&resultof%3D%2522%2577%2565%2562%2522%2520%2522%2573%2565%2572%2576%2565%2572%2522%2520

However, when I tried an install of the snapshot it failed and the SystemOut.log for Process center wasn't exactly bursting at the seams with useful information.

The only messages in the log were:
CWLLG0151I:  Standard hiring sample v801 Beginning login to remote server
CWLLG0152E:  Standard hiring sample v801 Remote login failed

I had a suspicion that despite setting <deploy-snapshot-using-https>true</deploy-snapshot-using-https>  in the Process server 10custom.xml that Process Center was still using http.  The reason I suspected this, was that in the ProcessCenter console, on the servers tab, when I clicked "Configure server" it tried to open a window to http://my-process-server:443/ProcessAdmin   i.e.   using an ssl port but on http.

A quick check in the LSW_SERVER table showed that the host name and port were correct  my-process-server:443  but there was no column in there that specified to use http or https.

So how could I prove what was happening?   I found the details of the trace that I need to set on the Process Center JVM:

com.ibm.bpm.fds.*=all:ProcessApplicationLifecycle=all:com.ibm.bpm.fds.repo.util.ContributionHelper=off:WLE.*=all

The trace output showed the following message:

com.lombardisoftware.server.ejb.repositoryservices.DeploytoServerSupport  deploySnapshot  Deploy Snapshot using :  http://

So it appears my suspicions were correct.  However, I had followed IBMs documents to the letter but PCenter was still not deploying using https.

One last thing to try was to update the 100custom.xml in Process Center so that now contained  <deploy-snapshot-using-https>true</deploy-snapshot-using-https>

A quick resynch of the node and restart of the JVMs and suddenly the deploy using https worked!

Whilst this works for me, the only downside I can see is that if you have Process Center talking to a mix of environments, they all need to be deployed to using http or all using https and doesn't appear as though you can have a mix

Connecting IBM Cognos BI to the IBM Business Monitor reporting database

The last few weeks I have been struggling to get Cognos within Business Monitor to connect to the Monitor database correctly.   After a lot of trial and error and some help from Cognos and BPM experts I have now got this working and managed to repeat the process on several environments.

The initial error appeared when trying to deploy a Monitor model - this was documented in this technote .   When logging onto the Cognos admin console, the test connection on the database would work for the thin client but fail for the thick client as this wasn't installed at the time.

As we were using a back end Oracle database I installed the Oracle instant client.   Even though I was running 64 bit WAS, Business Monitor and Oracle, the client that Cognos requires is a 32 bit version.

Once this was installed, I then set the LD_LIBRARY_PATH on the application server running the cognos dispatcher.   This is in the WAS admin console:   Application servers ==>  <server name> ==> Java and process management ==> Process definition ==> Environment entries

I added the <oracle_client_dir>/lib to the LB_LIBRARY_PATH.  Just adding the install dir didn't fix the problem and it appears you need to add the /lib so WAS knows where to find the .so file.

As the processes were running as non-root I needed to ensure that the JVMs could access the oracle directories so in the profile of the WAS user I added:

export ORACLE_HOME=/u001/app/oracle/product/11.2.0/client_1
export ORACLE_SID=myDB
export ORACLE_BASE=/u001/app/oracle
export TNS_ADMIN=/u001/app/oracle/product/11.2.0/client_1/network/admin
export TWO_TASK=myDB

After this a restart of the environment meant I could successfully run a test connection with the cognos admin console. 

WebSphere Request for Enhancments

If any of you have ever requested an enhancement to Websphere, you will know in the past this hasn't been the easiest process.  Having to go through your account manager and then not knowing whether anyone is actually looking at your request, well now the process has become a lot simpler and more transparent.

There is now a RFE page on the developerworks site:

https://www.ibm.com/developerworks/rfe/

You can submit your RFE to IBM as well as searching RFEs that have already been submitted.  There is also an ability to vote for RFEs so the product team know which features are really important to the end users.

Securing a thin client from WAS to Oracle

My current client is a large user of WAS and Oracle but had not until a few weeks ago set up any SSL connections between the 2 of them. When I came to do this there appeared to be very little documentation on turning on SSL for a thin client and the little I did find tended to be more about coding within an application than within an application server.

This blog proved a useful starting point:

http://blog.anowak.net/2009/09/configuring-encrypted-connection.html

However, it still didn't appear to work.

After a fair bit of testing and tracing it turns out it was in fact the oracle.net.ssl_version setting mentioned in Artur Novaks blog that resolved the
issue in the end but it appeared the negotiation of the SSL version between WAS and Oracle does not work as documented.

I am not sure if this how WAS and Oracle communicate at all versions or if it was just our specific set up.

We had enabled SSL on our Oracle DB which was running version 11.1.0.7.
Within our WAS server we had the latest ojdbc5 driver file which was v
11.2.0.2. The WAS version we were running was 6.1.0.31


Within the oracle DB, the SSL version was left unspecified which by default should have meant that the highest level of SSL should have been negotiated - starting at TLSv1, then SSLv3 and then SSLv2. Or at least that was how I read it in the Oracle documentation.

By setting the custom property in WAS to have a trust store but no ssl version specified we got the following error:

[23/09/11 14:11:13:593 BST] 0000002f DataSourceCon E DSRA8040I: Failed to connect to the DataSource. Encountered "": java.sql.SQLException: IO Error: The Network Adapter could not establish the connectionDSRA0010E: SQL State = 08006,
Error Code = 17,002

followed by

Caused by: oracle.net.ns.NetException: The ssl protocol specified is not
supported.
at
oracle.net.nt.TcpsConfigure.configureVersion(TcpsConfigure.java:181)
at
oracle.net.nt.TcpsNTAdapter.setSSLSocketOptions(TcpsNTAdapter.java:188)
at oracle.net.nt.TcpsNTAdapter.connect(TcpsNTAdapter.java:138)
at oracle.net.nt.ConnOption.connect(ConnOption.java:123)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:353)
... 101 more
Caused by: java.lang.IllegalArgumentException: SSLv2Hello
at com.ibm.jsse2.mb.a(mb.java:30)
at com.ibm.jsse2.lb.(lb.java:4)
at com.ibm.jsse2.jc.setEnabledProtocols(jc.java:570)
at
oracle.net.nt.TcpsConfigure.configureVersion(TcpsConfigure.java:177)


So for some reason TLS and SSL v3 were not tried.

I then tried WAS set to oracle.net.ssl_version=3.0 but that resulted in the following:

[23/09/11 14:12:36:529 BST] 00000031 DataSourceCon E DSRA8040I: Failed
to connect to the DataSource. Encountered "": java.sql.SQLException: IO Error: Received fatalalert: handshake_failureDSRA0010E: SQL State = 08006, Error Code = 17,002

But I didn't get much more in any of the standard output files so I turned on tracing with SSL=all and a custom property on the jvm of javax.net.debug
showed me the handshake going on. These couple of lines show WAS reading sslv3 but receiving a TLSv1 format but then going straight into a failure.

[23/09/11 14:16:42:389 BST] 00000033 SystemOut O WebContainer : 0, READ: SSLv3 Alert, length = 2
[23/09/11 14:16:42:390 BST] 00000033 SystemOut O WebContainer : 0, RECV TLSv1 ALERT: fatal, handshake_failure


So it appears Oracle was sending a handshake as TLSv1, WAS was expecting this in SSLv3 format, but rather than negotiating to use sslv3 it just gives us the handshake error.

Setting the SSL version to TLSv1 wasn't documented so I just set:

oracle.net.ssl_version=1.0;

and this worked as you see WAS reading TLSv1

[23/09/11 14:19:10:891 BST] 00000033 SystemOut O WebContainer : 0, READ: TLSv1 Handshake, length = 97

as well as allowing me to run a successful test connection!

So my connection url string in the end was just this:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=mydbhost)(PORT=2484))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=mydbservice)))

and then set a custom property of "connectionProperties" to the following value:

"oracle.net.ssl_version=1.0;javax.net.ssl.trustStore=/usr/wps6.1/WebSphere/ProcServer/profiles/myprofile/config/cells/myCell/trust.p12; javax.net.ssl.trustStoreType=PKCS12; javax.net.ssl.trustStorePassword=password"