Securing a thin client from WAS to Oracle

My current client is a large user of WAS and Oracle but had not until a few weeks ago set up any SSL connections between the 2 of them. When I came to do this there appeared to be very little documentation on turning on SSL for a thin client and the little I did find tended to be more about coding within an application than within an application server.

This blog proved a useful starting point:

http://blog.anowak.net/2009/09/configuring-encrypted-connection.html

However, it still didn't appear to work.

After a fair bit of testing and tracing it turns out it was in fact the oracle.net.ssl_version setting mentioned in Artur Novaks blog that resolved the
issue in the end but it appeared the negotiation of the SSL version between WAS and Oracle does not work as documented.

I am not sure if this how WAS and Oracle communicate at all versions or if it was just our specific set up.

We had enabled SSL on our Oracle DB which was running version 11.1.0.7.
Within our WAS server we had the latest ojdbc5 driver file which was v
11.2.0.2. The WAS version we were running was 6.1.0.31


Within the oracle DB, the SSL version was left unspecified which by default should have meant that the highest level of SSL should have been negotiated - starting at TLSv1, then SSLv3 and then SSLv2. Or at least that was how I read it in the Oracle documentation.

By setting the custom property in WAS to have a trust store but no ssl version specified we got the following error:

[23/09/11 14:11:13:593 BST] 0000002f DataSourceCon E DSRA8040I: Failed to connect to the DataSource. Encountered "": java.sql.SQLException: IO Error: The Network Adapter could not establish the connectionDSRA0010E: SQL State = 08006,
Error Code = 17,002

followed by

Caused by: oracle.net.ns.NetException: The ssl protocol specified is not
supported.
at
oracle.net.nt.TcpsConfigure.configureVersion(TcpsConfigure.java:181)
at
oracle.net.nt.TcpsNTAdapter.setSSLSocketOptions(TcpsNTAdapter.java:188)
at oracle.net.nt.TcpsNTAdapter.connect(TcpsNTAdapter.java:138)
at oracle.net.nt.ConnOption.connect(ConnOption.java:123)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:353)
... 101 more
Caused by: java.lang.IllegalArgumentException: SSLv2Hello
at com.ibm.jsse2.mb.a(mb.java:30)
at com.ibm.jsse2.lb.(lb.java:4)
at com.ibm.jsse2.jc.setEnabledProtocols(jc.java:570)
at
oracle.net.nt.TcpsConfigure.configureVersion(TcpsConfigure.java:177)


So for some reason TLS and SSL v3 were not tried.

I then tried WAS set to oracle.net.ssl_version=3.0 but that resulted in the following:

[23/09/11 14:12:36:529 BST] 00000031 DataSourceCon E DSRA8040I: Failed
to connect to the DataSource. Encountered "": java.sql.SQLException: IO Error: Received fatalalert: handshake_failureDSRA0010E: SQL State = 08006, Error Code = 17,002

But I didn't get much more in any of the standard output files so I turned on tracing with SSL=all and a custom property on the jvm of javax.net.debug
showed me the handshake going on. These couple of lines show WAS reading sslv3 but receiving a TLSv1 format but then going straight into a failure.

[23/09/11 14:16:42:389 BST] 00000033 SystemOut O WebContainer : 0, READ: SSLv3 Alert, length = 2
[23/09/11 14:16:42:390 BST] 00000033 SystemOut O WebContainer : 0, RECV TLSv1 ALERT: fatal, handshake_failure


So it appears Oracle was sending a handshake as TLSv1, WAS was expecting this in SSLv3 format, but rather than negotiating to use sslv3 it just gives us the handshake error.

Setting the SSL version to TLSv1 wasn't documented so I just set:

oracle.net.ssl_version=1.0;

and this worked as you see WAS reading TLSv1

[23/09/11 14:19:10:891 BST] 00000033 SystemOut O WebContainer : 0, READ: TLSv1 Handshake, length = 97

as well as allowing me to run a successful test connection!

So my connection url string in the end was just this:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=mydbhost)(PORT=2484))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=mydbservice)))

and then set a custom property of "connectionProperties" to the following value:

"oracle.net.ssl_version=1.0;javax.net.ssl.trustStore=/usr/wps6.1/WebSphere/ProcServer/profiles/myprofile/config/cells/myCell/trust.p12; javax.net.ssl.trustStoreType=PKCS12; javax.net.ssl.trustStorePassword=password"

Deployment error on WAS 6.1.0.39

Since upgrading our WAS environment a few weeks ago to WAS 6.1.0.39 from 6.1.0.31 we were getting errors anytime we tried to deploy an application ear file.

The error we were getting was:

Caused by: org.apache.commons.logging.LogConfigurationException: org.
apache.commons.logging.LogConfigurationException: org.apache.commons.
logging.LogConfigurationException: Class org.apache.commons.logging.
impl.Jdk14Logger does not implement Log

Which looked pretty much like this PMR https://www-304.ibm.
com/support/docview.wss?uid=swg21502693 although that only related to WAS 7 and the workaround documented wasn't available at 6.1

IBM did some investigation and a fix was targetted for FP 43 which was of course no use to me. They did provide a temp fix in the shape of IFPM45666 but that didn't resolve the issue.

After having initially to go back to FP37, we were finally provided a workaround that allows us to go back to FP39.

In /deploytool/itp/ejbdeploy.sh by adding -DNoCustomConverter=true we were able to succesfully deploy our applications again.

Display JMS connection pool contents

Recently I have been having problem with an application seeming to use up all the available connections in an MQ connection pool. To enable me to show what was going on to the developers I needed to create a jython script to show the pool contents and how long the connections had been in use for.

The connections in the pool did not appear to be getting freed up even when the purge timer was reached, so we set up the stuck timer under connection pool ==> advance settings.

This at least provided warnings in the logs that there were stuck threads - "A stuck connection is an active connection that is not responding or returning to the connection pool."

This provided some messages in the log to prove what we expected but as tends to be the case this wasn't sufficient. So I set up the following script to run at regular intervals:

ps = AdminControl.queryNames ('WebSphere:type=J2CConnectionFactory,process=server1,*').splitlines()
for p in ps:
print p
pc=AdminControl.invoke( p , "showPoolContents" )
print pc
This gave all sorts of useful information including details on each connection:

MCWrapper id 7d597d59 Managed connection com.ibm.ejs.jms.JMSManagedQueueConnection@6b376b37
managed connection factory = com.ibm.ejs.jms.WSJMSManagedQueueConnectionFactory@50f350f3
physical connection = com.ibm.mq.jms.MQXAQueueConnection@6beb6beb
credential = null
open connection handles = [com.ibm.ejs.jms.JMSQueueConnectionHandle@48324832] State:STATE_TRAN_WRAPPER_INUSE Start time inuse Thu Jun 16 18:21:24 BST 2011 Time inuse 53119 (seconds)

The proof I was after was the start time - this tended to be when the app was first run and would just hang around.

Even running the following to purge the pool did not help:

AdminControl.invoke( p , "purgePoolContents" )

The showpoolcontents would show the same information with the following line added:

Connection marked to be destroyed. Waiting for transaction end and connection close.

With this information to hand the developers finally agreed to check out there code!




n.b. If you need to view a database connection pool instead of a JMS pool, change the type in the querynames command to type=DataSource,*

Change order of JAAS login modules

I have recently been installing some 3rd party software into a WPS environment. Part of the install instructions was to add some new JAAS login modules but also to change the order of these and the existing modules.

Unfortunately, the order of these, even though you can change this via the amdin console, is not an attribute and is determined by the order they appear in the security.xml file.

With a bit of googling it is easy enough to find out how to create the new modules, so the only way to script the whole process that I could see was to delete the existing modules and then add them again but in the correct order:

import java.lang.System as sys
sec = AdminConfig.list("Security")
slc = AdminConfig.showAttribute(sec, "systemLoginConfig")
entries = AdminConfig.showAttribute(slc, "entries")
entries = entries.replace('[','')
entries = entries.replace(']','')
entries = entries.split(' ')
for entry in entries:
print entry
alias = AdminConfig.showAttribute(entry, "alias")
if ( alias != "WEB_INBOUND" ) :
print "Not the module we are interested in"
else:
print "Changing mods"
loginMods = AdminConfig.showAttribute(entry, "loginModules")
loginMods = loginMods.replace('[','')
loginMods = loginMods.replace(']','')
loginMods = loginMods.split(' ')
for loginMod in loginMods:
print "Deleting login module " + loginMod
AdminConfig.remove(loginMod)

nmid="com.myco.1st.jaas.class.name"
newModuleId = AdminConfig.create("JAASLoginModule", entry, [["moduleClassName", nmid ]])
AdminConfig.modify( newModuleId , [["authenticationStrategy", "REQUIRED" ]] )
#

nmid="com.myco.2nd.jaas.class.name"
newModuleId = AdminConfig.create("JAASLoginModule", entry, [["moduleClassName", nmid ]])
AdminConfig.modify( newModuleId , [["authenticationStrategy", "REQUIRED" ]] )
#

AdminConfig.save()

IBM WebSphere Application Server support on POWER7 hardware

I have just had a fun few days istalling WebSphere process server on a client site. They were installing this on Power7 hardware which caused us an issue. Simply trying to run any install commands on Process server 6.1 caused a java core and a snap dump and output similar to this:

Unhandled exception Type=Segmentation error vmState=0x00000000 J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033 Handler1=09001000A03A89F0 Handler2=09001000A0574DB8 R0=09000000014F1FE8 R1=0FFFFFFFFFFFE4E0 R2=07000000002DBE30 R3=000000000000000B R4=0000000000000000 R5=09000000014F1FE8 R6=000000011191E673 R7=000000011191E668 R8=000000011191E670 R9=0000000037CE0000 R10=000000011181BB70 R11=0900000001507F50 R12=090000000140B90C R13=000000011000EB20 R14=000000011191E658 R15=00000001104E4800 R16=0000000110D7B230 R17=00000001118199F8 R18=09001000A03AC2C8 R19=0000000000000021 R20=0000000111819AB8 R21=000000011191E690 R22=0700000000447E70 R23=0700000000447EA0 R24=0700000000448100 R25=0000000112067450 R26=07000000002D8188 R27=07000000002D8878 R28=0000000000000000 R29=0000000000000001 R30=07000000002D8A68 R31=07000000002DBE30 IAR=0900000001507F50 LR=00000001105A4EE0 MSR=A00000000000D032 CTR=0900000001507F50 CR=4254543420000004 FPSCR=8200000000000000 XER=2000000482000000 FPR0 0000001800000018 (f: 24.000000, d: 5.092790e-313)

It turns out it wasn't the WPS install but the underlying WAS java version that was causing the issue. The quick and easy way around this was to set:

export JAVA_COMPILER=NONE for the user that was running the install.

This link gives more details of supported versions of WebSphere on Power 7 kit http://www-01.ibm.com/support/docview.wss?uid=swg21422150

Redirecting heapdumps for WAS on zOS

I have recently been doing a fair bit of WAS work on zOS, going back to where it started for me in the WebSphere world.

On one of the systems we had an application that was running out of memory regularly, creating heapdumps and then filling the filesystems so WAS wasn't able to restart itself. To get round this we created a new large filesystem and redirected the heapdumps to this FS using the following generic JVM argument for the servant process:

-Xdump:heap:defaults:file=/WASdumps/heapdump.%Y%m%d.%H%M%S.%pid.phd

Where /WASdumps was the new filesystem