Deployment error on WAS 6.1.0.39

Since upgrading our WAS environment a few weeks ago to WAS 6.1.0.39 from 6.1.0.31 we were getting errors anytime we tried to deploy an application ear file.

The error we were getting was:

Caused by: org.apache.commons.logging.LogConfigurationException: org.
apache.commons.logging.LogConfigurationException: org.apache.commons.
logging.LogConfigurationException: Class org.apache.commons.logging.
impl.Jdk14Logger does not implement Log

Which looked pretty much like this PMR https://www-304.ibm.
com/support/docview.wss?uid=swg21502693 although that only related to WAS 7 and the workaround documented wasn't available at 6.1

IBM did some investigation and a fix was targetted for FP 43 which was of course no use to me. They did provide a temp fix in the shape of IFPM45666 but that didn't resolve the issue.

After having initially to go back to FP37, we were finally provided a workaround that allows us to go back to FP39.

In /deploytool/itp/ejbdeploy.sh by adding -DNoCustomConverter=true we were able to succesfully deploy our applications again.

Display JMS connection pool contents

Recently I have been having problem with an application seeming to use up all the available connections in an MQ connection pool. To enable me to show what was going on to the developers I needed to create a jython script to show the pool contents and how long the connections had been in use for.

The connections in the pool did not appear to be getting freed up even when the purge timer was reached, so we set up the stuck timer under connection pool ==> advance settings.

This at least provided warnings in the logs that there were stuck threads - "A stuck connection is an active connection that is not responding or returning to the connection pool."

This provided some messages in the log to prove what we expected but as tends to be the case this wasn't sufficient. So I set up the following script to run at regular intervals:

ps = AdminControl.queryNames ('WebSphere:type=J2CConnectionFactory,process=server1,*').splitlines()
for p in ps:
print p
pc=AdminControl.invoke( p , "showPoolContents" )
print pc
This gave all sorts of useful information including details on each connection:

MCWrapper id 7d597d59 Managed connection com.ibm.ejs.jms.JMSManagedQueueConnection@6b376b37
managed connection factory = com.ibm.ejs.jms.WSJMSManagedQueueConnectionFactory@50f350f3
physical connection = com.ibm.mq.jms.MQXAQueueConnection@6beb6beb
credential = null
open connection handles = [com.ibm.ejs.jms.JMSQueueConnectionHandle@48324832] State:STATE_TRAN_WRAPPER_INUSE Start time inuse Thu Jun 16 18:21:24 BST 2011 Time inuse 53119 (seconds)

The proof I was after was the start time - this tended to be when the app was first run and would just hang around.

Even running the following to purge the pool did not help:

AdminControl.invoke( p , "purgePoolContents" )

The showpoolcontents would show the same information with the following line added:

Connection marked to be destroyed. Waiting for transaction end and connection close.

With this information to hand the developers finally agreed to check out there code!




n.b. If you need to view a database connection pool instead of a JMS pool, change the type in the querynames command to type=DataSource,*

Change order of JAAS login modules

I have recently been installing some 3rd party software into a WPS environment. Part of the install instructions was to add some new JAAS login modules but also to change the order of these and the existing modules.

Unfortunately, the order of these, even though you can change this via the amdin console, is not an attribute and is determined by the order they appear in the security.xml file.

With a bit of googling it is easy enough to find out how to create the new modules, so the only way to script the whole process that I could see was to delete the existing modules and then add them again but in the correct order:

import java.lang.System as sys
sec = AdminConfig.list("Security")
slc = AdminConfig.showAttribute(sec, "systemLoginConfig")
entries = AdminConfig.showAttribute(slc, "entries")
entries = entries.replace('[','')
entries = entries.replace(']','')
entries = entries.split(' ')
for entry in entries:
print entry
alias = AdminConfig.showAttribute(entry, "alias")
if ( alias != "WEB_INBOUND" ) :
print "Not the module we are interested in"
else:
print "Changing mods"
loginMods = AdminConfig.showAttribute(entry, "loginModules")
loginMods = loginMods.replace('[','')
loginMods = loginMods.replace(']','')
loginMods = loginMods.split(' ')
for loginMod in loginMods:
print "Deleting login module " + loginMod
AdminConfig.remove(loginMod)

nmid="com.myco.1st.jaas.class.name"
newModuleId = AdminConfig.create("JAASLoginModule", entry, [["moduleClassName", nmid ]])
AdminConfig.modify( newModuleId , [["authenticationStrategy", "REQUIRED" ]] )
#

nmid="com.myco.2nd.jaas.class.name"
newModuleId = AdminConfig.create("JAASLoginModule", entry, [["moduleClassName", nmid ]])
AdminConfig.modify( newModuleId , [["authenticationStrategy", "REQUIRED" ]] )
#

AdminConfig.save()

IBM WebSphere Application Server support on POWER7 hardware

I have just had a fun few days istalling WebSphere process server on a client site. They were installing this on Power7 hardware which caused us an issue. Simply trying to run any install commands on Process server 6.1 caused a java core and a snap dump and output similar to this:

Unhandled exception Type=Segmentation error vmState=0x00000000 J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033 Handler1=09001000A03A89F0 Handler2=09001000A0574DB8 R0=09000000014F1FE8 R1=0FFFFFFFFFFFE4E0 R2=07000000002DBE30 R3=000000000000000B R4=0000000000000000 R5=09000000014F1FE8 R6=000000011191E673 R7=000000011191E668 R8=000000011191E670 R9=0000000037CE0000 R10=000000011181BB70 R11=0900000001507F50 R12=090000000140B90C R13=000000011000EB20 R14=000000011191E658 R15=00000001104E4800 R16=0000000110D7B230 R17=00000001118199F8 R18=09001000A03AC2C8 R19=0000000000000021 R20=0000000111819AB8 R21=000000011191E690 R22=0700000000447E70 R23=0700000000447EA0 R24=0700000000448100 R25=0000000112067450 R26=07000000002D8188 R27=07000000002D8878 R28=0000000000000000 R29=0000000000000001 R30=07000000002D8A68 R31=07000000002DBE30 IAR=0900000001507F50 LR=00000001105A4EE0 MSR=A00000000000D032 CTR=0900000001507F50 CR=4254543420000004 FPSCR=8200000000000000 XER=2000000482000000 FPR0 0000001800000018 (f: 24.000000, d: 5.092790e-313)

It turns out it wasn't the WPS install but the underlying WAS java version that was causing the issue. The quick and easy way around this was to set:

export JAVA_COMPILER=NONE for the user that was running the install.

This link gives more details of supported versions of WebSphere on Power 7 kit http://www-01.ibm.com/support/docview.wss?uid=swg21422150

Redirecting heapdumps for WAS on zOS

I have recently been doing a fair bit of WAS work on zOS, going back to where it started for me in the WebSphere world.

On one of the systems we had an application that was running out of memory regularly, creating heapdumps and then filling the filesystems so WAS wasn't able to restart itself. To get round this we created a new large filesystem and redirected the heapdumps to this FS using the following generic JVM argument for the servant process:

-Xdump:heap:defaults:file=/WASdumps/heapdump.%Y%m%d.%H%M%S.%pid.phd

Where /WASdumps was the new filesystem

CONM6009E: The database is unable to get a connection to the database from DataSource

Another day, another test, another error!

We were running a stress test through our WAS systems that connect to an Oracle database on AIX. When we got a large number of concurrent requests we ended up getting the following error in our WAS logs:

CONM6009E: The database is unable to get a connection to the database from DataSource

We assumed at first that we had not sized our connection pools correctly. We turned on PMI and checked the size of the connection pools and found we weren't hitting the connection pool limits. We then checked the Oracle database which seemed correct but had logged a message stating the maximum user procs limit had been reached.

So on the DB server we ran the following:

lsattr -EH -l sys0 | grep -i maxuproc

which resulted in the following:

maxuproc 1024 Maximum number of PROCESSES allowed per user True

1024 was less than the total number of connection pool threads we had set in WAS. A quick chat with a friendly AIX administrator to increase this setting then resolved the issue.

Testing WAS app without creating a session

Since writing a post (here) on in-memory session count, I have been doing endless amounts of work on sessions, tracing them to see how the reaper script works as well as how frequently it runs.

One of the big issues we were facing is the number of in memory sessions we were creating. Due to memory limitations and an app that was creating large sessions we have limited number of sessions available so understanding the ins and outs of session management has been useful.

In front of our IHS and WAS servers we had a load balancer that was firing a request through to the front screen of the logon to see if the application we up and running. Getting the load balancer to test a static page on the web servers wasn't sufficient for our requirements. Given the frequency of the LB requests though and the fact every time they accessed the front page they were allocating a session, it would mean we would often end up with overflowed sessions.

Instead of hitting the app front page we tried to hit a simple jsp within the app but then WAS would create a session for that request rather than anything explicit in the application. After a bit of digging I found a line of code I could add to a jsp

<%@page session="false" %>

This also means the stats I was producing in my previous post were more accurate and did not inclue the LB requests in the session count!