IBM WebSphere Application Server support on POWER7 hardware

I have just had a fun few days istalling WebSphere process server on a client site. They were installing this on Power7 hardware which caused us an issue. Simply trying to run any install commands on Process server 6.1 caused a java core and a snap dump and output similar to this:

Unhandled exception Type=Segmentation error vmState=0x00000000 J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033 Handler1=09001000A03A89F0 Handler2=09001000A0574DB8 R0=09000000014F1FE8 R1=0FFFFFFFFFFFE4E0 R2=07000000002DBE30 R3=000000000000000B R4=0000000000000000 R5=09000000014F1FE8 R6=000000011191E673 R7=000000011191E668 R8=000000011191E670 R9=0000000037CE0000 R10=000000011181BB70 R11=0900000001507F50 R12=090000000140B90C R13=000000011000EB20 R14=000000011191E658 R15=00000001104E4800 R16=0000000110D7B230 R17=00000001118199F8 R18=09001000A03AC2C8 R19=0000000000000021 R20=0000000111819AB8 R21=000000011191E690 R22=0700000000447E70 R23=0700000000447EA0 R24=0700000000448100 R25=0000000112067450 R26=07000000002D8188 R27=07000000002D8878 R28=0000000000000000 R29=0000000000000001 R30=07000000002D8A68 R31=07000000002DBE30 IAR=0900000001507F50 LR=00000001105A4EE0 MSR=A00000000000D032 CTR=0900000001507F50 CR=4254543420000004 FPSCR=8200000000000000 XER=2000000482000000 FPR0 0000001800000018 (f: 24.000000, d: 5.092790e-313)

It turns out it wasn't the WPS install but the underlying WAS java version that was causing the issue. The quick and easy way around this was to set:

export JAVA_COMPILER=NONE for the user that was running the install.

This link gives more details of supported versions of WebSphere on Power 7 kit http://www-01.ibm.com/support/docview.wss?uid=swg21422150

Redirecting heapdumps for WAS on zOS

I have recently been doing a fair bit of WAS work on zOS, going back to where it started for me in the WebSphere world.

On one of the systems we had an application that was running out of memory regularly, creating heapdumps and then filling the filesystems so WAS wasn't able to restart itself. To get round this we created a new large filesystem and redirected the heapdumps to this FS using the following generic JVM argument for the servant process:

-Xdump:heap:defaults:file=/WASdumps/heapdump.%Y%m%d.%H%M%S.%pid.phd

Where /WASdumps was the new filesystem

CONM6009E: The database is unable to get a connection to the database from DataSource

Another day, another test, another error!

We were running a stress test through our WAS systems that connect to an Oracle database on AIX. When we got a large number of concurrent requests we ended up getting the following error in our WAS logs:

CONM6009E: The database is unable to get a connection to the database from DataSource

We assumed at first that we had not sized our connection pools correctly. We turned on PMI and checked the size of the connection pools and found we weren't hitting the connection pool limits. We then checked the Oracle database which seemed correct but had logged a message stating the maximum user procs limit had been reached.

So on the DB server we ran the following:

lsattr -EH -l sys0 | grep -i maxuproc

which resulted in the following:

maxuproc 1024 Maximum number of PROCESSES allowed per user True

1024 was less than the total number of connection pool threads we had set in WAS. A quick chat with a friendly AIX administrator to increase this setting then resolved the issue.

Testing WAS app without creating a session

Since writing a post (here) on in-memory session count, I have been doing endless amounts of work on sessions, tracing them to see how the reaper script works as well as how frequently it runs.

One of the big issues we were facing is the number of in memory sessions we were creating. Due to memory limitations and an app that was creating large sessions we have limited number of sessions available so understanding the ins and outs of session management has been useful.

In front of our IHS and WAS servers we had a load balancer that was firing a request through to the front screen of the logon to see if the application we up and running. Getting the load balancer to test a static page on the web servers wasn't sufficient for our requirements. Given the frequency of the LB requests though and the fact every time they accessed the front page they were allocating a session, it would mean we would often end up with overflowed sessions.

Instead of hitting the app front page we tried to hit a simple jsp within the app but then WAS would create a session for that request rather than anything explicit in the application. After a bit of digging I found a line of code I could add to a jsp

<%@page session="false" %>

This also means the stats I was producing in my previous post were more accurate and did not inclue the LB requests in the session count!

javax.net.ssl.SSLHandshakeException: bad certificate

We have been doing some testing on WAS recently where our app makes a call to a 3rd party which hosts some static images. In our test environments though we were getting a "bad certificate" error.

Our key stores and trust stores all appeard to have the valid certs in that we thought were reuqired. Unfortunately, even when we turned on tracing in WAS we couldn't see what the certificate was that was causing the issues.

Due to firewalls and proxies, we couldn't hit the url dirrect from a PC so we couldn't check it out manually. So to allows us to see what ceriticates were being served we used the openssl command which listed the certs served by the target site we were trying to hit:

/usr/linux/bin/openssl s_client -connect www.ourtargethost.com:443 -showcerts

Thisn showed the certifcate chain and the issues highlighted what the issues with the certs was:

CONNECTED(00000003)
depth=0 /C=GB/ST=Somewhere/L=Warrington/O=My company Ltd/OU=HS4/CN=www.ourtargethost.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=GB/ST=Somewhere/L=Warrington/O=My company Ltd/OU=HS4/CN=www.ourtargethost.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=GB/ST=Somewhere/L=Warrington/O=My company Ltd/OU=HS4/CN=www.ourtargethost.com
verify error:num=21:unable to verify the first certificate
verify return:1

WebSphere Application Server version 8

IBM have recently announced an Alpha release of WAS v8

The product can be downloaded from:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/wsasoa/support.shtml

wsadmin and WAS commands hanging

In the last week we were having all sorts of problems getting any commands working even though they were running as root. I first notcied that when ever I treid to get into a wsadmin session, it would just hang.

There were no error messages and nothing obvious. We then discovered that all commands that end up running java under the covers were having the same issue, so startServer.sh , stopManager.sh , serverStatus.sh and pretty much all the supplied WAS scripts.

After an age looking around at the setupcmdline and seeing if the OSGI bundels were causing an issue. We also found a fix in fp29 that seemed in a similar area but that didn't resolve it. Just as we were about to log a call with IBM, we decided to take a javacore of the processes we were running whilst they were hung (why we didn't do this sooner I have no idea!)

We took several javacores, 30 seconds apart. Although there were no blocking threads, in each javacore, the main thread appeared to be looking up the localhost:

at java/net/Inet6AddressImpl.getLocalHostName(Native Method)
at java/net/InetAddress.getLocalHost(InetAddress.java:1463)

These same entried were in each of the javacores so it appeared there was an issue getting the localhost name. Once we spotted this, it didn't take long to find out there was an issue contacting our dns servers. We removed the /etc/resolv.conf whilst we looked into this, so WAS would now go back to using the hosts file on the server and everything then jumped back into life.

*Added 19th Mar 2010

I have just been directed to this page from IBM which may well have resolved my problem. If you can't simply turn off DNS then this might be a preferred option:

IBM link swg21170467