IBM Installation Manager - Installing across the 'web' - no packages found

Recently my client decided to try and set up Installation Manager for IBM products on a webserver.   I guessed this was possible but just to make sure I spoke to my go-to expert, Dave Hay,  who pointed me at his blog  http://planetlotus.org/profiles/dave-hay_131315

Perfect!

I followed his steps to the letter but when I ran:

./imcl listAvailablePackages -repositories  http://jslwebserver/repos/WebSphere/8.5.5/Base/repository

I received the message "No packages were found"

Hmmm, how odd.  When I used IM to test connections to the repository it worked ok.  A quick look at the webserver logs showed that IM was trying to find repository.xml as well as repository.config and getting a 404 error.

A but more reading and I spotted this http://www-01.ibm.com/support/knowledgecenter/SSDV2W_1.8.2/com.ibm.cic.auth.ui.doc/topics/t_introduction.html    which states

Restriction: Installing from an HTTP server requires by using the Packaging Utility. You must use Packaging Utility to create the repository that is on the HTTP server.

Ok, so I downloaded the packaging utility and then carried out the next few steps:

1  Copied pu.offering.disk.linux.gtk.x86_64_1.8.2000.20150303_1543.zip   to /home/wasadm/PU

2   Unzipped the above file

3.  cd disk_linux.gtk.x86_64/InstallerImage_linux.gtk.x86_64/

4.    ./userinstc -acceptLicense

"Installed com.ibm.cic.packagingUtility_1.2.2000.20150303_1543 to the /home/wasadm/IBM/PackagingUtility directory.

5.    cd   /home/wasadm/IBM/PackagingUtility

6.    ./IBMPU


At which point a GUI starts up.  I selected "Copy repository" which allowed me to point at my exisitng repository.config file and provide a new location for the updated repository.

The utility then ran and in my new location, the repository now included a few new files, one of which was repository.xml

So now when I ran the following I was able to see the package


./imcl listAvailablePackages -repositories  http://jslwebserver/repos/WebSphere/8.5.5/BasePU/repository

com.ibm.websphere.BASE.v85_8.5.5000.20130514_1044


I was then able to run my install using the repository on hosted on the webserver.

Business Monitor and Cognos performance problem

Recently we have been seeing some issues with our Business Monitor servers on start up.   After running IBM Business Monitor 7.5.1 for over a year, every time we restarted the environment there was a spike in CPU consumption that lasted for over half an hour.

On closer inspection it was the cognos process spawned by Monitor that was consuming all this CPU.    A quick call to IBM support pointed us at the NC tables used by cognos.  A few of our tables had over half a million rows, where as the following document suggestes anything over 1000 could impact performance:

http://www-01.ibm.com/support/docview.wss?uid=swg21637944

So we followed the steps in the technote and cleared down the NC tables and all of a sudden there was no issue following the next restart.

We have been keeping an eye on these tables since then and noticed they do continue to grow.   Further questions to IBM support suggest this is normal behavior as long as there is no rapid increase in the size of the tables.  It just means every now and then we may have to clear down these tables again.

BPM portal images not displaying

Another battle with BPM today, this time to do with the images in Process Portal not displaying correctly.

When I tried to access the url for Process Portal  the screen displayed but some of the images/css files were missing so the page didn't look complete.

After finding this page it seemed quite a common problem

http://www-01.ibm.com/support/docview.wss?uid=swg21587668

I followed the instructions to the letter but still no joy.  A colleague mentioned it might be an issue with hostnames.  If BPM is set to a short name e.g.   mybpm.host  but the network then appends the fully qualified host name mybpm.host.mydomain.com this may cause issues.  I made sure the 99local and 100custom xml files were all using the FQDN.

Still no joy then I noticed something odd.  When logging on from my desktop the page worked correctly but when using the browser on the BPM host VM the error was still there.  A look in the http log showed that both requests looked identical and both showed up http respone code 200 for the CSS and images.

At this point I realised it might be the browser.  Good old IE.  Simply adding mybpm.host.mydomain.com  to the trusted sites fixed the problem and css files suddenly displayed correctly.

Well that was a few hours wasted!!!!

Snapshot install error

Wow, has it really been a year since my last blog post!

I have recently been having issues on my BPM 8.0.1.1 environment with some snapshot installs.
I had set up my Process server environment and initially set up a single Process server environment, using http ports and direct (not via a LB).  In this scenario everything worked as expected.

I then extended my Process server environment to a second node, and fronted it all with a webserver that was only enabled for https.

I followed the instructions in this useful infocenter document to ensure I was making the correct changes in the xml files:

http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0mx/index.jsp?topic=%2Fcom.ibm.wbpm.imuc.sbpm.doc%2Ftopics%2Ftconfig_custom_cluster_env.html&resultof%3D%2522%2577%2565%2562%2522%2520%2522%2573%2565%2572%2576%2565%2572%2522%2520

However, when I tried an install of the snapshot it failed and the SystemOut.log for Process center wasn't exactly bursting at the seams with useful information.

The only messages in the log were:
CWLLG0151I:  Standard hiring sample v801 Beginning login to remote server
CWLLG0152E:  Standard hiring sample v801 Remote login failed

I had a suspicion that despite setting <deploy-snapshot-using-https>true</deploy-snapshot-using-https>  in the Process server 10custom.xml that Process Center was still using http.  The reason I suspected this, was that in the ProcessCenter console, on the servers tab, when I clicked "Configure server" it tried to open a window to http://my-process-server:443/ProcessAdmin   i.e.   using an ssl port but on http.

A quick check in the LSW_SERVER table showed that the host name and port were correct  my-process-server:443  but there was no column in there that specified to use http or https.

So how could I prove what was happening?   I found the details of the trace that I need to set on the Process Center JVM:

com.ibm.bpm.fds.*=all:ProcessApplicationLifecycle=all:com.ibm.bpm.fds.repo.util.ContributionHelper=off:WLE.*=all

The trace output showed the following message:

com.lombardisoftware.server.ejb.repositoryservices.DeploytoServerSupport  deploySnapshot  Deploy Snapshot using :  http://

So it appears my suspicions were correct.  However, I had followed IBMs documents to the letter but PCenter was still not deploying using https.

One last thing to try was to update the 100custom.xml in Process Center so that now contained  <deploy-snapshot-using-https>true</deploy-snapshot-using-https>

A quick resynch of the node and restart of the JVMs and suddenly the deploy using https worked!

Whilst this works for me, the only downside I can see is that if you have Process Center talking to a mix of environments, they all need to be deployed to using http or all using https and doesn't appear as though you can have a mix

Connecting IBM Cognos BI to the IBM Business Monitor reporting database

The last few weeks I have been struggling to get Cognos within Business Monitor to connect to the Monitor database correctly.   After a lot of trial and error and some help from Cognos and BPM experts I have now got this working and managed to repeat the process on several environments.

The initial error appeared when trying to deploy a Monitor model - this was documented in this technote .   When logging onto the Cognos admin console, the test connection on the database would work for the thin client but fail for the thick client as this wasn't installed at the time.

As we were using a back end Oracle database I installed the Oracle instant client.   Even though I was running 64 bit WAS, Business Monitor and Oracle, the client that Cognos requires is a 32 bit version.

Once this was installed, I then set the LD_LIBRARY_PATH on the application server running the cognos dispatcher.   This is in the WAS admin console:   Application servers ==>  <server name> ==> Java and process management ==> Process definition ==> Environment entries

I added the <oracle_client_dir>/lib to the LB_LIBRARY_PATH.  Just adding the install dir didn't fix the problem and it appears you need to add the /lib so WAS knows where to find the .so file.

As the processes were running as non-root I needed to ensure that the JVMs could access the oracle directories so in the profile of the WAS user I added:

export ORACLE_HOME=/u001/app/oracle/product/11.2.0/client_1
export ORACLE_SID=myDB
export ORACLE_BASE=/u001/app/oracle
export TNS_ADMIN=/u001/app/oracle/product/11.2.0/client_1/network/admin
export TWO_TASK=myDB

After this a restart of the environment meant I could successfully run a test connection with the cognos admin console. 

WebSphere Request for Enhancments

If any of you have ever requested an enhancement to Websphere, you will know in the past this hasn't been the easiest process.  Having to go through your account manager and then not knowing whether anyone is actually looking at your request, well now the process has become a lot simpler and more transparent.

There is now a RFE page on the developerworks site:

https://www.ibm.com/developerworks/rfe/

You can submit your RFE to IBM as well as searching RFEs that have already been submitted.  There is also an ability to vote for RFEs so the product team know which features are really important to the end users.

Securing a thin client from WAS to Oracle

My current client is a large user of WAS and Oracle but had not until a few weeks ago set up any SSL connections between the 2 of them. When I came to do this there appeared to be very little documentation on turning on SSL for a thin client and the little I did find tended to be more about coding within an application than within an application server.

This blog proved a useful starting point:

http://blog.anowak.net/2009/09/configuring-encrypted-connection.html

However, it still didn't appear to work.

After a fair bit of testing and tracing it turns out it was in fact the oracle.net.ssl_version setting mentioned in Artur Novaks blog that resolved the
issue in the end but it appeared the negotiation of the SSL version between WAS and Oracle does not work as documented.

I am not sure if this how WAS and Oracle communicate at all versions or if it was just our specific set up.

We had enabled SSL on our Oracle DB which was running version 11.1.0.7.
Within our WAS server we had the latest ojdbc5 driver file which was v
11.2.0.2. The WAS version we were running was 6.1.0.31


Within the oracle DB, the SSL version was left unspecified which by default should have meant that the highest level of SSL should have been negotiated - starting at TLSv1, then SSLv3 and then SSLv2. Or at least that was how I read it in the Oracle documentation.

By setting the custom property in WAS to have a trust store but no ssl version specified we got the following error:

[23/09/11 14:11:13:593 BST] 0000002f DataSourceCon E DSRA8040I: Failed to connect to the DataSource. Encountered "": java.sql.SQLException: IO Error: The Network Adapter could not establish the connectionDSRA0010E: SQL State = 08006,
Error Code = 17,002

followed by

Caused by: oracle.net.ns.NetException: The ssl protocol specified is not
supported.
at
oracle.net.nt.TcpsConfigure.configureVersion(TcpsConfigure.java:181)
at
oracle.net.nt.TcpsNTAdapter.setSSLSocketOptions(TcpsNTAdapter.java:188)
at oracle.net.nt.TcpsNTAdapter.connect(TcpsNTAdapter.java:138)
at oracle.net.nt.ConnOption.connect(ConnOption.java:123)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:353)
... 101 more
Caused by: java.lang.IllegalArgumentException: SSLv2Hello
at com.ibm.jsse2.mb.a(mb.java:30)
at com.ibm.jsse2.lb.(lb.java:4)
at com.ibm.jsse2.jc.setEnabledProtocols(jc.java:570)
at
oracle.net.nt.TcpsConfigure.configureVersion(TcpsConfigure.java:177)


So for some reason TLS and SSL v3 were not tried.

I then tried WAS set to oracle.net.ssl_version=3.0 but that resulted in the following:

[23/09/11 14:12:36:529 BST] 00000031 DataSourceCon E DSRA8040I: Failed
to connect to the DataSource. Encountered "": java.sql.SQLException: IO Error: Received fatalalert: handshake_failureDSRA0010E: SQL State = 08006, Error Code = 17,002

But I didn't get much more in any of the standard output files so I turned on tracing with SSL=all and a custom property on the jvm of javax.net.debug
showed me the handshake going on. These couple of lines show WAS reading sslv3 but receiving a TLSv1 format but then going straight into a failure.

[23/09/11 14:16:42:389 BST] 00000033 SystemOut O WebContainer : 0, READ: SSLv3 Alert, length = 2
[23/09/11 14:16:42:390 BST] 00000033 SystemOut O WebContainer : 0, RECV TLSv1 ALERT: fatal, handshake_failure


So it appears Oracle was sending a handshake as TLSv1, WAS was expecting this in SSLv3 format, but rather than negotiating to use sslv3 it just gives us the handshake error.

Setting the SSL version to TLSv1 wasn't documented so I just set:

oracle.net.ssl_version=1.0;

and this worked as you see WAS reading TLSv1

[23/09/11 14:19:10:891 BST] 00000033 SystemOut O WebContainer : 0, READ: TLSv1 Handshake, length = 97

as well as allowing me to run a successful test connection!

So my connection url string in the end was just this:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=mydbhost)(PORT=2484))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=mydbservice)))

and then set a custom property of "connectionProperties" to the following value:

"oracle.net.ssl_version=1.0;javax.net.ssl.trustStore=/usr/wps6.1/WebSphere/ProcServer/profiles/myprofile/config/cells/myCell/trust.p12; javax.net.ssl.trustStoreType=PKCS12; javax.net.ssl.trustStorePassword=password"

Deployment error on WAS 6.1.0.39

Since upgrading our WAS environment a few weeks ago to WAS 6.1.0.39 from 6.1.0.31 we were getting errors anytime we tried to deploy an application ear file.

The error we were getting was:

Caused by: org.apache.commons.logging.LogConfigurationException: org.
apache.commons.logging.LogConfigurationException: org.apache.commons.
logging.LogConfigurationException: Class org.apache.commons.logging.
impl.Jdk14Logger does not implement Log

Which looked pretty much like this PMR https://www-304.ibm.
com/support/docview.wss?uid=swg21502693 although that only related to WAS 7 and the workaround documented wasn't available at 6.1

IBM did some investigation and a fix was targetted for FP 43 which was of course no use to me. They did provide a temp fix in the shape of IFPM45666 but that didn't resolve the issue.

After having initially to go back to FP37, we were finally provided a workaround that allows us to go back to FP39.

In /deploytool/itp/ejbdeploy.sh by adding -DNoCustomConverter=true we were able to succesfully deploy our applications again.

Display JMS connection pool contents

Recently I have been having problem with an application seeming to use up all the available connections in an MQ connection pool. To enable me to show what was going on to the developers I needed to create a jython script to show the pool contents and how long the connections had been in use for.

The connections in the pool did not appear to be getting freed up even when the purge timer was reached, so we set up the stuck timer under connection pool ==> advance settings.

This at least provided warnings in the logs that there were stuck threads - "A stuck connection is an active connection that is not responding or returning to the connection pool."

This provided some messages in the log to prove what we expected but as tends to be the case this wasn't sufficient. So I set up the following script to run at regular intervals:

ps = AdminControl.queryNames ('WebSphere:type=J2CConnectionFactory,process=server1,*').splitlines()
for p in ps:
print p
pc=AdminControl.invoke( p , "showPoolContents" )
print pc
This gave all sorts of useful information including details on each connection:

MCWrapper id 7d597d59 Managed connection com.ibm.ejs.jms.JMSManagedQueueConnection@6b376b37
managed connection factory = com.ibm.ejs.jms.WSJMSManagedQueueConnectionFactory@50f350f3
physical connection = com.ibm.mq.jms.MQXAQueueConnection@6beb6beb
credential = null
open connection handles = [com.ibm.ejs.jms.JMSQueueConnectionHandle@48324832] State:STATE_TRAN_WRAPPER_INUSE Start time inuse Thu Jun 16 18:21:24 BST 2011 Time inuse 53119 (seconds)

The proof I was after was the start time - this tended to be when the app was first run and would just hang around.

Even running the following to purge the pool did not help:

AdminControl.invoke( p , "purgePoolContents" )

The showpoolcontents would show the same information with the following line added:

Connection marked to be destroyed. Waiting for transaction end and connection close.

With this information to hand the developers finally agreed to check out there code!




n.b. If you need to view a database connection pool instead of a JMS pool, change the type in the querynames command to type=DataSource,*

Change order of JAAS login modules

I have recently been installing some 3rd party software into a WPS environment. Part of the install instructions was to add some new JAAS login modules but also to change the order of these and the existing modules.

Unfortunately, the order of these, even though you can change this via the amdin console, is not an attribute and is determined by the order they appear in the security.xml file.

With a bit of googling it is easy enough to find out how to create the new modules, so the only way to script the whole process that I could see was to delete the existing modules and then add them again but in the correct order:

import java.lang.System as sys
sec = AdminConfig.list("Security")
slc = AdminConfig.showAttribute(sec, "systemLoginConfig")
entries = AdminConfig.showAttribute(slc, "entries")
entries = entries.replace('[','')
entries = entries.replace(']','')
entries = entries.split(' ')
for entry in entries:
print entry
alias = AdminConfig.showAttribute(entry, "alias")
if ( alias != "WEB_INBOUND" ) :
print "Not the module we are interested in"
else:
print "Changing mods"
loginMods = AdminConfig.showAttribute(entry, "loginModules")
loginMods = loginMods.replace('[','')
loginMods = loginMods.replace(']','')
loginMods = loginMods.split(' ')
for loginMod in loginMods:
print "Deleting login module " + loginMod
AdminConfig.remove(loginMod)

nmid="com.myco.1st.jaas.class.name"
newModuleId = AdminConfig.create("JAASLoginModule", entry, [["moduleClassName", nmid ]])
AdminConfig.modify( newModuleId , [["authenticationStrategy", "REQUIRED" ]] )
#

nmid="com.myco.2nd.jaas.class.name"
newModuleId = AdminConfig.create("JAASLoginModule", entry, [["moduleClassName", nmid ]])
AdminConfig.modify( newModuleId , [["authenticationStrategy", "REQUIRED" ]] )
#

AdminConfig.save()

IBM WebSphere Application Server support on POWER7 hardware

I have just had a fun few days istalling WebSphere process server on a client site. They were installing this on Power7 hardware which caused us an issue. Simply trying to run any install commands on Process server 6.1 caused a java core and a snap dump and output similar to this:

Unhandled exception Type=Segmentation error vmState=0x00000000 J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033 Handler1=09001000A03A89F0 Handler2=09001000A0574DB8 R0=09000000014F1FE8 R1=0FFFFFFFFFFFE4E0 R2=07000000002DBE30 R3=000000000000000B R4=0000000000000000 R5=09000000014F1FE8 R6=000000011191E673 R7=000000011191E668 R8=000000011191E670 R9=0000000037CE0000 R10=000000011181BB70 R11=0900000001507F50 R12=090000000140B90C R13=000000011000EB20 R14=000000011191E658 R15=00000001104E4800 R16=0000000110D7B230 R17=00000001118199F8 R18=09001000A03AC2C8 R19=0000000000000021 R20=0000000111819AB8 R21=000000011191E690 R22=0700000000447E70 R23=0700000000447EA0 R24=0700000000448100 R25=0000000112067450 R26=07000000002D8188 R27=07000000002D8878 R28=0000000000000000 R29=0000000000000001 R30=07000000002D8A68 R31=07000000002DBE30 IAR=0900000001507F50 LR=00000001105A4EE0 MSR=A00000000000D032 CTR=0900000001507F50 CR=4254543420000004 FPSCR=8200000000000000 XER=2000000482000000 FPR0 0000001800000018 (f: 24.000000, d: 5.092790e-313)

It turns out it wasn't the WPS install but the underlying WAS java version that was causing the issue. The quick and easy way around this was to set:

export JAVA_COMPILER=NONE for the user that was running the install.

This link gives more details of supported versions of WebSphere on Power 7 kit http://www-01.ibm.com/support/docview.wss?uid=swg21422150

Redirecting heapdumps for WAS on zOS

I have recently been doing a fair bit of WAS work on zOS, going back to where it started for me in the WebSphere world.

On one of the systems we had an application that was running out of memory regularly, creating heapdumps and then filling the filesystems so WAS wasn't able to restart itself. To get round this we created a new large filesystem and redirected the heapdumps to this FS using the following generic JVM argument for the servant process:

-Xdump:heap:defaults:file=/WASdumps/heapdump.%Y%m%d.%H%M%S.%pid.phd

Where /WASdumps was the new filesystem

CONM6009E: The database is unable to get a connection to the database from DataSource

Another day, another test, another error!

We were running a stress test through our WAS systems that connect to an Oracle database on AIX. When we got a large number of concurrent requests we ended up getting the following error in our WAS logs:

CONM6009E: The database is unable to get a connection to the database from DataSource

We assumed at first that we had not sized our connection pools correctly. We turned on PMI and checked the size of the connection pools and found we weren't hitting the connection pool limits. We then checked the Oracle database which seemed correct but had logged a message stating the maximum user procs limit had been reached.

So on the DB server we ran the following:

lsattr -EH -l sys0 | grep -i maxuproc

which resulted in the following:

maxuproc 1024 Maximum number of PROCESSES allowed per user True

1024 was less than the total number of connection pool threads we had set in WAS. A quick chat with a friendly AIX administrator to increase this setting then resolved the issue.

Testing WAS app without creating a session

Since writing a post (here) on in-memory session count, I have been doing endless amounts of work on sessions, tracing them to see how the reaper script works as well as how frequently it runs.

One of the big issues we were facing is the number of in memory sessions we were creating. Due to memory limitations and an app that was creating large sessions we have limited number of sessions available so understanding the ins and outs of session management has been useful.

In front of our IHS and WAS servers we had a load balancer that was firing a request through to the front screen of the logon to see if the application we up and running. Getting the load balancer to test a static page on the web servers wasn't sufficient for our requirements. Given the frequency of the LB requests though and the fact every time they accessed the front page they were allocating a session, it would mean we would often end up with overflowed sessions.

Instead of hitting the app front page we tried to hit a simple jsp within the app but then WAS would create a session for that request rather than anything explicit in the application. After a bit of digging I found a line of code I could add to a jsp

<%@page session="false" %>

This also means the stats I was producing in my previous post were more accurate and did not inclue the LB requests in the session count!

javax.net.ssl.SSLHandshakeException: bad certificate

We have been doing some testing on WAS recently where our app makes a call to a 3rd party which hosts some static images. In our test environments though we were getting a "bad certificate" error.

Our key stores and trust stores all appeard to have the valid certs in that we thought were reuqired. Unfortunately, even when we turned on tracing in WAS we couldn't see what the certificate was that was causing the issues.

Due to firewalls and proxies, we couldn't hit the url dirrect from a PC so we couldn't check it out manually. So to allows us to see what ceriticates were being served we used the openssl command which listed the certs served by the target site we were trying to hit:

/usr/linux/bin/openssl s_client -connect www.ourtargethost.com:443 -showcerts

Thisn showed the certifcate chain and the issues highlighted what the issues with the certs was:

CONNECTED(00000003)
depth=0 /C=GB/ST=Somewhere/L=Warrington/O=My company Ltd/OU=HS4/CN=www.ourtargethost.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=GB/ST=Somewhere/L=Warrington/O=My company Ltd/OU=HS4/CN=www.ourtargethost.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=GB/ST=Somewhere/L=Warrington/O=My company Ltd/OU=HS4/CN=www.ourtargethost.com
verify error:num=21:unable to verify the first certificate
verify return:1

WebSphere Application Server version 8

IBM have recently announced an Alpha release of WAS v8

The product can be downloaded from:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/wsasoa/support.shtml

wsadmin and WAS commands hanging

In the last week we were having all sorts of problems getting any commands working even though they were running as root. I first notcied that when ever I treid to get into a wsadmin session, it would just hang.

There were no error messages and nothing obvious. We then discovered that all commands that end up running java under the covers were having the same issue, so startServer.sh , stopManager.sh , serverStatus.sh and pretty much all the supplied WAS scripts.

After an age looking around at the setupcmdline and seeing if the OSGI bundels were causing an issue. We also found a fix in fp29 that seemed in a similar area but that didn't resolve it. Just as we were about to log a call with IBM, we decided to take a javacore of the processes we were running whilst they were hung (why we didn't do this sooner I have no idea!)

We took several javacores, 30 seconds apart. Although there were no blocking threads, in each javacore, the main thread appeared to be looking up the localhost:

at java/net/Inet6AddressImpl.getLocalHostName(Native Method)
at java/net/InetAddress.getLocalHost(InetAddress.java:1463)

These same entried were in each of the javacores so it appeared there was an issue getting the localhost name. Once we spotted this, it didn't take long to find out there was an issue contacting our dns servers. We removed the /etc/resolv.conf whilst we looked into this, so WAS would now go back to using the hosts file on the server and everything then jumped back into life.

*Added 19th Mar 2010

I have just been directed to this page from IBM which may well have resolved my problem. If you can't simply turn off DNS then this might be a preferred option:

IBM link swg21170467

*sys-package-mgr*: can't create package cache dir

We hit another issue today after applying a fix pack to our WAS system, like the osgi bundle issue, it was due to file permissions.

When we were running some jython scripts, we were trying to import some packages but got an error:

from org.python.modules import re
WASX7015E: Exception running command: "from org.python.modules import re"; exception information:
com.ibm.bsf.BSFException: exception from Jython:
Traceback (innermost last):
File "", line 1, in ?
ImportError: no module named org

To recreate the error, rather than running this in a script I just started up a wsadmin session and on doing so I got the following error:

*sys-package-mgr*: can't create package cache dir, '/temp/cachedir/packages'

After a bit of investigation, it turned out the "cachedir" directory was owned as root but we run our scripts as a WASAdmin user. It looks like the permissions were changed to root after applying some fixpacks a couple of weeks ago, so a simple chown on cachedir resolved the issue.

Websphere in memory session count

I have recently been working on an application that uses in memory sessions. Our previous systems had always put session data on a database so it was easy to find out how many active sessions we had at any one time which is always useful when we have issues with a system so we can state how many users have been affected.

To allow us to view this sort of information within WAS, I first had to enable PMI on each WAS server.

This article discusses the overhead of PMI

Now that PMI was enabled, it was simply a case of writing a jython script to run at regular intervals to get the data:

The commands are as follows (I have taken out the commands to strip the data to the format I was specifically after so I will leave that up to your own jython skills to sort out)


servers = AdminTask.listServers( '[-serverType APPLICATION_SERVER]').splitlines()
for server in servers:
# Now just get the app server name - not the whole jytoh config id
newserver = server.split('(')
# get the session manager mbean
ps = AdminControl.queryNames ('WebSphere:type=SessionManager,process=' + newserver[0] + ',*')
# now get the stats for the mbean
AdminControl.getAttribute(ps, 'stats')


And hopefully you will get some output like this:

['', 'Stats name=My_WAR_FILE_NAME, type=servletSessionsModule', '{', 'name=SessionObjectSize, ID=18, description=The average size of the session objects at session level, including only serializable attributes in the cache., unit=BYTE, type=AverageStatistic, avg=1762.5, min=1713, max=1812, total=200925, count=114, sumSq=4.0370855625E10, type=TimeStatistic, avg=1762.5, min=1713, max=1812, total=200925, count=114, sumSq=4.0370855625E10', '}']

As well as the current count I could also check out the session object size which might also be useful if you have a large number of sessions and a small heap size

WMSG1603E - An error occurred trying to read the bundle

I encountered a strange error this weekend whilst installing multiple fix packs across numerous WAS systems. Despite installing these fixes numerous times, this error only occurred on one system.

I was upgrading from WAS 6.1.0.21 to 6.1.0.27 - that included WAS, SDK, IHS and Plugin fixes.

After installing the fix packs when I restarted the server I got the following error:

WMSG1603E: An internal error occurred. It was not possible to register the WebSphere MQ JMS client with the application serve
r due to exception org.osgi.framework.BundleException: An error occurred trying to read the bundle

followed by a java stack which included the following:

WMSG1603E: An internal error occurred. It was not possible to register the WebSphere MQ JMS client with the application serve
r due to exception org.osgi.framework.BundleException: An error occurred trying to read the bundle

A quick search and the reason was obvious. This WAS system does not run as root but when I checked the file permissions on the org.osgi.framework bundles in {WAS_INSTALL_DIR}/profiles/{PROFILE_NAME}/configuration the bundle in question was owned by root:

drwxr-x--- 2 wasadm wasadm 256 07 Feb 10:25 org.eclipse.update
drwxr-xr-x 4 root system 256 07 Feb 10:26 org.eclipse.osgi
drwxr-x--- 3 wasadm wasadm 256 04 Jan 11:24 org.eclipse.core.runtime

A quick change of permissions on the directory and all sub directories followed by a restart and everything came up fine.