Wednesday 25 November 2009

IBM tech exchange webcasts

In case any of the few people who visit the site are not aware, IBM run technical webcasts on a frequent basis, covering a broad range of topics within the WebSphere arena.

If you can't dial in at the time, then podcasts of the meeting are usually available a few days afterwards

Take a look here

Tuesday 24 November 2009

Default profile

Another quick tip....

This is documented well in the Info Center but always worth reiterating how it can be done.

If like me you have several WAS profiles on a machine, it is always worth knowing which one is the default profile and how to change it.

There are a couple of ways to find the default profile:

1. Run a script form the {WAS_INSTALL)/bin such as serverStatus.sh and examine the output. It mentions the profile that is being used

./serverStatus.sh -all
ADMU0116I: Tool information is being logged in file
/usr/was6/WebSphere/AppServer/profiles/myMgr/logs/serverStatus.log
ADMU0128I: Starting tool with the mymgr profile
ADMU0503I: Retrieving server status for all servers
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: dmgr
ADMU0508I: The Deployment Manager "dmgr" is STARTED


The profile is seen in the ADMU0128I message - in this case it is myMgr

2. Run the manageprofiles script with the getdefaultname keyword:

./manageprofiles.sh -getDefaultName
myMgr

So now we have the defaul profile, if you want to change it just run the manageprofiles command with the setdefault keyeword:

./manageprofiles.sh -setDefaultName -profileName myNode
INSTCONFSUCCESS: Success: The default profile has been set.

Then double check it has been set correctly:

./manageprofiles.sh -getDefaultName
myNode

Friday 20 November 2009

WAS and 64 bit MQ

Just another quick post while I remember.

A few weeks ago we had an error when starting up our app servers - we were running 64 bit WAS ND 6.1.0.21 and 64 bit MQ 6.0.2.4

The error in the WAS logs was:

java.lang.UnsatisfiedLinkError: mqjbnd05 (Not found in java.library.path)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:986)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:950)
at java.lang.System.loadLibrary(System.java:453)
at com.ibm.mq.MQSESSION.loadLib(MQSESSION.java:1028)
at com.ibm.mq.server.MQSESSION$1.run(MQSESSION.java:246)

It appears that although MQ was 64 bit, WAS was tryingto use the 32 bit libraries. It needed 2 steps to reoslve this:

1. On the app server in the JVM custom properties, we created a new prop called java.library.path and set this to /usr/mqm/java/lib64

2. On the AIX server as root in /usr/mqm/bin I ran the dltmqlnk command to remove links to the 32 bit files.

That is me done for the week, time to go home and relax until Monday. Well, when I say relax, I mean get jumped on and my ear drums perforated by my noisy kids and then woken up by them before 6am at the weekend! hmmm, roll on Monday and the relative peace and quiet of the office :-)

Websphere v7 MQ issue - WSVR0017E

We have recently started upgrading some of our WAS systems from 5.1.1.9 (yes I know these are well out of support - they were never even patched from fp 9 after they went in as they were stable) to v7.0.0.5

After migrating our MQ defs we had the following error when starting up our app servers

WSVR0017E: Error encountered binding the J2EE resource,....java.lang.IllegalStateException: QUEUE_DEFINED

Initially we thought it was to do with the migration but when defining a new queue in WAS 7 and setting some of the properties to QUEUE DEFINED we still got the error.

After a bit of searching we thought we would upgrade to 7.0.0.7 as there seemed to be a few fixes in this area although not specifically for this error but this didn't fix the issue.

At this point we went to IBM, after a bit of to-ing anf fro-ing, they said the fix should have been in fp7 as we had thought but for some reason it was left out and it targetted now for fp9. In the meaintime they provided a fix PK89856 for us to apply. The only issue is that this doesn't install on fp 7 as WAS already thinks it is there, so we had to drop back to fp5 then install it.

Tuesday 17 November 2009

CCSID zero or null?

hmmm, another config problem with the environment I am working on. The application deployed was sending messages via MQ and the data arriving at the far end was in the wrong CCSID. It certinaly wasn't MQ doing this so I thought it had to be the application but the aplpication vendors were certina it was WAS.

I had a look at the MQ definitions within the WAS console and the CCSID was blank so WAS couldn't have been doing anything odd, could it?

I decided to have a look at the resources.xml file in my envorinment rather than relying n the console and surpsise surprise there it was CCSID="0"!!!! So it appears zero shows up as null within the console.

I was unable to update this field via jython as whenever I added in a null value it said the field had to be numeric. However, when delting this queue definition and then readding it with a null value it worked fine and in resources.xml there was no CCSID field at all

Monday 16 November 2009

CORBA.NO_RESPONSE

I seem to be spending a fair bit of time recently tidying up systems that have been built for a while but don't quite work as expected once the testers get onto them.

My latest error was a CORBA.NO_REPSONSE error. The app server came up cleanly although there were suspicions in my mind when a war file was being started that it always waited for exactly 3 minutes before it completed initialisation.

Then when trying to access the app, we found the following in SystemErr.log:

java.rmi.RemoteException: CORBA NO_RESPONSE 0x4942fb01 Maybe; nested exception is:
org.omg.CORBA.NO_RESPONSE: Request 20 timed out vmcid: IBM minor code: B01 completed

A quick look at my app server ports showed that I had my ORB_LISTENER_ADDRESS set to "my node name" and the port was "0", so I assumed it was then using the noadagent port. My NA ORB_LISTENER_ADDRESS was set to "localhost" and port "9900". While this seems ok, in the hosts file, localhost resolved to the loopback address of 127.0.0.1

To resolve the issue, within WAS, I updated the Node agent ORB Listner to "myhost" port "9900" so that when WAS tried to connect to the host, it had a proper IP address to talk to.

Problem solved!

Thursday 12 November 2009

Webservers on unmanaged nodes

My current client is running IHS but managing them from within WAS as running on an unmanaged node (all this is on AIX)

A lot of the Infocenter documentation is based around running on managed nodes so the troubleshooting has been fun. There was never anything major, just fiddly.

We run IHS as non root, so when we first added the webserver to WAS we were unable to start it as we had the port set to 80. A non root user cannot use a port less than 1024 . The first thought was to change the port to port 8080 (see the issues in the vhost matching port below about the issues we had with this and our load balancer). Of course the easier way was to update the user and group settings within the httpd.conf.

When trying to propagate the plugin we got errors about file permissions on the admin console and also on the dmgr log. This took a while to track down - the WAS user had full access to update the plugin file on the unmanaged node so I just couldn't work this out. Eventually when checking the set up between enviornments, I clicked on "webservers ==> my server ==> Configuration file" which should open up the httpd.conf. This failed to show up due to a permissions problem.

So I gave the WAS user access to httpd.conf on the remote box and as well as allowing me to view the httpd.conf I could also propagte the plugin. Why the was users needs to check the httpd.conf before sedning out the file I don't know as the location of the plugin file, although mentioned in httpd.conf is also part of the WAS config.

The next issue I had was startting and stopping the webservers. When I clicked start, it took a minute before we got an error saying failed to start. Further investigation showed the server had started and the error was due to WAS not being able to get the server status. Just through various bits of testing we were doing, we found out this was due to a firewall issue. It seems that as there is no serverStatus script that can be run, that WAS does some sort of telnet or other network check on the port that IHS is listening on (in this case 8080). Oncer we opened up port 8080 on the firewall the start request worked correctly and the status showed up as well.

One of the drawbacks of running no an unmanaged node is that there is no stopServer or startServer script that can be run which would be the case on an unmanaged node. The only way to script from the dmgr box is to run a jython script like the following:

Stop

AdminControl.invoke('WebSphere:name=WebServer,process=dmgr,platform=common,node=myMgr,version=6.1.0.21,type=WebServer,mbeanIdentifier=WebServer,cell=myCell,spec=1.0', 'stop', '[myCell mynode mywebserver]', '[java.lang.String java.lang.String java.lang.String]')

Start

AdminControl.invoke('WebSphere:name=WebServer,process=dmgr,platform=common,node=myMgr,version=6.1.0.21,type=WebServer,mbeanIdentifier=WebServer,cell=myCell,spec=1.0', 'start', '[myCell mynode mywebserver]', '[java.lang.String java.lang.String java.lang.String]')

Anyway, off now to change some performce settings - thread pools and oracle connection pools! The fun never stops

Wednesday 4 November 2009

Virtual Host matching

Ran into another issue last week although this was easy to work around it was a bit of an annoyance.

A user was hitting our Load Balancer on port 8080 which we could then see this being forwarded onto our HTTP Server on port 8080 as well, but by the time this request got through to the WAS plugin, the plugin was trying to match it to en entry with port 80 which didn't exist in our plugin so the request failed.

For some reason the header was being updated by our LB, although I could just add port 80 to the vhost entry so that the plugin forwarded the reuqest onto WAS, I was keen to figure out what was going on and if there was any other way around this.

Our LB team provided little help, but the guys at IBM pointed me at the "Virtual host matching" setting in the plugin properties screen which maps to the VHostMatchingCompat setting in the plugin.

From the info center:

"By default, this field is not checked, and matching is done logically using the port number contained in the host header. Select this field if you want virtual host mapping performed by physically using the port number for which the request was received."

I checked this field, redployed the plugin and now the plugin would try to match on port 8080 (the port the request actualy came in on) rather than port 80 which was in the header.


Jython command assistance

I am still amazed at how many people don't know about the command assistance that was introduced in WAS 6. It is great for those who sometimes struggle to get their heads around jython commands and are trying to script some config work.

In the admin console ==> System Administration ==> Console preferences

Check the "enable command assistance notifications" and if you want to keep a more persistent log of the comands also check "Log command assistance commands"

Now when you make certain changes, on the right hand siode of the admin console in the Help pane you should see "Command assistance - View administrative scripting command for last action"

If you click here you will see a new window with the jython command for the action you just performed.

If you checked the option to log these commands, you will see a new log file created under the dmgr log root file called commandAssistanceJythonCommands_.log