The LawsonGuru Letter, brought to you by Decision Analytics  
   

November 2006

)
       
 

Click here to forward to a colleague

   
       
 

In this issue:
1. Improving RMI Stability and Reliability
2. Guest Spot: Lawson Web Applications Performance and Stability
3. Worthwhile Reading
4. Lawson Tips & Tricks

The LawsonGuru Letter is a free periodic newsletter providing provocative commentary on issues important to the Lawson Software community.  The LawsonGuru Letter is published by—and is solely the opinion of—John Henley of Decision Analytics.

Visit Decision Analytics at https://www.danalytics.com. For subscription information, see the bottom of this message.  The LawsonGuru Letter is not affiliated with Lawson Software.

   
       
  1. Improving RMI Stability and Reliability )  
       
 

Alas, there is no silver bullet.  If you're one of those clients who doesn't experience RMI-related issues, consider yourself lucky!

Last month (see https://www.danalytics.com/guru/letter/archive/2006-10.htm), we looked at some of the RMI-related issues, as reported by a number of Lawson clients.  Now that the 8.0.3 Environment (on Unix and Windows) has an official decommission date (July, 31, 2008), many of you are starting to plan your move to Lawson System Foundation (LSF) 9.0.  And you really do want to go to LSF 9.0.  But so do a lot of other people—so you better get in line. I’m told that Lawson has a 6-9 month waiting list.   I suppose you could go it alone.  But you can’t.  Well, you can and you can’t.  According to Lawson’s marketing you can--with a catch:

 

When you try getting support if you run into an issue, you’ll find out the hard way that you really can’t do it without Lawson.  They're not making this easy.  Remember how Lawson encouraged you to use their consultants for your 8.0 upgrades?  “Use Lawson consulting or never get your project done.  If someone else installs your software you risk losing your support.”   Turn the clock back four years (!), and read “Will Lawson deny support for non-Lawson installed software?”  (see https://www.danalytics.com/guru/letter/archive/2002-09.htm).  Sound familiar? 

Now, where was I?  Oh yeah--you still have to deal with 8.0.3 RMI issues.  So, this month we'll delve into some of the ways in which you can improve RMI stability. 

Unfortunately, there doesn't seem to be any universal fix that you can make, but there are some ways you can improve RMI reliability and stability.  These tips are culled from various other consultants as well as my personal experience.  Alas, there is no silver bullet, and it takes a lot of iterative tuning and experimentation.  It's extremely painful.  If you're one of those clients who doesn't experience RMI-related issues, consider yourself lucky!  For the rest of you, here goes.  For the most part, these hints are meant for clients running RMI on Unix or Windows and using Tomcat as their servlet container.

First and foremost, if you can schedule a very short downtime (assuming you're not running a 24x7 and/or global operation), take the pro-active step of a daily re-cycle of RMI, Tomcat, and your web server.  Remember that the shutdown (and more importantly, the startup) order are important: Startup RMI, then Tomcat, then your web server. Then ProcessFlow and BCI. Shutdown in the reverse order.  Having a quick nightly script that does this on a scheduled basis sure beats unscheduled downtime!
Check, double-check, and then triple-check your .jar files.  Heck, check 'em four times! The jar files in GENDIR/java/jar need to match the ones deployed in WebSphere’s "ear" file, or in CATALINA_BASE/webapps/ROOT/WEB-INF/lib.  Likewise, the versions of the jar files in CATALINA_BASE/webapps/ROOT/WEB-INF/lib and CATALINA_BASE/webapps/ROOT/WEB-INF/classes should match as closely as possible.  You would be amazed how many clients I know who have missed those steps (as well as the step to remove the xerces.jar file) in the post-install instructions, and have 8.0.3 files mixed with 8.0.2 files.  Simply fixing those can make a world of difference.  Also, there are
If you have a version 8.0.2 lawsonrt.jar on the IOSRE of an 8.0.3 IOS install, remove it. It is not automatically de-installed with the upgrade to 8.0.3.
Tuning the RMI settings in PoolMgr.properties is an art.  For each connection, 14 MB of RAM is used, so that means for those 50 users, 140MB of RAM is required.  I'm told that this is flawed.  In reality there are 2 PTS processes, 2 streamDME’s and however many are streamIDA’s configured. The streamIDA’s increase with the Number of min and max are configured upwards in the PoolMgr.properties file.  Based on observation it is much higher and also the reason RMI becomes unresponsive many times and requires the restart/reboot.  Tune for the maximum expected load, and make the initial settings equal to ~25% of the maximum, e.g. if maximum is 10, set initial to 2.  The pts processes are declared globally across all product lines, but the streamida and streamdme processes are created specific to each product line. If you expect a peak load of 50 concurrent connections, this equates to 50 / 5, so you need to configure of 10 as the maximum number of concurrent connections. 
"Pooled DME" (aka streamdme) Although the settings are typically not set up when Lawson was installed, you can configure the initial and maximum number of DME connections (initconns_dme and maxconns_dme).
See Lawson Knowledgebase article 938683 for OS-specific issues, patches and environment variables settings.
Kept 100% current at all times on your Java releases.  If you're using 1.3.1, then it needs to be at the latest release of 1.3.1; if 1.4.2 then at the latest release of 1.4.2 and depending on the release all the available patches applied.  If you're lucky enough to be on AIX (which bundles Java with the OS patches), after you've applied the latest AIX maintenance release, apply the latest java fixes beyond the fix pack; they are available from the IBM patch site.
Medium and larger clients need to pay particular attention to the "Xms###m" and "Xmx###m" (upper and lower Java Memory Stack) settings; adjust them higher for both lower and upper ends based on the configured connections in the PoolMgr.properties file.  Change them in the startps.sh in $CGIDIR/rmi on Unix or in startps.bat for Windows Systems. There are also registry updates that are listed in the Windows Registry for Windows installations.  The stable installations with large clients have Xmx and Xms set at 1000m (i.e. 1 GB of RAM min and max in the Java Stack).  Also make the adjustments in the logan.env configuration files.
For larger implementations where a single Tomcat is handling the load for large numbers of users, there has to be an adjustment to the number of processes available for that one Tomcat in the CATALINA_BASE/conf/server.xml file. The default is 75 and it needs to be moved higher to 100 or 150 (or even higher based on the number of users that are actually accessing the system). This is necessary for large numbers of Lawson Portal and Requisition Self Service users.
Currently, the most reliable supported RMI / servlet combination seems to be running IBM Websphere 6.0, which is now supported by 8.0.3 ESP7.
Another way to achieve high reliability is by running IIS 6.0 and Tomcat 5.0.28 in a VMWare Image. (It's not supported but very reliable and makes the best use of available resources.) This way you can have FOUR Tomcats on 1 Server with Four IIS web server instances. It works great and makes complete use of the resources on that server. 4 CPU’s and 4 Images with 4 GB of RAM. All hooked to one Lawson instance. Needs to be load leveled via a network appliance.
Most robust Tomcat installation on UNIX is a free-standing Apache 2.0.X web server and Tomcat 5.0.28, running on a Unix server by itself running remotely.
   
     
  2. Guest Spot: Lawson Web Applications Performance and Stability )  
       
     
       
  Introduction

Unlike my previous articles, I start this article with the conclusion, because it may answer some of the questions asked in the recent LawsonGuru Letter about RMI stability. Technical details of my test process are quite complex, though the results might appeal to the larger audience.

I have measured stability and performance of Lawson web interface on multiple installations over the last two years on both UNIX and Windows platforms. I have applied Lawson configuration changes before measurements, which may have increased stability, and provided good response time with the large number of concurrent sessions.

Test results

  • Starting with, I believe, ESP4, where streamdme ("Pooled DME") was introduced, Lawson can sustain large number of concurrent users (I’ve tested up to 500) on Portal and self-service applications without failures of Lawson components. Lawson’s recent formula of non-concurrent-to-concurrent users’ ratio on one of the installations was 20:1. So, that translates to 10,000 non-concurrent users. More scientific approach I’ve developed was acquired during large transaction volume experiments related to SEA and benefit enrollment. There I came up with 30:1 or more formula instead (the actual number is based on the test, and is different every time). So, Lawson’s formula was even more conservative than the one I calculated.


  • The failures usually appear first in non-Lawson components – if Lawson is configured correctly.


  • The failures usually appear first in Tomcat. The Websphere testing did not include stress component that would find the first point of failure. On the Windows platform, the first failed componentproviding configuration of all software is correctis the ISAPI filter for Tomcat.  I used the ISAPI DLL from Tomcat 3.3 (128Kb in size).  I will do testing with a newer version of ISAPI DLL in the future with Tomcat 5.  Interestingly, Tomcat 4.1 ISAPI did not work with the other versions correctly.


  • The servlets can easily sustain 4-times or more concurrent users than CGI-based calls. Hence, any custom modifications designed for large concurrent number of users must use servlets rather than CGI calls for DME, AGS, etc..


  • Environment version 8.0.3 on UNIX and Windows handles prolonged servlet usage much better than 8.0.2. Installing ESP6 for 8.0.3 environment is highly recommended for stability reasons.


  • Lawson web server gets about 5-10% of the CPU load, while the Lawson application server gets 80-85%. The rest is the database server (for SEA/Portal work without batch jobs and Crystal Reports).


  • If majority of the calls use servlets, there is no need to keep databases in a shared access mode (e.g. setting up Oracle in a shared mode for large concurrent user numbers).


  • One needs to tweak Lawson settings related to the number of concurrent LATM users for the most popular forms, and monitor form usage, so no more than 10 instances of the same form would be in memory. With the high volume of user switching form cleanup processes seem to get “stuck” sometimes if this tweak is not done.


  • If you encounter non-Tomcat-related failures on the web side, most probably it’s related to either the authentication process, especially if you have a custom one, or a CGI DME call to LOBKMARK table during the initial login process. Then IE Windows Refresh will allow a user to continue. Providing configuration is correct, you will not encounter the DME-related issue until you will have more than 200 concurrent users (6000 non-concurrent), hence most people do not see it, even at the peak usage.

Technical Details

In the article published in May 2006 “Securing Lawson Self-Service” (see https://www.danalytics.com/guru/letter/archive/2006-05.htm) I have a diagram describing the data path between the users’ browser and lawson application server. There are two data paths – one via RMI, and another via CGI calls.

When RMI starts, it keeps RMI processes (pts, streamdme and streamida) in memory, and connections open to the database at all times. Hence, when a user makes a servlet request, no new program starts on the application server or the database server, except when a new instance of the servlet program needs to be allocated due to the large number of concurrent requests.
When a call is made via CGI, a CGI stub program starts on the web server, and “real” CGI program starts on the Lawson application server, as well as Lawson database driver (e.g. oradb9). If you use Oracle in a dedicated mode (default), you also have a separate database process on the Lawson application server, and one on the database server. That starting and stopping I/O and memory reshuffling is what appears to kill Lawson sessions and/or increase response time due to various reasons, some not related to Lawson components.

On a web side of things, it appears Tomcat ISAPI filter on Windows was not designed with more than 200-250 concurrent users in mind, so it starts failing under larger loads. The solution is setting multi-homed load-balanced web server, possibly on the same physical server, for each 150 concurrent users planned. That improves stability of the web calls dramatically.

I’ve got better stability results with JK2 connector than JK. It’s a pity it’s not developed anymore by the Internet community (hence, Lawson does not support it). The difference appears to be in a thread handling. I’ve observed better stability with Tomcat 4.1 than 4.0, possibly for the same reason. You should NOT be using Tomcat 3 anymore. Besides the fact Lawson does not support it now, it exhibited poor stability with high number of concurrent user sessions.

Interestingly, high concurrent number of LID users did NOT perform as well as high number of concurrent Portal users on some large forms (e.g. HR11).

In conclusion, if you’re looking to have a large number of users using Lawson Portal interface, evaluation of your existing Lawson and web setup, and performance and stability testing is a key. Most of the time, performance and stability issues occurred due to the inadequate setup, mismatch of the components (e.g. Environment ESP6 with IOS SP4), poor database setup and insufficient server hardware power (Lawson's latest sizing process gives pretty accurate hardware estimates based on the expected user counts).
   
       
  3. Worthwhile Reading )  
       
  IT Jobs Special Report

- QUOTE OF THE ISSUE –

“'Just because something doesn't do what you
planned it to do doesn't mean it's useless.”

-- Thomas Edison

Looking for IT work? Looking for IT workers? Look no further for a wealth of advice and trend reports that will help improve your position.
InfoWorld, September 18, 2006
http://www.infoworld.com/reports/38SRjobs.html

Numbers Crunch
Think reporting has changed since Enron? Just wait.
CFO Magazine, September 2006
http://www.cfo.com/article.cfm/7851805/c_7873404

The Greenest Office in America
Adobe has turned its headquarters into a towering example of environmentalism - and is saving millions of dollars in the process.
Business 2.0 Magazine, September 2006
http://money.cnn.com/magazines/business2/business2_archive/2006/09/01/8384321/index.htm
   
       
  4. Lawson Tips & Tricks )  
  New Feature to insert a column in Excel Upload Wizard

For years, one of the Lawson tips I’ve been sharing with users has been how to add a new column to an upload mapping you’d previously saved. This is often needed when you add a new column to a spreadsheet and want to include it in your upload.

Now, Lawson has added it. Download version 2.0.5 of the Lawson Add-Ins for Microsoft Office. Then, it’s just a click away!

   
       
     
Please share The LawsonGuru Letter in whole or in part as long as copyright and attribution are always included.
 

Decision Analytics is an independent consultancy, focusing on Lawson technical projects, and specializing in reporting, customization/modification, data conversion, and integration/interfaces. Please visit https://www.danalytics.com for more information.