Zero Risk Legacy System Re-engineering

 How To Squeeze 200+ Concurrent Users
Onto One Little i486

David Newall
david.newall@tellurian.com.au

October 20, 1998


We helped develop a system that provides users with a new GUI application, which they use in place of several old legacy systems.  Instead of replacing the old legacy systems, the new application adds value to them, and does so without them being altered in any way.  The new application hides the different legacy systems from the users, and gives an illusion of a single, integrated system.  A useful side-effect of this approach is that the life of the legacy systems has been extended: it achieves performance targets for more concurrent users than it could have had those users performed equivalent work without using the GUI.

Introduction

This is a short paper.  It is part technical discussion and part narrative for a system development project.

We were asked to help design and implement a new customer support system for a major corporation.  Growth of the number of different computer systems that their support staff needed to use concerned the corporation.  They wanted to reduce the number of separate systems that their staff needed to learn.  They also wanted to empower their staff to be able to answer questions about any and all products, which meant their staff needed access to equivalent systems.  Whilst about it, they wanted to enhance their level of customer support by being able to track contacts with their customer and also the problems their customers' reported.  Finally, they felt that the business rules embodied in the legacy systems were hard to define, and that replacing those systems with something new would be a long-term project: They wanted the new functions immediately but could not risk discarding or altering the old systems.  Oh; and the project was to be a proto-type, to prove the concept, and consequently had a very low budget.

Design

The application was designed using a client-server model, where the client (that is the GUI) communicated with servers using a request/reply interface.  The system architect divided the project into four major components: The GUI; contact and problem tracking; the legacy systems interfaces; and something to co-ordinate all of the components.  The GUI had to run on Macintosh-that was the standard desktop.  Most of the legacy systems were hosted on the IBM mainframes, and were designed for use via 3270 terminals.  Some further legacy systems were hosted on Digital Vaxen running VMS.  Other systems ran on Tandems.  Obviously only Unix had the technical features, stability, interoperability, and attractive price/performance ratio needed for a complicated system such as this.  Since the budget was very small, Intel architecture PCs were selected.  SCO Unix was chosen for the operating system, which provided its own, unique challenge.

Our task was to provide infrastructure to interface to the legacy systems, and to do so in such a way that future changes to the legacy systems would not result in major changes to our system.  As things turned out, our brief grew to encompass the complete system, however the design of the GUI, and therefore its 'taste', was "set in concrete" before we took it over.  Similarly, we were given the design of the contact and problem subsystem.

A request/reply model is similar to a transaction paradigm, and the system architect nicely followed that paradigm in the design of the server.  The server, or "back-end" as it came to be called, was written as a series of transaction-servers.  Tuxedo was selected as a transaction monitor; its use was to route requests and replies, and to balance transaction load amongst the available server processes.

Communication between the GUI and the back-end takes the form of a simple text-based stream, and uses TCP/IP.  There was no Tuxedo interface for Macintosh (although I believe this is no longer the case), and so we wrote a simple program, which ran on the back-end as a TCP server, that the GUI connected to.  It acts as a gateway between the GUI and Tuxedo, and is technically the Tuxedo client (Tuxedo provides a RPC type of client/server mechanism.)

There often is confusion over names we use.  'GUI' refers to the program running on the Macintosh.  The 'client' refers to the Tuxedo client, which the GUI connects to, and which serves to translate requests from the GUI into Tuxedo service calls, and to format Tuxedo replies appropriately for the GUI.  The 'servers' are the various programs that act on the GUI's requests.  The 'legacy systems' are the various systems running on the IBM hosts, the Vaxen, the Tandems and what have you.  I describe our system architecture as multi-level client/server.

I shan't describe the GUI in great detail.  It's not particularly interesting, and in fact it's not even particularly beautiful.  I should go further and say an expert user who had strong ideas about look and feel designed the appearance and operation of the GUI.  Without intending to denigrate the design skills of users, the ease of use of the GUI, and in fact the performance of the whole system, did suffer a little from lack of design experience.  On the positive side, this user was also the system champion: the project would have failed without having had someone willing to sell the idea to management; to guide our understanding of the problem; to feed us pizza during the long nights; and to stand up to upper-management when we had some early disasters.  I suppose the user is pleased that, far from losing the money that was budgeted (which is what the IT department predicted would happen), the final solution was approximately on-time, within budget, and according to specifications.

I also shall not thoroughly describe the contact and problem reporting subsystem.  It is quite uninteresting.  Data is stored in a relational database (if you are interested it uses Informix, but is soon to be Oracle) and the GUI can send requests to add, list, retrieve and update data.

A caution as an aside: This system was to be a prototype.  Five years later it is still in operation, and it has outlived two serious attempts to replace it.  The replacement systems were expensive and cumbersome, and there is now a movement to make this 'prototype' be a permanent part of the corporations IT infrastructure.  Lesson to be learned: Never believe anybody who tells you that it's only got to be a throw-away piece of code.  Throwaway code is used forever.  Another lesson, albeit one of a somewhat political nature: Large software projects take longer to produce less effective results than small projects.  If it's really critical to your business, leave it to a small team, say around five people, and give them a sensible budget; this will save millions of dollars, but more important: you'll get results that work.

Legacy system integration

Most of the legacy systems are traditional character based applications.  In the style of these things, they guide operators from screen to screen, collecting information and providing results as they go.  This really does not fit with the request/response design of the GUI.  Design of the new system included deciding what features (functions) of the legacy systems would need to be mirrored.  These features were described by the result, not by the navigation or menu options; for example, "retrieve the invoice for a particular date," or "change the account type for this account to that type."

The descriptions turned into the interface for the server programs, thus we were able to define transaction-oriented interfaces for an interactive, character based program.  Before we could use this transaction interface we needed two things: We needed to document how to operate the legacy applications in order to achieve the appropriate result; and we needed some way to connect to the legacy system that made us look like a person sitting at a terminal.

SNA Gateway

Our application had to appear to the legacy system as a 3270 terminal, however we did not want to tie our system to closely to 3270.  We realised that our system (despite being a prototype) might well outlast the legacy system, and their was no reason to suspect that legacy system's replacement would use 3270.  We defined a gateway protocol that encapsulated the essence of a terminal, without being specific to 3270.  A SNA stack with HLLAPI was purchased, and our protocol was implemented.  That first implementation was written for Macintosh; it's odd how decisions get to be made, for example, the standard desktop platform was Macintosh, so of course all server platforms should be Macintosh where possible.

The Macintosh version of the 3270 gateway was replaced with a Unix version, again running on i486 computers and SCO, and using an SNA protocol with HLLAPI which was written by Philips.  This was much more satisfactory from our point of view: In addition to working properly (which the Macintosh version really did not do), this version was easier to administer, particularly given that the hardware resides in the eastern states and we reside in the central state.

The gateway protocol is quite simple.  The purpose of the gateway is to make our system appear to be a 3270 terminal; only a few commands are needed.  Commands include press keys; return a section of the screen; and wait for the legacy application to stop updating the screen.

Legacy control

It might be tempting to imagine that sequencing of legacy systems is easy and can be table driven: Assuming we are at a known point, say the main menu, you can define a table that shows what keys you press to go from screen to screen; and to define the screens you need to navigate through; and to define the data you need to type in order to perform the work.  Actually this is far from sufficient.  It certainly does not cover complicated situations, such as when you need to go to one part of an application, capture some data, go somewhere else in the application and enter the data which you previously captured.  Character based applications often require this sort of interaction.  Not only does the table based approach fail to solve more than the simple interaction, it fails to help with exceptions, meaning when some error occurs.

We wrote a new language to sequence interaction with the legacy system.  The language grammar is simple.  It provides the usual features, such as variables, numeric and string expressions, conditional and unconditional loops and branches, procedure calls, and also provides features specific to this application domain, namely pressing keys, reading sections of the screen, waiting for key phrases to appear on the screen.  We find that writing programs to control the legacy application is quick and easy.

Each program written in our control language has a name, a list of input parameters, and a list of output parameters.  The name corresponds to the type of request that the GUI makes; the input parameters are the provided with that request; and the output parameters constitute the reply that is returned to the GUI.

The programs are compiled to a psuedo-code that is interpreted by a server program.  The server program is a Tuxedo server that selects and interprets the appropriate compiled psuedo-code needed to service each request that it receives.

Performance

The system works well.  Surprisingly well!  We have abstracted other applications, and perform scripted operation of those other applications, and the result is usually much faster than an expert operator would achieve sitting at a terminal using that other application.  Not only is it faster, but the number of logins to the other applications is much smaller than the number of people using our system.  Our system functions like a multiplexor, allowing many users to funnel through a small number of sessions.  It keeps those small number of sessions very busy, however we are told by the administrators of the legacy system that the impact on their system is much less than would be if each of our users were logged directly into theirs.  They have told us that costly upgrades to the IBM host have been avoided through the use of our technology.

We observe a tremendous economy of facilities in our approach.  We support almost 600 concurrent users during peak periods, and use a total of seven small machines.  The particular machines have changed over the life of the project.  We started with two 50MHz i486 machines, each with 64MB of RAM, one for running the client and the other for the back-end and database.  (This ignores the SNA gateway machines.)  We chose to limit the number of client connections to the client machine to 220.  As our user-base grew we expanded our hardware to give us three i486 machines.  At around 500 concurrent users we found that CPU requirements for legacy system control became more than could be handled by a single i486, so we added a second i486 giving us database on one server and legacy systems on the second.  Later we replaced those two i486 servers with a single Pentium-120 with 96MB RAM.

Our system is very reliant on correct operation of the legacy systems, and in fact we are usually the first to discover problems with the legacy systems.  It is almost a truism that problems reported with our system can be used as a barometer for problems with the legacy systems.  In fact, the infrastructure that we developed has proven so successful that the corporation has built a number of new legacy-style systems with the express intention that they would only be accessed via our GUI.  This makes sense because it allows the mainframe programmers to stay in their comfortable environment, writing the same style of code that they have written for the past 20 years.

Conclusion

Providing an integrated application on the desktop allowed the customer's staff to focus on their work, rather than on the computer systems.  The system was designed around what jobs needed to be done even though there was no single computer system which supported those jobs directly.

Without this system, the legacy systems would not have been able to cope with the businesses growth in user load.  Likewise, the training expense, had all users needed to learn all legacy systems, would have been prohibitive.

The system has proven to be very efficient, supporting hundreds of concurrent users on small computers.  It has given broad access to large parts of the customer's total computing environment, and done so without requiring legacy system changes or upgrades.