Has anyone here been involved in any projects designed to eliminate COBOL or mitigate the risk of COBOL (for example, writing in-house compilers or using COBOL to generate NetBeans, etc.)? If so, what tools/strategies were adopted? Were you successful? What would you do differently?
Anecdotal stories welcome. Just trying to understand the approaches people use and the complexity of the problem.
Can't contribute with personal stories, but I guess this resource could be useful: COBOL to Java Automatic Migration with GPL'ed Tools
COBOL to Anything Else is fraught with problems based on two things.
The ALTER statement.
The REDEFINES clause.
Other than that, it's actually really easy.
Here's how you have to do it.
Find ALL your file definitions. Every "COPYBOOK" with Every Single File Definition. No exceptions. No "this is minor".
Positively identify the files that represent persistent business records. Sometimes these are called "Master Files". Sometimes they're not specially identified. COBOL systems are filled with temporary, transient files. You need to distinguish the files that represent persistent business state from other files which represent transient data passed from COBOL to Sort to COBOL to system utility to blah-blah-blah.
For each piece of persistent business information, locate all REDEFINES clauses. These are a terrible, terrible problem. Odds are good that these do not do what you think they do. Try to identify the various subclasses of data represented by the various REDEFINES alternatives. You WILL get this wrong. Get over it now. At some point, you will find mystery data that you cannot explain.
At this point, you need to survey your files to find out what's really in each field. For this, you can buy a tool. Or you can write a parser for COBOL DDE record definitions. These are not that hard to parse.
Now that you have the survey of the actual contents of each field, revise your understanding of the REDEFINES and what really exists on the files. This, BTW, can take a long time. Months. Seriously. Your data will be a mess. At some point, you will have to identify some records that you cannot process because you cannot determine what they mean.
Create your revised data model. Do this right. Create the correct data model. The one you should have used.
Create your conversion programs. Extract from COBOL files to load the new data model.
Write unit tests for your conversions. Be able to Prove that what's in the revised data model matches the original COBOL data as near as possible. Recall that there's mystery data. Recall also that you can't fully comprehend every nuance of the legacy COBOL reporting. You'll have a lot of notes and exceptions and caveats and hand-waving. Get close enough that you're confident you understand the data.
Write high-level summaries of what the COBOL is supposed to be doing. Do not analyze what it really does except in an emergency. The COBOL is going to be crazy, unreadable, unrelaible, contradictory, nearly useless.
Write your revised COBOL system from scratch, using the revised summary use cases. Write proper unit tests. Be sure the new system actually works by converting the legacy data, running the new stuff in parallel.
This is cheaper than converting COBOL code. Most COBOL programs are wordy, filled with useless trash to handle badly designed special-case and exception handling. Once you start rewriting them in Python or Java, you'll see that a good language (like Python) with a good set of libraries can trim a big COBOL program down to something quite small. Also, once you rationalize all the exceptions and goofy special cases that accreted over time, you'll find that there's a very simple thing hiding under all those PEFORM statements.
Do this "convert and run in parallel" operation as many times as it takes to be sure you like the new software better than the old software.
Have an official decommissioning event where you officially turn off the COBOL system AND DELETE THE FILES. This is very, very important. You will not go forward if there is some lingering doubt about the new stuff. If everyone likes the new stuff enough, they will be happy to delete the old stuff. If they don't like the new stuff enough, keep converting and fixing until they like it enough to delete the old files.
The "risk" of COBOL programs IMHO is the fact that such programs are running large business critical applications, and if you break them the company's owners will break you. Such programs have to evolve anyway to meet ongoing demands, so the real problem is how to understand how the application is organized, how to make changes, and how to validate the resulting programs. While you can argue endlessly about the merits of COBOL as a programming langauge, these problems aren't any different than it is for other langauges in which large and mission critical applications have been implemented.
One problem that makes COBOL a bit more problematic is a more general lack of tools to help handle it. Java and C# have modern IDEs, lots of searching and indexing capabilities, test coverage tools, class diagram extraction tools, and these all help. Most of the COBOL vendors also have IDEs but I'm not sure they are as capable.
Semantic Designs provides a variety of COBOL tools, including tools for search huge source code bases using COBOL-sensitive searches, test coverage tools, style-checking, clone detection tools, system architecture extraction, etc.
You still have to understand the changes required and make the changes carefully.
Should you decide you want to process the COBOL code to extract other facts or build COBOL-to-X conversion tools, you may find the SD COBOL Front End of use. It is built on top of DMS, an engine for building custom analysis and translation rules.
Yes, building such analysis or translation tools are hard. Another response talks about some of the issues you might face. The SD front ends solve the key problem of, "can you parse COBOL accurately", without which solving the other problems is moot, as you aren't going to translate a large COBOL system by hand within any reasonable timeframe or budget, or risk of scaring management to death. Remember, the application is mission-critical.
The problem may not be COBOL, it might be us. Generally, COBOL applications are large and poorly understood. This leads some to the conclusion that the application is a mess and the only solution is a re-write, or, as in your case, refreshed with a new language.
Consider this: COBOL is generally used to implement complex business applications where the problem space defies bounding through any resonable set of abstractions. Business transactions, and the rules required to implement them, tend to be highly arbitrary, occasionally illogical and sometimes inconsistent. There are exceptions to nearly every rule. Worse yet, business rules can be like fashion: All the rage today and trash tomorrow. Even the strongest foundations crumble when the ground continually shifts (think continental drift).
For another analogy, think of a business application as a patchwork quilt put together from a whole lot of odds and ends over a long period of time. Sometimes a pattern emerges, sometimes not. COBOL is pretty good for building patchwork quilts. Converting your application to some other language 'X' will not change the fact that what you have is a patchwork quilt.
Granted, it may be difficult to recruit new programmers to support a COBOL application. I know when I got my first COBOL job 25 years ago I thought I had died and gone to hell. However, I have since come to respect the language and the complexity of big business. I have also had the opportunity to see what happens when Java is employed in developing large complex business applications. The picture isn't any better. All you end up doing is substituting one set of problems (eg. large monolithic chunks of procedural code) with new ones (eg. Class bashing to accommodate unanticipated hairpin turns in an object model).
Changing your application language as a means of attracting the young and the bright may not work as well as hoped for either. Initially, all problems are new and interesting. Attracting new talent should not be difficult, however, the novelty of building and maintaining a business application wears off in time. At that point the 'young and the restless' will be moving on leaving you with your original staffing problem. The thing that many programmers do not understand is that they are employed to support a business, not the other way round. This can lead to a lot of frustration when something they have put their hearts into is sidelined or redefined beyond recognition. The ultimate solution might be convincing the young and the bright that there is “life” in COBOL after all, it ain’t sexy but it can get the job done as well as anything else.
I find that the majority of COBOL programmers I have worked with are business subject area specialists first and programmers second. They tend to be good at understanding business rules and interactions, but not so good at (or less concerned with) application architecture. Where I work, this problem was solved to some extent through the use of "Bassett Frame Technology". Programmers fall into two broad groups, the application developers and the architecture/infrastructure developers. The application architecture group build reusable "frames" that capture the essence of the architecture and many of the standard system behaviours. Then application developers copy these frames into their programs whenever a particular feature or behaviour is required, leaving them to concentrate on the business. This is a gross over-simplification (check it out in wikipedia or just GOOGLE Bassett Frame Technology).
My advice is to stick with COBOL but consider evolving toward a better support environment and tool sets. Think evolution as opposed to revolution. In evolution there is no particular agenda beyond survival, but over time, this strategy is adaptable to any situation. Revolution, on the other hand, sometimes succeeds but more often than not, results in a huge amount of collateral damage and then fails anyway.
If you still want to convert your COBOL application to another language, I suggest the following: Spend a lot of time analyzing and refactoring the existing code base because there are a lot of “interesting” problems to solve before you should even consider attempting converting COBOL to anything else. Pay particular attention the way COBOL implements flow-of-control (eg. COBOL PERFORM does not follow the call-return semantics that you might expect – see this). This link also points to a few good papers on COBOL restructuring. Finally, another poster (Ira Baxter) has done a lot of good work in this area as well.
Note: On the evolution front, don’t think dinosaur – they were highly successful until a meteor hit and just about wiped the slate clean! Think shark, just as relevant today as 150 million years ago.
One possible approach that I have used is to keep the COBOL core but expand around the edges.
Microfocus provide a tool called Enterprise Server which allows COBOL to interact with web services.
If you have a COBOL program A and another COBOL program B and A calls B via the interface section, the tool allows you to expose B's interface section as a web service.
For program A, you then generate a client proxy and A can now call B via a web service.
Of course, because B now has a web service any other type of program (command line, Windows application, Java, ASP etc.) can now also call it.
Also, they have a product called COBOL.NET which runs inside Visual Studio and translates COBOL to MSIL. This means that you can then link in any .NET components.
So the approach is to keep the COBOL core but interface via web services and do new developments in any CLR compliant language (C#, VB etc.)
This is a very simple question with a quite complex answer. There may be many approaches to migrating a COBOL application to other language, but there are many variables involved. I will try to be pragmatic:
I. Is the System still evolving/being fixed?
If there is a large number of changes in the source throughout your project, you are trying to hit a moving target. Don't. First stabilize the COBOL system by, unfortunatly, spending money improving it.
II. Check the gap:
Beware that if you choose more than one alternative, break down your project into distinct phases. If you try more than one of those at a time, the difficulty will be the same as creating a new system and migrating its data.
III. Design the Target
Based on your answer to the previous question, you will decide what technology to use on the target system. In relation to the previous:
III. Remember Collating Sequence
If your source is mainframe, beware. A sorted file on the mainframe will be unsorted in ASCII systems.
IV. Chose your tools wisely
There isn't a perfect tool to migrate COBOL. IBM will say theirs is, Micro Focus will also, but all of then have pitfalls and many, many problems.
V. Data Strong
COBOL rely heavily on data. So you should spend more than half your project analyzing COBOL data structures. And the other half you will spend fixed what you couldn't get from analysis.
VI. Don't use in-house compilers
Using your own compiler doesn't mitigate any risk, but adds TCO. Buy one instead. The most reliable one will be from IBM, given its size. Micro Focus is investing in mainframe downsizing so it's another good choice.
VII. Buy tools for data analysis
Yeah.
VIII. Make sure you have time
This is perhaps the worse part. Testing a batch system take take hours to run and produces millions of business records will take days. Every fix you do on the source will take that many amount of hours to run again, and a lot of mandays to check.
*IX. Establish Acceptance Tests DURING ANALYSIS PHASE
Most COBOL systems are very old, which, unfortunately means that few people know how they work, and probably none knows every business rule implemented. Which means your acceptance you be hell, unless you make a very clear agreement with your client on how the both of you will consider the system functional and bug-free.
X. Migrate bugs
Always, whenever migrating an application, make sure you reproduce bugs.
XI. Beware of Other Languages
Mainframe has EZtrieve, Natural, Assembler living together with COBOL. If your source system is mainframe, make sure you only have COBOL sitting there.