views:

1126

answers:

19

I have a bunch of scripts - some in perl and some in bash - which are used for:

  • Creating a database (tables, indexes, constraints, views)
  • Parsing spreadsheets and loading the data into the database
  • Getting info about a bunch of files and loading that into the
    database.

These scripts are used in conjunction with a much larger application that is written in java, and my manager has requested that I rewrite the scripts in java. His reasoning is that it is easier to work with, port, manage, understand, and support if it's all in one language, and that too many separate pieces is a design issue.

My initial reaction is that this is a bad idea. The scripts are beautifully concise and fast, and tasks that are trivial in the scripts - such as using regexs to find and replace invalid values - will be so much more verbose and very likely slower when done in java.

The one drawback of the scripts is that when they run on windows they require cygwin in order to run. Therefore I would like to give a counter proposition that I port all the bash scripts to perl so that they can run on windows without cygwin, and that I spend time organizing and documenting the scripts.

The problem is that a "gut reaction" type of response is not going to be enough to convince my manager. I come from a linux background, he from Windows, and we have some of the classic linux vs. windows differences in approaches.

So I have two questions:

  1. Is my "gut reaction" correct? Is java slower, more verbose, and harder to maintain for database management, spreadsheet parsing, & file processing tasks?
  2. If the answer to the first question is yes, what is the best way to present my case?


EDIT: Thanks everyone for the insights. I'd like to make one clarification: the scripts are not full-blown apps hidden away in obfuscated scripts. They are, for the most part, tasks that had been done manually that I automated via scripts and later embellished as the requirements developed. And the reason I used a scripting language instead of java to start with is because these tasks were so much easier to do in scripts. For example, one script runs a bunch of queries, formats the results, and outputs them to a file. How many LOC do you think it would take to do that in java?

+2  A: 

It's not clear from your description: are the shell scripts being called from the Java app, or are they just working side-by-side? If the former is the case, then you have to consider the overhead of having Java fork a process each time to run the scripts, which reduces the performance gain from using them in the first place.

Alex
The scripts are working side-by-side. They are used to get all the pieces in place before the java app can run.
MCS
A: 

Is my "gut reaction" correct? Is java slower, more verbose, and harder to maintain for database management, spreadsheet parsing, & file processing tasks?

No.

It seems like your manager is tasking the wrong person to do this. Its clear that you're not comfortable writing Java and that you shouldn't be doing so. Why doesn't one of the developers from the "java side" help you out?

Kevin
No, I am comfortable writing java - I wrote most of the java app that these scripts support. And there is no other developer on the team.
MCS
Will it take longer to Execute? Probably not, like this author says. Will it take longer to write in java? You bet. For manipulating files and moving stuff around, you want the OS on your side.
Robert P
+23  A: 

The trouble is, your Gut reaction might be right, but that doesn't mean your manager is necessarily wrong - he probably has very good reasons for wanting it all done in java. Not least, if you fall under a bus, finding a replacement who knows java, perl and bash is going to be a lot harder than finding someone who knows java. And that's leaving aside the "they can only be run on a PC with cygwin installed" issue. And in all likelihood, performance isn't as big an issue as you think it is.

Having said that, your best bet is to spend a bit of time estimating the time it will take to port them all to java, so he can make an informed decision. And while you're at it, estimate how long it would take to port the bash scripts to perl and document them. Then let him decide. Remember - he doesn't get to spend the majority of his time coding, like you do, so it's only fair that he gets to make some decisions instead.

If he decides to proceed with the java option, port one of the scripts as well as you can, then report back with the two versions and, if you're right about the concision of the perl/bash scripts, you should be able to get some mileage from examining the two versions side by side.

EDIT: MCS, to be honest, it sounds to me as if those scripts are better implemented in perl and/or bash, rather than java, but that's not really the point - the point is how do you demonstrate that to your manager. If you address that, you address both the "gut reaction" question (btw, here's a tip - start referring to your gut reactions as "judgement, based on experience") and the "best way to present my case" question.

Now, the first thing you have to realise is that your manager is (probably) not going down this path just to piss you off. He almost certainly has genuine concerns about these scripts. Given that they're probably genuine concerns (and there's no point in going any further if they're not - if he's made his mind up to do this thing for some political reason then you're not going to change his mind, no matter what, so just get on with it and add it to your CV) it follows that you need to provide him with information that addresses his concerns if you're going to get anywhere. If you can do that then you're more than halfway to getting your own way.

So, what are his concerns? Based on your post, and on my judgement and experience :-) I'd say they are:

  • maintainability
  • that's it, just maintainability

I would also guess that his concerns are not:

  • performance

I might be wrong about this last one, of course; in the last place I worked we had a SQL Server performance problem to do with replication that impacted the business's ability to provide customer support, so performance was an issue, so we addressed it. But generally speaking performance isn't as much of an issue as programmers think. If he's actually told you that performance is an issue, then factor it in. But if he hasn't mentioned it, forget it - it's probably only you that thinks the fact that these scripts run faster in perl/bash than they probably will in java matters at all.

So, maintainability. This comes down to answering the question "who will maintain these scripts if MCS falls under a bus?" and the supplementary question "will that cause me (i.e. your manager) problems?" (Aside: don't get hung up on the whole bus thing. "Falling under a bus" is a useful and diplomatic shorthand for all sorts of risks, e.g. "what happens if someone lures him away with a salary my company can't match?", "what happens if he decides to emigrate to Bermuda?", "what happens if I want to fire him?", "what happens if I want to promote him?", and, of course, "what happens if just he stops turning up for work one day for some unknown, possibly bus-related, reason?")

Remember, it's your manager's job to consider and mitigate these risks.

So, how to do that?

First, demonstrate how maintainable these scripts actually are. Or at least how maintainable they can be. Document them (in proper documents, not in the code). Train a colleague to maintain them (pick someone who would like to acquire/improve their perl and bash skills, and who your manager trusts). Refactor them to make them more readable (sacrificing performance and clever scripting tricks if necessary). If you want to continue using bash, create a document that provides step-by-step instructions for installing cygwin and bash. Regardless, document the process of installing perl, and running the scripts.

Second, pick one of the scripts and port it to java. Feel free to pick the script that best demonstrates the advantages of perl/bash over java, but do the best job you can of porting it. Use java.util.regex to do the same clever things you do in your perl. Document it to the standard that other in-house java utilities are documented. If performance is actually a factor, measure its performance relative to the perl/bash script.

Third, having been through that exercise, be honest with yourself about their relative maintainability. Ask the guy you trained what he thinks. If you still think the perl/bash scripts are more or less as maintainable as java versions would be, estimate the work involved in porting the remaining scripts to java as accurately as you can (you'll be able to do this pretty accurately now, because you'll have actually ported one). Then take the comparative scripts and the documentation and the estimates (and the performance figures, if appropriate) to your manager and go through them with him. Present your counter-proposals (a. leave them in perl and bash but document them and train a colleague, and b. port the bash scripts to perl, document them and train a colleague).

Finally, let your manager weigh up all the information and decide, and abide by his decision. In fact, don't just abide by his decision, accept the fact that he might be right. Just because you know more about perl/bash/java than him doesn't mean you necessarily know more about managing the team/department than he does. And if his decision is to stick with perl/bash, or port to perl, rejoice! Because you have not only got your own way, you have gone up in your manager's estimation and learned an invaluable lesson along the way.

gkrogers
Yes, the bus hit factor is very high here.
MCS
is it? I consider a working knowledge of those 3 languages a prereq for pretty much any useful work.
rmeador
Definite bus-hit factor, agreed.At my last job, I was tasked with porting my perl to vb6. Unfortunately, I accurately estimated 4000 hours (twice the time it took to write the perl, which I wrote almost non-stop for 2 years).Never happened, now they all hate me. Oh well.
Just because you port them to java, doesn't mean you reduce the bus-hit factor entirely. Bash and Perl scripts ported to java are likely to need a lot of Process management which isn't trivial in java (especially when compared to Bash and Perl). Maybe that's why they were scripted?
Adrian Pronk
+7  A: 

Personally I find db, file management harder to do with java, but it may be easier to maintain once they are written.

But is it worth it? If it works, don't 'fix' it.

Personally I don't care - If I get work to do, I debate pros and cons with my manager and if she insists, I do it and get payed. Usually she comes to her senses though, and gives me more important work to do.

Spikolynn
+6  A: 

I certainly agree that it is easier for everyone if you work with a set of tools that most of you know. However, since you have both Java and Perl code I am assuming that at least some of you know both, and as such I honestly do not see the big problem with having both Java and Perl code.

If the Perl scripts work as expected and can be maintained I would not spend time rewriting them in Java. Scripting is a lot easier in Perl than it is in Java imo, so unless you really need to convert, I don't see the point. I would prefer to spend the hours on something that actually adds value to whatever you're doing.

You say that the scripts need cygwin to run. I have done a lot of Perl on both Unix/Linux and Windows, and unless you're doing a lot of specific Unix stuff my experience is that scripts can easily be converted to run under a Windows Perl like ActiveState. Maybe that could be an option in your case.

Brian Rasmussen
+4  A: 

Should they be rewritten? That depends. The strongest argument that your boss has is that the rest of the application is written in Java and sounds like that might be the way the organization is headed. Reducing the number of different langues that must be supported by the organization is actually a pretty smart long term decision. I know, I know, right tool for the right job, but from a cost perspective,it is entirely possible that it will cost the organization more money to hire someone who knows both PERL and JAVA than just Java. Even if the scripts are beautiful they still have to be supported, and that means he has to keep at least one person on staff who knows how to do that. It's another thing that he (and the organization) has to worry about at the end of the day.

Kevin
A: 

For me, it depends on how badly written the Perl is (I've never seen Perl that I would say was "WELL" written), and whether you'll ever need to READ the Perl.

Perl is often a Write Once, Read Never language. If it all works, and you're not likely to need to alter it, I would say don't touch it.

Brian Postow
If Perl is WORN, Java is a Never Even Finish Writing Once, Read Never language. Isn't this level of discourse intelligent?
jrockway
I once read (may even have been on SO) that Java was a "Write Once, Debug Everywhere" language ;-)
RET
+3  A: 

I can see what you are saying, but being short and concise isn't always maintainable--sometimes verbose and explicit is maintainable.

Also, once it's all in Java, you'll be more likely to have a UI/Control Console feel which might be an improvement.

If you really like the scripting language feel, maybe you could counter-propose groovy. It's syntax is very easy for Java programmers to pick up and it's 100% java compatible (including extending java classes in groovy and the like), yet it's a scripting language--as powerful as any--with all the power and lack of compiling that implies.

By the way, Java handles regular expressions fine.

Also by the way, If you wrote all these scripts and are the only one familiar with them, you might want to start looking around for a new job. Sorry to say it, but asking you to make your "Special little tricks" documented and maintainable is often something they don't think about until just before a layoff.

Bill K
Java reg ex: one caveat, they behave as if they started with "^" and ended with "$", frequently requiring ".*" to be added to the edges if you are looking for a pattern *somewhere within* the searched string, rather than a "complete" match.
Roboprog
+6  A: 

I think your first reaction is right. One argument is If it works, don't 'fix' it. Another argument is that one developer can write almost same amount of SLOC independently of language which used. It sounds strange if you know how is Java verbose but think about how carefully you must design your Java code to get same result using perl features as closures, dynamic generated code, instant regexps and other. And now when Java to Perl SLOC ratio to same result is more than 10:1. Each line of code you must read, understand and maintain. Java is faster. Yes. Java is faster for some thinks as number crunching and some sort of text processing. Perl is faster for regexps and some other text processing and far far more productive than Java generally. Perl is worse maintainable if compared by SLOC but same or better than Java if compared by feature. If Perl is written using best practices and keep coding style than can beat Java in maintainability especially if used for short scripts.

Hynek -Pichi- Vychodil
+5  A: 

From my own experience (which includes mixing Java and Perl in a single system), I'd suggest the following:

1) "Java is slower" is not necessarily true, but also isn't relevant (even if true) unless the additional run time interferes with some time-critical workflow.

2) Long-term maintainability is a legitimate issue. Having e.g. a single DAO layer that doesn't have to be maintained in two languages can pay back in the long haul. How much of your Java code and current scriptage would have to be modified (twice) to cover a refactoring in the database?

3) If you really have a preference for lighter-weight notation, but your manager wants Java, could you compromise on Java libraries (from previous point) combined with one of the interoperable scripting-like languages that runs on the JVM and could share use of the standard libs you write for e.g. database access? I'm thinking of something in the JRuby-Groovy-Scala-Jython spectrum.

joel.neely
+5  A: 

In general, I understand your manager's desire to minimize and standardize on the different languages/platforms used in your environment.

However, there are certain tasks for which a scripting language is much better suited than a language like Java. If you feel that's the case with the scripts you're being asked to rewrite, maybe rather than proposing using Perl as a one-off language for this particular task, you could propose adopting Perl (or another scripting language if you think you'd get better buy-in) as the "supported" language for scripting tasks.

That said, depending on what you mean by "used in conjunction with" (that is, how tightly coupled the different bits are), it may simply be the case that these tasks would make more sense written as Java libraries to be called by the rest of the application.

John Price
+2  A: 

Just one point. In many ways, he has a point, but...

Perl (or bash scripting) is a glue language. It is one of the best languages out there for sticking to systems and making them work better. Perl is a fully-interpreted language, which affords it significant power for run-time-code-rewrite and more dynamic programming styles. You can pass perl code blocks around as data, and modify them up until the moment you call "eval" on the string. Whether or not there's native java functionality to embed perl, you can easily create such embedding yourself, making for an immensely powerful system.

You might want to make clear to your supervisor what potential you will lose if you remove the perl. At my last job, two of the developers got IronPython added to our "legal language list" so that we could implement libraries and trivially pass them through the database for a massive-scale automation project that turned into a very simple, very tiny project, with a bunch of python code gluing and being glued to compiled modules.

All in all, there are times when a million lines of Java cannot do what 10 lines of Bash script does. That's when you want to use it. The rest of the time, your boss is right, so long as you're afforded the time to do it.

Perl is not "fully-interpreted". shell is. Perl is compiled to an op tree and then run by a VM.
tsee
That's completely semantics.It could be converted to pascal behind the scenes, compiled with debug extensions, then run with any errors converted back to the perl.it still acts interpreted and can utilize the benefits of if.
It may be semantics but it's true. It's just as (pick one) compiled OR interpreted as Java. Except that perl is compiled much more quickly, so it doesn't (much) matter that it's recompiled every time you run it.
runrig
+12  A: 

It depends. I've found that text processing in Java can take up to 8 or 9 times the amount of code as in Perl. If these scripts need to be tightly integrated into the application then I would agree with your manager but if there just background tasks I'd look into using ActiveState on windows and rewriting the bash scripts in Perl.

Jared
For a more streamlined CPAN experience on Windows, consider Strawberry Perl
jplindstrom
+2  A: 

If you build a shed and use a hammer 80-90% of the time, does it follow that you should use only hammers to build sheds? No, you use the most appropriate tools for each part of the job, just as you have done!

Also the average skills/experience level of the IT work-force has increased in recent years. E.g. This SO Poll showed the medium SO programmer is in their 30's with over 10 years experience.

You boss will have no problem recruiting programmers with a broad mix of skills and experience.

snoopy
+2  A: 

Have you considered Ant? I have to admit I never tried, but always wanted to port my scripts to Ant. File operations are easy and there are even tasks to create SQL statements. Of course if your scripts are more like programs, i.e. many loop constructs, then this is not the way to go. Just a suggestion.

Familiarity breeds contempt: I have to use Ant. It often sucks. Ant is no Make.
Roboprog
+4  A: 

Convert to all Perl

Your right to think that Java's Regexp is slower. Perl's Regexp variant has undergone many changes to make sure that it is as fast as possible.

Converting from BASH to Perl should prove easy to accomplish, Perl can easily do what you are doing in BASH.

By getting rid of the BASH files, you can also get rid of Cygwin.

Brad Gilbert
+3  A: 

Just do as you said: convert your shell to Perl and document it

The code you mention seems not to be part of the application, it seems to be "setup" code or "maintenance" code. As one answer notice, "one job = one tool":

  • for your app, it's Java,
  • for packaging your app, it's ant or maven or make,
  • for setting up environment, fill in the db, making reports from logs, it's a scripting language (Perl, Python, shell).

To convince your boss:

  1. http://en.wikipedia.org/wiki/Golden_hammer
  2. migrate from one language to another is risky: you'll have to spend lot of time to check for regression errors
  3. In my experience, one line of Perl = 20 lines of Java (give a try: migrate one of your Perl script). So the codebase will be multiplied by 20, and more code to maintain is more headakes
  4. Perl maintain all its modules and doc in the same place (cpan.org). For Java, there is no "reference point". You'll have to waste time on the net to make a choice between java spreadsheet parsers, learn to use it (hope the doc will be ok), and make some java-cryptic-glue-code:

    SheetHolder = ParserFactory .newInstance(Configuration.asProperties()) .parse(SheetReader.asStream());

+1  A: 

On a project in the past Perl code was ported to Java resulting in significant speed increase. The company had mostly Java programmers and our tools Eclipse, Ant, JUnit and Maven were not suited for Perl development. I have seen Perl code in lots of companies, but most of the time it was only meant as temporary solution, quick fix, prototype, demo etc .. It makes sense to rewrite, but you should look at it on a case by case business, sometimes time or manpower would not allow it.

Navi
A: 

Remember Java isn't the only JVM language - perhaps something like Groovy or Jython would be a compromise that would keep everyone happy.

Gaius