Interactive Data Language, IDL: Does anybody care?

Question

Anyone use a language called Interactive Data Language, IDL? It is popular with scientists. I think it is a poor language because it is proprietary (every terminal running it has to have an expensive license purchased) and it has minimal support (try searching for IDL, the language, right now on stack) . I am trying to convince my colleagues to stop using it and learn C/C++/Python/Fortran/Java/Ruby. Does anybody know about or even care about IDL enough to have opinions on it? What do you think of it? Should I tell my colleagues to stop wasting their time on it now? How can I convince them?

Edit: People are getting the impression that I don't know or use IDL. Also, I said IDL has minimal support which is true in one sense, so I must clarify that the scientific libraries are indeed large. I use IDL all the time, but this is exactly the problem: I am only using IDL because colleagues use it. There is a file format IDL uses, the .sav, which can only be opened in IDL. So I must use IDL to work with this data and transfer the data back to colleagues, but I know I would be more efficient in another language. This is like someone sending you a microsoft word file in an email attachment and if you don't understand how wrong that is then you probably write too many words not enough code and you bought microsoft word.

Edit: As an alternative to IDL Python is popular. Here is a list of The Pros of IDL (and the cons) from AstroBetter:

Pros of IDL

• Mature many numerical and astronomical libraries available
• Wide astronomical user base
• Numerical aspect well integrated with language itself
• Many local users with deep experience
• Faster for small arrays
• Easier installation
• Good, unified documentation
• Standard GUI run/debug tool (IDLDE)
• Single widget system (no angst about which to choose or learn)
• SAVE/RESTORE capability
• Use of keyword arguments as flags more convenient

Cons of IDL

• Narrow applicability, not well suited to general programming
• Slower for large arrays
• Array functionality less powerful
• Table support poor
• Limited ability to extend using C or Fortran, such extensions hard to distribute and support
• Expensive, sometimes problem collaborating with others that don’t have or can’t afford licenses.
• Closed source (only RSI can fix bugs)
• Very awkward to integrate with IRAF tasks
• Memory management more awkward
• Single widget system (useless if working within another framework)
• Plotting:
  • Awkward support for symbols and math text
  • Many font systems, portability issues (v5.1 alleviates somewhat)
  • not as flexible or as extensible
  • plot windows not intrinsically interactive (e.g., pan & zoom)

Pros of Python

• Very general and powerful programming language, yet easy to learn. Strong, but optional, Object Oriented programming support
• Very large user and developer community, very extensive and broad library base
• Very extensible with C, C++, or Fortran, portable distribution mechanisms available
• Free; non-restrictive license; Open Source
• Becoming the standard scripting language for astronomy
• Easy to use with IRAF tasks
• Basis of STScI application efforts
• More general array capabilities
• Faster for large arrays, better support for memory mapping
• Many books and on-line documentation resources available (for the language and its libraries)
• Better support for table structures
• Plotting
  • framework (matplotlib) more extensible and general
  • Better font support and portability (only one way to do it too)
  • Usable within many windowing frameworks (GTK, Tk, WX, Qt…)
  • Standard plotting functionality independent of framework used
  • plots are embeddable within other GUIs
  • more powerful image handling (multiple simultaneous LUTS, optional resampling/rescaling, alpha blending, etc)
• Support for many widget systems
• Strong local influence over capabilities being developed for Python

Cons of Python

• More items to install separately
• Not as well accepted in astronomical community (but support clearly growing)
• Scientific libraries not as mature:
  • Documentation not as complete, not as unified
  • Not as deep in astronomical libraries and utilities
  • Not all IDL numerical library functions have corresponding functionality in Python
• Some numeric constructs not quite as consistent with language (or slightly less convenient than IDL)
• Array indexing convention “backwards”
• Small array performance slower
• No standard GUI run/debug tool
• Support for many widget systems (angst regarding which to choose)
• Current lack of function equivalent to SAVE/RESTORE in IDL
• matplotlib does not yet have equivalents for all IDL 2-D plotting capability (e.g., surface plots)
• Use of keyword arguments used as flags less convenient
• Plotting:
  • comparatively immature, still much development going on
  • missing some plot type (e.g., surface)
  • 3-d capability requires VTK (though matplotlib has some basic 3-d capability)

Answer 1

To defeat your enemy you must learn it, though not necessarily master it.

I would take a favorite language of yours and compare and contrast it to IDL. What advantages does IDL have? What disadvantages? Ask your colleagues to show you some of their favorite code written in IDL and compare it with some of your favorite code snippets. It may not be all bad - people are paying big bucks and using it after all.

Trying to replace something that is a favorite may not be a good way to win over friends or win an argument. If you conclude that IDL offers only slim or no advantages then try to build a gateway from a newer language (such as Ruby) to IDL so people can use either in conjunction. You may be trying to remove an entrenched culture that's very-happy-with-IDL-thank-you-very-much, though if they take to a hybrid approach and eventually replace their IDL with another language because the other language is actually superior then you'll have made a positive impact.

Whatever the case, I'm never a fan of paradigm shifts and losing time and money invested in one tool for ideological benefits of another is a bad business decision. However old proprietary technology must be phased out when it makes sense to do so - though a phased approach is key.

Answer 2

Hi, Being a Ph.D. student in astronomy I started using IDL about a year ago. I'm also not entirely convinced of this language and certainly quite annoyed by it being proprietary (but check out the open-source variant GDL: http://gnudatalanguage.sourceforge.net/). Nevertheless it is a very powerful language that has many tools and functions built in that scientists use a lot, e.g. it can do all sorts of fits out of the box and has a large number of graphical plotting tools ready to use. Also, there are many tools that built upon IDL, like the IDL Astronomy User's Library (http://idlastro.gsfc.nasa.gov/). So, while it might have been better to implement all these tools into an open language in the beginning, I'm afraid there is no back now that so many people are using and -- and have gotten used to it.

Answer 3

Well, I did search for IDL on stack and that's how I got here! :-)

I've been programming for almost 30 years and am just learning IDL. Thus far, I admit that I'm not overly fond of it. However, it does have some things that many other languages don't have (e.g. mathematical array operations can be done with a single command, not with loops or some other device).

As others said, IDL's popularity is somewhat cultural and a matter of being entrenched. I first heard of IDL almost 20 years ago. At that point in time, it seemed to me that Stallman had some nice ideas and a few useful tools out, and Linus had not yet put out his famous comp.os.minix message. Thus IDL had a good head start on anything open source that would be competitive; to my knowledge nothing in the open source community is competitive yet (and I know about GDL, but if I'm not mistaken, it's way behind IDL - I'd be happy to be corrected on that). Eventually, it may be supplanted, but I do not expect that to happen soon.

Answer 4

Hi, Correct me if I'm wrong, but are you asking if you should tell your colleagues to "stop wasting their time" on a language, you don't know, but don't like because it's proprietary????

Well, I think it's a bit short sighted. First, you should ask your colleagues "why they are using IDL". I've been using IDL on and off for about fifteen years, and I think they'll tell you "because I can do what I have to do quickly". I've been programming in IDL/C++/LabVIEW/Python/Pascal for 20 years and I think you should use the language/environment that's best suited for the job. I wouldn't use IDL for a flashy user-interface application, but for analyzing and visualization of Gigabytes of data it's hard to beat. (I certainly wouldn't use Python or Ruby for that!)

And about IDL being proprietary software. Well that's true (though you don't need expensive licenses to run an IDL application. You can use the Virtual Machine to run applications, you need an expensive (true) license to develop applications). But, what's wrong with being proprietary? My car is a proprietary product (and I guess yours is too :-) ), so are my tv set, telephone, etc. etc. So IDL is proprietary which means you can't change the language (other than asking ITTVIS for changes), but you don't even know the language! So what's the problem? (BTW the open source version GDL was mentioned and there are other (open source) alternatives). How much have you contributed to C++/Python/Ruby etc?

I hope the proprietary argument isn't (mis-)used because you don't like the (high) license fees? It's true there are free (read: no money being transfered) C++/Python/Java compilers, but ITTVIS is a commercial company, who wants to make money. Well I'm a professional programmer, and though I support the open source thought, I do like being paid at the end of the month (guess where that money has to come from :-) ). (BTW I'm no ITTVIS employee.)

So, in short. If you think IDL is too expensive, that's ok (but don't use the proprietary argument). There are alternatives, you're free to choose! But before you (ask your colleagues to) switch, ask yourself what the consequences are for your productivity! You might save a couple of thousand dollars on license fees, but if it takes 10 times as long to complete a job.........

Kind regards

Answer 5

I'm an fMRI researcher at Children's Hospital in Cincinnati, and for years radiology researchers here have used IDL to develop an image processing package (called CCHIPS). It is a very well developed package, and has had quite a few people expanding its utility over the years. Even though I'm pretty much a hardcore MATLAB user, and therefore have a tendency to lean towards packages like SPM for fMRI image processing, I still end up using CCHIPS and writing/editing a few IDL scripts quite often. It's not as "comfortable" to me as MATLAB (we all have out favorite "blankies", after all!), but it's fairly easy to learn.

I guess my point is this... If a programming resource works well, is well established, is used well by many, AND (most importantly!) there are still people within your organization willing to maintain/debug/modify/expand the resource, then there's no need to take an alarmist stance that changing to something new must be done immediately. If there were no one at all at your institution willing to maintain a resource any more, that may be cause for urgent concern. Otherwise, I would suggest getting a little more familiar with what you don't like, then applying gentle pressure to introduce what you feel is a better option, backing it up with clear reasons for change.

Answer 6

I've been using IDL for 8 years in a medical imaging research laboratory. I also use MATLAB, LabVIEW and Visual C++.

IDL Cost: It's true that programming terminals need IDL licenses. However, you can run your IDL applications under their free "Virtual Machine" on any terminal if you can put up with a splash screen. Also, a lot of other languages/development environments cost as much as IDL (if you use them legally). Visual Studio is more expensive at this university per license than IDL.

IDL/MATLAB vs Visual C++/etc: You can write a program or GUI application in a day in IDL that would take a week to write in C++/Visual C++ -- that's a quote from our expert Visual C++ programmer. IDL only takes two weeks to learn, and there is an excellent book to learn from. Of course the C++ program will run faster and you will have more controls to play with if you add a Visual GUI. However, if you want to prototype an algorithm or an application with a user interface to analyze data, IDL (or MATLAB) will save you a lot of time.

IDL vs MATLAB: IDL is a little more verbose than MATLAB, and does not have the user base or the number of toolboxes, but the main user forum it does have is excellent, with a number of responsive experts. It used to be that the IDL GUI programming interface was superior, although MATLAB may have caught up -- I'm still much more comfortable programming IDL "widgets." Also, IDL's built-in functions sometimes seem to have a little more "built-in," which may make up for its smaller user base. A good example is the convolve command: "convol" in IDL vs "conv" in MATLAB. The command is a longer word in IDL, but there are also included flags for normalizing the result, as well as for dealing with invalid data and edge effects. MATLAB syntax is more elegant and concise, and it's nice to be able to directly return more than one value from a function.

Trust me: learning "scientific data" languages like IDL and MATLAB is worthwhile if you want to spend more time working with data than working with code. I'm not going to say one is better than the other, but languages like this can be indispensible in the lab, especially an imaging lab.

Answer 7

I had similar thoughts when I started learning IDL -- and I'm still not a fan of it, but I now have some code out there in the wild that I support in SolarSoft (a software distribution system for solar physics ... and most of the software's in IDL).

IDL's good at processing large data cubes and digital images, and it's fairly easy to learn if you know Fortran. (which most of the older scientists do, and I had to take in college as an engineering student who wasn't CompE)

The thing is -- there's a LOT of code out there in IDL. The cost of converting it all to some other language, and giving it the necessary testing is just astronomical. My boss's estimate is that it'd take over a dozen man years, and we'd need people who understood IDL, whatever new language, and the actual science so they understood why the logic of the code is there (and not just some oddity of the language). ... and then you have to consider retraining all of the scientists. At our current funding level, that's just not going to happen.

All of that being said, I'd love for their Regex engine to use PCRE or at least supported zero-width assertions. They've finally made it so I can pass a string into the XML parser without having to write it to a file first, but I'm still waiting for real SOAP support, and not my hack attempts at it. I have to work around restrictions like no zero-element arrays (I use pointers to arrays, so I can leave a null pointer), namespaces (I fake it with objects) and lack of XPath support (I've seen some odd problems when after walking DOM tree, the cleanup time quadruples for every doubling of elements in the tree ... I've had to return large recordsets as tax delim, rather than XML, but hope to test if their VOTable implementation has the same issues)

At this point, I'm not suggesting that people stop using IDL -- but I am campaigning to get people to stop storing and distributing their data catalogs as IDL save files, due to the restriction on reading them with anything other than IDL. And I know of a few efforts to make services for scientists to run their data prep, so that hopefully they can get away from requiring scientists to have IDL to be able to use the data.

Answer 8

I'm a post-doc in earth sciences, and have used IDL extensively for processing large datasets. I have to say it is very EASY to work with (for me and many others anyway), and if it saves me hundreds of hours of debugging c memory errors, than it is easily worth the license cost (I have a CS degree as well, so I know "real" programming languages).

It sounds like your problem is not so much with IDL as it is with the file format. Rather than convincing your colleagues to use a different language, convince them to use a different file format. I have almost never used .sav files, I always use common formats for data such as geotiff, HDF, netCDF, or even simple binary or ASCII as appropriate. That way, all of my colleagues can use their language of choice to read it in.

I got here looking for info on using IDL with SOAP, and, sadly, the comment above won the google contest. Now if only I could find some actual info on it :)

Answer 9

I'm a climate scientist, and I use IDL every day. It's a love-hate relationship. But I'm going to come to IDL's defense here because I don't think you've articulated good reasons for the scientists you work with to make the switch.

The only viable replacement for IDL in that list is Python coupled with the full set of scientific libraries. Even then, many things are harder, or more verbose in python than in IDL. Languages like C and Fortran are just too low level for tooling around analysing a dataset or making some figures, and they lack an interactive shell.

Most scientists are interested in getting answers to questions, and tools like IDL, Matlab, and NCL are purpose built to help us get answers faster.

Answer 10

Perhaps this might help someone: GDL does handle (read & write) the IDL .sav files. The SAVE and RESTORE routines are implemented with the help of the free CMSV library (written in IDL).

Furthermore, GDL can be built as a Python module- one can then use the .sav files from Python as well. The Python support in GDL still uses the numarray package, so it might not be very convenient, though.

Answer 11

If the con of expense is the main concern, and you need only the IDL language interpreter itself, not the fancy packages, you could be happy with Fawlty Language, f free clone of IDL.

http://fl.net23.net/

I've actually done real work released to the public with this instead of IDL, w/o telling the boss, as a test, and succeeded, but then I mostly run non-GUI programs and do interactive command line work. GUI widgets weren't complete last time I checked. However, Fawlty was way ahead of GDL in terms of straightforward programming.

Answer 12

I'm an astronomer and have used IDL for many years. There are some things that are very nice about it - e.g. array handling including string arrays - and quite a bit of astronomy related routines are available. On the other hand, as a language I prefer python. Cost is an issue for IDL licenses - not cheap even for a single user. There are other annoyances as well. The PostScript plots that IDL makes by default are not very good. It's annoying to have specify thick lines every time and the font is kinda ugly. I've started using the matplotlib python module and it has much to recommend it. For example, one doesn't need to remake a plot to change an axis title. I only wish that I had all those handy IDL astronomy library routines written in python for matplotlib.