I plan to add a better search feature to my site, so I thought that I would write it in C and use the CGI as a means to access it. But it seems that Perl is the most popular language when it comes to CGI-based stuff. Why is that? Wouldn't it be faster programmed in C or machine code?
What advantages, if any, are there to writing it in a scripting language?
Thanks.
String manipulation, often a large part of web development, is rather painful and error-prone in C, in parts due to the lack of automatic memory management. Keep in mind that often, script execution time isn't the bottleneck or can be circumvented by proper caching mechanisms. In many cases, it's a good idea to choose a language which maximizes developer productivity instead of unnecessarily sacrificing development time for performance gains which will go unnoticed by the user of the site.
This general principle does not, however, fully apply in your case, as a search engine might well benefit from optimized low-level code. This doesn't mean that you'll have to do everything in C, though: the PHP interpreter has been known to be painfully slow, but as most library functions are implemented in C, you can get away with it. I'd recommend to write the app in the high-level language of your choice, and only re-implement the parts in C which have been identified as botlenecks.
Security, for one thing. If you write in C, you have to be very careful to make sure all your string handling is correct so you don't introduce buffer overflows, etc. In any decent scripting language, someone else has already done that for you. You may be able to have other security holes, but unless there's a bug in the runtime or an extension module, you won't have a buffer overflow. This benefit is not limited to scripting languages; compiled languages such as Java and C# provide it as well, and it is obtainable (albeit frequently more difficultly) in C++ with std::string and C with a good string library.
Securitywise, Perl has another useful feature not seen in many other systems: "taint" mode. This keeps you from blindly passing user input to other systems as part of a database query, command line, etc. This is an excellent benefit when writing CGI scripts, as your script will die cleanly before it passes uninspected user input off to the shell for execution. Taint mode is not perfect, as the untainting process depends on the programmer doing things correctly, but it at least helps catch code paths you missed.
Also, at this point, Perl has been used for a long time for CGI scripting, so there is a large body of libraries, frameworks, etc. already in existence to make writing new scripts easier. Plus CPAN has code to do just about anything.
In addition to the answers mentioned already, for basic web applications network transfer speed is a more common bottleneck than language choice. It is usually easier to write web applications in Perl than C, so a small difference in runtime speed is not worth the extra effort needed to create the application. C is in fact sometimes used for certain parts of very computation-intensive web applications.
OK, the rest of the answers gave pretty good objective reasons. Just for completeness, here's a subjective evaluation to give it some color:
I wrote:
Based on those experiences, the pure C one gave the most satisfaction as far as "Look at this cool technical achievement I made" angle. Especially as that was in the days when CGI was brand shiny new and static HTML was the main content everywhere.
The roll-my-own Perl CGI was a lot easier technically than C one due to all the objective reasons listed in other answers.
And the CPAN Perl projects were the only ones that provided fairly decent delivery turnaround time and allowed me to concentrate on building the business logic as opposed to the plumbing.
Back in the day when CGI was becoming popular, Perl was the easiest language to use. People could pick up "baby Perl" very quickly, and since the program was a text file, they could easily upload it and pass it around. Since Perl started life as a system administration language, lots of servers already had it installed. When it came time to make a CGI script on some hosting service, Perl was most likely already there. Not only that, a Perl script is pretty much the same on any platform, so what you wrote locally most likely worked exactly the same on a different machine.
It was faster to program for "accidental programmers" in the big scheme of things because they had less to learn before they could make a useful program; they could start with nothing and have a Perl program running in an hour, even if they were just cargo-culting it. They didn't have to worry about all the things that come with writing and compiling a C program, then transferring it to another host (which might be a different platform).
Perl got a quick foothold, and you still see the effects of that today. If Perl had to start from scratch today, I don't think it would necessarily win out over anything else. PHP has certainly taken over the low-end, quick-startup crowd (and for most of them, it's probably the right tool at first).
It didn't hurt that Perl had a lot of text processing features, either. Some people talk about CPAN, but that barely existed when Perl started to get noticed for CGI programming.
However, Perl's not as special for CGI programming as it used to be. It still does all of the great things it always has, but now various other languages have caught up in both functionality, availability, and community awareness.
I started programming CGI stuff in 1994, and I still see how amazingly and mind-boggling hard most frameworks make it. I really wish we had Seaside back then because you never even know about all the stupid things other frameworks make you do. How much better the world would have been if we'd all learned Smalltalk instead. :)
At the time CGI was invented back in the early days of the web, it was the only way do any kind of dynamic processing of web requests, such as responding to forms being submitted or clicking on an imagemap. The web server software itself could deliver only static content, so external programs were needed to handle the interactive stuff.
The first webmasters were probably also system administrators who were often well-versed in Perl. I remember the first NCSA httpd servers came with sample CGI programs written in Perl, C, and shell. The shell scripts were discarded pretty quickly because they were insecure and not good for anything other than really short CGI programs. The C programs worked fine, but Perl was so much more convenient.
My guess is Perl took off as the de facto standard language to use with CGI for several reasons:
There's no reason why Perl has to be used; any language that can work with Unix environment variables is suitable.
That said, CGI has fallen out of favor because it's very slow relative to the languages that run in the web server's address space, such as PHP.
I think the benefit for using a scripting language, is most people are much more productive using a higher level dynamic language than they are using C.
A lot of people seem to worry about speed, but in reality it's normally fine...and if it does become an issue most scripting languages have a extension mechanism where you can write modules in C and still use them in the higher level scripting langauge (Like XS in perl, or pythons c-api)