Strengths of Shell Scripting compared to Python

tags:

python
shell

views:

2002

answers:

+11 Q:

Strengths of Shell Scripting compared to Python

I tried to learn shell(bash) scripting few times but was driven away by the syntax. Then I found Python and was able to do most of the things a shell script can do in Python. I am now not sure whether I should invest my time in learning shell scripting anymore. So I want to ask

What are strengths of shell scripting that make it an indispensable tool as compared to Python?

I am not a system administration by profession but am interested in setting up Linux systems for home users and hence think learning shell scripting can become necessary.

+16 A:

Shell scripting has simpler notations for I/O redirection.
It is simpler to create pipelines out of existing programs in shell.
Shell scripting reuses entire programs.
Shell is universally available (on anything like Unix) - Python is not necessarily installed.

'Tis true that you can do everything in Python that you can do in shell; 'tis also true that there are things that are easy in Python that are hard in shell (just as there are things that are easy in shell but hard in Python). Knowing both will be best in the long term.

Jonathan Leffler 2009-04-28 05:32:10

1 and 2, good points. #3 is a weakness, since it leads to a lot of overhead that Python can avoid. #4 might be largely untrue. Python is part of most Linux distros now.

S.Lott 2009-04-28 10:54:14

I don't see the overhead. Do you have some numbers?

Svante 2009-04-28 11:39:16

Regarding Python - yes, it is pretty much universally available on Linux; all the world is not a Linux box, though. Having said which, it is available for every Unix-like platform. Nevertheless, on my Solaris box, it is installed somewhat out of the way (/usr/sfw/bin/python); I only found the directory at all in the last year or so.

Jonathan Leffler 2009-04-28 12:18:31

@Svante: write a simple shell loop to do a number of operations using commands like "test" and "expr" and compare the run times with a Python script that uses the "os" module. Commands like test and expr often involve forking a subprocess to do the real work.

S.Lott 2009-04-28 13:23:49

@S. Lott: 'test' is a built-in in all modern shells. 'expr' is a separate executable. Incidentally, the good thing about reusing programs is that you don't have to write the code.

Jonathan Leffler 2009-04-28 13:39:47

I see point 3 as a huge advantage, not a disadvantage. For example, consider resizing a directory full of images with the convert program.

Glenn 2009-05-01 23:24:17

Also notable with respect to #4 is that the `bash` shell specifically is not universally available on all linux distros, in particular those for embedded systems, and others running the `busybox` executable. Scripts which target the Bourne shell itself, with which `bash` is backwards-compatible, will run on all POSIX-compliant systems, however that language is a much more restricted one than that implemented by `bash`.

intuited 2010-05-12 01:16:09

#3 is really a reiteration of #1: In most, if not all, high-level languages like Python, it is possible (if somewhat awkward) to call an external program and capture its output. The advantage of using a shell language is that since this is the primary function of the language, it's been optimized for that kind of task, so working with external programs can be done with a minimal amount of typing. So it's incorrect to list #3 as an advantage. But with Python you can also potentially call a routine from a module, which is more probably as likely to be portable as whatever program you would run.

intuited 2010-05-12 01:27:28

@intuited: eval "$(history 2 | head -1 | perl -pe 's/(?<=which is )more probably/at least/')"; history -d $(history 2 | head -1 | sed 's/^\s*$[0-9]\+$/\1/')

intuited 2010-05-12 01:49:59

+3 A:

There's nothing you can do with shell scripts that you can't do with python. The big advantage of shell scripts is that you use the same commands as you do when you use the shell, so if you're a heavy shell user, shell scripting will at some point become a very quick and easy way to automate your shell work.

I also find it easier to deal with pipes of data in shell scripts than in python, though it's absolutely doable from python.

And, finally, you don't have to fire up an additional interpeter to run the shell scripts, giving you a very small, but sometimes maybe noticeable speed and memory usage advantage.

But then again, Python scripts are a lot more maintainable, I'm trying to migrate from big ugly shell scripts to Python scripts for that very reason. It's also easier to do exception handling and QA with Python.

elzapp 2009-04-28 05:32:27

+2 A:

The shell is available everywhere. If you stick to a relatively basic set of portable functionality, your scripts can run on cell phones, wireless routers, DVRs, netbooks, workstations, big iron servers, and the like. Python is not necessarily included out of the box on lots of systems, and depending on the environment it may be hard to get it installed.

Learning some shell scripting can also help you learn some command line tricks, since the command line is, well, the shell. It's also good for taking some fairly long and complicated command line, and converting that into a more general script after you realize you'll need it some more.

The shell also has some pretty powerful features; pipelines are a really interesting control construct that is native only to the shell, as far as I know.

Brian Campbell 2009-04-28 05:33:20

+5 A:

"What are strengths of shell scripting that make it an indispensable tool as compared to Python?"

The shell is not indispensable. Why do you think there are so many? bash, tcsh, csh, sh, etc., etc.,

Python is a shell. Not the one you'd use for running all commands, but for scripting, it's ideal.

Python is a more-or-less standard part of all Linux distro's.

The more traditional shells do too many things.

They have a handy user interface for running commands. This includes one-line commands where the shell searches your PATH, forks and execs the requested program. It also includes pipelines, sequences and concurrent programs (using ;, | and &) as well as some redirection (using > and <).
They have a crummy little programming-language-like capability for running scripts. This language is rather hard to use and extremely inefficient. Most statements in this language require forking one or more additional processes, wasting time and memory.

Running programs from the shell, redirecting stderr to a log file and that kind of thing is good. Do that in the shell.

Almost everything else can be done more efficiently and more clearly as a Python script.

You need both. However, you should never write a script with if-statements or loops in a traditional shell language.

S.Lott 2009-04-28 11:07:43

-1. A) the presence of a proliferation of different shells serves to attest to, rather than detract from, the assertion that a shell of some variety is essential to use of a UNIX system. B) the OP specified "UNIX shell", not just "shell". The Python interpreter is not a UNIX shell, it's a Python "shell". The Python programming language is oriented to procedural programming and data manipulation; shell languages are oriented to executing programs and manipulating their output. C) Your point #2 might be true if none of the script's statements are control flow statements like `if` or `while`.

intuited 2010-05-12 01:37:20

@intuited. A) The proliferation of different shells attests to the number of features that accrete into "shell"; not it's essential role, but it's catch-all for features. B) The python interpreter is a proper unix shell, it uses #!. C) A "simple" shell `if` statement often involves running the `test` program. The shell is a crummy programming language in all respects.

S.Lott 2010-05-12 10:54:47

@S. Lott: Seems like some semantic dissonance is in the circuitry here. My definition of a UNIX shell would be an interpreter whose language is primarily geared towards controlling, at a high level, the operation of a UNIX system. By this definition, a shell of some sort is certainly essential to the operation of a UNIX system, and Python doesn't qualify. How would you define that term?

intuited 2010-05-12 20:33:05

@S.Lott: I'm not a huge fan of the syntax that is used in traditional shell scripting, but I find it quite useable, and appropriate in many situations where the main goal is to control processes and the flow of their output. This often requires making decisions and iterating. Modern shells (`bash`, at least) have builtins to handle decision-making (eg `test` is a `bash` builtin, in addition to more versatile constructs). It certainly has its weaknesses, but also its strengths. It's also worth learning in detail because it is of a very different paradigm than other sorts of languages.

intuited 2010-05-12 20:47:28

@intuited: "find it quite usable" is a salute to your skill, not the crummy language. Python can be more usable by people not as skilled.

S.Lott 2010-05-12 22:23:59

@intuited: "shell" means it responds to the #! protocol correctly. It can be interpreted strictly, or byte-code-compiled and interpreted, or (like awk) compiled to binary and executed. To be a shell, the app has to take an executable which begins with #! and behave correctly.

S.Lott 2010-05-12 22:26:08

@S.Lott: I would use that as the definition for a UNIX "scripting language" rather than a "shell". I am actually starting to use python a bit more as a system shell; can you recommend some useful packages to provide things like pipeline functionality?

intuited 2010-05-19 22:19:20

@intuited: "scripting language" rather than a "shell" is distinction I can't fathom. I can't see any difference. Pipelines -- as I said -- are hard in Python and the **only** thing the shell does well. Look at http://jwilk.net/software/python-pipeline for something that might work.

S.Lott 2010-05-20 16:05:34

@S.Lott: One way to define the distinction would be to say that a shell is a scripting language suitable as a way to interactively control system functionality. EG php is a scripting language which follows the #! protocol, but isn't particularly convenient as a way to control the system at a high level. The same is true, but more so, for awk or sed. Strictly speaking, not all scripting languages follow the #! protocol, but most do. The term 'scripting language' is itself pretty vaguely defined; I think at this point it just means that the source code files can be executed directly.

intuited 2010-05-20 20:12:32

@S.Lott: python-pipeline does seem pretty useful, and seems like it could be combined very effectively with the iterpipes module. I'm expecting the lack of bash tab completion to be annoying though, especially when using programs like `git` that provide completion on things other than filenames. Maybe there is a module to provide that too. It would probably need to start a bash instance and move it to the current directory.. unless there is a bash interpreter written in python, which I doubt.

intuited 2010-05-21 01:24:25

@intuited: "the lack of bash tab completion"? Are you thinking of interactive Python replacing the interactive shell? Bad plan. Python can replace much of shell **scripting**. The interactive read-execute loop of the bash shell doesn't need replacing. You cannot simply ditch the existing shells and replace them with python. You should ditch all your shell **scripts** and replace them with Python (and Python-shell hybrids in a few cases.)

S.Lott 2010-05-21 10:21:20

@S.Lott: I tend to use a lot of loops and other complex structures from the interactive shell. If I end up doing the same thing a few times, I drop it into a shell script and parameterize it. It starts to get complicated when there are common dependencies among various commands: environment variables, procedures, etc. So I build that out into a "module", a sourceable bash file. But then I want to be able to get documentation on the various functions. I wrote a semi-functional framework to provide this, but it's a bit sketchy, which is why I'd like to start using Python for this instead.

intuited 2010-05-22 20:52:56

@S.Lott: but anyway the point is that for me there's no real clear line between interactive shell usage and construction of scripts/modules. In any case it's often a lot less work to put things into a sourceable bash file and deal with the eccentricities than it is to rewrite everything in a more modular language. But if I could use Python from the start, then the entire process would probably end up being a lot faster... so I'm hoping to find a way to make it practical to use Python as an interactive system shell. The bits and pieces, other than tab-completion, seem to be mostly there.

intuited 2010-05-22 20:58:07

@intuited: "no real clear line between interactive shell usage and construction of scripts/modules". Makes no sense. I cannot comment. That is as foreign to me as if you started talking about operating a drill press. I cannot see any connection between the two. I apologize for being an idiot but **designing** a program/script/shell script and running a program are two independent, almost unrelated tasks. I cannot see how there is any overlap at all. I'm sorry, but you're making no sense claiming there's an overlap between design and operations.

S.Lott 2010-05-25 02:23:45

@S.Lott: Interesting, you have a very different philosophy than I. I feel that design and operations are basically chickens and eggs. This is to say that a shell script is a program which runs programs. Many of those programs were initially implemented as increasingly complex shell scripts. At some point it became worthwhile, probably as much for the sake of efficiency as modularity, for some of them to be rewritten in C, and later, Python. For example: at some point `cat "$1" | ssh "$2" "cat - >\"$1\""` begat `scp`. They knew how best to design it because they were already using it.

intuited 2010-05-25 06:22:52

@intuited. As I said, they are not at all "chickens and eggs". They are more like architectural drawings and buildings, or DNA and chickens, to me. I cannot understand "At some point it became worthwhile, probably as much for the sake of efficiency as modularity, for some of them to be rewritten in C, and later, Python" at all. Consequently, I'm rereading your comments and realizing I probably did not understand any of your comments, either. Indeed, I cannot understand how to write code without careful, meticulous, up-front design. I admire your skills.

S.Lott 2010-05-27 18:48:58

+6 A:

The shell makes common and simple actions really simple, at the expense of making more complex things much much more complex.

Typically, a small shell script will be shorter and simpler than the corresponding python program, but the python program will tend to gracefully accept modifications, whereas the shell script will tend to get less and less maintainable as code is added.

This has the consequence that for optimal day-to-day productivity you need shell-scripting, but you should use it mostly for throwaway scripts, and use python everywhere else.

Paul Hankin 2009-05-02 08:29:36

+1 A:

one doesn't have to learn shell scripting, as all the previous answers indicate; but learning is never a bad thing. it's really a question of personal priorities. it's very hard for someone else to tell you what is and isn't worth your time.

most programmers find that learning new languages gets incrementally easier each time. (the same is largely true of natural languages too.) and the earlier you start, the better.

plus: having learned a language enables you to extravagantly diss its limitations from a position of complete knowledge and familiarity. this probably won't get you laid, but might earn you a beer from your peers!

adrianrf 2010-06-19 06:35:17

ansaurus

tags:

views:

answers:

Strengths of Shell Scripting compared to Python

related questions