How to become a great open source programmer of python?

Question

Hi,

I'm a C# programmer of web applications.

Weeks ago, I asked a question on stackoverflow about What should a software engineer (web) start by learning - Erlang, Haskell, Python, C++, F#. Thanks to all of those who suggested their thoughts and help to make a decision. I've found Python should be the one I should start looking forward. I've also got interested in Erlang. I'll also try to manage some time in a month for Erlang but Python is what I started over. I've started learning from Dive into Python which is doing good so far.

There was a reason I picked up Python over many other choices. In parallel to my post (mentioned above) and I was also doing an investigation. I had in mind before raising the question over here. Why most of the great engineers (I don't use developers - they deserved more respect to call them Engineers) were/are contributors of open source projects? And the open source project has its format. All the great softwares ever invented has some direct or indirect connection to to open source. They created a great open source project, before created this they worked for a open project, etc. etc.

I also wanted to be like them. To be like them I must plan and do, accordingly. So, this is the first reason that makes me to pick Python over others.

As I started seen things about Python, it has amazing chemistry with C. And most of its customers layered their hardcore process in C instead of Python. That make me curious. I went off and started looking more about this fact and found some big names that uses Python with C. Even, some great open source project (enterprise) are also designed in such manner.

Why all the good names backend their process with C and use Python for script? Doesn't Python good enough to be used for backend process? If this is being good practice of keep the design way then why we haven't seen any good book on this yet? Or do we have? I've to keep this in plan so if you have any good reading stuff please do mention here.

I'm going to plan broader my scope of learning. The goal was to learn a language now it turning out to be synthesis on open source platform. I've a DELL OPTIPLEX desktop on which I am thinking to setup an open source environment with UNIX, VIM, PYTHON. There are many open source gurus working on some great open source stuff. I've seen Sir Alex Martelli here on stackoverflow. I wish I could get something from him on how should I design an open source development environment? But, there are many other hero and I also like their experience on this.

They concept of learning now learning to become a good open source developer (unix, python with c). For this I need how to setup a development environment? What are the great development of open source development environment? Please, keep the answer in context with Python.

Thanks.

Answer 1

Start coding! Seriously, create some sample apps. If you run into pain points, that's where you'll learn. When you're done one, then read best practices books to learn how you could have done better. Otherwise you're not learning, you're just following a recipe.

Answer 2

Read lots of code. Develop your own coding style, but work on a lot of projects so that:

a) your style can inherit the best parts of theirs

b) you won't get into the bad habit of always doing things the way you've first learned (as python is constantly changing, and so the 'best' idiom has a habit of switching about

Ask lots of questions, and good luck!!

Answer 3

Wow, plenty of questions you ask. Let me try to address them...

Why all the good names backend their process with C and use Python for script? Doesn't Python good enough to be used for backend process?

I'm not sure what you mean. Python is used a lot in back-end work; if and when some part proves to be a CPU bottleneck, extracting a stable core and using C code for it (or C++, or Fortran, etc -- they all interface well to Python) is a reasonable first stab at optimization (Cython is also often a feasible alternative) -- just as, at a "last" resort, one can delve down into platform-specific assembly language to extract even more performance (GMP, the main library I wrap in gmpy, does it when warranted - the really good assembly parts are actually specific to particular CPU models, not even "platforms"). Why do you think (e.g.) that having CoreDuo-specific assembly for all-out CPU optimization mean general x86 assembly "doesn't good enough", or having generic assembly mean C "doesn't good enough", and so on, and so forth? In the (rare but not unheard of) cases where squeezing each CPU cycle is really important, for that tiny part of the code for which it is important, going down one level of abstraction is an excellent strategy and in no way implies the higher layer of abstraction "doesn't good enough".

BTW, that's why I put "last" in scare-quotes -- because when even the most specific assembly still requires more optimization, one can (for some CPUs) dip down to microcode, or FPGAs, or fully custom ICs. Of course that does emphatically not mean software solutions aren't (to use the right verb for once;-) good enough (for the remaining 99.999% of your needs): it means real engineers understand performance trade-offs...!-)

If this is being good practice of keep the design way then why we haven't seen any good book on this yet? Or do we have? I've to keep this in plan so if you have any good reading stuff please do mention here.

What subject(s) are you thinking of, that on their own are worth a whole book? I don't think there's a large potential market for a book on (say) how and when to pick parts of Python to recode in C, parts of C to recode in generic assembly for your chosen platform(s), parts of generic assembly to recode in CPU model-specific assembly, and so on down to microcode, FPGA, custom ICs -- that's my motivation for not devoting (say) a year of my life to writing such a book. I think the Nutshell does a decent job in outlining the very basics -- how you use profiling to pick parts that are CPU hogs (if any), optimize them at a Python level, and recode in lower-level languages if needed (no mention of Cython -- wasn't there yet when I last rewrote the Nutshell). But of course that part's 10% of the book, or less.

I'm going to plan broader my scope of learning. The goal was to learn a language now it turning out to be synthesis on open source platform. I've a DELL OPTIPLEX desktop on which I am thinking to setup an open source environment with UNIX, VIM, PYTHON. There are many open source gurus working on some great open source stuff. I've seen Sir Alex Martelli here on stackoverflow. I wish I could get something from him on how should I design an open source development environment? But, there are many other hero and I also like their experience on this.

Hard to beat Ubuntu (though I personally use mostly Macs, that's because Mac OS X still offers a superior laptop experience, and my personal machines are mostly laptops by now -- my workstation, at work, does run Ubuntu) -- though I'm pretty sure other modern Linux distros would be shoulder and shoulder with it. My personal quirk: code that I consider crucial to my (open source) work, I always try to rebuilt, from scratch, from sources -- Python, lots of Python extensions, GMP, svn, hg, Cython... I admit these days I make an exception for the OS itself, the gcc suite, X11 & friends, gvim -- one does have to draw a line some where to get any spare cycles to get some useful work done;-).

They concept of learning now learning to become a good open source developer (unix, python with c). For this I need how to setup a development environment? What are the great development of open source development environment? Please, keep the answer in context with Python.

I don't think there's much to it.

Some people swear by IDEs (eric, for example, with PyQt -- or Eclipse plus pydev); me, I stick to gvim (and while I use iPython occasionally, plain python is still what I use most). Similarly for VCSs such as svn and hg, or DBs such as PostgreSQL, etc, there are plenty of GUIs around, but I find the command-line tools more suited to my personal preferences.

Mostly, I recommend picking a project in an area you know passably well (say, web apps, in your case), studying and using it enough to form a supported opinion, and once you identify things it's missing (or open bugs on its tracker, etc), dive in and start contributing. If and when you do a good job at that, you'll eventually be accepted as a project committer.

The key is picking something you really, personally care about -- whether you then work on it with Ubuntu, or Gentoo, or with Gvim, or Emacs, or Eric, etc, etc, it's really a very minor consideration IMNSHO.

Answer 4

There are two major reasons to write C code instead of Python: speed, and glue.

Python is significantly slower than C. Most of the time, this won't matter at all, but sometimes it does. If you were to write a Fourier Transform function in pure Python, you would not be happy with the speed; so the SciPy guys have provided a C implementation of the Fourier Transform, which you can just call from Python.

But unless you know you are tackling something really complicated, such as encoding compressed video or the Fourier Transform, you should always start out by just writing it in Python. Much of the time it will run fast enough. If so, you are done!

If you write your Python prototype, and it seems to work but is slow, you can release it as free software anyway. A slow solution is better than no solution, and people may thank you for it. But if you are motivated, you can recode your Python into C to speed it up. (Measure your Python first, so you know what parts need the speedup.) And, as a bonus, you will have already implemented the algorithm correctly once (in Python) so you have some good idea how to handle the problem in C.

The other major reason to write C modules for Python is to provide a "glue" layer that adapts Python to be able to use some external resource. For example, in the Windows version of Python, you might use a C module to allow you to use COM objects. The whole COM API is written for C, so a C module is the best way to adapt COM for Python.

So, in summary, my advice is just to write some code that solves some problem you have, and see if you think the result is valuable enough to share with the world.

P.S. This is a great resource for starting Python programmers:

http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html