Personal Project - Next practical language/tech to learn

Question

I'm working on a personal project doing some finance analysis. It's a totally new field for me, and I'm really having fun with it so far, plus working in the high-level language arena is a great break from my embedded systems daytime work.

I have a MySQL backend on a non-local server with a pile of stock data. My task now is to do some analysis of the stocks and produce something approximating a useful result. There are a couple technical difficulties.

(1) I have a lot of records. To be precise, I believe I'm near 100K records right now, and this number grows by 6.1K each weekday. I need to create a way to rummage through these fields and do data analysis - based on a given computation, go look at this other set. Fine and dandy, nothing too outre. But this means I could really use a straightforward API for talking to MySQL.

(2) Ideally, it runs on OS X 10.4.11. No Windows/Linux machine at home.

(3) I can use PHP, C++, Perl, etc. I even have an R installation. I'm pretty flexible with stuff, so long as it runs on OS X. (Lots of options here, pick water, H20, or dihydrogen monoxide ;-) )

(4)Lack of hassle. While I like clever and fun ways of doing things, I'm trying to get some analysis done, not spend ten hours doing installation work and scratching my head figuring out a theoretical syntax question needed to spout out "hello world".

What's the question?

I'd like to dig into something different than my usual PHP/C++/C toolset. I'm looking for recommendations for languages/technologies that will assist me and meet the above requirements.

In particular, I've heard a lot of buzz about F# and Python on SO. I've used CLISP for small problems before, and kinda liked it. I'm seeking opinions about those in particular.

edit:since I rent the DB server and have a limited amount of CPU time online, I'm trying to do the analysis on a local machine.

Answer 1

How about going with something completely different, free, and interesting? WebObjects, included with Xcode on your Mac. There's an Eclipse plug-in to provide an IDE for WebObjects development, or you can use Xcode/Textmate/etc.

It'll also simplify some of your development tasks, as you mention in your question. From the Wikipedia article: "Given a data model, WebObjects can automatically create a customizable Web application without writing a single line of code."

Answer 2

What about Ruby, Haskell or Erlang?

Answer 3

I would suggest Python, although in a few months (after some percolation time) F# might be a better choice.

Answer 4

I'd stick with CLISP, frankly. If you don't know it well yet, then learn that. You may end up writing "PHP in Lisp", but that's not so bad, and CLISP has any language feature you can possibly want, being that it's Common Lisp from which all known language features seem to have derived from at some point or another :-).

Play with CLISP, play with your project, learn some macros. You'll have a good time.

Answer 5

The easiest would be Python. it's already installed on mac, and has more than reasonably performance for anything SQL related.

if you liked LISP, there's PLT Scheme for mac; but i don't know how well it interfaces to MySQL. Another interesting LISP is arc, i'm still checking it myself, hope to get it to 'practical' uses.

another not so mainstream option is Lua, which does have a SQL libraries, is by far the fastest scripting language, has a very neat and clean syntax, and stays close to language design principles, almost like Scheme.

Answer 6

Many developers in small shops, or that move between startups a lot, end up learning a handful of different types of languages, some are unavoidable (javascript, java, C#), some are fun, some are special purpose and very powerful. i alluded to that here:

http://stackoverflow.com/questions/196792/next-step-for-system-administrator#225652

You could look at python and ruby with C extensions, scala, clojure, F#, ocaml, haskell, erlang, all the languages that the wide finder submissions are in

http://eigenclass.org/hiki/widefinder2-conclusions

Answer 7

If you're looking into Haskell (and I can't promise that you won't bump into some "theoretical syntax questions"), check out this paper:

Composing contracts: an adventure in financial engineering
http://research.microsoft.com/~simonpj/papers/papers.html

Answer 8

I would recommend to use R for that. It can handle data matrices well (SQL is not suitable for that). I don't have personal experience with it, but I had a colleague who was also dabbling in financial analysis and was very happy with R. But I believe that with other (more general) languages, you would soon run into some library problems, but with R you have everything you need for statistics and large datasets handling. As a next choice, I would use Python, but it's a personal preference and not necessarily better than the others.

Answer 9

Rebol is great for Finance, it's a multi-paradigm (Functional and OOP) language like F# so learning Rebol first could help you tackle F# afterwards.

You can find here a few financial examples:

Derivatives Option Pricer (based on Black & Scholes Formula): http://reboltutorial.com/blog/derivative-options-pricer/

As for data, you can use the free MySQL Driver or even a pure text data file format which can cope with millions of rows: http://reboltutorial.com/blog/mysql/ or http://reboltutorial.com/blog/rebol-sqllinq-like-text-database-engine-blaze-through-millions-of-rows-a-second/

Rebol is also especially great at parsing data see for example parsing google finance csv file http://reboltutorial.com/blog/parse-csv/