I share your desire for fast prototyping AND fast runtimes,
and am similarly dissatisfied with kludging together {R
or Python} + {Fortran or C or C++}. I don't really want a
full-out Scheme-style (or worse, Haskell-style) functional
language though. (Been there, done that; they sound pretty,
but aren't practical.) I just want an imperative language
that doesn't suck, with a few functional and maybe OOP
features.
That said, I'm now looking into Scala, with the Scalala
MATLAB-like library. It does linear algebra and I think,
LAPACK bindings (through some Java/JNI library),
and also some plotting. Since Scala has an interactive
interpreter (albeit a bit immature), you can use it in an
R/MATLAB interactive style, which is nice.
This is fairly new, so way less library support than R.
Scala is like Java but less sucky and more functional-y and
type-inference-y. Supports lightly typed, fast prototyping
better than either C++ or Java. In terms of language
features, I think it's similar to OCaml or F#. I find it
easier to grok, personally, though it's a somewhat complex
language.
I've found Scala to be faster than Python (supposedly it's
as fast as Java) which I avoid in the same situations you
stated -- sometimes you need to write your own inner loops
and it would be nice to not have to bounce in and out of C.
(FWIW, Python's ctypes is a lot nicer than R's C API,
but you do still get the basic annoyances of C/C++ world
like segfaults.) I know some folks who write L-BFGS
and other performance-crucial sort of numeric code directly
in Scala and they seem happy. But I have not tried this
myself yet.
The Scalala author tells me he does all his algorithms and
analysis in Scala now. Of course, he'd be the first one to
do that :) but since you mentioned complete integration in
a single language as a goal, that is one success story.
For the longer term hardware innovation problems you
mentioned, like GPU usage, the JVM (Scala, Clojure,
Java) seems like a bad bet because it has such insufficient
C/C++ integration. I wonder if any functional language is
very good here. I suspect hardware innovations will always
require close-to-the-metal C++ coding, at least for several
years after they come out.
JVM is good at multiple cores though. Scala has some nice
parallelism libraries, though I suspect any functional JVM
language should have good ones.
Positive sides about the JVM: good JIT compilation, garbage
collection and runtime type safety (no segmentation faults),
yet only slightly slower than C++. (Supposedly at least.)
Very widespread in corporate programming, so it'll be around
for years to come on open source platforms.
Negative sides:
insufficient C/C++ integration (JNI, though it gets the job done)
clunky integration into Unix-land. Just using Java's I/O
library is such a pain. (ScalaNLP's Pipes.scala make
this substantially easier. i need to bother those guys
again about putting it on github so i could link to that
file, it's great). And every time you want a commandline
script you have to write a shell loop that adds dozens of
jars to the CLASSPATH. Argh. This is NOT acceptable.
insufficient package management compared to Python's or
Ruby's. I always thought RubyGems and eggs/easy_install
were clunky, but watching Maven 2.x crash and burn is a
disheartening experience. This is a general problem with
the Java ecosystem. On the other hand people don't seem
to have as much of a problem as me.
any JVM language will always be slower than well-tuned C++ or Fortran
On the other languages people are talking about...
F#: If I had Windows for free, I'd try C#/F# because
supposedly they have very good C++ integration. But I
don't, and Mono doesn't look widely enough used to bet on
as a platform. Despite the annoyance the JVM is, it's
definitely here to stay.
Clojure: I don't understand what the big deal is. It's
cool because it has irritating parentheses?