Advice on implementing low-level libraries in D (as opposed to C/C++)

Question

I need some advice on selecting the D programming language for a project. The project is a low-level library akin to a database with many associative containers and so forth. Hence efficiency is quite important to me.

I need to provide a C API for the library for compatibility with other languages like C++ and Python and I also anticipate that some sections might need to be written in plain C for tuning performance.

D seems very appealing for this job: Are there any pitfalls I should be aware of considering these requirements? How does the performance of D containers compare to the std::(map, vector, unordered_map, etc...), taking manual performance-tuning into account (e.g. using std::map::lower_bound for search/insert and so forth).

Answer 1

Are you thinking short term, i.e. tight deadline, get this up and running next week, or long term, i.e. early planning stages of a large multi-year project?

If you're interested in the short term, I'd recommend against D. It's still too bleeding edge.

Long term, however, D is starting to stabilize. Version 2 of the language will likely be stable in 6 months. Andrei Alexandrescu is releasing a book entitled "The D Programming Language" in March, and a huge push is being made to stabilize D2 for it.

As far as pitfalls, I'd say the biggest one is that the idiomatic D way to do most things is with lots of templates, which makes it difficult to create stable ABIs for things. It can be done, it just isn't idiomatic. The other is that there's no good container library for version 2 of the language yet, though apparently this is being worked on.

Performance-wise, DMD, which is the reference implementation, has an old optimizer. If you are truly obsessed with performance, this may be a problem. GDC, a D compiler for GCC, has a better optimizer but is always a few releases behind. LDC, a D compiler for LLVM, has a terrific optimizer, but only supports version 1 of the language. D is, however, supposed to be as fast as C++, and D compiled with DMD really is as fast as C++ compiled with the Digital Mars C++ compiler according to benchmarks I've done.

Answer 2

In addition to dsimcha's answer I would note that writing good performance apps in D requires in the first place to play nice with the GC. Garbage collection in D is not as fast as in Java or C#, so you need to know when and how to avoid or minimize it. Luckily, you can 1) make much better use of stack allocation with raii and 2) use manual memory management when required.

Here is a (somewhat old) presentation about how Tango has taken advantage of D arrays and slicing for a high-performance library, Array slicing fo' shizzle: http://video.google.com/videoplay?docid=-4010965350602541568&hl=en#

google 'd conference 2007' for the slides. (I'm new here and can only post one link, sorry)