How to profile an Erlang program in terms of memory usage?

Question

I would like to further enhance the efficiency of an existing Erlang program. First I would like to identify bottlenecks and then decide on where to further optimize.

I have tryed fprof, but it only gives information on total and average runtime. I would most like to see a log similar to the output of fprof, but in terms of average and total memory usage with respect to functions and processes.

---

For starters it would be enough to profile a single module, that does not spawn processes, only it's functions would be called. This would already help, for I could separate the program to distinct modules for testing.

---

Typical suspicious points are, where bigger lists are being handled.

Here the usage of ++ has been resolved by lists:reverse([Head|Tail]) like syntax.

I am also considering using ETS tables instead of Lists for cases with more than a few hundred elements.

Thank You in advance!

Answer 1

The perfect starting point is the Profiling section from the Erlang Efficiency Guide:

http://www.erlang.org/doc/efficiency_guide/profiling.html

Answer 2

Doing some advertising for my own sake: I wrote a little erlang gen_server a while ago, that records and logs system statistics, combined with a little perl script that parses them and outputs pretty charts.

I've found it pretty useful for doing memory watching etc. under load, as it allows you to continuously monitor a detailed view of the memory usage, while for instance testing different things.

The erlang part is fairly non-intrusive, a simple gen_server that you can start from anywhere, you can just put it under your supervision tree. You can configure the poll frequency etc, and it will write statistics to a file in a simple json format.

The perl script then runs over it, and aggregates the logs to draw charts. There are base-classes, and if you know a little perl, you can easily write a class to log and chart any custom parameter you want.

The script can be obtained from: http://github.com/whoppix/erlang-statistics

Sample chart (Erlang node that leaks atoms):

Hope this helps you :)