I've tested octave and R too.
Regarding octave: I was very impressed with the similarity of octave syntax. It doesn't took me many time to transpose my matlab scripts to octave. Meanwihile I have a particular problem on printing markers jointly with errorbar wich was fixed by Jarno Rajahalme at nabble and to change the xtick font size, which workaround I got in a question response at nabble. So it still have some bugs which with some effort can be overcomed. If you experiences some problems you may try nabble mailing forum: [email protected]. By the way my team cannot adapt (user friendly) to it such they adapt to matlab, so we're still using matlab. Since matlab is built under gnuplot, another way to correct its bugs is editing the generated gnuplot file. The best IDE I found to it was QtOctave, that I made a short review in "Remember Blog".
Regarding R: according to a research made by SciViews, R performance is superior to matlab and octave. I don't have much experience with R. I studied mclust package to wrote a wikibook chapter about EM Clustering in R. By the way, they seems to have a very active community. So you may found third party packages to infinite proposals, which are not IMO so standardized. The best IDE I found was StatET plugin for eclipse, JGR (Java GUI for R) and emacs. Despite the time cost to learn a new programming language, if I would choose an open source platform to make my experiment graphics and some data mining analysis I would try R.