Prototyping neural networks

Question

Hi all,

from your experience, which is the most effective approach to implement artificial neural networks prototypes? It is a lot of hype about R (free, but I didn't work with it) or Matlab (not free), another possible choice is to use a language like C++/Java/C#. The question is mainly targeting the people that tried to test some neural networks architectures or learning algorithms.

If your choice is to use a programming language different from the three mentioned above, can you tell me their names and some explanations concerning your choice (excepting: this is the only/most used language known by me).

Thanks.

Answer 1

I have used Matlab's Neural networking tool box to some extent, and enjoyed it as a prototyping tool due to its graphic interface to allow you to set up the network, experiment with sizes of training vs test input data, and its post-training testing all built in. It just seemed nice and natural to get started with...

I also experimented with Joone (Java Object Oriented Neural Engine). It is set up quite nicely, and at the time I was a bit of an amateur and still had no problems getting various networks up and running, with testing. It is Java, so it may not have the performance that you might want if you are training large complex systems, but its API was extremely approachable.

I have also seen some work done with Flood in C++. It has a bunch of classes set up with networks that are appropriate to solve a range of problems. It is worth checking out.

Answer 2

I tried using both higher-abstraction level languages (matlab, java) and lower ones (C). Both using toolboxes and libraries and coding them myself. The overall response? Neither is the perfect tool. Please keep in mind that:

• a prototype might not be enough: often you need to run the network over large samples, or several times over a subset of samples (in case of evolving neural networks) to get decent results. If you have to run a network a million times, even a small performance gain might be a huge help and time-saver (i.e. C over matlab);

• if, on the other hand, you need ease of coding you might want to use one of the many pre-packaged libraries (javaNN, etc);

• what kind of neural network are you using? continuous-time recurring neural networks (CTRNN)? Backprop? How do you do train them? How do you verify their results? Is accuracy important? (i.e. are you running them on a small device, such as a memory-limited robotic control board, like Arduinos?)

If you have time to spare, I'd suggest

1. have them learn the concepts using a higher-level language, or even pseudocode first;
2. once they are familiar with all intricacies, especially if using evolving neural networks, have them chose a language they are already familiar with
3. then you might want to start researching how to optimize for speed, memory footprint, etc.

Hope this helps.

Answer 3

Since neural networks are used a lot by scientists, and not so much by programmers, choose a programming language that targets scientists and provides good libraries for neural networks.

Python is a reasonable choice since it is widely used by scientists. Two distributions to start with are

http://www.pythonxy.com/

http://code.enthought.com/

These Python distributions include a lot of add-on modules that are not in the standard Python library but which are very useful for the type of programming that scientists do. Searching the Python package index comes up with a few neural network packages that may be OK if you are just learning about neural networks.

http://pypi.python.org/pypi?%3Aaction=search&term=neural&submit=search

However, if you are doing serious work with Neural Networks you will want something like the Fast Neural Network library. This comes with Python bindings so that you can do your programming in Python, use the wide variety of Python modules for graphing, visualisation, data manipulation and so on. But, your neural networks will run using optmised compiled code from the FANN library. The best of both worlds.

In other words, for running the actual neural network code, you need C, not Java. Since C libraries don't integrate that well with Java, choose a language that does integrate smoothly with C libraries. Python does that and is also rather more productive than Java since there is a lot less lines of code required to explain your algorithms. Some people have found a 10 times increase in productivity over Java.

You mentioned R, perhaps because it has statistical functions that you would need to use, or perhaps you have people available who can write R code. Again, choosing Python versus R is not an either-or decision. You can use both.

The RPY library allows Python programs to access R libraries and code. Using this you would write your main programs in Python and treat R as a tool to provide libraries, in the same way that you make use of the FANN library which was written in C.

http://rpy.sourceforge.net/

There is another module called RSPython that works both ways, so that R programs can access libraries written in Python. This would be useful if you have an expert R programmer helping you.

http://www.omegahat.org/RSPython/

And that's not all. You can leverage Python to simplify Java programming. If you have a Java neural network engine, you can still write most of your program in Python using the Jython version which runs on the Java VM and allows you to use any Java libraries and classes in your code. And you can still use the fast FANN library as well since they provide bindings for Java programs.

The main reason that I recommend Python for your work is that it is used by an awful lot of scientists which is why there are two science oriented distributions available. A second reason is that Python is very easy for novice programmers to get started with, and in exploring neural networks you will probably start with simpler simulations and work up to more complex ones with more data manipulation and analysis of results. Python will allow you to build up your own library of code, and become an expert Python programmer so that you can focus more of your attention on the neural network problems. If you bounce around with a bit of Java, and a bit of C++ and a bit of R, then you will be able to spend less time on neural networks. That strategy might be good for someone who wants a career in programming, but not for someone who wants to achieve some significant results with neural networks.

Even if your neural network work extends into specialised hardware, so-called neuromorphic chips, you can still leverage Python as this paper from the NIH demonstrates:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701676/

Answer 4

Typically when I was tinkering with these kinds of algorithms, I found that using the open-source Weka Toolkit was a great way to prototype and discover a whole bunch of different learning algorithms (not just artificial neural networks). These days, it looks as though they have bindings from a bunch of different languages, so you shouldn't be tied to Java if you want to interact with Weka at the code level.

Once I found and understood something that was cool/doing a pretty good classification job, I wrote the algorithm in C or C++ partly for fun and partly to gain the necessary speed gains required to work with larger data sets.

Answer 5

Scilab is an open source alternative to Matlab.

You may experiment with neural networks using the ANN Toolbox for Scilab.

Focus on the theory and experiments. Did you choose discriminant features for your samples? What's the state of your training and test sets. For each experiment, study the confusion matrix. Do you have an idea why a sample is misclassified? Does it seem logical to you? If not, which feature you don't use would help?

Implementing comes next, use the language you're familiar with. A managed language like Java or C# is likely to be less error prone: at least you're less likely to screw up things because of pointer or memory allocation bugs. Optimizing comes last (after some decent profiling as always).

Answer 6

Use C++ and if your neural network is simple enough don't use weird frameworks out there.

Maybe 5 years ago I did a Sokoban solver using Reinforcement Learning. At the time we decided to go with Java and use some Agent Framework developed by an Italian University.

First bad decision here was to use this Framework. It basically had bugs here and there which caused us to waste loads of time debugging the framework's code.

Once we got to the point where everything was stable, training the Neural Network was just dead slow. We left it running overnight in a quite powerful machine and still it solved very few puzzles. Basically Java object allocation and garbage collection were damaging the whole performance of the system quite badly. We tweaked a bit the app by creating pools of objects instead of allocating them all the time but the program's performance was still an order of magnitude less than a similar assignment that was implemented in C++ using sockets directly.

Hope this helps!

Answer 7

This depends on your current setup. When I used to work on them back in college days, I had to use C++ + MPI+numerical recipes. This was done because I had to load share on big beowulf cluster.

If your computation needs are not big, anything would do. Prepackaged libraries are available on all the platforms (R, Python(numPy, scipy), C/C++(Numerical recipes) etc). If you are comfortable programming in any one of them, it should not be a big deal.

If I had to prototype anything now, I'd probably go with Python (just because I find it much more easier for prototyping)

Answer 8

In my nn classes in school we used matlab and then I used java for my thesis.

I would suggest java or matlab. Matlab because it probably already has alot of what you might need. And java because it is quick to implement what might be missing in open source projects. And besided since besides implementing the neural networks you might need some way to visualize them. And for that I personally believe that java or matlab is quite easy.

Answer 9

I like flood. It's free, comprehensive, and written in C++.

Answer 10

I would also highly recommend python. For a prototype, python is a great choice: it is easier and faster to program in, there are a huge number of libraries available, and it widely used in the scientific community.

Also, if you are using python, you can also take advantage of the excellent PyBrain package which contains algorithms for neural networks, reinforcement learning, unsupervised learning, and other machine learning tasks, which should help you build a prototype quickly.

Answer 11

The implementations in Matlab are sophisticated and complete. I have found it to be sufficient for evaluating different types of networks. It is also very programmable using external interfaces.

However, since the implementations of the algorithms are not open source, it is sometimes more difficult when you need to move a particular piece of code into an application outside of Matlab, as my hand coded implementations of different neural network types produced different results.

Answer 12

I have started writing a NN implementation using C++ and found that I didn't know enough about the maths involved in the beginning. What ended up hapenning was that it was too hard to refactor the code as I was tweaking the calculation model.

Eventually I gave into MATLAB as it was definitely a better companion to learning how neural networks work. I was able to make huge changes to the algorithm using a few key strokes and graph the results too.

Perhaps my experience would have been better if I had used an already built matrix computation framework. Considering that's how you do 3D there should be a few really optimized libraries out there for most languages. Heck you might as well leverage Direct3D or OpenGL for that, but I am sure there is something more suitable for Neural Networks out there.

I guess I am not providing much information on what you should do. I can however tell what you should not do - and that's trying to write matrix manipulation code yourself.

Answer 13

You might want to give Weka a look. It has some built-in tools for things like data visualization, and it's been around for years now (some screenshots).

Answer 14

In my view in working with neural networks the key is getting the training set right not so much how the network itself is actualised in code. I would choose a language based on the type of problem you are trying to solve using the network. For the network itself c++, c#, python and java are all viable.

Are you using this in connection with a problem requiring image processing? In which case you will probably want something that hooks up to an image processing library like OpenCV easily. Or is there some audio processing involved?

You may need to easily visualise the training sets so how easy would this be with the language of choice? Can you work with either OpenGL or DirectX libraries either directly or using a wrapper of some kind? For DirectX the choices are C++ and C#. Will a higher level of abstraction say using WPF work?

I have used C# because I am familiar with it and can leverage the numerous data handling technologies in .net and use wpf for any visualisations needed.