I am not a mathematician. I enjoy a good math puzzle, but I admit my weaknesses whole heartedly. That said, I've always had an interest in Neural Networks, and while I understand them enough to implement them from scratch, I hit a wall when I need to understand any concept that I can only find mathematic proofs for. Where is the programmer's guide to neural networks, using code instead of formula to explain the practical reasonings?
+1
A:
Unfortunately, I don't know if there's a good single "programmers source" that will give you all of the concepts. I liked Neural and Adaptive Systems: Fundamentals through Simulations.
The best way to have a "programmer's understanding" of neural networks is not so much by examining the code, but in the problem and the correct results. So, if you don't want to look at math, I recommend you look at a given problem. For example, consider the XOR problem as an example of why you need non-linear activation functions, look at the number of variables and their possible values for understanding why a neural network needs to be of a certain size and toplogy to be effective, and split your data into train/test regimes and do studies to see why overfitting is dangerous. Examine the code with the data.
I also recommend not getting too hung up, but reading further. Certain practices in feed-forward networks become more clear once you see their generalization in recurrent and constructive neural networks. I also recommend going wider: Bayesian networks, fuzzy cognitive maps, SOM, Boltzman machines, simulated annealing, and reinforcement learning all have intuitions.
Does this go towards answering your question?
John the Statistician
2008-11-23 15:50:31
I understand it better with a small problem like a XOR network. My grasping of the networks behavior falls apart with problems I couldn't solve myself, and backpropagation becomes my wall.I could write it, but I don't understand the details of how it works.
ironfroggy
2008-11-23 15:53:57
I was in the same boat w/ backprop, I couldn't understand it, but over time I saw so many related ideas, that I finally got it. Let me try a couple of explanations.
John the Statistician
2008-11-23 16:04:13
The internal nodes are features. So, first you learn what inputs together make a feature, and then then which features influence an output. The problem is at the output, you don't know if the feature is unrelated, or whether the feature is poorly determined.
John the Statistician
2008-11-23 16:06:07
So, if it is unrelated, that means other examples will compensate with correcting the weights of the inputs later. But, if the feature is poorly determined, then the other examples will not compensate, and the features will change.
John the Statistician
2008-11-23 16:06:48
There is a closely related technique in reinforcement learning called value propagation. In reinforcement learning, you are trying to optimize some reward, but if you do something good, you don't know which action led to your reward, so you have to spread the reward back, and average it out.
John the Statistician
2008-11-23 16:09:35
Backprop can be seen as a kind of reinforcement learning, where you don't know whether your mistake was to form the feature, or to associate the feature with that output.
John the Statistician
2008-11-23 16:10:10
I looked at Neural and Adaptive Systems: Fundamentals through Simulations at the library today, and it seems *very* mathy, (like every other NN book I've seen)
Tristan Havelick
2008-12-03 07:24:01
There is certainly math there, there's no denying it; however, I thought it also had reasonable explanation, and it's nice to have an implementation to try things on.
John the Statistician
2008-12-03 15:43:02
A:
You need a bit of understanding in either Pascal or Delphi but this overview from ThinkQuest is pretty helpful from a programming stand-point. It also explains some of the difficulties and why the mathematics looks a bit intimidating. (I'm not a mathematician either.)
I was pretty interested in this sort of thing a while back (still am for the most part) and in search of some walk-throughs that I can follow pretty quickly.
Hope that helps a little at least.
Mat Nadrofsky
2008-11-23 16:30:29
+4
A:
Another alternative is a non-math, non-programming explanation. The book Blondie24: Playing at the Edge of AI contains a really great explanation of neural networks. It's about a checkers-playing AI developed by the author. It's not completely without programming references, but it does a great job of explaining how the algorithms work without getting into the code of the solution.
Bill the Lizard
2008-11-23 17:17:50
You know, I've seen you recommend this a few times, so I'm putting it on my wishlist.
John the Statistician
2008-11-23 18:03:48
I want a non-math, pro-programmers explanation. So, the programming references are not a problem. Thanks for the suggestion!
ironfroggy
2008-11-30 16:32:55
A:
I have personally used:
Practical Neural Network Recipes in C++
The author in my opinion does not fully utilize the more powerful functionality of C++, in many cases it reads more like traditional C with classes. The book is also a little dated by now.
HOWEVER - if you need explanations of the algorithms and techniques used in neural networks, explained in a way that an intelligent layperson could understand, so that you can go away and try these things for yourself, then I would certainly give this book a try. Not much navel-gazing goes on here, which is what I liked.
It takes you through all the main things needed to program a neural network - how to compare the actual output with the desired in order to obtain an error signal, and then use this error signal in conjunction with back propagation algorithms to modify the network link connection strengths, doing this iteratively so that gradually the neural network 'learns' the task.
AndyUK
2008-12-03 16:07:54