Neural Network Output Grouping 0.5?

Question

I tried to write a Neural Network system, but even running through simple AND/OR/NOR type problems, the outputs seem to group around 0.5 (for a bias of -1) and 0.7 (for a bias of 1). It doesn't look exactly "wrong"... The 1,1 in the AND pattern does seem higher than the rest and the 0,0 in the OR looks lower, but they are still all grouped so it's debatable.

I was wondering a) if there's some obvious mistake I've made or b) if there's any advice for debugging Neural Nets... seeing as you can't always track back exactly where an answer came from...

Thanks!

Mike

Answer 1

for a) it's impossible to tell without more information.

for b) the best way I have found is to have a pre-calculated set of vectors and weights for a small test version of my network architecture and to have some debug code that dumps the network state out to a csv file so I can look at it in Excel.

The forward (and backward) passes of your network should be completely calculable manually for a single pattern, and a known network initial state, so it's a good idea to have a candidate pattern, or a couple if you can bear doing it manually, fully enumerated for your tests. It's not actually as bad as it sounds.

Doing this has the added advantage that it reinforces your understanding of how the algorithm works and will cause you to check your implementation again from a different standpoint.

good luck.

P.S. shameless plug on backprop blog entry here...

Answer 2

I answered a similar question and I posted some values you can test against:

http://stackoverflow.com/questions/2359571/artificial-neural-networks-benchmark/2375593#2375593

Use those values in order to test your neural network. Your outputs should match the values VERY closely, even if you're using different languages... you should be able match at least 6 digits after the decimal without a problem.

Answer 3

Thanks to everyone on this page :) I tried everything on this page and it all seemed fine.

In the end, Simon's page was very informative, but Lirik helped the most :) His link lead me to the http://www.generation5.org/content/2001/xornet.asp link. I checked all of my values all the way through and found they were... spot on... magically, the next time I ran the program it magically worked :) (admitedly it was a little less efficient than I would've liked, taking a lot more Epochs than I'd hoped for, but it works and this is the important thing :))

So thanks to everyone who replied :)