Prerequisites Needed to Read Books on Neural Networks (and understand them)

Question

I've been trying to learn about Neural Networks for a while now, and I can understand some basic tutorials online, and I've been able to get through portions of Neural Computing - An Introduction but even there, I'm glazing over a lot of the math, and it becomes completely over my head after the first few chapters. Even then its the least book "math-y" I can find.

Its not that I'm afraid of the math or anything, its just I haven't learned what I need, and I'm not sure what I need exactly. I'm currently enrolled at my local university, working on catching up on classes I need to enter the MS in Comp. Sci program (my BA is in Business/Info. Sys.) and I haven't gotten very far. According to the university's little course descriptions, NN's are actually covered in a Electrical Engineering course on Pattern Recognition (seems odd to me that this course is EE), which has a few EE prereq's that I don't need to get into the MS Comp. Sci. Program.

I'm extremely interested in this topic, and know I eventually want to learn a lot more about it, the problem is, I don't know what I need to know first. Here are topics I think I might need, but this is just speculation from ignorance:

• Single Variable Calculus (I've had Calc I and II, so I think I'm covered here, just listing for completeness)
• Multi Variable Calculus
• Linear Algebra (I've not taken this formally yet, but can actually understand many of the concepts from what I've managed to grok on Wikipedia and other sites)
• Discrete Mathematics (Another I've not taken formally, but learned a portion of on my own
• Graph Theory
• Probability Theory
• Bayesian Statistics
• Circuit Design
• Other maths?
• Other comp sci topics

Obviously there is a neuroscience component here as well, but I actually haven't had any trouble understanding books when they talk about it as applied to NN's, largely because its conceptual

In short, Can someone lay out a semi-clear path that one needs to really understand, read book on and eventually implement Neural Networks?

Answer 1

You can't implement "neural networks" -- you'll end up implementing a specific kind of NN (e.g. perceptron). There are many different kinds of NNs, each more suitable for some specific kind of task, and each kind uses some math (and not only math) concepts that are specifically only to that particular kind. For example, Boltzmann machines use concepts from statistical thermodynamics (founded by Boltzmann).

As for your question: without a clear goal, there is no clear (not even "semi-clear") path.

Answer 2

I second zvrba's idea that you set yourself a clear goal. A few guiding questions: a. Do you want to study NNs as a model of biological networks or as a computational tool? b. Are you interested in their learning aspect? associative memory? signal processing? c. Do you want to understand complex theory? or just enough to write simulation software?

Also, I would start small: implement a perceptron in your favorite programming language. The math is not that bad, and it will probably focus you on your next steps. Use a binary classification dataset, say UCI's tic-tac-toe endgame.

Answer 3

If you want a list of college courses that you'll need to understand the book, here it is:

• Calculus (I, II and III)
• Differential Equations
• Linear Algebra
• Statistics (or a good covering of Bayes)

However, I did just fine in my NN classes without Diff. Eq. and just had to look up concepts I hadn't studied yet.

You can take the black box approach as above, but if you really want to understand the math and implementation of the networks, you'll have to study. It's going to be a steep learning curve to fully grasp the more advanced networks no matter what you do. You can either take the above classes first, or you can start reading the book and look up everything you don't grasp on wikipedia, and then from those articles read whatever you have to read to understand them, etc. You will find that, either way, you'll eventually get past that initial peek and things will be easier.

It would be good if you told us why you want to learn neural networks. I've not found a single use for them in my professional career, though I'm not a game developer or telecommunications developer.

Answer 4

For basic, back-prorogation neural networks, the most important things are:

• Calculus

• Linear Algebra

• Basic Stats / Probability

If you are just looking for more specific topics (you said you already took Calc, so I'll leave that out), here are some subjects that will be useful to know, if not necessarily directly applicable to building a neural network:

• Solving Linear Systems of Equations (you'd learn this in a Linear Algebra course)

• Least Squares Regression

• Optimization theory

You should realize that there are several other methods that can be used to solve certain problems along with neural networks.

Often the most difficult aspect of solving a problem is determining the best method to use.

Answer 5

This software will help you to understand basic aspects of Neural Networks, see:

Sharky Neural Network - Neural networks in action