Neural networks, how do they look in coding?

Question

Basically i know the concept of a neural network and what it is, but i can't figure out how it looks when you code it or how do you store the data, i went through many tutorials that i found on google but couldn't find any piece of code, just concepts and algorithms.

Can anyone give me a piece of code of a simple neural network something like "Hello World!"?

Answer 1

What you mainly need is an object representing a single neuron with the corrispective associations with other neurons (that represent synapses) and their weights.

A single neuron in a typical OOP language will be something like

class Synapse
{
  Neuron sending;
  Neuron receiving;
  float weight;
}

class Neuron
{
  ArrayList<Synapse> toSynapses;
  ArrayList<Synapse> fromSynapses;

  Function threshold;
}  

where threshold represents the function that is applied on weighted sum of inputs to see if the neuron activates itself and propagates the signal.

Of course then you will need the specific algorithm to feed-forward the net or back-propagate the learning that will operate on this data structure.

The simplest thing you could start implement would be a simple perceptron, you can find some infos here.

Answer 2

AI-Junkie has a great tutorial on (A)NNs and they have the code posted there.

Here is a neuron (from ai-junkie):

struct SNeuron
{

   //the number of inputs into the neuron
   int m_NumInputs;

   //the weights for each input
   vector<double> m_vecWeight;

   //ctor
   SNeuron(int NumInputs);

};

Here is a neuron layer (ai-junkie):

struct SNeuronLayer
{
  //the number of neurons in this layer
  int m_NumNeurons;

  //the layer of neurons
  vector<SNeuron> m_vecNeurons;

  SNeuronLayer(int NumNeurons, int NumInputsPerNeuron);
};

Like I mentioned before... you can find all of the code with the ai-junkie (A)NN tutorial.

Answer 3

You said you're already familiar with neural networks, but since there are many different types of neural networks of differing complexity (convolutional, hebbian, kohonen maps, etc.), I'll go over a simple Feed-forward neural network again, just to make sure we're on the same page.

A basic neural network consists of the following things

1. Neurons
   1. Input Neuron(s)
   2. Hidden Neurons (optional)
   3. Output Neuron(s)
2. Links between Neurons (sometimes called synapses in analogy to biology)
3. An activation function

The Neurons have an activation value. When you evaluate a network, the input nodes' activation is set to the actual input. The links from the input nodes lead to nodes closer to the output, usually to one or more layers of hidden nodes. At each neuron, the input activation is processed using an activation function. Different activation functions can be used, and sometimes they even vary within the neurons of a single network.

The activation function processes the activation of the neuron into it's output. The early experiments usually used a simple threshold function (i.e. activation > 0.5 ? 1 : 0), nowadays a Sigmoid function is often used.

The output of the activation function is then propagated over the links to the next nodes. Each link has an associated weight it applies to its input.

Finally, the output of the network is extracted from the activation of the output neuron(s).

I've put together a very simple (and very verbose...) example here. It's written in Ruby and computes AND, which is about as simple as it gets.

A much trickier question is how to actually create a network that does something useful. The trivial network of the example was created manually, but that is infeasible with more complex problems. There are two approaches I am aware of, with the most common being backpropagation. Less used is neuroevolution, where the weights of the links are determined using a genetic algorithm.

Answer 4

This is the NUPIC programmer's guide. NuPIC is the framework to implement their theory (HTM) based on the structure and operation of the neocortex

This is how they define HTM

HTM technology has the potential to solve many difficult problems in machine learning, inference, and prediction. Some of the application areas we are exploring with our customers include recognizing objects in images, recognizing behaviors in videos, identifying the gender of a speaker, predicting traffic patterns, doing optical character recognition on messy text, evaluating medical images, and predicting click through patterns on the web.

this is a simple net with nument 1.5

from nupic.network import *
from nupic.network.simpledatainterface import WideDataInterface 
def TheNet():
    net=SimpleHTM(
          levelParams=[
            { # Level 0 
            },
            { # Level 1
              'levelSize': 8, 'bottomUpOut': 8,
              'spatialPoolerAlgorithm': 'gaussian',
              'sigma': 0.4, 'maxDistance': 0.05,
              'symmetricTime': True, 'transitionMemory': 1,
              'topNeighbors': 2, 'maxGroupSize': 1024,
              'temporalPoolerAlgorithm': 'sumProp'
            },
            { # Level 2
               'levelSize': 4, 'bottomUpOut': 4,
               'spatialPoolerAlgorithm': 'product',
               'symmetricTime': True, 'transitionMemory': 1,
               'topNeighbors': 2, 'maxGroupSize': 1024,
               'temporalPoolerAlgorithm': 'sumProp'
             },
             { # Level 3
               'levelSize': 1,
               'spatialPoolerAlgorithm': 'product',
               'mapperAlgorithm': 'sumProp'
             },],)

    Data=WideDataInterface('Datos/__Categorias__.txt',    'Datos/Datos_Entrenamiento%d.txt', numDataFiles = 8)#

    net.createNetwork(Data)
    net.train(Datos)

if __name__ == '__main__':
    print "Creating HTM Net..."
    TheNet()