Approximating z = y^2 + x^2 with Neural Network

tags:

neural-network

views:

answers:

+1 Q:

Approximating z = y^2 + x^2 with Neural Network

Hello,

How quickly is a neural network expected to approximate the z = y^2 + x^2 function? Mine seems to struggle when I make the inputs negative as well and all the weights become really small ( *10^-16! if use 2x40x1) or all become of same number (like -0.16 and 0.16 if use 2x20x1). I am using 2000 input examples per epoch.

However it seems to learn ok if all inputs are positive. What could this mean?

What number of input pairs per epoch, architecture, and number of epochs should I use to solve this do you think?

I am using backprop, neural network without bias with 1 hidden layer (and my inputs are all between -1 and +1 and desired outputs [0, 1]).

Thank you,

With positive inputs, the function is monotonic. As soon as you cross the origin that is no longer the case.

I think you may need to allow non-zero bias in order to develop a neural network with non-monotonic output.

Also, make sure that your implementation isn't doing bounded optimization, many optimization algorithms incorporate a non-negativity assumption.

Ben Voigt 2010-09-23 01:13:36

Hi Ben, this is the code I've designed so there is no 'bounded optimization or non-negativity assumptions' (which were intended at least!lol). I will code in the bias, as so far I haven't included it as I don't know how to update the bias throught backpropagation. How would I do that? So I would include an extra input of 1 into each neuron with a random weight attached to that input (ie the bias) - but I am supposed to update this weight right? Just like the other weights? It'll be tricky to code it in the way I've done it so far.. Will have to think.

Katya 2010-09-23 01:28:50

@Katya: I would first try just having some neurons with zero bias, some with +1, some with -1.

Ben Voigt 2010-09-23 01:31:19

Just to check please: this code is actually designed to test financial market prediction (I was just trying to test it on the 2D function first). Do you think that I will need the bias in order to do the predictions intended? (Monotonic means that as x increases y increases, so I guess a complex function like that is not monotonic right?). In ideal case how should the bias be adjusted throughout backpropagation please?

Katya 2010-09-23 01:44:45

I don't think that adjustment of bias is actually necessary. The problem with requiring zero bias is that you can only represent an linear transformation, by allowing non-zero bias you permit affine transformation. But the magnitude of the bias can be fixed without loss of generality as all the other coefficients will scale to accommodate whatever bias you choose. So you just need to initialize some nodes with negative, some zero, some positive bias. But be aware I'm making suggestions rather than speaking from experience.

Ben Voigt 2010-09-23 04:18:30

ansaurus

tags:

views:

answers:

Approximating z = y^2 + x^2 with Neural Network

related questions