Neural networks – Java based neural networks – how to implement back propagation
I'm building a test neural network. It won't work My main problem is back propagation According to my research, I know it's easy to use sigmoid function So I update each weight through (1-output) (output) (target – output), but the question is, if my output is 1 but my target is not? If it is one at some time, the weight update will always be 0... Now I just want to try to add input from 2 input neurons, so the best weight should be only 1. As an output neuron, just add its input I'm sure I screwed up this in many places, but here's my code:
public class Main {
public static void main(String[] args) {
Double[] inputs = {1.0,2.0};
ArrayList<Double> answers = new ArrayList<Double>();
answers.add(3.0);
net myNeuralNet = new net(2,1,answers);
for(int i=0; i<200; i++){
myNeuralNet.setInputs(inputs);
myNeuralNet.start();
myNeuralNet.backpropagation();
myNeuralNet.printOutput();
System.out.println("*****");
for(int j=0; j<myNeuralNet.getOutputs().size(); j++){
myNeuralNet.getOutputs().get(j).resetinput();
myNeuralNet.getOutputs().get(j).resetOutput();
myNeuralNet.getOutputs().get(j).resetNumCalled();
}
}
}
}
package myneuralnet;
import java.util.ArrayList;
public class net {
private ArrayList<neuron> inputLayer;
private ArrayList<neuron> outputLayer;
private ArrayList<Double> answers;
public net(Integer numInput,Integer numOut,ArrayList<Double> answers){
inputLayer = new ArrayList<neuron>();
outputLayer = new ArrayList<neuron>();
this.answers = answers;
for(int i=0; i<numOut; i++){
outputLayer.add(new neuron(true));
}
for(int i=0; i<numInput; i++){
ArrayList<Double> randomWeights = createRandomWeights(numInput);
inputLayer.add(new neuron(outputLayer,randomWeights,-100.00,true));
}
for(int i=0; i<numOut; i++){
outputLayer.get(i).setBackConn(inputLayer);
}
}
public ArrayList<neuron> getOutputs(){
return outputLayer;
}
public void backpropagation(){
for(int i=0; i<answers.size(); i++){
neuron IoUt = outputLayer.get(i);
ArrayList<neuron> IoUtBack = IoUt.getBackConn();
Double iSigDeriv = (1-IoUt.getOutput())*IoUt.getOutput();
Double iError = (answers.get(i) - IoUt.getOutput());
System.out.println("Answer: "+answers.get(i) + " IoUt: "+IoUt.getOutput()+" Error: "+iError+" Sigmoid: "+iSigDeriv);
for(int j=0; j<IoUtBack.size(); j++){
neuron jNeuron = IoUtBack.get(j);
Double ijWeight = jNeuron.getWeight(i);
System.out.println("ijWeight: "+ijWeight);
System.out.println("jNeuronOut: "+jNeuron.getOutput());
jNeuron.setWeight(i,ijWeight+(iSigDeriv*iError*jNeuron.getOutput()));
}
}
for(int i=0; i<inputLayer.size(); i++){
inputLayer.get(i).resetinput();
inputLayer.get(i).resetOutput();
}
}
public ArrayList<Double> createRandomWeights(Integer size){
ArrayList<Double> iWeight = new ArrayList<Double>();
for(int i=0; i<size; i++){
Double randNum = (2*Math.random())-1;
iWeight.add(randNum);
}
return iWeight;
}
public void setInputs(Double[] is){
for(int i=0; i<is.length; i++){
inputLayer.get(i).setInput(is[i]);
}
for(int i=0; i<outputLayer.size(); i++){
outputLayer.get(i).resetinput();
}
}
public void start(){
for(int i=0; i<inputLayer.size(); i++){
inputLayer.get(i).fire();
}
}
public void printOutput(){
for(int i=0; i<outputLayer.size(); i++){
System.out.println(outputLayer.get(i).getOutput().toString());
}
}
}
package myneuralnet;
import java.util.ArrayList;
public class neuron {
private ArrayList<neuron> connections;
private ArrayList<neuron> backconns;
private ArrayList<Double> weights;
private Double threshold;
private Double input;
private Boolean isOutput = false;
private Boolean isInput = false;
private Double totalSignal;
private Integer numCalled;
private Double myOutput;
public neuron(ArrayList<neuron> conns,ArrayList<Double> weights,Double threshold){
this.connections = conns;
this.weights = weights;
this.threshold = threshold;
this.totalSignal = 0.00;
this.numCalled = 0;
this.backconns = new ArrayList<neuron>();
this.input = 0.00;
}
public neuron(ArrayList<neuron> conns,Double threshold,Boolean isin){
this.connections = conns;
this.weights = weights;
this.threshold = threshold;
this.totalSignal = 0.00;
this.numCalled = 0;
this.backconns = new ArrayList<neuron>();
this.input = 0.00;
this.isInput = isin;
}
public neuron(Boolean tf){
this.connections = new ArrayList<neuron>();
this.weights = new ArrayList<Double>();
this.threshold = 0.00;
this.totalSignal = 0.00;
this.numCalled = 0;
this.isOutput = tf;
this.backconns = new ArrayList<neuron>();
this.input = 0.00;
}
public void setInput(Double input){
this.input = input;
}
public void setOut(Boolean tf){
this.isOutput = tf;
}
public void resetNumCalled(){
numCalled = 0;
}
public void setBackConn(ArrayList<neuron> backs){
this.backconns = backs;
}
public Double getOutput(){
return myOutput;
}
public Double getinput(){
return totalSignal;
}
public Double getRealinput(){
return input;
}
public ArrayList<Double> getWeights(){
return weights;
}
public ArrayList<neuron> getBackConn(){
return backconns;
}
public Double getWeight(Integer i){
return weights.get(i);
}
public void setWeight(Integer i,Double d){
weights.set(i,d);
}
public void setOutput(Double d){
myOutput = d;
}
public void activation(Double myInput){
numCalled++;
totalSignal += myInput;
if(numCalled==backconns.size() && isOutput){
System.out.println("Total Sig: "+totalSignal);
setInput(totalSignal);
setOutput(totalSignal);
}
}
public void activation(){
Double activationValue = 1 / (1 + Math.exp(input));
setInput(activationValue);
fire();
}
public void fire(){
for(int i=0; i<connections.size(); i++){
Double iWeight = weights.get(i);
neuron iConn = connections.get(i);
myOutput = (1/(1+(Math.exp(-input))))*iWeight;
iConn.activation(myOutput);
}
}
public void resetinput(){
input = 0.00;
totalSignal = 0.00;
}
public void resetOutput(){
myOutput = 0.00;
}
}
OK, so this is a lot of code, so allow me to explain The network is now simple, just an input layer and an output layer - I want to add a hidden layer later, but I'm taking baby steps now Each layer is an ArrayList of neurons The input neuron loads the input, in this case 1 and a 2 These neurons are triggered. It calculates the sigmoid input and output to the output neuron. It adds them and stores the value Then back propagation is adopted (response output) (output) (1-output) (output of specific input neuron), and the weight is updated accordingly Many times, it loops through and I get infinity, which seems to be related to negative weight or sigmoid When it doesn't happen, it converges to 1. Because (1 output of 1) is 0, my weight stops updating
Numcalled and totalsignal values are exactly what the algorithm waits for all neuron inputs before continuing I know it's strange for me to do this, but neurons have a neuron called a connection, which is used to keep it connected forward Another ArrayList named backconns holds the reverse connection I should update the correct weights because I get all the reverse connections between neurons I and j, but all neurons J (the upper layer) I just pull I. I apologize for this confusion - I've been trying a lot of things for hours, but I still can't understand Any help is greatly appreciated!
Solution
Some of the best textbooks on neural networks are generally Chris bishop and Simon Haykin Try reading the chapter on BackProp and understand why the terms in weight update rules are their way The reason I ask you to do this is that BackProp is more subtle than it looks at first If you use the linear activation function on the output layer, things will change (think about why you might want to do this. Tip: post processing), or if you add a hidden layer When I really read the book, it became clearer