LMS Network

The LMS Network is a two layer feed-forward network that implements the standard Least Mean Squares rule for learning. Its neurons and synapses behave like a standard network, and are updated according to their own neuron rules. However, a dialog can be called up that trains the synapses according to the Least Mean Squares rule.

The LMS rule is a form of supervised learning, which means that the user must supply desired output values for each of a list of input values. See training files.

The LMS rule works as follows. The change in a weight is equal to the product of a learning rate ε, the pre-synaptic source activation, and the difference between the post-synaptic activation aj and a desired activation tj. The error is the difference between the desired and actual activation of the target neuron.

Repeated application of this rule mimizes reduces mean squared error on a set of training data.

This rule is also known as the "Widrow-Hoff" rule, and the "Delta Rule." Networks that use these rules are sometimes called "adalines" or "madalines" (for the multilayer case, which these networks do not currently implement). They are descendents of an early form of network studied by Rosenblatt called a "perceptron."

Initialization

Since these are two layer networks, they are initialized with a set number of input and output units. The resulting layer will be two layers of the specified number of neurons with feed-forward connections.

Parameters

Learning Rate: Learning rate is denoted by ε. This parameter determines how quickly synapses change.

Training

To train the LMS network select "Train" in the tab-context menu. (Right click on the subnetwork tab and select "train"). You must set an input and output file.

Input File: Use this button to select an input file for training (See training files).

Output File: Use this button to select an output file for training.

Randomize Network: Randomize weights of the network.

User / Play: This repeatedly applies the LMS learning algorithm

User / Step: This applies the LMS learning algorithm once to the entire network.

Batch / Epochs: In batch mode you specify a specific number of epochs for the algorithm to perform. An epoch is one iteration through the entire dataset. The number of epochs is the number of such complete iterations to perform.

Batch / Train: Begin iterating for the number of iterations specified in the epochs field.

Props: Set parameters for this network: momentum and learning rate, described above.

RMS Error: Displays the current root mean squared error.