Trace Neuron

The Trace neuron is a special type of neuron designed primarily to interact with the trace synapse. It is based on Sutton and Barto's 1981, "Towards a Modern Theory of Adaptive Networks: Expectation and Prediction," which prefigures a broad body of contemporary research into reinforcement learning, and is, to some extent, superceded by "temporal difference" models.

The neuron involves two variables, the standard activation variable and what is called a trace, or eligibility trace, computed in a manner similar to the running average neuron. The synapses connected to a Trace neuron are only updated when the trace neurons they are connected to have a positive non-zero trace, which described as being "eligible." The trace rises and decays with stimulus activity, and with this behavior in mind, the trace can be described as a moving window of opportunity for synaptic change.

The neuron's activation is its weighted input W .

The trace y is updated as follows:

It is assumed that c1 and c2 are between 0 and 1, and that c2 = 1 - c1. (Note that these parameters allow one to implement Sutton and Barto's Equations (4) as well as (3), in the latter case by setting c2 to 0.)

The trace of a neuron can be observed in a trace neuron's tooltip, and can also be observed by attaching a trace neuron to a linear neuron via a signal synapse, and placing "get trace" in its label field.

The trace neuron also contains special methods to get the difference between the neuron's trace and its activation, which functions as a kind of measure of surprise--how different is the neuron's current value from the running trace of its past values? Two methods gauge this: getDifference and getDifferenceHistory, which can also be plugged in to the label field of a signal synapse. The trace synapse uses getDifferenceHistory.

See Ralph Sutton and Andrew Barto (1981), "Towards a Modern Theory of Adaptive Networks: Expectation and Prediction," Psychological Review.

c1

Parameter scaling trace value.

c2

Parameter scaling addition of past activation to the trace.