Neural network design for J function approximation in dynamic programming

Abstract
This paper shows that a new type of artificial neural network (ANN) -- the Simultaneous Recurrent Network (SRN) -- can, if properly trained, solve a difficult function approximation problem which conventional ANNs -- either feedforward or Hebbian -- cannot. This problem, the problem of generalized maze navigation, is typical of problems which arise in building true intelligent control systems using neural networks. (Such systems are discussed in the chapter by Werbos in K.Pribram, Brain and Values, Erlbaum 1998.) The paper provides a general review of other types of recurrent networks and alternative training techniques, including a flowchart of the Error Critic training design, arguable the only plausible approach to explain how the brain adapts time-lagged recurrent systems in real-time. The C code of the test is appended. As in the first tests of backprop, the training here was slow, but there are ways to do better after more experience using this type of network.

arXiv:adap-org/9806001v1 3 Jun 1998
Neural Network Design for J Function
Approximation in Dynamic Programming
Xiaozhong Pang Paul J. Werbos
Abstract
This paper will show that a new neural network design can solv e an
example of diﬃcult function approximation problems which a re crucial to
the ﬁeld of approximate dynamic programming(ADP). Althoug h conven-
tional neural networks have been proven to approximate smoo th functions
very well, the use of ADP for problems of intelligent control or planning
requires the approximation of functions which are not so smo oth. As an
example, this paper studies the problem of approximating th e J function
of dynamic programming applied to the task of navigating maz es in gen-
eral without the need to learn each individual maze. Convent ional neural
networks, like multi-layer perceptrons(MLPs), cannot lea rn this task. But
a new type of neural networks, simultaneous recurrent netwo rks(SRNs),
can do so as demonstrated by successful initial tests. The pa per also ex-
amines the ability of recurrent neural networks to approxim ate MLPs and
vice versa.
Keywords: Simultaneous recurrent networks(SRNs), multi-layer percep-
trons(MLPs), approximate dynamic programming, maze navigation , neural net-
works.
1 Introduction
1.1 Purpose
This paper has three goals:
First, to demonstrate the value of a new class of neural network w hich pro-
vides a crucial component needed for brain-like intelligent control s ystems for
the future.
Second, to demonstrate that this new kind of neural network pro vides better
function approximate ability for use in more ordinary kinds of neural network
applications for supervised learning.
Third, to demonstrate some practical implementation techniques n ecessary
to make this kind of network actually work in practice.

X(t) Y(t)Supervised Learning
System
Actual Y(t)
^
Figure 1: What is supervised learning?
1.2 Background
At present, in the neural network ﬁeld perhaps 90% of neural net work applica-
tions involve the use of neural networks designed to performance a task called
supervised learning(Figure 1). Supervised learning is the task of lea rning a non-
linear function which may have several inputs and several outputs based on some
examples of the function. For example, in character recognition, t he inputs may
be an array of pixels seen from a camera. The desired outputs of th e network
may be a classiﬁcation of character being seen. Another example wo uld be for
intelligent sensing in the chemical industry where the inputs might be s pectral
data from observing a batch of chemicals, and the desired outputs would be
the concentrations of the diﬀerent chemicals in the batch. The pur pose of this
application is to predict or estimate what is in the batch without the ne ed for
expensive analytical tests.
The work in this paper will focus totally on certain tasks in supervised
learning. Even though existing neural networks can be used in supe rvised learn-
ing, there can be performance problems depending on what kind of f unction is
learned. Many people have proven many theorems to show that neu ral networks,
fuzzy logic, Taylor theories and other function approximation have a universal
ability to approximate functions on the condition that the functions have certain
properties and that there is no limit on the complexity of the approxim ation.
In practice, many approximation schemes become useless when the re are many
input variables because the required complexity grows at an expone ntial rate.
For example, one way to approximate a function would be to constru ct a
table of the values of the function at certain points in the space of p ossible
inputs. Suppose there are 30 input variables and we consider 10 pos sible values
of each input. In that case, the table must have 10 30 numbers in it. This
is not useful in practice for many reasons. Actually, however, man y popular

approximation methods like radial basis function(RBF) are similar in sp irit to
a table of values.
In the ﬁeld of supervised learning, Andrew Barron[30] has proved s ome func-
tion approximation theorems which are much more useful in practice . He has
proven that the most popular form of neural networks, the multi- layer percep-
tron(MLP), can approximate any smooth function. Unlike the case with the
linear basis functions (like RBF and Taylor series), the complexity of t he net-
work does not grow rapidly as the number of input variables grows.
Unfortunately there are many practical applications where the fu nctions to
be approximated are not smooth. In some cases, it is good enough j ust to add
extra layers to an MLP[1] or to use a generalized MLP[2]. However, th ere are
some diﬃcult problems which arise in ﬁelds like intelligent control or image
processing or even stochastic search where feed-forward netw orks do not appear
powerful enough.
1.3 Summary and Organization of This Paper
The main goal of this paper is to demonstrate the capability of a diﬀer ent
kind of supervised learning system based on a kind of recurrent net work called
simultaneous recurrent network(SRN). In the next chapter we w ill explain why
this kind of improved supervised learning system will be very importan t to
intelligent control and to approximate dynamic programming. In eﬀe ct this
work on supervised learning is the ﬁrst step in a multi-step eﬀort to b uild more
brain-like intelligent systems. The next step would be to apply the SRN to static
optimization problems, and then to integrate the SRNs into large sys tems for
ADP.
Even though intelligent control is the main motivation for this work, t he
work may be useful for other areas as well. For example, in zip code r ecog-
nition, AT &T [3] has demonstrated that feed-forward networks can achieve a
high level of accuracy in classifying individual digits. However, AT &T and the
others still have diﬃculty in segmenting the total zip codes into individ ual dig-
its. Research on human vision by von der Malsburg[4] and others has suggested
that some kinds of recurrency in neural networks are crucial to t heir abilities
in image segmentation and binocular vision. Furthermore, research ers in image
processing like Laveen Kanal have showed that iterative relaxation algorithms
are necessary even to achieve moderate success in such image pro cessing tasks.
Conceptually the SRN can learn an optimal iterative algorithm, but th e MLP
cannot represent any iterative algorithms. In summary, though w e are most
interested in brain-like intelligent control, the development of SRNs c ould lead
to very important applications in areas such as image processing in th e future.
The network described in this paper is unique in several respects. H owever,
it is certainly not the ﬁrst serious use of a recurrent neural netwo rk. Chapter
3 of this paper will describe the existing literature on recurrent net works. It
will describe the relationship between this new design and other desig ns in the

literature. Roughly speaking, the vast bulk of research in recurre nt networks
has been academic research using designs based on ordinary diﬀere ntial equa-
tions(ODE) to perform some tasks very diﬀerent from supervised learning —
tasks like clustering, associative memory and feature extraction. The simple
Hebbian learning methods[13] used for those tasks do not lead to th e best per-
formance in supervised learning. Many engineers have used anothe r type of
recurrent network , the time lagged recurrent network(TLRN), where the recur-
rency is used to provide memory of past time periods for use in forec asting the
future. However, that kind of recurrency cannot provide the ite rative analysis
capability mentioned above. Very few researchers have written ab out SRNs,
a type of recurrent network designed to minimize error and learn an optimal
iterative approximation to a function. This is certainly the ﬁrst use o f SRNs to
learn a J function from dynamic programming which will be explained more in
chapter 2. This may also be the ﬁrst empirical demonstration of the need for
advanced training methods to permit SRNs to learn diﬃcult functions .
Chapter 4 will explain in more detail the two test problems we have use d
for the SRN and the MLP, as well as the details of architecture and le arning
procedure.
The ﬁrst test problem was used mainly as an initial test of a simple form of
SRNs. In this problem, we tried to test the hypothesis that an SRN c an always
learn to approximate a randomly chosen MLP, but not vice versa. Alt hough
our results are consistent with that hypothesis, there is room for more extensive
work in the future, such as experiments with diﬀerent sizes of neur al networks
and more complex statistical analysis.
The main test problem in this work was the problem of learning the J
function of dynamic programming. For a maze navigation problem, ma ny neural
network researchers have written about neural networks which learn an optimal
policy of action for one particular maze[5]. This paper will address the more
diﬃcult problem of training a neural network to input a picture of a ma ze and
output the J function for this maze. When the J function is known, it is a trivial
local calculation to ﬁnd the best direction of movement. This kind of n eural
network should not require retraining whenever a new maze is encou ntered.
Instead it should be able to look at the maze and immediately ”see” the optimal
strategy. Training such a network is a very diﬃcult problem which has never
been solved in the past with any kind of neural network. Also it is typic al of the
challenges one encounters in true intelligent control and planning. T his paper
has demonstrated a working solution to this problem for the ﬁrst tim e. Now
that a system is working on a very simple form for this problem, it would be
possible in the future to perform many tests of the ability of this sys tem to
generalize its success to many mazes.
In order to solve the maze problem, it was not suﬃcient only to use an SRN.
There are many choices to make when implementing the general idea o f SRNs
or MLPs. Chapter 5 will describe in detail how these choices were mad e in this
work. The most important choices were:

1. Both for the MLP and for the feed-forward core of the SRN we u sed the
generalized MLP design[2] which eliminates the need to decide on the nu mber
of layers.
2. For the maze problem, we used a cellular or weight-sharing archite cture
which exploits the spatial symmetry of the problem and reduces dra matically
the number of weights. In eﬀect we solved the maze problem using on ly ﬁve
distinct neurons. There are interesting parallels between this netw ork and the
hippocampus of the human brain.
3. For the maze problem, an adaptive learning rate(ALR) procedur e was
used to prevent oscillation and ensure convergence.
4. Initial values for the weights and the initial input vector for the S RN were
chosen essentially at random, by hand. In the future, more syste matic methods
are available. But this was suﬃcient for success in this case.
Finally Chapter 6 will discuss the simulation results in more detail, give th e
conclusions of this paper and mention some possibilities for future wo rk.
2 Motivation
In this chapter we will explain the importance of this work. As discuss ed above,
the paper shows how to use a new type of neural network in order t o achieve
better function approximation than what is available from the types of neu-
ral networks which are popular today. This chapter will try to expla in why
better function approximation is important to approximate dynamic program-
ming(ADP), intelligent control and understanding the brain. Image processing
and other applications have already been discussed in the Introduc tion. These
three topics — ADP, intelligent control and understanding the brain — are all
closely related to each other and provide the original motivation for the work
of this paper.
The purpose of this paper is to make a core contribution to developin g the
most powerful possible system for intelligent control.
In order to build the best intelligent control systems, we need to co mbine
the most suitable mathematics together with some understanding o f natural
intelligence in the brain. There is a lot of interest in intelligent control in the
world. Some control systems which are called intelligent are actually v ery quick
and easy things. There are many people who try to move step by ste p to add
intelligence into control , but a step-by-step approach may not be enough by
itself.
Sometimes to achieve a complex diﬃcult goal, it is necessary to have a p lan,
thus some parts of the intelligent control community have develope d a more
systematic vision or plan for how it could be possible to achieve real int elligent
control. First, one must think about the question of what is intelligen t control.
Then, instead of trying to answer this question in one step, we try t o develop a
plan to reach the design. Actually there are two questions:

1. How could we build an artiﬁcial system which replicates the main capa -
bilities of brain-like intelligence, somehow uniﬁed together as they are uniﬁed
together in the brain?
2. How can we understand what are the capabilities in the brain and ho w
they are organized in a functional engineering view? i.e. how are thos e circuits
in the human brain arranged to learn how to perform diﬀerent tasks ?
It would be best to understand how the human brain works before b uilding
an artiﬁcial system. However, at the present time, our understa nding of the
brain is limited. But at least we know that local recurrency plays critic al rule
in the higher part of the human brain[6][7][8][4].
Another reason to use SRNs is that SRNs can be very useful in ADP m ath-
ematically. Now we will discuss what ADP can do for intelligent control a nd
understanding the brain.
The remainder of this chapter will address three questions in order :
1. What is ADP?
2. What is the importance of ADP to intelligent control and understa nding the
brain?
3. What is the importance of SRNs to ADP?
2.1 What is ADP and J Function?
To explain what is ADP, let us consider the original Bellman equation[9]:
J(R(t)) = max
u(t)
(U (R(t), u (t))+ < J (R(t + 1)) >)/ (1 + r) − U0 (1)
where r and u0 are constants that are used only in inﬁnite-time-horizon problems
and then only sometimes, and where the angle brackets refer to ex pectation
value. In this paper we actually use:
J(R(t)) = max
u(t)
(U (R(t), u (t))+ < J (R(t + 1)) >) (2)
since the maze problem do not involve an inﬁnite time-horizon.
Instead of solving for the value of J in every possible state, R(t), we can use
a function approximation method like neural networks to approxima te the J
function. This is called approximate dynamic programming(ADP). This paper
is not doing true ADP because in true ADP we do not know what the J function
is and must therefore use indirect methods to approximate it. Howe ver, before
we try to use SRNs as a component of an ADP system, it makes sense to ﬁrst
test the ability of an SRN to approximate a J function, in principle.
Now we will try to explain what is the intuitive meaning of the Bellman
equation(Equation(1)) and the J function according to the treatment taken
from[2].
To understand ADP, one must ﬁrst review the basics of classical dy namic
programming, especially the versions developed by Howard[28] and B ertsekas.

Classical dynamic programming is the only exact and eﬃcient method t o com-
pute the optimal control policy over time, in a general nonlinear sto chastic envi-
ronment. The only reason to approximate it is to reduce computatio nal cost, so
as to make the method aﬀordable (feasible) across a wide range of a pplications.
In dynamic programming, the user supplies a utility function which may take
the form U (R(t), u (t)) — where the vector R is a Representation or estimate of
the state of the environment (i.e. the state vector) — and a stoch astic model of
the plant or environment. Then ”dynamic programming” (i.e. solution of the
Bellman equation) gives us back a secondary or strategic utility func tion J(R).
The basic theorem is that maximizing U (R(t), u (t)) + J(R(t + 1)) yields the
optimal strategy, the policy which will maximize the expected value of U added
up over all future time. Thus dynamic programming converts a diﬃcu lt prob-
lem in optimizing over many time intervals into a straightforward proble m in
short-term maximization. In classical dynamic programming, we ﬁnd the exact
function J which exactly solves the Bellman equation. In ADP, we learn a kind
of ”model” of the function J; this ”model” is called a ”Critic.” (Alternatively,
some methods learn a model of the derivatives of J with respect to the variables
Ri ; these correspond to Lagrange multipliers, λ i , and to the ”price variables”
of microeconomic theory. Some methods learn a function related to J, as in the
Action-Dependent Adaptive Critic (ADAC)[29].
2.2 Intelligent Control and Robust Control
To understand the human brain scientiﬁcally, we must have some suit able math-
ematical concepts. Since the human brain makes decisions like a cont rol system,
it is an example of an intelligent control system. Neuroscientists do n ot yet un-
derstand the general ability of the human brain to learn to perform new tasks
and solve new problems even though they have studied the brain for decades.
Some people compare the past research in this ﬁeld to what would hap pen
if we spent years to study radios without knowing the mathematics o f signal
processing.
We ﬁrst need some mathematical ideas of how it is possible for a compu ting
system to have this kind of capability based on distributed parallel co mputation.
Then we must ask what are the most important abilities of the human b rain
which unify all of its more speciﬁc abilities in speciﬁc tasks. It would be s een
that the most important ability of brain is the ability to learn over time h ow to
make better decisions in order to better maximize the goals of the or ganism. The
natural way to imitate this capability in engineering systems is to build s ystems
which learn over time how to make decisions which maximize some measur e
of success or utility over future time. In this context, dynamic pro gramming
is important because it is the only exact and eﬃcient method for maxim izing
utility over future time. In the general situation, where random dis turbances
and nonlinearity are expected, ADP is important because it provides both the
learning capability and the possibility of reducing computational cost to an

aﬀordable level. For this reason, ADP is the only approach we have to imitating
this kind of ability of the brain.
The similarity between some ADP designs and the circuitry of the brain has
been discussed at length in [10] and [11]. For example, there is an impor tant
structure in the brain called the limbic system which performs some kin ds of
evaluation or reinforcement functions, very similar to the function s of the neural
networks that must approximate the J function of dynamic programming. The
largest part of the limbic system, called the hippocampus, is known to possess
a higher degree of local recurrency[8].
In general, there are two ways to make classical controllers stable despite
great uncertainty about parameters of the plant to be controlled . For example,
in controlling a high speed aircraft, the location of the center of the gravity
is not known. The center of gravity is not known exactly because it d epends
on the cargo of the air plane and the location of the passengers. On e way to
account for such uncertainties is to use adaptive control method s. We can get
similar results, but more assurance of stability in most cases[16] by u sing related
neural network methods, such as adaptive critics with recurrent networks. It is
like adaptive control but more general. There is another approach called robust
control or H∞ control, which trys to design a ﬁxed controller which remains
stable over a large range in parameter space. Baras and Patel[31] h ave for
the ﬁrst time solved the general problem of H∞ control for general partially
observed nonlinear plants. They have shown that this problem redu ces to a
problem in nonlinear, stochastic optimization. Adaptive dynamic prog ramming
makes it possible to solve large scale problems of this type.
2.3 Importance of the SRN to ADP
ADP systems already exist which perform relatively simple control ta sks like
stabilizing an aircraft as it lands under windy conditions [12]. However t his
kind of task does not really represent the highest level of intelligenc e or planning.
True intelligent control requires the ability to make decisions when fu ture time
periods will follow a complicated, unknown path starting from the initia l state.
One example of a challenge for intelligent control is the problem of nav igating a
maze which we will discuss in chapter 4. A true intelligent control syst em should
be able to learn this kind of task. However, the ADP systems in use to day
could never learn this kind of task. They use conventional neural n etworks to
approximate the J function. Because the conventional MLP cannot approximate
such a J function, we may deduce that ADP system constructed only from
MLPs will never be able to display this kind of intelligent control. Theref ore,
it is essential that we can ﬁnd a kind of neural network which can per form
this kind of task. As we will show, the SRN can ﬁll this crucial gap. The re
are additional reasons for believing that the SRN may be crucial to in telligent
control as discussed in chapter 13 of [9].

3 Alternative Forms of Recurrent Networks
3.1 Recurrent Networks in General
There is a huge literature on recurrent networks. Biologists have u sed many
recurrent models because the existence of recurrency in the bra in is obvious.
However, most of the recurrent networks implemented so far hav e been clas-
sic style recurrent networks, as shown on the left hand of Figure 2 . Most of
these networks are formulated from ordinary diﬀerential equatio n(ODE) sys-
tems. Usually their learning is based on a restricted concept of Hebb ian learn-
ing. Originally in the neural network ﬁeld, the most popular neural ne tworks
were recurrent networks like those which Hopﬁeld[14] and Grossbe rg[15] used
to provide associative memory.
FEATURE EXTRACTION
ART,SOM, ...
MINIMIZATION
HOPFIELD, CAUCHY
CLASSICAL RECURRENT NETWORKS
RECURRENT NETWORKS
TLRN
SRN
(Dynamic Systems,
Prediction)
(Better function
approximation)
ASSOCIATIVE MEMORY
STATIC FUNCTION
CLUSTERING
HOPFIELD, HASSOUN
Figure 2: Recurrent networks
Associative memory networks can actually be applied to supervised le arning.
But in actuality their capabilities are very similar to those of look-up ta bles and
radial basis functions. They make predictions based on similarity to p revious

examples or prototypes. They do not really try to estimate genera l functional
relationships. As a result these methods have become unpopular in p ractical
applications of supervised learning. The theorems of Barron discus sed in the
Introduction show that MLPs do provide better function approxim ation than
do simple methods based on similarity.
There has been substantial progress in the past few years in deve loping new
associative memory designs. Nevertheless, the MLP is still better f or the speciﬁc
task of function approximation which is the focus of this paper.
In a similar way, classic recurrent networks have been used for tas ks like
clustering, feature extraction and static function optimization. B ut these are
diﬀerent problems from what we are trying to solve here.
Actually the problem of static optimization will be considered in future
stages of this research. We hope that the SRN can be useful in suc h appli-
cations after we have used it for supervised learning. When people u se the
classic Hopﬁeld networks for static optimization, they specify all th e weights
and connections in advance[14]. This has limited the success of this kin d of
network for large scale problems where it is diﬃcult to guess the weigh ts. With
the SRN we have methods to train the weights in that kind of structu re. Thus
the guessing is no longer needed. However, to use SRNs in that applic ation
requires reﬁnement beyond the scope of this paper.
There have also been researchers using ODE neural networks who have tried
to use training schemes based on a minimization of error instead of He bbian
approaches. However, in practical applications of such networks , it is important
to consider the clock rates of computation and data sampling. For t hat reason,
it is both easier and better to use error minimizing designs based on dis crete
time rather than ODE.
3.2 Structure of Discrete-Time Recurrent Networks
If the importance of neural networks is measured by the number o f words pub-
lished, then the classic networks dominate the ﬁeld of recurrent ne tworks. How-
ever, if the value is measured based on economic value of practical a pplication,
then the ﬁeld is dominated by time-lagged recurrent networks(TLR Ns). The
purpose of the TLRN is to predict or classify time-varying systems u sing re-
currency as a way to provide memory of the past. The SRN has some relation
with the TLRN but it is designed to perform a fundamentally diﬀerent t ask.
The SRN uses recurrency to represent more complex relationships between one
input vector X(t) and one output Y (t) without consideration of the other times
t. Figure 3 and Figure 4 show us more details about the TLRN and the SR N.
In control applications, u(t) represents the control variables which we use to
control the plant. For example, if we design a controller for a car en gine, the X(t)
variables are the data we get from our sensors. The u(t) variables would include
the valve settings which we use to try to control the process of co mbustion. The
R(t) variables provide a way for the neural networks to remember pas t time

Z -1
X(t)
R(t)
X(t+1)
u(t)
TLRN
Figure 3: Time lagged recurrent network(TLRN)
X
f
y
Figure 4: Simultaneous recurrent network(SRN)

cycles, and to implicitly estimate important variables which cannot be o bserved
directly. In fact, the application of TLRNs to automobile control is t he most
valuable application of recurrent networks ever developed so far.
A simultaneous recurrent network(Figure 4) is deﬁned as a mapping :
ˆY (t) = F (X(t), W ) (3)
which is computed by iterating over the following equation:
y(n+1)(t) = f (y(n)(t), X (t), W ) (4)
where f is some sort of feed-forward network or system, and ˆY is deﬁned as:
ˆY (t) = lim
n→∞
y(n)(t) (5)
When we use ˆY in this paper, we use n = 20 instead of ∞ here.
In Figure 4, the outputs of the neural network come back again as inputs
to the same network. However, in concept there is no time delay. Th e inputs
and outputs should be simultaneous. That is why it is called a simultaneo us
recurrent network(SRN). In practice, of course, there will alwa ys be some phys-
ical time delay between the outputs and the inputs. However if the S RN is
implemented in fast computers, this time delay may be very small comp ared to
the delay between diﬀerent frames of input data.
In Figure 4, X refers to the input data at the current time frame t. The vector
y represents the temporary output of the network, which is then r ecycled as an
additional set of inputs to the network. At the center of the SRN is actually the
feed-forward network which implements the function f . (In designing an SRN,
you can choose any feed-forward network or system as you like. T he function f
simply describes which network you use). The output of the SRN at a ny time
t is simply the limit of the temporary output y.
In Equation (3) and (4), notice that there are two integers — n and t —
which could both represent some kind of time. The integer t represents a slower
kind of time cycle, like the delay between frames of incoming data. The integer
n represents a faster kind of time, like the computing cycle of a fast e lectronic
chip. For example, if we build a computer to analyze images coming from a
movie camera, ” t” and ” t + 1” represent two successive incoming pictures with
a movie camera. There are usually only 32 frames per second. (In th e human
brain, it seems that there are only about 10 frames per second com ing into the
neocortex.) But if we use a fast neural network chip, the computa tional cycle
— the time between ” n” and ” n + 1” — could be as small as a microsecond.
In actuality, it is not necessary to choose between time-lagged rec urrency
(from t to t + 1) and simultaneous recurrency (from n to n + 1). It is possible to
build a hybrid system which contains both types of recurrency. This could be
very useful in analyzing data like movie pictures, where we need both memory
and some ability to segment the images. [9] discusses how to build such a hybrid.

However, before building such a hybrid, we must ﬁrst learn to make S RNs work
by themselves.
Finally, please note that the TLRN is not the only kind of neural netwo rk
used in predicting dynamical systems. Even more popular is the time d elayed
neural network(TDNN), shown in Figure 5. The TDNN is popular beca use it
is easy to use. However, it has less capability, in principle, because it h as no
ability to estimate unknown variables. It is especially weak when some o f these
variables change slowly over time and require memory which persists o ver long
time periods. In addition, the TLRN ﬁts the requirements of ADP dire ctly,
while the TDNN does not[9][16].
X(t)
X(t-1)
X(t-k)
TDNN
X(t+1)
u(t-k)
u(t-1)
u(t)
Figure 5: Time delayed neural network(TDNN)
3.3 Training of SRNs and TLRNs
There are many types of training that have been used for recurre nt networks.
Diﬀerent types of training give rise to diﬀerent kinds of capabilities fo r diﬀerent
tasks. For the tasks which we have described for the SRN and the T LRN, the
proper forms of training all involve some calculation of the derivative s of error
with respects to the weights. Usually after these derivatives are k nown, the
weights are adapted according to a simple formula as follows:
newWi,j = oldWi,j − LR ∗ ∂Error
∂W i,j
(6)
where LR is called the learning rate.
There are ﬁve main ways to train SRNs, all based on diﬀerent method s for
calculating or approximating the derivatives. Four of these method s can also
be used with TLRNs. Some can be used for control applications. But the
details of those applications are beyond the scope of this paper. Th ese ﬁve

types of training are listed in Figure 6. For this paper, we have used t wo of
these methods: Backpropagation through time(BTT) and Trunca tion.
Types of SRN
Simultaneous Backpropagation
Backpropagation Through Time
Forward Propagation
Training
Truncation
Error Critics
Figure 6: Types of SRN Training
The ﬁve methods are:
1. Backpropagation through time(BTT). This method and forward propa-
gation are the two methods which calculate the derivatives exactly. BTT is also
less expensive than forward propagation.
2. Truncation. This is the simplest and least expensive method. It us es only
one simple pass of backpropagation through the last iteration of th e model.
Truncation is probably the most popular method used to adapt SRNs even
though the people who use it mostly just call it ordinary backpropag ation.
3. Simultaneous backpropagation. This is more complex than trunca tion,
but it still can be used in real time learning. It calculates derivatives w hich are
exact in the neighborhood of equilibrium but it does not account for t he details
of the network before it reaches the neighborhood of equilibrium.
4. Error critics(shown in Figure 7). This provides a general approx imation
to BTT which is suitable for use in real-time learning[9].

TLRN
TLRN
CriticError
CriticError
Error
Error
R(t)
X(t)
u(t)
X(t) u(t)
X(t+1) X(t+1) X(t=1)
u(t+1)
R(t+1)
λ^ (t)
λ^
λ (t)
(t+1)
Figure 7: Error Critics

5. Forward propagation. This, like BTT, calculates exact derivative s. It
is often considered suitable for real-time learning because the calcu lations go
forward in time. However, when there are n neurons and m connections, the
cost of this method per unit of time is proportional to n ∗ m. Because of this
high cost, forward propagation is not really brain-like any more than BTT.
3.3.1 Backpropagation through time(BTT)
BTT is a general method for calculating all the derivative of any outc ome or
result of a process which involves repeated calls to the same networ k or net-
works used to help calculate some kind of ﬁnal outcome variable or re sult E.
In some applications, E could represent utility, performance, cost or other such
variables. But in this paper, E will be used to represent error. BTT was ﬁrst
proposed and implemented in [17]. The general form of BTT is as follows :
for k = 1 to T do forward
calculation(k);
calculate result E;
calculate direct derivatives of E with respect to outputs of forward calculations;
for k = T to 1 backpropagate through forwards calculation(k), calculating run-
ning totals where appropriate.
These steps are illustrated in Figure 8. Notice that this algorithm can be
applied to all kinds of calculations. Thus we can apply it to cases where k
represents data frames t as in the TLRNs, or to cases where k represents internal
iterations n as in the SRNs. Also note that each box of calculation receives input
from some dashed lines which represent the derivatives of E with respect to the
output of the box. In order to calculate the derivatives coming out of each
calculation box, one simply uses backpropagation through the calcu lation of
that box starting out from the incoming derivatives. We will explain in m ore
detail how this works in the SRN case and the TLRN case.
So far as we know BTT has been applied in published working systems fo r
TLRNs and for control, but not yet for SRNs until now. However, R umelhart,
Hinton and Williams[18] did suggest that someone should try this.
The application of BTT for TLRNs is described at length in [2] and [9]. The
procedure is illustrated in Figure 9. In this example the total error is actually
the sum of error over each time t where t goes from 1 to T . Therefore the
outputs of the TLRN at each time t (t < T ) have two ways of changing total
errors:
(1)A direct way when the current predictions ˆY (t) are diﬀerent from the
current targets Y (t);
(2)An indirect way based on the impact of R(t) on errors in later time
periods.
Therefore the derivative feedback coming into the TLRN is actually t he sum
of two feedbacks from two diﬀerent sources. As a technical deta il, note that
R(0) needs to be speciﬁed somehow. However, we will not discuss this point
here because the focus of this paper is on SRNs.

k=T
k=1
k=2
Result
(Error, Utility
Calculation
Calculation
Error
Error
Error
Calculation
)
Figure 8: Backpropagation through time(BTT)

X(T) Y(T) Y(T)
Y(1)Y(1)X(1)
X(2) Y(2) Y(2)
R(0)
R(T-1)
R(2)
R(1)
TLRN
TLRN
TLRN
^
^
^
Figure 9: BTT for TLRN

y(1)
y(0)X(t)
F_y(2)
F_y(1)
F_y(N)
Error
Y
Y
= y(n)^
Figure 10: BTT for SRN

Figure 10 shows the application of BTT to training an SRN. This ﬁgure
also provides some explanation of our computer code in the appendix . In this
ﬁgure, the left-hand side(the solid arrows) represents the neur al network which
predicts our desired output Y . (In our example, Y represents the true values of
the J function across all points in the maze). Each box on the left repres ents a
call to a feed-forward system. The vector X(t) represents the external inputs to
the entire system. In our case, X(t) consists of two variables, indicating which
squares in the maze contain obstacles and which contains the goal r espectively.
For simplicity, we selected the initial vector y(0) as a constant vector as we
will describe below. Each call to the feed-forward system includes c alls to a
subroutine which implements the generalized MLP.
On the right-hand side of Figure 10, we illustrate the backpropagat ion cal-
culation used to calculate the derivatives. For the SRN, unlike the TL RN, the
ﬁnal error depends directly only on the output of the last iteration . Therefore
the last iteration receives feedback only from the ﬁnal error but t he other iter-
ations receive feedback only from the iterations just after them. Each box on
the right-hand side represents a backpropagation calculation thr ough the feed-
forward system on its left. The actual backpropagation calculatio n involves
multiple calls to the dual subroutine F net2, which is similar to a subroutine in
chapter 8 of [2].
Notice that the derivative calculation here costs about the same am ount as
the forward calculation on the left-hand side. Thus BTT is very inexp ensive in
terms of computer time. However, the backpropagation calculatio ns do require
the storage of many intermediate results. Also we know that the hu man brain
does not perform such extended calculations backward through t ime. Therefore
BTT is not a plausible model of true brain-like intelligence. We use it here
because it is exact and therefore has the best chance to solve this diﬃcult
problem never before solved. In future research, we may try to s ee whether this
problem can also be solved in a more brain-like fashion.
3.3.2 T runcation for SRNs
Truncation is probably the most popular method to train SRNs even t hough the
term truncation is not often used. For example, the ”simple recurr ent networks”
used in psychology are typically just SRNs adapted by truncation[19 ].
Strictly speaking there are two kinds of truncation — ordinary one- step trun-
cation(Figure 11) and multi-step truncation which is actually a form o f BTT.
Ordinary truncation is by far the most popular. In the derivative ca lculation
of ordinary truncation, the memory inputs to the last iteration are treated as if
they were ﬁxed external inputs to the network. In truncation th ere is only one
pass of ordinary backpropagation involving only the last iteration of the net-
work. Many people have adapted recurrent networks in this simple w ay because
it seems so obvious. However, the derivatives calculated in this way a re not
exactly the same because they do not totally represent the impact of changing

the weights on the ﬁnal error. The reason for this is that changing the weights
will change the inputs to the ﬁnal iteration. It is not right to treat t hese inputs
as constants because they are changed when the weights are cha nged.
y(1)
y(0)X(t)
F_y(N)
Error
Y
Y
= y(n)^
Figure 11: Truncation
The diﬀerence between truncation and BTT can be seen even in a simp le
scalar example, where n=2 and the feed-forward calculation is linear . In this
case, the feed-forward calculation is:
y(1) = A ∗ y(0) + B ∗ X (7)
y(2) = A ∗ y(1) + B ∗ X (8)
In additon,
E = Error = 1
2 (Y − y(2))2 (9)

∂E
∂y (2) = y(2) − Y (10)
In truncation, we use Equation (8) and deduce:
∂E
∂B = ∂E
∂y (2) ∗ ∂y (2)
∂B = (y(2) − Y ) ∗ X (11)
But for a complete calculation, we substitute (7) into (8), deriving:
y(2) = A2 ∗ y(0) + A ∗ B ∗ X + B ∗ X (12)
which yields:
∂E
∂B = (y(2) − Y ) ∗ (A ∗ X + X) (13)
The result in Equation(11) is usually diﬀerent from the result in Equa-
tion(13), which is the true result, and comes from BTT. Depending o n the
value of A, these results could even have opposite signs.
In this paper, we have tried to used truncation because it is so easy and so
popular. If truncation had worked, it would be the easiest way to so lve this
problem. However, it did not work.
3.3.3 Simultaneous Backpropagation
Simultaneous backpropagation is a method developed independently in diﬀer-
ent forms by Werbos, Almeida and Pineda[20][21][22]. The most general form
of this method for SRNs can be found in chapter 3 of[9] and in [23]. This
method is guaranteed to converge to the exact derivatives for th e neighborhood
of the equilibrium y(∞ ) in the case where the forward calculations have reached
equilibrium[20].
As with BTT, the derivative calculations are not expensive. Unlike BTT
there is no need for intermediate storage or for calculation backwa rd through
time. Therefore simultaneous backpropagation could be plausible as a model
of learning in the brain. On the other hand, these derivative calculat ions do
not account for the details of what happened in the early iterations . Unlike
BTT, they are not guaranteed to be exact in the case where the ﬁn al y(n) is
not an exact equilibrium. Even in modeling the brain there may be some n eed
to train SRNs so as to improve the calculation in early iterations. In su mmary,
though simultaneous backpropagation may be powerful enough to solve this
problem, there was suﬃcient doubt that we decided to wait until late r before
experimenting with this method.

3.3.4 Error Critic
The Error Critic, like simultaneous backpropagation, provides appr oximate deriva-
tives. Unlike simultaneous backpropagation, it has no guarantee of yielding ex-
act results in equilibrium. On the other hand, because it approximate s BTT
directly in a statistically consistent manner, it can account for the e arly itera-
tions. Chapter 13 of [9] has argued that the Error Critic is the only p lausible
model for how the human brain adapts the TLRNs in the neocortex. It would
be straightforward in principle to apply the Error Critic to training SR Ns as
well.
Figure 7 shows the idea of an Error Critic for TLRNs. This ﬁgure shou ld be
compared with Figure 10. The dashed input coming into the TLRN in Figu re
7 is intended to be an approximation of the same dashed line coming into the
TLRN in the Figure 9. In eﬀect, the Error Critic is simply a neural netw ork
trained to approximate the complex calculations which lead up to that dashed
line in the Figure 8. The line which ends as the dashed line in Figure 7 begins
as a solid line because those derivatives are estimated as the ordinar y output of
a neural network, the Error Critic. In order to train the Error Cr itic to output
such approximations, we use the error calculation illustrated on the lower right
of Figure 7. In this case, the output of the Error Critic from the pr evious
time period is compared against a set of targets coming from the TLR N. These
targets are simply the derivatives which come out of the TLRN after one pass of
backpropagation starting from these estimated derivatives from the later time
period. This kind of training may seem a bit circular but in fact it has an e xact
parallel to the kind of bootstrapping used in the well known designs f or adaptive
critics or ADP.
As with simultaneous backpropagation, we intend to explore this kind of
design in the future, now that we have shown how SRNs can in fact so lve the
maze problem.
3.3.5 F orward Propagation
The major characteristics of this method have been described abo ve. This
method has been independently rediscovered many times with minor v ariations.
For example, in 1981 Werbos called it conventional perturbation[2]. W illiams
has called it the Williams – Zipser method[5]. Narendra has called it dynamic
backpropagation.
Nevertheless, because this method is more expensive than BTT, ha s no per-
formance advantage over BTT, and is not plausible as a model of lear ning in
the brain, we see no reason to use this method.

4 Two Test Problems
In this paper we use two examples to show that the SRN design has mo re general
function approximation capabilities than does the MLP. Our primary f ocus was
on the maze problem because of its relation to intelligent control as d iscussed
in Chapters 1 and 2. However, before studying this more specialized example,
we performed a few experiments on a more general problem which we call Net
A/Net B. This chapter will discuss these two problems in more detail.
4.1 Net A/Net B
In the Net A/Net B problem, our fundamental goal is to explore the idea that
the functions that an MLP can approximate are a subset of what an SRN can.
In other words, we hypothesize that an SRN can learn to approxima te any
functions which an MLP can represent without adding too much comp lexity,
but not vice versa. To consider the functions which an MLP can repr esent, we
can simply sample a set of randomly selected MLPs, and then test the ability
of SRNs to learn those functions. Similarly we can generate SRNs at r andom
and test the ability of MLPs to learn to approximate the SRNs.
In order to implement this idea, we used the approach shown in Figure 12.
The ﬁrst step in the process was to pick Net A at random. In some ex periments,
Net A was an SRN, while in the other experiments, it was an MLP. In all t hese
experiments, Net B was chosen to be the opposite kind of network f rom Net A.
In picking Net A, we always used the same feed-forward structure . But we used
a random number generator to set the weights. After Net A was ch osen and
Net B was initialized, we generated a stream of random inputs betwee n -1 and
+1 following a uniform distribution. For each set of inputs, we trained Net B
to try to imitate the output of Net A. Of course Net A was ﬁxed. The results
gave an indication of the ability of Net B to approximate Net A.
Net B
Net A
Random Inputs
Figure 12: Net A/Net B

Our preliminary experiments did show that the SRNs have some advan tage
over the MLPs. However, in all of these experiments, the SRN was t rained
with truncation, not BTT. To fully explore all the theoretical issues would
require a much larger set of computer runs. Still, these initial exper iments were
very useful in testing some general computer code in order to pre pare for the
complexities of the maze problem.
4.2 The Maze Problem
In the classic form of the maze problem, a little robot is asked to ﬁnd t he
shortest path from the starting position to a goal position on a two -dimensional
surface where there are some obstacles. For simplicity, this surfa ce is usually
represented as a kind of chess board or grid of squares in which eve ry square is
either clear or blocked by an obstacle. In formal terms, this means that we can
describe the state of the maze by providing three pieces of informa tion:
(1) An array A[ix][iy] which has the value 0 when the square is clear and 1 when
it is covered by an obstacle;
(2) The coordinates of the goal;
(3) The coordinates of the starting square.
In our case, we used a large number to represent the obstacles.
As discussed in the introduction, many researchers have trained n eural net-
works to learn an individual maze[5]. Our goal was to train a network t o input
the array A and to output J[ix][iy] for all the clear squares. According to dy-
namic programming, the best strategy of motion for a robot is simply to move
to that neighboring square which has the smallest J.
This more general problem has not been solved before with neural n etworks.
For example, Houillon etc[24] initially attempted to solve this problem wit h
MLPs, but were unsuccessful. Widrow in several plenary talks has r eported
that his neural truck backer upper has some ability to see and avoid obstacles.
However, this ability was based on an externally developed potential function
which was not itself learned by neural networks. Such potential fu nctions are
analogous to the J function which we are trying to learn.
In fact, this maze problem can always be solved directly and economic ally
by dynamic programming. Why then do we bother to use a neural net work on
this problem? The reason for using this test is not because this simple maze
is important for its own sake, but because this is a very diﬃcult proble m for
a learning system, and because the availability of the correct solutio n is very
useful for testing. It is one thing for a human being to know the ans wer to a
problem. It is a diﬀerent thing to build a learning system which can ﬁgur e out
the answer for itself. Once the simple maze problem is fully conquered , we can
then move on to solve more diﬃcult navigation problems which are too c omplex
for exact dynamic programming.
In order to represent the maze problem as a problem for supervise d learning,

we need to generate both the inputs to the network(the array A) and the desired
outputs(the array B)(Refer to the Appendix). For this basic experiment, we
chose to study the example maze shown in Figure 13. In this ﬁgure, G represents
the goal position, which is assigned a value of ”1”; the other number s represent
the true values of the J function as calculated by dynamic programming (sub-
routine ”Synthesis” in the attached code in the appendix). Intuitiv ely each J
value represents the length of the shortest path from that squa re to the goal.
10 2 3 4 5 6
4 35
7 8
G 2
2 3
Figure 13: Desired J function of a maze
Initially we chose to study this particular maze because it poses some very
unique diﬃculties. In particular there are four equally good direction s starting
from one of these squares in this maze — a feature which can be very confusing
to neural networks, even human. If we had used a fully connected conventional
neural network, then the use of a single test maze would have led to over-training
and meaningless results. However, as we will discuss later in this chap ter, we
constrained all of our networks to prevent this problem. Neverth eless, a major
goal of our future research will be to test the ability of SRNs to pre dict new
mazes after training on diﬀerent mazes.
This problem of maze navigation has some similarity to the problem of co n-
nectedness described by Minsky[25]. Logically we know that the desir ed output
in any square can depend on the situation in any other square. Ther efore, it
is hard to believe that a simple feed-forward calculation can solve this kind of
problem. On the other hand, the Bellman equation(Equation(1)) its elf is a sim-
ple recurrent equation based on relationships between ”neighborin g”(successive)

states. Therefore it is natural to expect that a recurrent stru cture could approx-
imate a J function. The empirical results in this paper conﬁrm these expectio ns.
5 Details of Architecture and Learning Proce-
dure
The architecture and learning used for the net A/Net B problem wer e all very
standard. They will be discussed brieﬂy in section 5.1. The bulk of this chapter
will then describe the two special feature – cellular architecture an d adaptive
learning rate(ALR) used for the maze problem.
5.1 Details for the Net A/Net B Problem
In all these experiments, the MLP network and the feed-forward network f in
the SRN was a standard MLP with two hidden layers. The input vector X
consisted of six numbers between -1 and +1. The two hidden layers a nd the
output layers all had three neurons. The initial weights were chose n at random
according to a uniform distribution between -1 and +1. Training was d one by
standard backpropagation with a learning rate of 0 . 1.
5.2 Weight-sharing and Cellular Architecture
5.2.1 What is Weight-sharing?
In theoretical terms, weight-sharing is a generalized technique fo r exploiting
prior knowledge about some symmetry in the function to be approxim ated.
Weight-sharing has sometimes been called ”windowing” or ”Lie Group” tech-
niques.
Weight-sharing has been used almost exclusively for applications like c har-
acter recognition or image processing where the inputs form a two- dimensional
array of pixels[3][26]. In our maze problem the inputs and outputs also form
arrays of pixels. Weight-sharing leads to a reduction in the number o f weights.
Fewer weights lead in turn to better generalization and easier learnin g.
As an example, suppose that we have an array of hidden neurons wit h volt-
ages net[ix][iy], while the input pixels form an array X[ix][iy]. In that case, the
voltages for a conventional MLP would be determined by the equatio n:
net[i][j] =
∑
ix,iy
W (i, j, ix, iy ) ∗ X(ix, iy ) (14)
Thus if each array has a size 20 ∗ 20, the weights form an array of size
20 ∗ 20 ∗ 20 ∗ 20. This means 160,000 weights — a very big problem. In basic
weight-sharing, this equation would be replaced by:

net[i][j] =
∑
d1,d2
W (d1, d 2) ∗ X(i + d1, i + d2) (15)
Furthermore, if d1 and d2 are limited to an range like [ − 5, 5], then the
number of weights can be reduced to just over 100. Actually this wo uld make
it possible to add two or three additional types of hidden neurons wit hout
exceeding 1,000 weights. This trick was used by Guyon etc[3]. They us ed it to
develop the most successful zip code digit recognizer in existence.
Intuitively AT &T justiﬁed this idea by arguing that similar patterns in dif-
ferent locations have similar meanings. However, there is a more rigo rous math-
ematical justiﬁcation for this procedure as we will see.
5.2.2 Lie Group Symmetry and Weight-sharing
The technique of weight-sharing in neural networks is really just a s pecial case
of the Lie-group method pioneered much earlier by Laveen Kanal an d others
in image processing. Formally speaking, if we know that the function F to be
approximated must obey a certain symmetry requirement then we c an impose
the same symmetry on the neural network which we use to approxim ate F .
More preciously, if Y = F (x) always implies that M Y = F (M x), where M is
some kind of simple linear transformation, then we can require that t he neural
network possess the same symmetry.
Both in image processing and in the maze problem, we can use the symm etry
with respect to those transformations M which move all the pixels by the same
distance to the left, to the right or up and down. In the language of physics,
these are called spatial translations.
Because we know that the best form of the neural network must a lso obey
this symmetry, we have nothing to lose by restricting our weights as required
by the symmetry.
5.2.3 How We implemented Weight-sharing
In order to exploit Lie group symmetry in a rigorous way, we ﬁrst ref ormulated
the task to be solved so as to ensure exact Lie group symmetry. To do this, we
designed our neural network to solve the problem of maze deﬁned o ver a torus.
For our purposes, a torus was simply an N by N square where the right-hand
neighbor of [ i, N ] is the point [ i, 0], and likewise for the other edges. This system
can still solve an ordinary maze as in Figure 13, where the maze is surr ounded
by walls of obstacles.
Next we used a cellular structure for our neural network including b oth the
MLPs and SRNs. A cellular structure means that the network is made up of a
set of cells each made up of a set of neurons. There is one cell for ea ch square
in the maze. The neurons and the weights in each cell are the same as those

in any other cell. Only the inputs and outputs are diﬀerent because t hey come
from diﬀerent locations.
The general idea of our design is shown in Figure 14. Notice that each cell is
made up of two parts: a connector part and a local memory part wh ich includes
4 neighbors and the memory from itself in Fingure 15. The connector part
receives the inputs to the cell and transmits its output to all four n eighboring
cells. In addition, the local memory part sends all its outputs back a s inputs,
but only to the same cell. Finally the forecast of J is based on the output of
the local memory part.
LocalLocalLocalLocal
Y(ix,iy)^
goal
CONNECTOR
obstacle
General Idea of Cellular Network
neigbourneigbour
Connector Connector
goal obstacle
Figure 14: General Idea of the Cellular Network
The exact structure which we used is shown completely in Figure 15. I n this
ﬁgure it can be seen that each cell receives 11 inputs on each iterat ion. Two of
these inputs represent the goal and obstacle variables, A[ix][iy] and B[ix][iy], for
the current pixel. The next four inputs represent the outputs of the connector
neuron from the four neighboring cells from the previous iteration. The ﬁnal
ﬁve inputs are simply the outputs of the same cell from the previous iteration.
Then after the inputs, there are only ﬁve actual neurons. The co nnector part
is only one neuron in our case. The local memory part is four neurons . The
prediction of J[ix][iy] results from multiplying the output of the last neuron by
W s, a weight used to rescale the output.
To complete this description, we must specify how the ﬁve active neu rons
work. In this case, each neuron takes inputs from all of the neuro ns to its left,
as in the generalized MLP design[2]. Except for ˆJ, all of the inputs and outputs

memory
from itself
memory
4 neighbors
Inputs
Ws
goal obstacle
Y^Figure 15: Inputs, Outputs and Memory of Each Cell
range between -1 and 1, and the tanh function is used in place of the usual
sigmoid function.
To initialize the SRN on iteration zero, we simply picked a reasonable look ing
constant vector for the ﬁrst four neurons out of the ﬁve. We se t the initial
starting value to -1. For the last neuron, we set it to 0. In future w ork, we shall
probably experiment with the adaptation of the starting vector y(0).
In order to backpropagate through this entire cellular structure , we simply
applied the chain rule for ordered derivatives as described in [2].
5.3 Adaptive Learning Rate
In our initial experiments with this structure, we used ordinary dyn amic pro-
gramming with only one special trick. The trick was that we set the nu mber of
iterations for SRN to only 1 on the ﬁrst 20 trials, and then to 2 for th e next 20
trials... and so on up until there were 20 iterations.
We found that ordinary weight adjustment led to extremely slow lear ning
due to oscillation. This was not totally unexpected because slow learn ing and
oscillation are a common result of simple steepest descent methods. There
are many methods available to accelerate the learning. Some of thes e like the
DEKF method developed by Ford Motor Company are similar to quasi-N ewton
methods[27] which are very powerful but also somewhat expensive . For this work
we chose to use a method called the adaptive learning rate(ALR) as d escribed
in chapter 3 of [9]. This method is relatively simple and cheap, but far mo re
ﬂexible and powerful than other simple alternatives.
In this method, we maintain a single adapted learning rate for each gr oup
of weights. In this case, we chose three groups of weights:
1. The weight W s used for rescaling of the output;
2. The constant or bias weights ww;

3. All the other weights W .
For each group of weights the learning rate is updated on each trial according
to the following formula:
LR(t + 1) = LR(t) ∗ (0. 9 + 0. 2 ∗
∑
k Wk(t) ∗ Wk(t − 1)∑
k Wk(t − 1) ∗ Wk(t − 1) ) (16)
where the sum over k actually refers to the sum over all weights in the same
group. In addition, to prevent overshoot, we would reset the lear ning rate to:
LR ∗ E
∑
k ( ∂E
∂W k
)
2 (17)
where the sum is taken over all weights. In this special case where t he error on
the next iteration would be predicted to be less than zero, i.e.:
E −
∑
k
(Wk(t + 1) − Wk(t)) ∗ ∂E
∂W k
(t)
= E −
∑
k
(LR ∗ ∂E
∂W k
(t)) ∗ ∂E
∂W k
(t)
= E − LR ∗
∑
k
( ∂E
∂W k
(t))
< 0 (18)
where Wk(t + 1) is the new value for the weights which would be used if the
learning rates were not reset. In our case, we modiﬁed this proced ure slightly
to apply it separately to each group of weights.
After the adaptive learning rates were installed the process of lear ning be-
came far more reliable. Nevertheless, because of the complex natu re of the
function J, there was still some degree of local minimum problem. For our pur-
poses, it was good enough to simply try out a handful of initial values which we
guessed at random. However, in future research, we would like to e xplore the
concept of shaping as described in [9].
6 Simulation Results and Conclusions
In this chapter, we will see some simulation results for the two test p roblems
discussed before. From analyzing the results, we can conclude tha t compared to
the MLPs, the SRNs are more powerful in nonsmooth function appr oximation.
In addition, our new design — the cellular structure — can really solve t he maze
problem.

6.1 Results for the Net A/Net B Problem
From Figure 16 to Figure 19 we can see that the SRN using the same th ree-
layered neural network structure(9 inputs, 3 outputs, and 3 ne urons for each
hidden layer) as the MLP can achieve better simulation result. The SR N not
only converged more rapidly than the MLP(Figure 16 and Figure 17, b ut also
reached a smaller error(Figure 18 and Figure 19), about 1 . 25 ∗ 10− 4, while the
MLP reached 5 ∗ 10− 4. Thus we can say that, in this typical case, an SRN has
better ability to learn an MLP than an MLP to learn an SRN.
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
0.002
0.004
0.006
0.008
0.01
0.012
Trials
Error
Figure 16: The MLP learned the SRN
6.2 Results for the Maze Problem
There are two parts of the results for the maze problem.
First, we compare the J function in each pixels of the same maze as predicted
by an SRN trained by BTT and an SRN trained by truncation respectiv ely with
the actual J function for the maze. Figure 20 and Figure 21 show th at the SRN
trained by BTT can really approximate the J function, but the SRN trained
by truncation cannot. Moreover, the SRN trained by BTT can learn the ability
to ﬁnd the optimal path from the start to the goal as calculated by dynamic
programming. Although there is some error in the approximation of J by the
SRN trained by BTT, the errors are small enough that a system gov erned by
the approximation of J would always move in an optimal direction.

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
0.2
0.4
0.6
0.8
1.2
1.4 x 10
−3
Trials
Error
Figure 17: The SRN learned the MLP
0 200 400 600 800 1000 1200
10 x 10
−4
Trials
Error
Figure 18: The last 1000 trials of ﬁgure 16

0 200 400 600 800 1000 1200
0.5
1.5
2.5
3 x 10
−4
Trials
Error
Figure 19: The last 1000 trials of ﬁgure 17
10 2 3 4 5 6
12.1 2.1
2.92.13.43.9
6.18.49.1
8.4
7.1
9.1 7.9
7.1
7.9
6.1
4.6
3.42.1
3.9
4.6
Figure 20: J function as predicted by SRN-BTT(I)

10 2 3 4 5 6
3.03.0
3.7 4.4
2.9
2.5 2.5
2.54.23.92.9
3.5 4.3
4.6
4.5 4.6 3.5 3.0
3.0
2.5
2.6
3.9
Figure 21: J function as predicted by SRN-Truncation(I)
Second, we show some error curves from Figure 22 to Figure 27. Fr om the
ﬁgures we can see the error curve of SRN trained by BTT not only co nverged
more rapidly than the curve of the SRN trained by truncation, but a lso reached
a much smaller level of error. The errors with the MLP did not improve at all
after about 80 trials(Figure 26 and Figure 27).
6.3 Conclusions
In this paper, we have described a new neural network design for J function
approximation in dynamic programming. We have tested this design in t wo test
problems: Net A/Net B and the maze problem. In the Net A/ Net B pro blem,
we showed that SRNs can learn to approximate MLPs better than ML Ps can
learn SRNs.
In the maze problem, a much more complex problem, we showed that w e
can achieve good results only by training an SRN with a combination of B TT
and adaptive learning rates. In addition, we needed to use a special design — a
cellular structure — to solve this problem. On the other hand, neithe r an MLP
nor an SRN trained by truncation could solve this problem.
Now that it has been proven that neural networks can solve these kinds
of problems, the next step in research is to consider many variation s of these
problems in order to demonstrate generalization ability and the ability to solve
optimization problems while the J function is not known.

0 0.5 1 1.5 2 2.5 3
x 10
Figure 22: Error curve of J function as predicted by SRN-BTT(II)
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3
x 10
0.85
0.9
0.95
1.05
1.1
1.15
1.2
1.25
1.3
1.35
Trials
Error
Figure 23: Error curve of J function as predicted by SRN-BTT(III )

0 0.5 1 1.5 2 2.5 3
x 10
0.5
1.5
2.5
3.5 x 10
Trials
Error
Figure 24: Error curve of J function as predicted by SRN-Truncat ion(II)
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3
x 10
Trials
Error
Figure 25: Error curve of J function as predicted by SRN-Truncat ion(III)

0 10 20 30 40 50 60 70 80 90 100
0.5
1.5
2.5 x 10
Trials
Error
Figure 26: Error curve of J function as predicted by MLP(I)
0 100 200 300 400 500 600 700 800 900 1000
0.5
1.5
2.5 x 10
Trials
Error
Figure 27: Error curve J function as predicted by MLP(II)

Acknowledgments
The work described herein was truly collaborative work. More than h alf
of it was performed when both authors worked together at the Un iversity of
Maryland, College Park, or a nearby location. Neither author receiv ed any ﬁ-
nancial support covering the time when this paper was written; how ever, this
work would have been impossible without prior support and ongoing co nnec-
tions involving: the University of Maryland (Prof. John S. Baras and Robert W.
Newcomb); the National University of Singapore; the IEEE Singapo re Section;
Scientiﬁc Cybernetics Inc.; the International Joint Conference o n Neural Net-
works; the National Science Foundation of the U.S. and the Nationa l Natural
Science Foundation of China.
References
[1] E. D. Sontag, “Feedback stabilization using two-hidden-layer ne ts”, IEEE
Trans. Neural Networks , Vol. 3, No.6, 1992.
[2] P. Werbos, The Roots of Backpropagation: From Ordered Derivatives to
Neural Networks and Political Forecasting , Wiley, 1994.
[3] I. Guyon, I. Poujaud, L. Personnaz, G. Dreyfus, J. Denker, and Y. Le Cun,
“Comparing diﬀerent neural network architectures for classifyin g handwrit-
ten digits”, Proceedings of the IEEE International Joint Conference on
Neural Networks , June 1989.
[4] von der Malsburg, C.& Schneider, W. Biol. Cybernetic, Vol. 54, pp. 29-40,
1986.
[5] W. Miller, R. Sutton & P. Werbos (eds.), Neural Networks for Control ,
MIT Press, 1990.
[6] V. B. Brooks, The Neural Basis of Motor Control , Oxford press.
[7] K. Pribram, Brain and Perception: Holonomy and Structure in Fi+gural
Processing, Erlbaum, 1991.
[8] H. Chang, W.J. Freeman, “Parameter optimization in models of the olfac-
tory neural system”, Neural Networks , Vol. 9, No. 1, pp 1-14,1996.
[9] D.White & D.Sofge (eds.), Handbook of Intelligent Control: Neural, Adap-
tive and Fuzzy Approaches , Van Nostrand, 1992.
[10] P. Werbos, “The brain as a neurocontroller: New hypotheses a nd new
experimental possibilities”, In K.Pribram (eds.), Origins: Brain and Self-
Organization, Erlbaum, 1994.

[11] P. Werbos, “Learning in the brain: engineering interpretation” , In K. Pri-
bram, (eds.), Learning as Self-organization , Erlbaum, 1996.
[12] D. Prokhorov, R. Santiago & D. Wunsch, “Adaptive critic design s: a case
study for neurocontrol”, Neural Networks , Vol.8, No.9, 1995.
[13] D.O.Hebb, Organization of Behavior , Wiley, New York, 1949.
[14] J. Hopﬁeld and D. Tank,”Computing with neural circuits: A model” , Sci-
ence, Vol. 233, pp. 625-633, 1986.
[15] S. Grossberg, The Adaptive brain I , North-Holland, 1987.
[16] P. Werbos, “Optimization methods for brain-like intelligent contr ol”, IEEE
Conference on Decision and Control , 1995.
[17] P. Werbos, Beyond regression: new tools for predictions and analysis i n the
behavioral science, Ph.D. dissertation, Committee on Applied Mathematics,
Harvard University, Cambridge, MA, Nov. 1974.
[18] D. Rumelhart, G. Hinton and R. Williams, “Learning internal repre senta-
tions by error propagation”, in D.Rumelhart and J.McClelland(eds.), Par-
allel Distributed Processing , Vol.1, MIT Press, 1986.
[19] L. Fausett, Fundamentals of Neural Networks: architectures, algorith ms
and applications, Prentice Hall, 1994.
[20] P. Werbos, “Generalization of backpropagation with application to a re-
current gas market model, neural networks”, Vol. 1, pp. 339-36 5, 1988.
[21] L. B. Almeida, “A learning rule for asynchronous perceptrons w ith feedback
in a combinatorial environment”, Proceedings of the IEEE International
Conference on Neural Networks , 1987.
[22] F. J. Pineda, “Generalization of backpropagation to recurren t and higher
oder networks” in it Proceedings of the IEEE International Confe rence on
Neural Information Processing Systems, 1987.
[23] P. Werbos, “Supervised learning: can it escape its local minimum” ,
WCNN93 Proceedings, Erlbaum, 1993. Reprinted in V. Roychowdhury et al
(eds.), Theoretical Advances in Neural Computation and Learning , Kluwer,
1994.
[24] P. Houillon and A. Caron, “Planar robot control in cluttered spa ce by
artiﬁcial neural network”, Math Modeling and Science Computing , Vol. 2,
pp. 498–502, 1993.
[25] M. L. Minsky and S. A. Papert, Perceptrons, MIT Press, 1990, expanded
edition.

[26] T. Maxwell, L. Giles and Y. C. Lee, “Generalization in neural netwo rks:
the contiguity problem”, IEEE First International Conference on Neural
Networks, 1987.
[27] P.K.H. Phua AND S.B.W. Chew, Symmetric rank-one update and quasi-
Newton methods, Optimization Techniques and Applications, Proceedings of
the International Conference on Optimization Techniques a nd Applications,
K.H. Phua et al., eds., World Scientiﬁc, 1992, Singapore, pp. 52–63.
[28] R. Howard, Dynamic Programming and Markhov Processes , MIT press,
Cambridge, MA, 1960.
[29] P. Werbos, Neural networks for control and system identiﬁc ation, IEEE
Conference on Decision and Control (Florida), IEEE, New York, 1989.
[30] A. Barron, “Asymptotically optimal functional estimation by min imum
complexity criteria”, Proceedings of 1994 IEEE International Symposium
on Information Theory , IEEE, New York, 1994.
[31] J. S. Baras and N. S. Patel, “Information state for robust co ntrol of set-
valued discrete time systems”, Proceedings of 34th Conference on Decision
and Control , IEEE, 1995, p. 2302.
A Appendix: The program of the maze problem
using SRN trained by BTT
/* The program of the maze problem using SRN trained by BTT*/
/* Using the SRN trained by BTT to learn the optimal path of a 5* 5 maze: */
/* Learning Rate Adaptive --- Lr_Ws,Lr_W,Lr_ww */
/* When change line 136 and 137 into p=0 then the program will b e MLP*/
/* When change line 243 into F_x[N+i]=0 then the program will be the
SRN trined by Truncation*/
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <time.h>
void F_NET2(double F_Yhat, double W[30][30],double x[30] ,int n,int m,int N,
double F_W[30][30], double F_net[30],double F_Ws[30],do uble Ws,double F_x[30]);
void NET(double W[30][30],double x[30],double ww[30],
int n, int m, int N, double Yhat[30]);

void synthesis(int B[30][30],int A[30][30],int n1,int n2 );
void pweight(double Ws,double F_Ws_T,double ww[30],doub le F_net_T[30],
double W[30][30], double F_W_T[30][30],int n,int N,int m) ;
int minimum(int s,int t,int u,int v);
int min(int k,int l);
double f(double x);
void main()
{
int i,j,it,iz,ix,iy,lt,m,maxt,n,n1,n2,nn,nm,N,p,q,po ,t,TT;
int A[30][30],B[30][30];
double a,b,dot,e,e1,e2,es,mu,s,sum,F_Ws_T,Ws,F_Yhat, wi;
double W[30][30],x[30],F_net_T[30],F_Ws[30],F_W_O[30 ][30],F_W[30][30];
double F_W_T[30][30],F_net[30],ww[30],yy[21][12][8][ 8];
double Yhat[30],F_y[21][12][8][8],F_x[30],F_Jhat[30] [30];
double S_F_W1,S_F_W2,Lr_W,S_F_net1,S_F_net2,Lr_ww,Lr _Ws,F_Ws_O;
double y[50][50],F_net_O[50], F_Ws1, F_Ws2,W_O[50][50] ,ww_O[50],Ws_O;
FILE *f;
/* Number of inputs,neurons and output:7,3,1 */
/* ’n’ is the number of the active neurons */
/* ’m’ and ’N’ both are the number of inputs */
/* ’nm’ is the number of memory is: 5 */
/* ’nn+1’*’nn+1’ is the size of the maze’ */
/* ’TT’ is the number of trials */
/* ’lt’ is the number of the interval time */
/* ’maxt’ is the max number for T in figure[8] */
/* Lr-Ws,Lr_ww and Lr_W are the learning rates for Ws,ww and W */
a=0.9; b=0.2;
n=5;m=11;N=11;nn=6;nm=5;TT=30000;lt=50;maxt=20;wi=25;Ws=40;
e=0;po=pow(2,31) -1;
/* Initial values of Old */
F_Ws_O=1;
for(i=m+1;i<N+n+1;i++)
{
for(j=1;j<i;j++)
F_W_O[i][j]=1;
F_net_O[i]=1;
}

Lr_W=Lr_ww=Lr_Ws=10;
/* Initial values of weights */
for (i=1;i<N+n+1;i++)
x[i]=0;
for (i=m+1;i<N+n+1;i++)
for (j=0;j<i;j++)
{
srand(rand());
W[i][j]=rand();
W[i][j]=0.2091;
}
for (i=m+1;i<N+n+1;i++)
{
srand(rand());
ww[i]=rand();
ww[i]=0.00678;
}
/* Input Maze */
n2=5*5;
n1=5*5-1;
for (i=0;i<7;i++)
for(j=0;j<7;j++)
{
if ((i==0)||(j==0)||(i==6)||(j==6))
B[i][j]=n2;
else
B[i][j]=n1;
}
/* Generate Obstacle */
B[2][2]=B[3][3]=B[4][4]=n2;
/* Generate Start */
B[2][4]=1;
for (i=0;i<7;i++)
for(j=0;j<7;j++)
{
A[i][j]=0;

}
/* Desired outputs */
synthesis(B,A,n1,n2);
if((f=fopen("results5","w"))==NULL) {
printf("Cannot open file");
exit(1);
}
/* Learning Pattern */
for(t=0;t<TT;t++)
{
for(i=m+1;i<N+n+1;i++)
{
for(j=1;j<i;j++)
{
F_W_T[i][j]=0;
}
F_net_T[i]=0;
}
for (i=1;i<n;i++)
for(ix=0;ix<nn+1;ix++)
for(iy=0;iy<nn+1;iy++)
yy[0][i][ix][iy]=-1;
for(ix=0;ix<nn+1;ix++)
for(iy=0;iy<nn+1;iy++)
yy[0][n][ix][iy]=0;
e=F_Ws_T=s=0;
/* If the next two lines are changed into p=0 then it is MLP */
p=(t/lt)+1;
p=(p<maxt ? p:maxt);
for(q=0;q<p+1;q++)
{
e=0;
for(ix=0;ix<nn+1;ix++)
for(iy=0;iy<nn+1;iy++)
{
if (B[ix][iy]==25)
x[1]=B[ix][iy];

else if (B[ix][iy]!=1)
x[1]=0;
x[2]=1;
if (ix!=0)
x[3]=yy[q][1][ix-1][iy];
else
x[3]=yy[q][1][nn][iy];
if (iy!=0)
x[4]=yy[q][1][ix][iy-1];
else
x[4]=yy[q][1][ix][nn];
if (ix!=nn)
x[5]=yy[q][1][ix+1][iy];
else
x[5]=yy[q][1][0][iy];
if (iy!=nn)
x[6]=yy[q][1][ix][iy+1];
else
x[6]=yy[q][1][ix][0];
for (i=1;i<n+1;i++)
x[6+i]=yy[q][i][ix][iy];
NET(W,x,ww,n,m,N,Yhat);
for (i=1;i<n+1;i++)
yy[q+1][i][ix][iy]=Yhat[i];
}
}
e=0;
for (ix=0;ix<nn+1;ix++)
for(iy=0;iy<nn+1;iy++)
{
if (t==(TT-1))
y[ix][iy]=yy[p+1][n][ix][iy];
if (B[ix][iy]!=25)
F_Jhat[ix][iy]=Ws*yy[p+1][n][ix][iy]
-A[ix][iy];
else
F_Jhat[ix][iy]=0;
e+=F_Jhat[ix][iy]*F_Jhat[ix][iy];
}
printf("\n t e %d %e",t,e);
fprintf(f,"\n%d %e",t,e);
/* Initialize F_y */

for(q=1;q<21;q++)
for(ix=0;ix<nn+1;ix++)
for(iy=0;iy<nn+1;iy++)
for(i=1;i<n+1;i++)
F_y[q][i][ix][iy]=0;
for(q=p;q>-1;q--)
{
for (ix=0;ix<nn+1;ix++)
for(iy=0;iy<nn+1;iy++)
{
if (B[ix][iy]==25)
x[1]=B[ix][iy];
else if (B[ix][iy]!=1)
x[1]=0;
x[2]=1;
if (ix!=0)
x[3]=yy[q][1][ix-1][iy];
else
x[3]=yy[q][1][nn][iy];
if (iy!=0)
x[4]=yy[q][1][ix][iy-1];
else
x[4]=yy[q][1][ix][nn];
if (ix!=nn)
x[5]=yy[q][1][ix+1][iy];
else
x[5]=yy[q][1][0][iy];
if (iy!=nn)
x[6]=yy[q][1][ix][iy+1];
else
x[6]=yy[q][1][ix][0];
for (i=1;i<n+1;i++)
x[6+i]=yy[q][i][ix][iy];
NET(W,x,ww,n,m,N,Yhat);
if (q==p)
{
F_Yhat=F_Jhat[ix][iy];
for(i=1;i<n+1;i++)
F_x[N+i]=0;
}
else
{
F_Yhat=0;
for(i=1;i<n+1;i++)

F_x[N+i]=F_y[q+1][i][ix][iy];
}
F_NET2(F_Yhat,W,x,n,m,N,F_W,F_net,
F_Ws,Ws,F_x);
if (ix!=0)
F_y[q][1][ix-1][iy]+=F_x[3];
else
F_y[q][1][nn][iy]+=F_x[3];
if (iy!=0)
F_y[q][1][ix][iy-1]+=F_x[4];
else
F_y[q][1][ix][nn]+=F_x[4];
if (ix!=nn)
F_y[q][1][ix+1][iy]+=F_x[5];
else
F_y[q][1][0][iy]+=F_x[5];
if (iy!=nn)
F_y[q][1][ix][iy+1]+=F_x[6];
else
F_y[q][1][ix][0]+=F_x[6];
for(i=1;i<n+1;i++)
F_y[q][i][ix][iy]+=F_x[6+i];
if (q==p) F_Ws_T+=F_Ws[1];
for(i=m+1;i<N+n+1;i++)
{
for(j=1;j<i;j++)
{
F_W_T[i][j]+=F_W[i][j];
}
F_net_T[i]+=F_net[i];
}
}
}
dot=0;
for(i=m+1;i<N+n+1;i++)
for(j=1;j<i;j++)
{
dot+=F_W_O[i][j]*F_W_T[i][j];
}
S_F_W1=S_F_W2=0;

for(i=m+1;i<N+n+1;i++)
for(j=1;j<i;j++)
{
S_F_W1 += F_W_O[i][j] * F_W_T[i][j];
S_F_W2 += F_W_O[i][j] * F_W_O[i][j];
s+=F_W_T[i][j]*F_W_T[i][j];
}
if ((S_F_W1>S_F_W2) || (S_F_W1==S_F_W2))
Lr_W=Lr_W*(a+b);
else if (S_F_W1<(-2)*S_F_W2)
Lr_W=Lr_W*(a-2*b);
else
Lr_W=Lr_W*(a+b*(S_F_W1/S_F_W2));
S_F_net1=S_F_net2=0;
for(i=m+1;i<N+n+1;i++)
{
s+=F_net_T[i]*F_net_T[i];
S_F_net1 +=F_net_O[i] *F_net_T[i];
S_F_net2 +=F_net_O[i] *F_net_O[i];
}
if ((S_F_net1>S_F_net2) || (S_F_net1==S_F_net2))
Lr_ww=Lr_ww*(a+b);
else if (S_F_net1<(-2)*S_F_net2)
Lr_ww=Lr_ww*(a-2*b);
else
Lr_ww=Lr_ww*(a+b*(S_F_net1/S_F_net2));
F_Ws1=F_Ws_O*F_Ws_T;
F_Ws2=F_Ws_O*F_Ws_O;
if ((F_Ws1>F_Ws2) || (F_Ws1==F_Ws2))
Lr_Ws=Lr_Ws*(a+b);
else if (F_Ws1<(-2)*F_Ws2)
Lr_Ws=Lr_Ws*(a-2*b);
else
Lr_Ws=Lr_Ws*(a+b*(F_Ws1/F_Ws2));
s+=F_Ws_T*F_Ws_T;
es=e/s;
if ((e-Lr_W*s)<0)
Lr_W=Lr_W*es;
if ((e-Lr_ww*s)<0)
Lr_ww=Lr_ww*es;
for(i=m+1;i<N+n+1;i++)
{
for(j=1;j<i;j++)

{
W_O[i][j]=W[i][j];
W[i][j]-=Lr_W*F_W_T[i][j];
}
ww_O[i]=ww[i];
ww[i]-=Lr_ww*F_net_T[i];
}
if ((e-Lr_Ws*s)<0)
Lr_Ws=Lr_Ws*es;
Ws_O=Ws;
Ws-=Lr_Ws*F_Ws_T;
sum=0;
for(i=m+1;i<N+n+1;i++)
{
for(j=1;j<i;j++)
{
F_W_O[i][j]=F_W_T[i][j];
}
F_net_O[i]=F_net_T[i];
}
F_Ws_O=F_Ws_T;
}
fclose(f);
}
void synthesis(int B[30][30],int A[30][30],int n1,int n2 )
{
int k,mini,no,i,j;
/* Initialization */
k=0;
for (i=0;i<7;i++)
for(j=0;j<7;j++)
{
A[i][j]=B[i][j];
}
/* Searching the optimal path */
/* Calculating the Utility */
no = n2-3-1;
while (k!=no)

{
k=0;
for(i=1;i<6;i++)
for(j=1;j<6;j++)
{
mini = 1 + minimum(A[i-1][j],A[i][j-1],A[i+1][j],A[i][j +1]);
if ((A[i][j]!=n2) && (A[i][j]!=1))
{
if ((A[i][j]==mini) && (A[i][j]!=n1))
k++;
else
if (mini!=n2)
A[i][j]=mini;
}
}
}
}
/* minimum: return the minimum value */
int minimum(int s,int t,int u,int v)
{
int mini;
mini=0;
mini=min(min(min(s,t),u),v);
return mini;
}
void NET(double W[30][30],double x[30],double ww[30],
int n, int m, int N, double Yhat[30])
{
int i,j;
double net;
for (i=m+1;i<N+n+1;i++)
{
net=0;
for (j=1;j<i;j++)
{
net += W[i][j] * x[j];
}
x[i]=f(net+ww[i]);

}
for (i=1;i<n+1;i++)
{
Yhat[i]=x[i+N];
}
}
double f(double x)
{
double z;
z=(1-exp(-x))/(1+exp(-x));
return z;
}
void F_NET2(double F_Yhat,double W[30][30],double x[30] ,int n,int m,int
N,double F_W[30][30],double F_net[30],double F_Ws[30], double Ws,
double F_x[30])
/* This subroutine calculates the F_W terms needed to adapt a fully connected */
/* generalized MLP. It does not backpropagate through the ne twork and it */
/* does not permit the switching off of weights. */
{
int i,j;
for (i=1;i<N+1;i++)
F_x[i]=0;
F_x[N+n]+=F_Yhat*Ws;
F_Ws[1]=F_Yhat*x[N+n];
F_net[N+n]=F_x[N+n]*(1-x[N+n]*x[N+n])*0.5;
for (j=1;j<N+n;j++)
F_W[N+n][j]=F_net[N+n]*x[j];
for (i=N+n-1;i>m;i--)
{
for (j=i+1;j<N+n+1;j++)
{
F_x[i]+=W[j][i]*F_net[j];
}
F_net[i]=F_x[i]*(1-x[i]*x[i])*0.5;
for (j=1;j<i;j++)
{
F_W[i][j]=F_net[i]*x[j];
}
}
for(i=m;i>0;i--)
for(j=m+1;j<N+n+1;j++)
F_x[i]+=W[j][i]*F_net[j];

}
int min(int k, int l)
{
int r;
if (k>l) r=l;
else r=k;
return r;
}
void pweight(double Ws,double F_Ws_T,double ww[30],doub le F_net_T[30],
double W[30][30], double F_W_T[30][30],int n,int N,int m)
{
int i,j;
for(i=m+1;i<N+n+1;i++)
{
for(j=1;j<i;j++)
{
printf("\n W[i][j] F_W_T %e %e",W[i][j],F_W_T[i][j]);
}
printf("\n ww F_net_T %e %e",ww[i],F_net_T[i]);
}
printf("\n Ws F_Ws_T %e %e",Ws,F_Ws_T);
}


On the 1/f fluctuations in the nonlinear systems affected by noise

Abstract
We investigate a problem of the necessary and sufficient conditions for appearance of the 1/f fluctuations in the simple systems affected by the external random perturbations, i.e. the power spectral density of the flux of particles moving in some contours and perturbed by the external forces. In some cases we observe the 1/f behavior but only in some range of frequencies and parameters of the systems.

arXiv:adap-org/9806002v1 5 Jun 1998
ON THE 1/f FLUCTUATIONS IN THE NONLINEAR SYSTEMS
AFFECTED BY NOISE
B. Kaulakys
Institute of Theoretical Physics and Astronomy, A. Goˇ stauto 12, 2600 Vilnius, Lithuania,
T. Meˇ skauskas
Department of Mathematics, Vilnius University, Naugarduk o 24, 2006 Vilnius, Lithuania
ABSTRACT
We investigate a problem of the necessary and suﬃcient conditions f or ap-
pearance of the 1 /f ﬂuctuations in the simple systems aﬀected by the external
random perturbations, i.e. the power spectral density of the ﬂux of particles moving
in some contours and perturbed by the external forces. In some cases we observe
the 1 /f δ (δ ≃ 1) behavior but only in same range of frequencies and parameters o f
the systems.
INTRODUCTION
Many processes of the greatest diversity exhibit temporal evolut ion charac-
terized by a power spectral density S(f ) which behaves like 1 /f δ (δ ≃ 1) at low
frequencies. Despite considerable eﬀorts, the generally accepte d theory of the 1 /f
noise is still lacking. Possibly, there are several physical mechanism s giving rise to
1/f noise.
Usually a study of 1 /f noise in materials starts from the approximate em-
pirical relation given in 1969 by Hooge which for the current I may be written as
[1]
S(f ) = ¯I
2 α
N f . (1)
Here ¯I
is the squared average current, α is a dimensionless constant which charac-
terizes the noise intensity and N is the total number of the free charge carriers or
ﬂuctuating particles. Since the current is proportional to the num ber of particles, the
power spectral density is proportional to the number of particles N too. Therefore,
it is likely that every particle inﬂuences on the spectral power additiv ely and it is
logical to search of the 1 /f ﬂuctuations in the few-particle systems aﬀected by some
perturbations.
MODELS AND RESULTS
One of the relatively universal and approved mechanisms which gene rates 1/f
noise is the intermittency [2]. Intermittent chaos is often observed phenomenon in
nonlinear dynamical systems. Investigating a transition from chao tic to nonchaotic
behavior in the randomly driven systems [3] we have observed the int ermittency
route to chaos too [4]. Therefore, it is interesting to analyze the po wer spectral

density of the ﬂux of small number of particles, or even one particle , moving in
some contours and perturbed by external random forces.
The intensity of one-dimensional ﬂux of small particles in some space point
xs may be expressed as
I(t) =
∑
k
sign(vk) δ(t − tk), (2)
where tk is a sequence of transit times when one of the particles crosses the point
xs and sign( vk) is the sign of velocity in this point. The power spectral density of
the ﬂux (2) is
S(f ) = 2
T
⟨ ⏐
⏐
⏐
⏐
⏐
∑
k
sign(vk)e−i2πf tk
⏐
⏐
⏐
⏐
⏐
2⟩
, (3)
where T is the whole observation time and the brackets ⟨... ⟩ denote averaging over
the realizations. When the averaged ﬂux ¯I is nonzero the dominant term in the
power spectral density at low frequencies regardless of the deta iled structure of the
ﬂux is Sdc(f ) = ¯I
/T (πf )2. To eliminate this term we divide the whole observation
time T into pieces of equal length and calculate the averaged power spect ral density
of the diﬀerence of two pieces ﬂuxes [6].
We analyze the power spectral density of particles moving in some po tential
and perturbed by the random noise modelled like in papers [3, 4] by the resets
of velocity of all particles after every time interval τ according to the replacement
vnew(kτ) = αv old(kτ) + βv ran
k , k = 1 , 2, . . . .Here α and β are positive parameters
and vran
k are random velocities from the Gaussian distribution of variance σ = 1.
First of all we have calculated the power spectral density of the ﬂu x of particles
bounded in the Duﬃng potential V (x) = x4−x2. Under some conditions a transition
from chaotic to nonchaotic behavior and synchronization of partic les in such system
may be observed [3-5]. The calculated spectral density of the curr ent of particles in
such systems usually was very close to the white noise. Only for α = 1 we observe
the 1 /f 2 noise, while for α close to 1 the noise dependence on the frequency is
between 1 /f 2 and 1 /f (see Fig. 1(a)). A very small component of 1 /f noise in the
ﬂux ﬂuctuations of such system may be explained on the basis of Eq. (1) according to
which the intensity of 1 /f noise component is proportional to the squared averaged
current ¯I
. But in such model the averaged current equals zero.
Therefore further we have analyzed the power spectral density of the current
of particles moving in the closed contour and perturbed by the comm on for all
particles noise. The simplest equations of motion for such model are of the form [5]
˙v = F (x) − γv, ˙x = v (4)
Here F (x) = F (x + L) is the (regular) periodic in the space force with L being the
length of the contour and γ is the friction coeﬃcient. Because of the directed drift the
averaged current in this model is nonzero. We have investigated th e dependence of
the power spectral density on the nonlinearity of equations of mot ion and on intensity
of the perturbation. Moreover, we have analyzed the ensemble of particles moving
with the friction in the periodically time-dependent force F (x + L, t + Tp) = F (x, t )
and aﬀected of the identical for all particles noise. We have observ ed the relatively
weak sensitivity of the spectral density dependence on the frequ ency to the nonlinear

and periodically time-dependent terms in the equation for the velocit y. Moreover,
the spectral density is not sensitive to the number of the particles N moving in the
contour too. Therefore, we show results of calculations for the s implest versions
of the model (4), i.e. for the motion of one particle aﬀected by the c onstant force
and for the free motion with perturbations modelled by the velocity r eplacements
every time interval τ (see Figs. 1 (b) and (d)). For some parameters τ when α ̸= 1
we observe the S(f ) dependence on the frequency f (in same range of frequencies)
close to the 1 /f -dependence. Although, when we analyze the current power spec tral
density in the large range of frequency we can, as a rule, discover t he Lorentzian-
type curves. However the increase of the S(f ) with the decrease of the frequency
usually is not like 1 /f 2 but often as 1 /f .
From the analyzes above it appears that the current power spect ral density in
the most degree depends not on the type of the potential (and fo rce F (x, t )) in which
the particles move but much more on the type and intensity of the pe rturbation.
Therefore, we have also analyzed the simplest model for the pertu rbed ﬂux: the
free motion of the particles with the Brownian-type additional ﬂuct uations of the
transition times. For such model the transit times may be calculated from the
iterative sequence tk = tk− 1 + 1 + βδt ran
k . Here the period of the free motion in
the closed contour is chosen to be equal 1 and δtran
k are random time increments
from the Gaussian distribution of variance σ = 1. Results of calculations of S(f )
for diﬀerent values of intensity of perturbation β are presented in Fig. 1 (c). With
increase of the perturbation intensity we observe transition from the white noise at
β ≤ 1 to the Lorentzian-type, 1 /f - and 1 /f 1. 3-type noise at β ≥ 2.
Summarizing, we have analyzed the problem of necessary and suﬃcie nt
conditions for appearance of the 1 /f -type ﬂuctuations in the current of the simple
systems of particles moving in some contour. Even in such simplest mo dels consisting
of few or even one particle moving in the closed contours and pertur bed by the
appropriate external noise we may observe the power spectral d ensity S(f ), which
behaves like 1 /f δ (δ ≃ 1) in same range of frequencies.
ACKNOWLEDGMENTS
The research described in this publication was supported in part by t he
Lithuanian State Science and Studies Foundation.
REFERENCES
1. F. N. Hooge, T. G. M. Kleinpenning and L. K. J. Vandamme. Rep. Prog. Phys.
44, 479 (1981).
2. H. G. Schuster. Deterministic chaos: an introduction. (VCH, Weinheim, 1989).
3. B. Kaulakys and G. Vektaris. Phys. Rev. E 52, 2091 (1995).
4. B. Kaulakys and G. Vektaris. In: Proc. 13th Int. Conf. on Noise in Physical
Systems and 1/f Fluctuations. Eds V. Bareikis and R. Katilius. (World Scientiﬁc:
Singapore, 1995) p.677.
5. B. Kaulakys, F. Ivanauskas and T. Meˇ skauskas. In: Proc. Intern. Conf. on
Nonlinearity, Bifurcation and Chaos: the Doors to the Futur e. (Lodz-Dobieszkow,
Poland, September 16-18, 1996) p.145.
6. T. Musha and H. Higuchi. Jap. J. Appl. Phys. 15, 1271 (1976).

/G14/G13/G10/G17 /G14/G13/G10/G16 /G14/G13/G10/G15 /G14/G13/G10/G14/G14/G13/G10/G16
/G14/G13/G10/G15
/G14/G13/G10/G14
/G14/G13/G13
/G14/G13/G14
/G14/G13/G15
/G14/G13/G16
/G0B/G44/G0C
/G49
/G36/G0B/G49/G0C
α /G03/G20/G03/G14
α /G03/G20/G03/G13/G11/G1C/G18
α /G03/G20/G03/G13/G11/G18
/G14/G12/G49
/G14/G12/G49/G15
/G14/G13/G10/G17 /G14/G13/G10/G16 /G14/G13/G10/G15 /G14/G13/G10/G14
/G14/G13/G10/G14
/G14/G13/G13
/G14/G13/G14
/G14/G13/G15
/G14/G13/G16
/G0B/G45/G0C
/G49
/G36/G0B/G49/G0C
τ/G03/G20/G03/G13/G11/G14
τ/G03/G20/G03/G14/G11/G17
α /G03/G20/G03/G13/G11/G18
α /G03/G20/G03/G14/G0F/G03τ/G03/G20/G03/G13/G11/G14
/G14/G12/G49
/G14/G12/G49/G15
/G14/G13/G10/G17 /G14/G13/G10/G16 /G14/G13/G10/G15 /G14/G13/G10/G14
/G14/G13/G10/G14
/G14/G13/G13
/G14/G13/G14
/G14/G13/G15
/G0B/G46/G0C
/G49
/G36/G0B/G49/G0C
β/G03/G20/G03/G1B
β/G03/G20/G03/G17
β/G03/G20/G03/G15
β/G03/G20/G03/G13/G11/G18
/G14/G12/G49
/G14/G12/G49/G14/G11/G16
/G14/G12/G49/G15
/G29/G4C/G4A/G11/G03/G14/G11/G03/G03/G33/G52/G5A/G48/G55/G03/G56/G53/G48/G46/G57/G55/G44/G4F/G03/G47/G48/G51/G56/G4C/G57/G4C/G48/G56/G03/G44/G56/G03/G49/G58/G51/G46/G57/G4C/G52/G51/G56/G03/G52/G49/G03/G49/G55/G48/G54/G58/G48/G51/G46/G5C/G03/G49/G52/G55/G03/G49/G4F/G58/G5B/G03/G52/G49/G03/G53/G44/G55/G57/G4C/G46/G4F/G48/G1D/G03/G0B/G44/G0C
/G4C/G51/G03/G57/G4B/G48/G03/G27/G58/G49/G49/G4C/G51/G4A/G03/G53/G52/G57/G48/G51/G57/G4C/G44/G4F/G03/G5A/G4C/G57/G4B/G03τ/G03/G20/G03/G16/G0F/G03β/G03/G20/G03/G14/G03/G44/G51/G47/G03/G47/G4C/G49/G49/G48/G55/G48/G51/G57/G03α /G1E/G03/G49/G52/G55/G03/G50/G52/G57/G4C/G52/G51/G03/G4C/G51/G03/G46/G4F/G52/G56/G48/G47
/G46/G52/G51/G57/G52/G58/G55/G03/G44/G46/G46/G52/G55/G47/G4C/G51/G4A/G03/G57/G52/G03/G28/G54/G11/G0B/G17/G0C/G1D/G03/G0B/G45/G0C/G03/G5A/G4C/G57/G4B/G03/G29/G0B/G5B/G0C/G03/G20/G03/G14/G0F/G03γ/G03/G20/G03/G13/G11/G14/G0F/G03/G2F/G03/G20/G03/G15/G0F/G03β/G03/G20/G03/G14/G1E/G03/G0B/G47/G0C/G03/G5A/G4C/G57/G4B
/G29/G0B/G5B/G0C/G03/G20/G03γ/G03/G20/G03/G13/G0F/G03/G2F/G03/G20/G03/G14/G0F/G03β/G03/G20/G03/G14/G1E/G03/G44/G51/G47/G03/G0B/G46/G0C/G03/G49/G52/G55/G03/G57/G4B/G48/G03/G4C/G57/G48/G55/G44/G57/G4C/G59/G48/G03/G56/G48/G54/G58/G48/G51/G46/G48/G03/G52/G49/G03/G57/G55/G44/G51/G56/G4C/G57/G03/G57/G4C/G50/G48/G56/G11
/G14/G13/G10/G15 /G14/G13/G10/G14
/G14/G13/G10/G14
/G14/G13/G13
/G14/G13/G14
/G14/G13/G15
/G14/G13/G16
/G49
/G36/G0B/G49/G0C
/G0B/G47/G0C
α /G03/G20/G03/G13/G11/G18/G0F/G03τ/G03/G20/G03/G14
α /G03/G20/G03/G13/G11/G1C/G0F/G03τ/G03/G20/G03/G13/G11/G16
α /G03/G20/G03/G13/G11/G1C/G0F/G03τ/G03/G20/G03/G14
α /G03/G20/G03/G14/G0F/G03τ/G03/G20/G03/G14
/G14/G12/G49/G14/G11/G18
/G14/G12/G49
/G14/G12/G49/G15