Optimal control

Optimal control theory is a branch of mathematical optimization that deals with finding a control for a dynamical system over a period of time such that an objective function is optimized. It has numerous applications in science, engineering and operations research. For example, the dynamical system might be a spacecraft with controls corresponding to rocket thrusters, and the objective might be to reach the moon with minimum fuel expenditure. Or the dynamical system could be a nation's economy, with the objective to minimize unemployment; the controls in this case could be fiscal and monetary policy. A dynamical system may also be introduced to embed operations research problems within the framework of optimal control theory.
Optimal control is an extension of the calculus of variations, and is a mathematical optimization method for deriving control policies. The method is largely due to the work of Lev Pontryagin and Richard Bellman in the 1950s, after contributions to calculus of variations by Edward J. McShane. Optimal control can be seen as a control strategy in control theory.
General method.
Optimal control deals with the problem of finding a control law for a given system such that a certain optimality criterion is achieved. A control problem includes a cost functional that is a function of state and control variables. An optimal control is a set of differential equations describing the paths of the control variables that minimize the cost function. The optimal control can be derived using Pontryagin's maximum principle (a necessary condition also known as Pontryagin's minimum principle or simply Pontryagin's principle), or by solving the Hamilton–Jacobi–Bellman equation (a sufficient condition).
We begin with a simple example. Consider a car traveling in a straight line on a hilly road. The question is, how should the driver press the accelerator pedal in order to "minimize" the total traveling time? In this example, the term "control law" refers specifically to the way in which the driver presses the accelerator and shifts the gears. The "system" consists of both the car and the road, and the "optimality criterion" is the minimization of the total traveling time. Control problems usually include ancillary constraints. For example, the amount of available fuel might be limited, the accelerator pedal cannot be pushed through the floor of the car, speed limits, etc.
A proper cost function will be a mathematical expression giving the traveling time as a function of the speed, geometrical considerations, and initial conditions of the system. Constraints are often interchangeable with the cost function.
Another related optimal control problem may be to find the way to drive the car so as to minimize its fuel consumption, given that it must complete a given course in a time not exceeding some amount. Yet another related control problem may be to minimize the total monetary cost of completing the trip, given assumed monetary prices for time and fuel.
A more abstract framework goes as follows. Minimize the continuous-time cost functional
formula_1
subject to the first-order dynamic constraints (the state equation)
formula_2
the algebraic "path constraints"
formula_3
and the endpoint conditions
formula_4
where formula_5 is the "state", formula_6 is the "control", formula_7 is the independent variable (generally speaking, time), formula_8 is the initial time, and formula_9 is the terminal time. The terms formula_10 and formula_11 are called the "endpoint cost " and the "running cost" respectively. In the calculus of variations, formula_10 and formula_11 are referred to as the Mayer term and the "Lagrangian", respectively. Furthermore, it is noted that the path constraints are in general "inequality" constraints and thus may not be active (i.e., equal to zero) at the optimal solution. It is also noted that the optimal control problem as stated above may have multiple solutions (i.e., the solution may not be unique). Thus, it is most often the case that any solution formula_14 to the optimal control problem is "locally minimizing".
Linear quadratic control.
A special case of the general nonlinear optimal control problem given in the previous section is the "linear quadratic" (LQ) optimal control problem. The LQ problem is stated as follows. Minimize the "quadratic" continuous-time cost functional
formula_15
Subject to the "linear" first-order dynamic constraints
formula_16
and the initial condition
formula_17
A particular form of the LQ problem that arises in many control system problems is that of the "linear quadratic regulator" (LQR) where all of the matrices (i.e., formula_18, formula_19, formula_20, and formula_21) are "constant", the initial time is arbitrarily set to zero, and the terminal time is taken in the limit formula_22 (this last assumption is what is known as "infinite horizon"). The LQR problem is stated as follows. Minimize the infinite horizon quadratic continuous-time cost functional
formula_23
Subject to the "linear time-invariant" first-order dynamic constraints
formula_24
and the initial condition
formula_17
In the finite-horizon case the matrices are restricted in that formula_20 and formula_21 are positive semi-definite and positive definite, respectively. In the infinite-horizon case, however, the matrices formula_20 and formula_21 are not only positive-semidefinite and positive-definite, respectively, but are also "constant". These additional restrictions on
formula_20 and formula_21 in the infinite-horizon case are enforced to ensure that the cost functional remains positive. Furthermore, in order to ensure that the cost function is "bounded", the additional restriction is imposed that the pair formula_32 is "controllable". Note that the LQ or LQR cost functional can be thought of physically as attempting to minimize the "control energy" (measured as a quadratic form).
The infinite horizon problem (i.e., LQR) may seem overly restrictive and essentially useless because it assumes that the operator is driving the system to zero-state and hence driving the output of the system to zero. This is indeed correct. However the problem of driving the output to a desired nonzero level can be solved "after" the zero output one is. In fact, it can be proved that this secondary LQR problem can be solved in a very straightforward manner. It has been shown in classical optimal control theory that the LQ (or LQR) optimal control has the feedback form
formula_33
where formula_34 is a properly dimensioned matrix, given as
formula_35
and formula_36 is the solution of the differential Riccati equation. The differential Riccati equation is given as
formula_37
For the finite horizon LQ problem, the Riccati equation is integrated backward in time using the terminal boundary condition
formula_38
For the infinite horizon LQR problem, the differential Riccati equation is replaced with the "algebraic" Riccati equation (ARE) given as
formula_39
Understanding that the ARE arises from infinite horizon problem, the matrices formula_18, formula_19, formula_20, and formula_21 are all "constant". It is noted that there are in general multiple solutions to the algebraic Riccati equation and the "positive definite" (or positive semi-definite) solution is the one that is used to compute the feedback gain. The LQ (LQR) problem was elegantly solved by Rudolf E. Kálmán.
Numerical methods for optimal control.
Optimal control problems are generally nonlinear and therefore, generally do not have analytic solutions (e.g., like the linear-quadratic optimal control problem). As a result, it is necessary to employ numerical methods to solve optimal control problems. In the early years of optimal control ( 1950s to 1980s) the favored approach for solving optimal control problems was that of "indirect methods". In an indirect method, the calculus of variations is employed to obtain the first-order optimality conditions. These conditions result in a two-point (or, in the case of a complex problem, a multi-point) boundary-value problem. This boundary-value problem actually has a special structure because it arises from taking the derivative of a Hamiltonian. Thus, the resulting dynamical system is a Hamiltonian system of the form
formula_44
where
formula_45
is the "augmented Hamiltonian" and in an indirect method, the boundary-value problem is solved (using the appropriate boundary or "transversality" conditions). The beauty of using an indirect method is that the state and adjoint (i.e., formula_46) are solved for and the resulting solution is readily verified to be an extremal trajectory. The disadvantage of indirect methods is that the boundary-value problem is often extremely difficult to solve (particularly for problems that span large time intervals or problems with interior point constraints). A well-known software program that implements indirect methods is BNDSCO.
The approach that has risen to prominence in numerical optimal control since the 1980s is that of so-called "direct methods". In a direct method, the state or the control, or both, are approximated using an appropriate function approximation (e.g., polynomial approximation or piecewise constant parameterization). Simultaneously, the cost functional is approximated as a "cost function". Then, the coefficients of the function approximations are treated as optimization variables and the problem is "transcribed" to a nonlinear optimization problem of the form:
Minimize
formula_47
subject to the algebraic constraints
formula_48
Depending upon the type of direct method employed, the size of the nonlinear optimization problem can be quite small (e.g., as in a direct shooting or quasilinearization method), moderate (e.g. pseudospectral optimal control) or may be quite large (e.g., a direct collocation method). In the latter case (i.e., a collocation method), the nonlinear optimization problem may be literally thousands to tens of thousands of variables and constraints. Given the size of many NLPs arising from a direct method, it may appear somewhat counter-intuitive that solving the nonlinear optimization problem is easier than solving the boundary-value problem. It is, however, the fact that the NLP is easier to solve than the boundary-value problem. The reason for the relative ease of computation, particularly of a direct collocation method, is that the NLP is "sparse" and many well-known software programs exist (e.g., SNOPT) to solve large sparse NLPs. As a result, the range of problems that can be solved via direct methods (particularly direct "collocation methods" which are very popular these days) is significantly larger than the range of problems that can be solved via indirect methods. In fact, direct methods have become so popular these days that many people have written elaborate software programs that employ these methods. In particular, many such programs include "DIRCOL", SOCS, OTIS, GESOP/ASTOS, DITAN. and PyGMO/PyKEP. In recent years, due to the advent of the MATLAB programming language, optimal control software in MATLAB has become more common. Examples of academically developed MATLAB software tools implementing direct methods include "RIOTS", "DIDO", "DIRECT", FALCON.m, and "GPOPS," while an example of an industry developed MATLAB tool is "PROPT". These software tools have increased significantly the opportunity for people to explore complex optimal control problems both for academic research and industrial problems. Finally, it is noted that general-purpose MATLAB optimization environments such as TOMLAB have made coding complex optimal control problems significantly easier than was previously possible in languages such as C and FORTRAN.
Discrete-time optimal control.
The examples thus far have shown continuous time systems and control solutions. In fact, as optimal control solutions are now often implemented digitally, contemporary control theory is now primarily concerned with discrete time systems and solutions. The Theory of Consistent Approximations provides conditions under which solutions to a series of increasingly accurate discretized optimal control problem converge to the solution of the original, continuous-time problem. Not all discretization methods have this property, even seemingly obvious ones. For instance, using a variable step-size routine to integrate the problem's dynamic equations may generate a gradient which does not converge to zero (or point in the right direction) as the solution is approached. The direct method "RIOTS" is based on the Theory of Consistent Approximation.
Examples.
A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the shadow price) formula_49. The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn. The marginal value is not only the gains accruing to it next turn but associated with the duration of the program. It is nice when formula_49 can be solved analytically, but usually, the most one can do is describe it sufficiently well that the intuition can grasp the character of the solution and an equation solver can solve numerically for the values.
Having obtained formula_49, the turn-t optimal value for the control can usually be solved as a differential equation conditional on knowledge of formula_49. Again it is infrequent, especially in continuous-time problems, that one obtains the value of the control or the state explicitly. Usually, the strategy is to solve for thresholds and regions that characterize the optimal control and use a numerical solver to isolate the actual choice values in time.
Finite time.
Consider the problem of a mine owner who must decide at what rate to extract ore from their mine. They own rights to the ore from date formula_53 to date formula_54. At date formula_53 there is formula_56 ore in the ground, and the time-dependent amount of ore formula_57 left in the ground declines at the rate of formula_58 that the mine owner extracts it. The mine owner extracts ore at cost formula_59 (the cost of extraction increasing with the square of the extraction speed and the inverse of the amount of ore left) and sells ore at a constant price formula_60. Any ore left in the ground at time formula_54 cannot be sold and has no value (there is no "scrap value"). The owner chooses the rate of extraction varying with time formula_58 to maximize profits over the period of ownership with no time discounting.
\end{align}
and using the initial and turn-T conditions, the formula_63 series can be solved explicitly, giving formula_64.
The manager maximizes profit formula_65:
formula_66
where the state variable formula_57 evolves as follows:
formula_68
Form the Hamiltonian and differentiate:
formula_69
As the mine owner does not value the ore remaining at time formula_54,
formula_71
Using the above equations, it is easy to solve for the differential equations governing formula_58 and formula_49
formula_74
and using the initial and turn-T conditions, the functions can be solved to yield
formula_75


Lyapunov stability

Various types of stability may be discussed for the solutions of differential equations or difference equations describing dynamical systems. The most important type is that concerning the stability of solutions near to a point of equilibrium. This may be discussed by the theory of Aleksandr Lyapunov. In simple terms, if the solutions that start out near an equilibrium point formula_1 stay near formula_1 forever, then formula_1 is Lyapunov stable. More strongly, if formula_1 is Lyapunov stable and all solutions that start out near formula_1 converge to formula_1, then formula_1 is said to be asymptotically stable (see asymptotic analysis). The notion of "exponential stability" guarantees a minimal rate of decay, i.e., an estimate of how quickly the solutions converge. The idea of Lyapunov stability can be extended to infinite-dimensional manifolds, where it is known as structural stability, which concerns the behavior of different but "nearby" solutions to differential equations. Input-to-state stability (ISS) applies Lyapunov notions to systems with inputs.
History.
Lyapunov stability is named after Aleksandr Mikhailovich Lyapunov, a Russian mathematician who defended the thesis "The General Problem of Stability of Motion" at Kharkov University in 1892. A. M. Lyapunov was a pioneer in successful endeavors to develop a global approach to the analysis of the stability of nonlinear dynamical systems by comparison with the widely spread local method of linearizing them about points of equilibrium. His work, initially published in Russian and then translated to French, received little attention for many years. The mathematical theory of stability of motion, founded by A. M. Lyapunov, considerably anticipated the time for its implementation in science and technology. Moreover Lyapunov did not himself make application in this field, his own interest being in the stability of rotating fluid masses with astronomical application. He did not have doctoral students who followed the research in the field of stability and his own destiny was terribly tragic because of his suicide in 1918 . For several decades the theory of stability sank into complete oblivion. The Russian-Soviet mathematician and mechanician Nikolay Gur'yevich Chetaev working at the Kazan Aviation Institute in the 1930s was the first who realized the incredible magnitude of the discovery made by A. M. Lyapunov. The contribution to the theory made by N. G. Chetaev was so significant that many mathematicians, physicists and engineers consider him Lyapunov's direct successor and the next-in-line scientific descendant in the creation and development of the mathematical theory of stability.
The interest in it suddenly skyrocketed during the Cold War period when the so-called "Second Method of Lyapunov" (see below) was found to be applicable to the stability of aerospace guidance systems which typically contain strong nonlinearities not treatable by other methods. A large number of publications appeared then and since in the control and systems literature.
More recently the concept of the Lyapunov exponent (related to Lyapunov's First Method of discussing stability) has received wide interest in connection with chaos theory. Lyapunov stability methods have also been applied to finding equilibrium solutions in traffic assignment problems.
Definition for continuous-time systems.
Consider an autonomous nonlinear dynamical system
where formula_9 denotes the system state vector, formula_10 an open set containing the origin, and formula_11 is a continuous vector field on formula_10. Suppose formula_13 has an equilibrium at formula_1 so that formula_15 then
Conceptually, the meanings of the above terms are the following:
The trajectory "formula_32" is (locally) "attractive" if
for all trajectories formula_35 that start close enough to formula_36, and "globally attractive" if this property holds for all trajectories.
That is, if "x" belongs to the interior of its stable manifold, it is "asymptotically stable" if it is both attractive and stable. (There are examples showing that attractivity does not imply asymptotic stability. Such examples are easy to create using homoclinic connections.)
If the Jacobian of the dynamical system at an equilibrium happens to be a stability matrix (i.e., if the real part of each eigenvalue is strictly negative), then the equilibrium is asymptotically stable.
System of deviations.
Instead of considering stability only near an equilibrium point (a constant solution formula_37), one can formulate similar definitions of stability near an arbitrary solution formula_32. However, one can reduce the more general case to that of an equilibrium by a change of variables called a "system of deviations". Define formula_39, obeying the differential equation:
This is no longer an autonomous system, but it has a guaranteed equilibrium point at formula_41 whose stability is equivalent to the stability of the original solution formula_32.
Lyapunov's second method for stability.
Lyapunov, in his original 1892 work, proposed two methods for demonstrating stability. The first method developed the solution in a series which was then proved convergent within limits. The second method, which is now referred to as the Lyapunov stability criterion or the Direct Method, makes use of a "Lyapunov function V(x)" which has an analogy to the potential function of classical dynamics. It is introduced as follows for a system formula_43 having a point of equilibrium at formula_44. Consider a function formula_45 such that
Then "V(x)" is called a Lyapunov function and the system is stable in the sense of Lyapunov. (Note that formula_54 is required; otherwise for example formula_55 would "prove" that formula_56 is locally stable.) An additional condition called "properness" or "radial unboundedness" is required in order to conclude global stability. Global asymptotic stability (GAS) follows similarly.
It is easier to visualize this method of analysis by thinking of a physical system (e.g. vibrating spring and mass) and considering the energy of such a system. If the system loses energy over time and the energy is never restored then eventually the system must grind to a stop and reach some final resting state. This final state is called the attractor. However, finding a function that gives the precise energy of a physical system can be difficult, and for abstract mathematical systems, economic systems or biological systems, the concept of energy may not be applicable.
Lyapunov's realization was that stability can be proven without requiring knowledge of the true physical energy, provided a Lyapunov function can be found to satisfy the above constraints.
Definition for discrete-time systems.
The definition for discrete-time systems is almost identical to that for continuous-time systems. The definition below provides this, using an alternate language commonly used in more mathematical texts.
Let ("X", "d") be a metric space and "f" : "X" → "X" a continuous function. A point "x" in "X" is said to be Lyapunov stable, if,
We say that "x" is asymptotically stable if it belongs to the interior of its stable set, "i.e." if,
Stability for linear state space models.
A linear state space model
where formula_60 is a finite matrix, is asymptotically stable (in fact, exponentially stable) if all real parts of the eigenvalues of formula_60 are negative. This condition is equivalent to the following one:
is negative definite for some positive definite matrix formula_63. (The relevant Lyapunov function is formula_64.)
Correspondingly, a time-discrete linear state space model
is asymptotically stable (in fact, exponentially stable) if all the eigenvalues of formula_60 have a modulus smaller than one.
This latter condition has been generalized to switched systems: a linear switched discrete time system (ruled by a set of matrices formula_67)
is asymptotically stable (in fact, exponentially stable) if the joint spectral radius of the set formula_67 is smaller than one.
Stability for systems with inputs.
A system with inputs (or controls) has the form
where the (generally time-dependent) input u(t) may be viewed as a "control", "external input",
"stimulus", "disturbance", or "forcing function". It has been shown that near to a point of equilibrium which is Lyapunov stable the system remains stable under small disturbances. For larger input disturbances the study of such systems is the subject of control theory and applied in control engineering. For systems with inputs, one must quantify the effect of inputs on the stability of the system. The main two approaches to this analysis are BIBO stability (for linear systems) and input-to-state stability (ISS) (for nonlinear systems)
Example.
This example shows a system where a Lyapunov function can be used to prove Lyapunov stability but cannot show asymptotic stability.
Consider the following equation, based on the Van der Pol oscillator equation with the friction term changed:
Let
so that the corresponding system is
The origin formula_74 is the only equilibrium point.
Let us choose as a Lyapunov function
which is clearly positive definite. Its derivative is
It seems that if the parameter formula_77 is positive, stability is asymptotic for formula_78 But this is wrong, since formula_79 does not depend on formula_80, and will be 0 everywhere on the formula_80 axis. The equilibrium is Lyapunov stable but not asymptotically stable.
Barbalat's lemma and stability of time-varying systems.
Assume that f is a function of time only.
Barbalat's Lemma says:
An alternative version is as follows:
In the following form the Lemma is true also in the vector valued case:
The following example is taken from page 125 of Slotine and Li's book "Applied Nonlinear Control".
Consider a non-autonomous system
This is non-autonomous because the input formula_114 is a function of time. Assume that the input formula_115 is bounded.
Taking formula_116 gives formula_117
This says that formula_118 by first two conditions and hence formula_119 and formula_120 are bounded. But it does not say anything about the convergence of formula_119 to zero. Moreover, the invariant set theorem cannot be applied, because the dynamics is non-autonomous.
Using Barbalat's lemma:
This is bounded because formula_119, formula_120 and formula_114 are bounded. This implies formula_126 as formula_84 and hence formula_128. This proves that the error converges.


Bilateral synchronization

In telecommunication, bilateral synchronization (or bilateral control) is a synchronization control system between exchanges A and B in which the clock at telephone exchange A controls the data received at exchange B and the clock at exchange B controls the data received at exchange A.
Bilateral synchronization is usually implemented by deriving the timing from the incoming bitstream.
Source: from Federal Standard 1037C in support of MIL-STD-188


Called-party camp-on

In telecommunication, a called-party camp-on is a communication system service feature that enables the system to complete an access attempt in spite of issuance of a user blocking signal. This is most often found in a switchboard system at a company. Instead of going to voicemail or simply sitting on hold until the line is free, this feature places you in a queue whereby the call will be put through as soon as the line clears.
Systems that provide this feature monitor the busy user until the user blocking signal ends, and then proceed to complete the requested access. This feature permits holding an incoming telephone call until the called party is free.


Closed-loop transfer function

In control theory, a closed-loop transfer function is a mathematical function describing the net result of the effects of a feedback control loop on the input signal to the plant under control.
Overview.
The closed-loop transfer function is measured at the output. The output signal can be calculated from the closed-loop transfer function and the input signal. Signals may be waveforms, images, or other types of data streams.
An example of a closed-loop transfer function is shown below:
The summing node and the "G"("s") and "H"("s") blocks can all be combined into one block, which would have the following transfer function:
formula_2 is called feedforward transfer function, formula_3 is called feedback transfer function, and their product formula_4 is called the open-loop transfer function.
Derivation.
We define an intermediate signal Z (also known as error signal) shown as follows:
Using this figure we write:
Now, plug the second equation into the first to eliminate Z(s):
Move all the terms with Y(s) to the left hand side, and keep the term with X(s) on the right hand side:
Therefore,


Distortion-limited operation

In telecommunication, distortion-limited operation is the condition prevailing when distortion of a received signal, rather than its attenuated amplitude (or power), limits performance under stated operational conditions and limits.
"Note:" Distortion-limited operation is reached when the system distorts the shape of the waveform beyond specified limits. For linear systems, distortion-limited operation is equivalent to bandwidth-limited operation.


Electromagnetic interference control

In Electrical systems, such as telecommunication, power electronics, industrial electronics, power engineering; electromagnetic interference "(EMI)" control is the control of radiated and conducted energy such that emissions that are unnecessary for system, subsystem, or equipment operation are reduced, minimized, or eliminated.
"Note:" Electromagnetic radiated and conducted emissions are controlled regardless of their origin within the system, subsystem, or equipment. Successful EMI control with effective susceptibility control leads to electromagnetic compatibility.


Supervised learning

Supervised learning (SL) is a paradigm in machine learning where input objects (for example, a vector of predictor variables) and a desired output value (also known as human-labeled "supervisory signal") train a model. The training data is processed, building a function that maps new data on expected output values. An optimal scenario will allow for the algorithm to correctly determine output values for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way (see inductive bias). This statistical quality of an algorithm is measured through the so-called generalization error.
Steps to follow.
To solve a given problem of supervised learning, one has to perform the following steps:
Algorithm choice.
A wide range of supervised learning algorithms are available, each with its strengths and weaknesses. There is no single learning algorithm that works best on all supervised learning problems (see the No free lunch theorem).
There are four major issues to consider in supervised learning:
Bias-variance tradeoff.
A first issue is the tradeoff between "bias" and "variance". Imagine that we have available several different, but equally good, training data sets. A learning algorithm is biased for a particular input formula_1 if, when trained on each of these data sets, it is systematically incorrect when predicting the correct output for formula_1. A learning algorithm has high variance for a particular input formula_1 if it predicts different output values when trained on different training sets. The prediction error of a learned classifier is related to the sum of the bias and the variance of the learning algorithm. Generally, there is a tradeoff between bias and variance. A learning algorithm with low bias must be "flexible" so that it can fit the data well. But if the learning algorithm is too flexible, it will fit each training data set differently, and hence have high variance. A key aspect of many supervised learning methods is that they are able to adjust this tradeoff between bias and variance (either automatically or by providing a bias/variance parameter that the user can adjust).
Function complexity and amount of training data.
The second issue is of the amount of training data available relative to the complexity of the "true" function (classifier or regression function). If the true function is simple, then an "inflexible" learning algorithm with high bias and low variance will be able to learn it from a small amount of data. But if the true function is highly complex (e.g., because it involves complex interactions among many different input features and behaves differently in different parts of the input space), then the function will only be able to learn with a large amount of training data paired with a "flexible" learning algorithm with low bias and high variance.
Dimensionality of the input space.
A third issue is the dimensionality of the input space. If the input feature vectors have large dimensions, learning the function can be difficult even if the true function only depends on a small number of those features. This is because the many "extra" dimensions can confuse the learning algorithm and cause it to have high variance. Hence, input data of large dimensions typically requires tuning the classifier to have low variance and high bias. In practice, if the engineer can manually remove irrelevant features from the input data, it will likely improve the accuracy of the learned function. In addition, there are many algorithms for feature selection that seek to identify the relevant features and discard the irrelevant ones. This is an instance of the more general strategy of dimensionality reduction, which seeks to map the input data into a lower-dimensional space prior to running the supervised learning algorithm.
Noise in the output values.
A fourth issue is the degree of noise in the desired output values (the supervisory target variables). If the desired output values are often incorrect (because of human error or sensor errors), then the learning algorithm should not attempt to find a function that exactly matches the training examples. Attempting to fit the data too carefully leads to overfitting. You can overfit even when there are no measurement errors (stochastic noise) if the function you are trying to learn is too complex for your learning model. In such a situation, the part of the target function that cannot be modeled "corrupts" your training data - this phenomenon has been called deterministic noise. When either type of noise is present, it is better to go with a higher bias, lower variance estimator.
In practice, there are several approaches to alleviate noise in the output values such as early stopping to prevent overfitting as well as detecting and removing the noisy training examples prior to training the supervised learning algorithm. There are several algorithms that identify noisy training examples and removing the suspected noisy training examples prior to training has decreased generalization error with statistical significance.
Other factors to consider.
Other factors to consider when choosing and applying a learning algorithm include the following:
When considering a new application, the engineer can compare multiple learning algorithms and experimentally determine which one works best on the problem at hand (see cross validation). Tuning the performance of a learning algorithm can be very time-consuming. Given fixed resources, it is often better to spend more time collecting additional training data and more informative features than it is to spend extra time tuning the learning algorithms.
Algorithms.
The most widely used learning algorithms are:
How supervised learning algorithms work.
Given a set of formula_4 training examples of the form formula_5 such that formula_6 is the feature vector of the formula_7-th example and formula_8 is its label (i.e., class), a learning algorithm seeks a function formula_9, where formula_10 is the input space and formula_11 is the output space. The function formula_12 is an element of some space of possible functions formula_13, usually called the "hypothesis space". It is sometimes convenient to represent formula_12 using a scoring function formula_15 such that formula_12 is defined as returning the formula_17 value that gives the highest score: formula_18. Let formula_19 denote the space of scoring functions.
Although formula_13 and formula_19 can be any space of functions, many learning algorithms are probabilistic models where formula_12 takes the form of a conditional probability model formula_23, or formula_24 takes the form of a joint probability model formula_25. For example, naive Bayes and linear discriminant analysis are joint probability models, whereas logistic regression is a conditional probability model.
There are two basic approaches to choosing formula_24 or formula_12: empirical risk minimization and structural risk minimization. Empirical risk minimization seeks the function that best fits the training data. Structural risk minimization includes a "penalty function" that controls the bias/variance tradeoff.
In both cases, it is assumed that the training set consists of a sample of independent and identically distributed pairs, formula_28. In order to measure how well a function fits the training data, a loss function formula_29 is defined. For training example formula_30, the loss of predicting the value formula_31 is formula_32.
The "risk" formula_33 of function formula_12 is defined as the expected loss of formula_12. This can be estimated from the training data as
Empirical risk minimization.
In empirical risk minimization, the supervised learning algorithm seeks the function formula_12 that minimizes formula_33. Hence, a supervised learning algorithm can be constructed by applying an optimization algorithm to find formula_12.
When formula_12 is a conditional probability distribution formula_41 and the loss function is the negative log likelihood: formula_42, then empirical risk minimization is equivalent to maximum likelihood estimation.
When formula_13 contains many candidate functions or the training set is not sufficiently large, empirical risk minimization leads to high variance and poor generalization. The learning algorithm is able to memorize the training examples without generalizing well. This is called overfitting.
Structural risk minimization.
Structural risk minimization seeks to prevent overfitting by incorporating a regularization penalty into the optimization. The regularization penalty can be viewed as implementing a form of Occam's razor that prefers simpler functions over more complex ones.
A wide variety of penalties have been employed that correspond to different definitions of complexity. For example, consider the case where the function formula_12 is a linear function of the form
A popular regularization penalty is formula_46, which is the squared Euclidean norm of the weights, also known as the formula_47 norm. Other norms include the formula_48 norm, formula_49, and the formula_50 "norm", which is the number of non-zero formula_51s. The penalty will be denoted by formula_52.
The supervised learning optimization problem is to find the function formula_12 that minimizes
The parameter formula_55 controls the bias-variance tradeoff. When formula_56, this gives empirical risk minimization with low bias and high variance. When formula_55 is large, the learning algorithm will have high bias and low variance. The value of formula_55 can be chosen empirically via cross validation.
The complexity penalty has a Bayesian interpretation as the negative log prior probability of formula_12, formula_60, in which case formula_61 is the posterior probability of formula_12.
Generative training.
The training methods described above are "discriminative training" methods, because they seek to find a function formula_12 that discriminates well between the different output values (see discriminative model). For the special case where formula_25 is a joint probability distribution and the loss function is the negative log likelihood formula_65 a risk minimization algorithm is said to perform "generative training", because formula_24 can be regarded as a generative model that explains how the data were generated. Generative training algorithms are often simpler and more computationally efficient than discriminative training algorithms. In some cases, the solution can be computed in closed form as in naive Bayes and linear discriminant analysis.
Generalizations.
There are several ways in which the standard supervised learning problem can be generalized:


Single-input single-output system

In control engineering, a single-input and single-output (SISO) system is a simple single-variable control system with one input and one output. In radio, it is the use of only one antenna both in the transmitter and receiver.
Details.
SISO systems are typically less complex than multiple-input multiple-output (MIMO) systems. Usually, it is also easier to make an order of magnitude or trending predictions "on the fly" or "back of the envelope". MIMO systems have too many interactions for most of us to trace through them quickly, thoroughly, and effectively in our heads.
Frequency domain techniques for analysis and controller design dominate SISO control system theory. Bode plot, Nyquist stability criterion, Nichols plot, and root locus are the usual tools for SISO system analysis. Controllers can be designed through the polynomial design, root locus design methods to name just two of the more popular. Often SISO controllers will be PI, PID, or lead-lag.


Sensors for arc welding

Sensors for arc welding are devices which – as a part of a fully mechanised welding equipment – are capable to acquire information about position and, if possible, about the geometry of the intended weld at the workpiece and to provide respective data in a suitable form for the control of the weld torch position and, if possible, for the arc welding process parameters.
Introduction.
The quality of a weld depends, besides the weld parameters which are important for the welding process (e.g. voltage, current, wire feed and weld speed) also mainly from the type of input of process energy and of the used filler material. The positioning of the torch exerts a direct influence on the material flow. The heat input for the melting of the component edges and the steady heat flow are, furthermore, directly connected with the torch guidance and exert substantial influence on the weld quality and on the resulting residual stresses. In fully mechanised and automated shielded gas welding, the inaccuracies of torch guidance, workpiece handling, groove preparation and thermal distortion are adding to the variations of the edge position and edge geometry. In fully mechanised welding, the information which is required for the weld quality is detected via sensors. Sensors are applied for checking the position of the component (detection of weld start and end of weld), for joint tracking and for the adaptation of the process parameters to changes of the joints/grooves. It is possible to use the sensors online (together/at the same time with the welding process) or offline (in a separate working step before welding). Sensors are mainly used in online joint tracking.
Principles.
All physical principles which are capable to provide information about the position of an object are suitable to serve as the starting basis for a sensor function. The ambient conditions prevailing during arc welding and also the requirements which are made by fully mechanised equipments have, however, many restrictions as a consequence. Figure 1 depicts the system overview. The monitoring strategy of the sensor (process or geometry) has been chosen as the superordinate criterion, the further subdivision is orientated on the measuring principle. A further distinctive feature of sensor systems is their design. Leading sensors are, thus, marked by the fact that measuring point and joining point are not located in the same position. Here, the measuring and joining process are mainly running in sequence. For making position-relevant statements about the welding process, those systems require calibration of the relative position. If process-oriented sensors are used, the measuring point and the joining point are identical.
What the measuring principles all have in common is the fact that through the evaluation of the sensor signal, geometrical information about the joint and its relative position to the measuring head is provided. The individual active principles allow different processing speed for acquiring the information.
Geometry-oriented.
Geometry-oriented sensors acquire their signals from the geometry of the groove or from an edge or area the course of which is in accordance with the groove.
Tactile sensors.
Electric contact sensors for joint tracking and/or work piece measurement are representing one type of tactile sensors. The sensor makes electric contact with the workpiece, the electrically conductive workpiece is included into the measuring circuit of the sensor.
The mechanical contact sensors belong to the second category of the tactile sensors. The mechanical deflection of a scanning element which makes contact with the workpiece is evaluated.
Electric contact sensors.
Following a determined searching strategy, the electric contact sensor systems are scanning the weld start or other track points via contacting the work piece with parts/components which have been subjected to voltage (direct voltage of several ten Volt up to 1 KV, depending on material and surface) of the welding equipment (shielding gas nozzle, welding electrode, stylus, or similar.) This means the offline-measuring of the weld start, the part position or part geometry before welding. Knowing the scheduled path, a transformation of the track points in accordance with the measured conditions is carried out. In this case, corrective action is not carried out during the welding process.
Thermal.
Here, the heat flow is measured with two thermo-couples which are arranged on the welding torch, the thermal flow is used for the side/lateral- and height control of the torch. The orientation of the torch towards the groove is detected via the comparison of the sensor temperature of the two thermo-couples. If the orientation of the torch is symmetrical, the difference of the radiated thermal flow equals to zero, so do the temperature differences of the thermo-couples. Dependent on the lateral misalignment of the torch the thermo-couples are subject to different heat flows, by the deformation of the arc and also by the changed position of the molten pool.
Mechanical contact.
Mechanical contact systems transform the deflection of the scanning element directly into electric control signals. The following transformer principles are differentiated:
Due to the required distance of the acting/break points in one level, transformers which are equipped with micro-switches have a control hysteresis in the working point which has the consequence of a restricted reproducible accuracy. Electric displacement of the working point is not possible. The other, above-mentioned transformer systems (the use of optical systems is probably limited due to design reasons) produce analogous signals in proportion to the scanning element deflection and allow thus the error-proportional weld head tracking and also the electric working point displacement through the superordinate control, e.g. in multiple layer welding. The output signals of the most commonly used inductive measuring transformer systems are between 0 and 10 V DC, depending on the scanning element deflection (Figure 2).
Boundary conditions.
Any impairment of the electric contact between sensor scanning element and workpiece is, in the case of electric-contact sensors, problematic, e.g. welding spatters at the shielding gas nozzle, scale and rolling skin on the workpiece surface or through a wire electrode end which has molten spheroidally and has adherence of slag.
When mechanical-contact sensors are used, the scanning elements must be adapted to the respective groove shapes. Butt welds with a square butt joint preparation must have a groove gap of more than 3 mm; in overlap joints the top plate must have a thickness of more than 3 mm.
The sensor must be mounted separately from the welding torch.
Thus, the groove scanning is mainly carried out in a leading position in front of the torch. If the welds are mainly straight, this adjustment is no problem. It is also possible to use scanning element arrays (e.g. fork callipers or separated scanning elements for height and lateral scanning which allow scanning in the torch level and thus weld scanning which is almost free from errors. Apart from the torch guidance along a weld groove, mechanical contact sensors can also be applied for detection of the weld start and end of the weld.
Optical.
Optical sensors belong to the group of non-contact measuring, geometry-oriented sensors (Figure 1). For information retrieval, the weld groove is scanned via a radiation detector which records the emitted optical radiation of the measured object. Semiconductor image sensors are applied for the detection of radiation. The optical measuring principles are differentiated into sensors with and without active structured lighting. If there is no active structured lighting, a camera is used for signal acquisition. The camera observes the workpiece and extracts the required information from the two-dimensional halftone picture. Active structured lighting means the application of a light source for the defined lighting of specified regions of the part. For the subsequent acquisition, single photo elements, lines or arrays can be used, depending on their design.
Operating mode.
For optical measurement without active structured lighting, a camera is directed on the region of the weld groove and the scene of interest is observed directly. This is used, for example, for SA welding processes in order to provide the welder with a live photograph of the weld groove on the monitor.
We know two semiconductor technologies for image sensing. The CCD camera (CCD: Charged Coupled Device) is the best known, most widely spread camera type, it is also used in standard video cameras.
If a CMOS image sensor has been used, the high input dynamics allow, even with a burning arc, to record a usable image of the weld groove.
The method of optical measuring technique with active structured lighting, mainly generated by a laser with a defined wavelength, is often used for the automation of welding processes. It is differentiated between 1, 2 and 3-D measuring systems. Since measuring in the arc directly is not possible, a defined distance (advance) which depends on the type and size of the arc itself must be maintained.
If one-dimensional measuring systems are used, the distance from sensor to workpiece surface is determined. This is carried out via measurement of the running time. A further, frequently used method is laser triangulation (Figure 4).
The distance of the workpiece is determined from the known dimensions of the sensor and the triangulation angle α.
This type of one-dimensional optical distance measurement systems is widely used in the field of industrial automation technique and is therefore offered by many companies. In automated welding, they are often used for the detection of the part and/or groove position before the start of the welding process.
There are different types of design of the two-dimensionally measuring sensor systems. From the 1D triangulation sensor, the two-dimensional laser scanner can be derived from the oscillation movement. Here, the groove geometry is detected via a scanning movement transverse to the groove (Figure 5). This is mainly carried out via a movable mirror unit which is integrated in the sensor head.
As an alternative, an oscillating movement of the entire sensor head can be carried out, this is, however, only considered a special application of a one-dimensionally measuring system. An advantage of the laser scanner is that, with according processing speed, the lighting conditions can be adapted for every single point-shaped distance measurement which results in illumination uniformity. Moreover, due to the point-shaped illumination, the laser point is through the concentrated laser power and also through appropriate optical filters, compared with the interfering arc radiation, easier to detect by the detection element.
The light-section sensor avoids the disadvantage of moving parts in the sensor head (Figure 6). Here, the surface is not scanned pointwise, the entire geometry is, moreover, captured in one image. For this purpose, the point-shaped laser beam is expanded via an optics to a line which is projected onto the surface of the workpiece transverse to the groove in accordance with the scan line of the scanner. The laser line is, in accordance with the same geometrical principle of triangulation, again acquired with a detector element, this time, however, two-dimensionally. For the acquisition, CCD and CMOS cameras with the above-mentioned properties can be used.
As the output signal after the pre-processing of sensor signals with a laser scanner and light-section sensor, the so-called height profile of the measured groove geometry is achieved. It represents the surface of the workpiece along the section at the projected laser line.
3 D measuring systems with active illumination are mainly using the light-section method in combination with the projection of several parallel laser lines. In doing so, each line generates a height profile. Through the arrangement of several lines along the weld groove, a further dimension is achieved which shows the change of the height profiles of the groove geometry. Through the number of the lines, the resolution in groove longitudinal direction is increasing, however, the data processing expenditure is also increasing. Similar to the projection of several parallel lines, the measurement via a projected circle or other geometrical figures on the workpiece surface is possible.
Boundary conditions.
What all optical measuring methods have in common is that the determined groove points must be transformed from the sensor coordinates of the cameras into machine and/or work piece coordinates. To this end, they must be calibrated on test work pieces before the welding process takes place and calibration matrices must be provided. Moreover, for the application of image processing algorithms, information about the groove profile must also be provided in advance. This is carried out via teaching of templates, input of geometrical parameters or teach-in via test work pieces. A more comprehensive image processing for 2 and 3 D sensor systems requires normally a PC system for the evaluation; this is why commercially available PC interfaces are used for data exchange, uniform sensor interfaces do, however, not yet exist.
Application problems.
In optical sensor systems, problems occur due to the operation principle through the scattered light of the open arc. Therefore, measuring in the working point directly is in most cases not possible when optical sensors are used, a certain advance/distance must be maintained. Further process trouble stems from weld spatters which may exert a negative influence on the detection results. Screening systems between sensor and torch provide a remedy to a certain extent. The direct observation of the arc with special cameras for process monitoring remain an exception.
The running of the sensor in front of the arc causes the limited accessibility of corners in the parts. In order to reduce this problem, a design/structure which is as compact as possible and a short advance distance are most important. The pre-defined orientation of the sensor is, moreover, restricting the working space of the robot. For untroubled operation of the optical components also stronger soiling/impurification (dust and deposition of weld fume particles) should be avoided, if possible. Exchangeable protective glasses and safety screens in the form of compressed air curtains provide a remedy. The quality of the surface which is to be measured has substantial influence on the measuring result. If the surface is strongly reflecting, unwanted reflection and faulty measurements may occur, lustreless surfaces are less difficult. Ever-changing surface qualities also lead to problems.
Since optical systems are equipped with semiconductor detectors and comprehensive electronics, it is most important to pay attention to safe electro-magnetic screening. This applies to the sensor, the image processing unit and the connecting cables thereof. Sensor systems with active laser illumination are reacting particularly sensitive to strong temperature fluctuations since the emitted light wavelength of the used laser diodes depends on the temperature of the laser. If the ambient temperature and thus the wavelength of the active illumination are changing, the light is no longer capable to penetrate through the narrow-band optical filter to the photodetector. Therefore, appropriate screening against the welding process or the cooling of the sensor head is required. Depending on the applied laser power, particular caution must be taken when sensors with active illumination are applied. The wavelength of the applied systems are often in the field of vision, which means the classification into the hazard classes 3A and 3B. The respective accident prevention regulations must be strictly adhered to.
The application of optical sensors demands the consideration of following points:
Inductive.
Inductive sensors evaluate the attenuation of a high-frequency electro-magnetic field which has been generated by eddy currents in the work piece. The application of single-coil design types allows side or height correction. Multiple-coil sensors allow correction in two coordinate directions and, moreover, influence on the weld torch orientation.
Capacitive.
Capacitive sensors measure the capacity between the work piece and an electrically conductive plate with small dimensions. They offer the possibility of distance measurement in media with unchanged dielectricity constant.
Process-oriented.
Process-oriented sensors are acquiring their signals from the primary or secondary process parameters.
Arc sensors are using the primary process parameters (weld current and/or voltage) of one moving or two unoscillated arcs for the generation of height and side/lateral correction signals.
These sensors require, of course, also a scannable groove geometry; the measuring and the joining point are, however, compared with geometry-oriented sensors, located in the same position.
Arc.
Stable working points in arc welding are developing as interface between process characteristic and power source characteristic (Figure 7). The process characteristic specifies the connection between a stable arc voltage and the appropriate current rating of the process under constant boundary conditions. A family of characteristics is achieved via the variation of the arc length / torch distance.
In TIG welding.
TIG welding belongs to the welding processes with non-melting electrode. Therefore, the process characteristic is often designated as arc characteristic. A direct change of the working distance is compensated via the length of the arc. As a result, the arc resistance is changing. Short arcs have a lower electric resistance than long arcs.
In TIG welding, typically power sources with a steeply drooping characteristic are applied. A change of the arc length leads, therefore, directly to the change of the process voltage. A comparative measurement allows the determination of the distance to the workpiece.
In GMA welding.
In GMA welding, the process characteristic in the voltage-current diagram is a result from the interaction of the electric properties of the wire stick-out and of the arc. In principle, stable working points are achieved through the application of suitable power source characteristics or through super-imposed control strategies.
There is a stable equilibrium in point 1 of Figure 8 where the energy which has been input into the process is sufficient for the melting of the continuously fed wire electrode. In the case of a rapid change of distance, the arc compensates the length change, point 2. The lower resistance of the short arc brings about the increase of the current intensity which leads to the faster melting of the wire stick-out until again a stable working point is reached, point 3. This compensation process takes approximately between 100 and 200 ms. The arc sensor evaluates the remaining change in current intensity between point 1 and point 3 in order to achieve a distance-proportionate parameter. In principle, this evaluation concept is also applicable to pulsed arc welding. The concept which has been specified above is, in the case of most arc sensors, extended by the transversal scanning of the groove geometry. The deflection of the process to the fusion faces allows the comparative measurement of the torch distance. By calculating the difference of the distance values, the lateral positioning of the torch can be evaluated. The mean value of both distance values indicates the height of the torch above the groove. Different concepts are applied for the deflection (Figure 9). Mechanical oscillation is most widely spread and is frequently used, especially with robots. Basically, fast deflectory systems, e.g. with magnetic or rotatory deflection offer the improvement of the signal rate and the signal quality, a higher apparatus expenditure must, however, be calculated when these systems are used. In double-wire technique, both fusion faces are scanned at the same time with one wire each.
Boundary conditions.
Arc sensors are evaluating the stable working points in arc welding. Disturbance variables of the process must be compensated via suitable filtering and evaluation strategies which are not susceptible to disturbances.
In the case of a simultaneous height and side control attention must be paid to the fact that only those groove geometries are suitable for arc sensor systems whose geometry allows the lateral position determination via comparative measurement of the fusion faces. V-type welds and fillet welds are suitable without any restrictions. Square butt welds without gap are not suitable for side/lateral control. Commercially available arc sensors are, so far, not applicable for aluminium materials.
Secondary process parameters.
Sensor types which are observing the molten pool are restricted in their applicability range by the fact that molten pool size and arc radiation are dependent on geometrical factors, e.g. material density or composition (alloying constituents). The optical observation of the molten pool region determines changes of the molten pool contour. The deflection from a contour which is defined as “ideal” is interpreted as malposition or as a change of the process behaviour and is compensated subsequently.
Spectral analysis.
The spectral analysis of the process signals compares the emission spectres of the arc or of the molten pool with assumed ideal values. Deflections point to a changed chemical composition or to energetic changes of the process zone.


Gradient descent

In mathematics, gradient descent (also often called steepest descent) is a iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a local maximum of that function; the procedure is then known as "gradient ascent".
It is particularly useful in machine learning for minimizing the cost or loss function. Gradient descent should not be confused with local search algorithms, although both are iterative methods for optimization.
Gradient descent is generally attributed to Augustin-Louis Cauchy, who first suggested it in 1847. Jacques Hadamard independently proposed a similar method in 1907. Its convergence properties for non-linear optimization problems were first studied by Haskell Curry in 1944, with the method becoming increasingly well-studied and used in the following decades.
A simple extension of gradient descent, stochastic gradient descent, serves as the most basic algorithm used for training most deep networks today.
Description.
Gradient descent is based on the observation that if the multi-variable function formula_1 is defined and differentiable in a neighborhood of a point formula_2, then formula_1 decreases "fastest" if one goes from formula_2 in the direction of the negative gradient of formula_5 at formula_6. It follows that, if
for a small enough step size or learning rate formula_8, then formula_9. In other words, the term formula_10 is subtracted from formula_2 because we want to move against the gradient, toward the local minimum. With this observation in mind, one starts with a guess formula_12 for a local minimum of formula_5, and considers the sequence formula_14 such that
We have a monotonic sequence
so, hopefully, the sequence formula_17 converges to the desired local minimum. Note that the value of the "step size" formula_18 is allowed to change at every iteration. With certain assumptions on the function formula_5 (for example, formula_5 convex and formula_21 Lipschitz) and particular choices of formula_18 (e.g., chosen either via a line search that satisfies the Wolfe conditions, or the Barzilai-Borwein method shown as following),
convergence to a local minimum can be guaranteed. When the function formula_5 is convex, all local minima are also global minima, so in this case gradient descent can converge to the global solution.
This process is illustrated in the adjacent picture. Here, formula_5 is assumed to be defined on the plane, and that its graph has a bowl shape. The blue curves are the contour lines, that is, the regions on which the value of formula_5 is constant. A red arrow originating at a point shows the direction of the negative gradient at that point. Note that the (negative) gradient at a point is orthogonal to the contour line going through that point. We see that gradient "descent" leads us to the bottom of the bowl, that is, to the point where the value of the function formula_5 is minimal.
An analogy for understanding gradient descent.
The basic intuition behind gradient descent can be illustrated by a hypothetical scenario. A person is stuck in the mountains and is trying to get down (i.e., trying to find the global minimum). There is heavy fog such that visibility is extremely low. Therefore, the path down the mountain is not visible, so they must use local information to find the minimum. They can use the method of gradient descent, which involves looking at the steepness of the hill at their current position, then proceeding in the direction with the steepest descent (i.e., downhill). If they were trying to find the top of the mountain (i.e., the maximum), then they would proceed in the direction of steepest ascent (i.e., uphill). Using this method, they would eventually find their way down the mountain or possibly get stuck in some hole (i.e., local minimum or saddle point), like a mountain lake. However, assume also that the steepness of the hill is not immediately obvious with simple observation, but rather it requires a sophisticated instrument to measure, which the person happens to have at the moment. It takes quite some time to measure the steepness of the hill with the instrument, thus they should minimize their use of the instrument if they wanted to get down the mountain before sunset. The difficulty then is choosing the frequency at which they should measure the steepness of the hill so not to go off track.
In this analogy, the person represents the algorithm, and the path taken down the mountain represents the sequence of parameter settings that the algorithm will explore. The steepness of the hill represents the slope of the function at that point. The instrument used to measure steepness is differentiation. The direction they choose to travel in aligns with the gradient of the function at that point. The amount of time they travel before taking another measurement is the step size.
Choosing the step size and descent direction.
Since using a step size formula_18 that is too small would slow convergence, and a formula_18 too large would lead to overshoot and divergence, finding a good setting of formula_18 is an important practical problem. Philip Wolfe also advocated using "clever choices of the [descent] direction" in practice. Whilst using a direction that deviates from the steepest descent direction may seem counter-intuitive, the idea is that the smaller slope may be compensated for by being sustained over a much longer distance.
To reason about this mathematically, consider a direction formula_31 and step size formula_32 and consider the more general update:
Finding good settings of formula_31 and formula_32 requires some thought. First of all, we would like the update direction to point downhill. Mathematically, letting formula_36 denote the angle between formula_37 and formula_31, this requires that formula_39 To say more, we need more information about the objective function that we are optimising. Under the fairly weak assumption that formula_5 is continuously differentiable, we may prove that:
This inequality implies that the amount by which we can be sure the function formula_5 is decreased depends on a trade off between the two terms in square brackets. The first term in square brackets measures the angle between the descent direction and the negative gradient. The second term measures how quickly the gradient changes along the descent direction.
In principle inequality  could be optimized over formula_31 and formula_32 to choose an optimal step size and direction. The problem is that evaluating the second term in square brackets requires evaluating formula_44, and extra gradient evaluations are generally expensive and undesirable. Some ways around this problem are:
Usually by following one of the recipes above, convergence to a local minimum can be guaranteed. When the function formula_5 is convex, all local minima are also global minima, so in this case gradient descent can converge to the global solution.
Solution of a linear system.
Gradient descent can be used to solve a system of linear equations
reformulated as a quadratic minimization problem.
If the system matrix formula_66 is real symmetric and positive-definite, an objective function is defined as the quadratic function, with minimization of
so that
For a general real matrix formula_66, linear least squares define
In traditional linear least squares for real formula_66 and formula_72 the Euclidean norm is used, in which case
The line search minimization, finding the locally optimal step size formula_18 on every iteration, can be performed analytically for quadratic functions, and explicit formulas for the locally optimal formula_18 are known.
For example, for real symmetric and positive-definite matrix formula_66, a simple algorithm can be as follows,
To avoid multiplying by formula_66 twice per iteration,
we note that formula_79 implies formula_80, which gives the traditional algorithm,
The method is rarely used for solving linear equations, with the conjugate gradient method being one of the most popular alternatives. The number of gradient descent iterations is commonly proportional to the spectral condition number formula_82 of the system matrix formula_66 (the ratio of the maximum to minimum eigenvalues of , while the convergence of conjugate gradient method is typically determined by a square root of the condition number, i.e., is much faster. Both methods can benefit from preconditioning, where gradient descent may require less assumptions on the preconditioner.
Solution of a non-linear system.
Gradient descent can also be used to solve a system of nonlinear equations. Below is an example that shows how to use the gradient descent to solve for three unknown variables, "x"1, "x"2, and "x"3. This example shows one iteration of the gradient descent.
Consider the nonlinear system of equations
Let us introduce the associated function
where
One might now define the objective function
which we will attempt to minimize. As an initial guess, let us use
We know that
where the Jacobian matrix formula_90 is given by
We calculate:
Thus
and
Now, a suitable formula_95 must be found such that
This can be done with any of a variety of line search algorithms. One might also simply guess formula_97 which gives
Evaluating the objective function at this value, yields
The decrease from formula_100 to the next step's value of
is a sizable decrease in the objective function. Further steps would reduce its value further until an approximate solution to the system was found.
Comments.
Gradient descent works in spaces of any number of dimensions, even in infinite-dimensional ones. In the latter case, the search space is typically a function space, and one calculates the Fréchet derivative of the functional to be minimized to determine the descent direction.
That gradient descent works in any number of dimensions (finite number at least) can be seen as a consequence of the Cauchy-Schwarz inequality. That article proves that the magnitude of the inner (dot) product of two vectors of any dimension is maximized when they are colinear. In the case of gradient descent, that would be when the vector of independent variable adjustments is proportional to the gradient vector of partial derivatives.
The gradient descent can take many iterations to compute a local minimum with a required accuracy, if the curvature in different directions is very different for the given function. For such functions, preconditioning, which changes the geometry of the space to shape the function level sets like concentric circles, cures the slow convergence. Constructing and applying preconditioning can be computationally expensive, however.
The gradient descent can be combined with a line search, finding the locally optimal step size formula_18 on every iteration. Performing the line search can be time-consuming. Conversely, using a fixed small formula_18 can yield poor convergence.
Methods based on Newton's method and inversion of the Hessian using conjugate gradient techniques can be better alternatives. Generally, such methods converge in fewer iterations, but the cost of each iteration is higher. An example is the BFGS method which consists in calculating on every step a matrix by which the gradient vector is multiplied to go into a "better" direction, combined with a more sophisticated line search algorithm, to find the "best" value of formula_104 For extremely large problems, where the computer-memory issues dominate, a limited-memory method such as L-BFGS should be used instead of BFGS or the steepest descent.
While it is sometimes possible to substitute gradient descent for a local search algorithm, gradient descent is not in the same family: although it is an iterative method for local optimization, it relies on an objective function’s gradient rather than an explicit exploration of a solution space.
Gradient descent can be viewed as applying Euler's method for solving ordinary differential equations formula_105 to a gradient flow. In turn, this equation may be derived as an optimal controller for the control system formula_106 with formula_107 given in feedback form formula_108.
It can be shown that there is a correspondence between neuroevolution and gradient descent.
Modifications.
Gradient descent can converge to a local minimum and slow down in a neighborhood of a saddle point. Even for unconstrained quadratic minimization, gradient descent develops a zig-zag pattern of subsequent iterates as iterations progress, resulting in slow convergence. Multiple modifications of gradient descent have been proposed to address these deficiencies.
Fast gradient methods.
Yurii Nesterov has proposed a simple modification that enables faster convergence for convex problems and has been since further generalized. For unconstrained smooth problems, the method is called the fast gradient method (FGM) or the accelerated gradient method (AGM). Specifically, if the differentiable function formula_5 is convex and formula_21 is Lipschitz, and it is not assumed that formula_5 is strongly convex, then the error in the objective value generated at each step formula_112 by the gradient descent method will be bounded by formula_113. Using the Nesterov acceleration technique, the error decreases at formula_114. It is known that the rate formula_115 for the decrease of the cost function is optimal for first-order optimization methods. Nevertheless, there is the opportunity to improve the algorithm by reducing the constant factor. The optimized gradient method (OGM) reduces that constant by a factor of two and is an optimal first-order method for large-scale problems.
For constrained or non-smooth problems, Nesterov's FGM is called the fast proximal gradient method (FPGM), an acceleration of the proximal gradient method.
Momentum or "heavy ball" method.
Trying to break the zig-zag pattern of gradient descent, the "momentum or heavy ball method" uses a momentum term in analogy to a heavy ball sliding on the surface of values of the function being minimized, or to mass movement in Newtonian dynamics through a viscous medium in a conservative force field. Gradient descent with momentum remembers the solution update at each iteration, and determines the next update as a linear combination of the gradient and the previous update. For unconstrained quadratic minimization, a theoretical convergence rate bound of the heavy ball method is asymptotically the same as that for the optimal conjugate gradient method.
This technique is used in stochastic gradient descent and as an extension to the backpropagation algorithms used to train artificial neural networks. In the direction of updating, stochastic gradient descent adds a stochastic property. The weights can be used to calculate the derivatives.
Extensions.
Gradient descent can be extended to handle constraints by including a projection onto the set of constraints. This method is only feasible when the projection is efficiently computable on a computer. Under suitable assumptions, this method converges. This method is a specific case of the forward-backward algorithm for monotone inclusions (which includes convex programming and variational inequalities).
Gradient descent is a special case of mirror descent using the squared Euclidean distance as the given Bregman divergence.


Process gain

In a spread-spectrum system, the process gain (or "processing gain") is the ratio of the spread (or RF) bandwidth to the unspread (or baseband) bandwidth. It is usually expressed in decibels (dB).
For example, if a 1 kHz signal is spread to 100 kHz, the process gain expressed as a numerical ratio would be / = 100. Or in decibels, 10 log10(100) = 20 dB.
Note that process gain does not reduce the effects of wideband thermal noise. It can be shown that a direct-sequence spread-spectrum (DSSS) system has exactly the same bit error behavior as a non-spread-spectrum system with the same modulation format. Thus, on an additive white Gaussian noise (AWGN) channel without interference, a spread system requires the same transmitter power as an unspread system, all other things being equal.
Unlike a conventional communication system, however, a DSSS system does have a certain resistance against narrowband interference, as the interference is not subject to the process gain of the DSSS signal, and hence the signal-to-interference ratio is improved.
In frequency modulation (FM), the processing gain can be expressed as
where:


IBM Spectrum LSF

IBM Spectrum LSF (LSF, originally Platform Load Sharing Facility) is a workload management platform, job scheduler, for distributed high performance computing (HPC) by IBM.
Details.
It can be used to execute batch jobs on networked Unix and Windows systems on many different architectures. LSF was based on the "Utopia" research project at the University of Toronto.
In 2007, Platform released "Platform Lava", which is a simplified version of LSF based on an old version of LSF release, licensed under GNU General Public License v2. The project was discontinued in 2011, succeeded by OpenLava.
In January, 2012, Platform Computing was acquired by IBM. The product is now called IBM Spectrum LSF.
IBM Spectrum LSF Community Edition is a no-charge community edition of the IBM Spectrum LSF workload management platform.


Charge control

Charge control is a technology that lets an electric utility control, in real time, the charging of a "gridable" (plug-in) vehicle, such as a plug-in hybrid (PHEV) or a battery electric vehicle (BEV). Through charge control, the utility is able to postpone charging of the vehicle during time of peak demand. Additionally, this technology may enable the owner and the power company to track the vehicle's usage and performance, while on the road and while charging.
Comparison to V2G.
In both V2G and charge control, the electric utility can control the power flow between a plug-in vehicle and the power grid. However, in charge control power only flows from the grid to the vehicle, while in V2G power can flow in both directions.
Peak load leveling.
Disabling charging in charge control vehicles helps balance the loading on the power grid by "valley filling" (charging at night when demand is low) and "peak shaving" (not charging when demand is high). It can enable utilities new ways to provide regulation services (keeping voltage and frequency stable).


Algebraic Riccati equation

An algebraic Riccati equation is a type of nonlinear equation that arises in the context of infinite-horizon optimal control problems in continuous time or discrete time.
A typical algebraic Riccati equation is similar to one of the following:
the continuous time algebraic Riccati equation (CARE):
or the discrete time algebraic Riccati equation (DARE):
"P" is the unknown "n" by "n" symmetric matrix and "A", "B", "Q", "R" are known real coefficient matrices.
Though generally this equation can have many solutions, it is usually specified that we want to obtain the unique stabilizing solution, if such a solution exists.
Origin of the name.
The name Riccati is given to these equations because of their relation to the Riccati differential equation. Indeed, the CARE is verified by the time invariant solutions of the associated matrix valued Riccati differential equation. As for the DARE, it is verified by the time invariant solutions of the matrix valued Riccati difference equation (which is the analogue of the Riccati differential equation in the context of discrete time LQR).
Context of the discrete-time algebraic Riccati equation.
In infinite-horizon optimal control problems, one cares about the value of some variable of interest arbitrarily far into the future, and one must optimally choose a value of a controlled variable right now, knowing that one will also behave optimally at all times in the future. The optimal current values of the problem's control variables at any time can be found using the solution of the Riccati equation and the current observations on evolving state variables. With multiple state variables and multiple control variables, the Riccati equation will be a matrix equation.
The algebraic Riccati equation determines the solution of the infinite-horizon time-invariant Linear-Quadratic Regulator problem (LQR) as well as that of the infinite horizon time-invariant Linear-Quadratic-Gaussian control problem (LQG). These are two of the most fundamental problems in control theory.
A typical specification of the discrete-time linear quadratic control problem is to minimize
subject to the state equation
where "y" is an "n" × 1 vector of state variables, "u" is a "k" × 1 vector of control variables, "A" is the "n" × "n" state transition matrix, "B" is the "n" × "k" matrix of control multipliers, "Q" ("n" × "n") is a symmetric positive semi-definite state cost matrix, and "R" ("k" × "k") is a symmetric positive definite control cost matrix.
Induction backwards in time can be used to obtain the optimal control solution at each time,
with the symmetric positive definite cost-to-go matrix "P" evolving backwards in time from formula_6 according to
which is known as the discrete-time dynamic Riccati equation of this problem. The steady-state characterization of "P", relevant for the infinite-horizon problem in which "T" goes to infinity, can be found by iterating the dynamic equation repeatedly until it converges; then "P" is characterized by removing the time subscripts from the dynamic equation.
Solution.
Usually solvers try to find the unique stabilizing solution, if such a solution exists. A solution is stabilizing if using it for controlling the associated LQR system makes the closed loop system stable.
For the CARE, the control is
and the closed loop state transfer matrix is
which is stable if and only if all of its eigenvalues have strictly negative real part.
For the DARE, the control is
and the closed loop state transfer matrix is
which is stable if and only if all of its eigenvalues are strictly inside the unit circle of the complex plane.
A solution to the algebraic Riccati equation can be obtained by matrix factorizations or by iterating on the Riccati equation. One type of iteration can be obtained in the discrete time case by using the "dynamic" Riccati equation that arises in the finite-horizon problem: in the latter type of problem each iteration of the value of the matrix is relevant for optimal choice at each period that is a finite distance in time from a final time period, and if it is iterated infinitely far back in time it converges to the specific matrix that is relevant for optimal choice an infinite length of time prior to a final period—that is, for when there is an infinite horizon.
It is also possible to find the solution by finding the eigendecomposition of a larger system. For the CARE, we define the Hamiltonian matrix
Since formula_13 is Hamiltonian, if it does not have any eigenvalues on the imaginary axis, then exactly half of its eigenvalues have a negative real part. If we denote the formula_14 matrix whose columns form a basis of the corresponding subspace, in block-matrix notation, as
then
is a solution of the Riccati equation; furthermore, the eigenvalues of formula_17 are the eigenvalues of formula_18 with negative real part.
For the DARE, when formula_19 is invertible, we define the symplectic matrix
Since formula_21 is symplectic, if it does not have any eigenvalues on the unit circle, then exactly half of its eigenvalues are inside the unit circle. If we denote the formula_22 matrix whose columns form a basis of the corresponding subspace, in block-matrix notation, as
where formula_24 and formula_25 result from the decomposition
then
is a solution of the Riccati equation; furthermore, the eigenvalues of formula_28 are the eigenvalues of formula_29 which are inside the unit circle.


Backstepping

In control theory, backstepping is a technique developed 1990 by Petar V. Kokotovic and others for designing stabilizing controls for a special class of nonlinear dynamical systems. These systems are built from subsystems that radiate out from an irreducible subsystem that can be stabilized using some other method. Because of this recursive structure, the designer can start the design process at the known-stable system and "back out" new controllers that progressively stabilize each outer subsystem. The process terminates when the final external control is reached. Hence, this process is known as "backstepping."
Backstepping approach.
The backstepping approach provides a recursive method for stabilizing the origin of a system in strict-feedback form. That is, consider a system of the form
where
Also assume that the subsystem
is stabilized to the origin (i.e., formula_11) by some known control formula_12 such that formula_13. It is also assumed that a Lyapunov function formula_14 for this stable subsystem is known. That is, this subsystem is stabilized by some other method and backstepping extends its stability to the formula_15 shell around it.
In systems of this "strict-feedback form" around a stable subsystem,
The backstepping approach determines how to stabilize the subsystem using formula_21, and then proceeds with determining how to make the next state formula_22 drive formula_21 to the control required to stabilize . Hence, the process "steps backward" from out of the strict-feedback form system until the ultimate control is designed.
Recursive Control Design Overview.
This process is known as backstepping because it starts with the requirements on some internal subsystem for stability and progressively "steps back" out of the system, maintaining stability at each step. Because
then the resulting system has an equilibrium at the origin (i.e., where formula_64, formula_65, formula_66, ..., formula_67, and formula_68) that is globally asymptotically stable.
Integrator Backstepping.
Before describing the backstepping procedure for general strict-feedback form dynamical systems, it is convenient to discuss the approach for a smaller class of strict-feedback form systems. These systems connect a series of integrators to the input of a
system with a known feedback-stabilizing control law, and so the stabilizing approach is known as "integrator backstepping." With a small modification, the integrator backstepping approach can be extended to handle all strict-feedback form systems.
Single-integrator Equilibrium.
Consider the dynamical system
 = f_x(\mathbf{x}) + g_x(\mathbf{x}) z_1\\
\dot{z}_1 = u_1
\end{cases}
where formula_2 and formula_21 is a scalar. This system is a cascade connection of an integrator with the subsystem (i.e., the input enters an integrator, and the integral formula_21 enters the subsystem).
We assume that formula_72, and so if formula_73, formula_11 and formula_75, then
So the origin formula_77 is an equilibrium (i.e., a stationary point) of the system. If the system ever reaches the origin, it will remain there forever after.
Single-integrator Backstepping.
In this example, backstepping is used to stabilize the single-integrator system in Equation  around its equilibrium at the origin. To be less precise, we wish to design a control law formula_29 that ensures that the states formula_79 return to formula_80 after the system is started from some arbitrary initial condition.
g_x(\mathbf{x})-k_1(\underbrace{z_1-u_x(\mathbf{x})}_{e_1})}^{v_1} \, + \, \overbrace{\frac{\partial u_x}{\partial \mathbf{x}}(\underbrace{f_x(\mathbf{x})+g_x(\mathbf{x})z_1}_{\dot{\mathbf{x}} \text{ (i.e., } \frac{\operatorname{d}\mathbf{x}}{\operatorname{d}t} \text{)}})}^{\dot{u}_x \text{ (i.e., } \frac{ \operatorname{d}u_x }{\operatorname{d}t} \text{)}}
So because this system is feedback stabilized by formula_144 and has Lyapunov function formula_145 with formula_146, it can be used as the upper subsystem in another single-integrator cascade system.
Motivating Example: Two-integrator Backstepping.
Before discussing the recursive procedure for the general multiple-integrator case, it is instructive to study the recursion present in the two-integrator case. That is, consider the dynamical system
 = f_x(\mathbf{x}) + g_x(\mathbf{x}) z_1\\
\dot{z}_1 = z_2\\
\dot{z}_2 = u_2
\end{cases}
where formula_2 and formula_21 and formula_22 are scalars. This system is a cascade connection of the single-integrator system in Equation  with another integrator (i.e., the input formula_41 enters through an integrator, and the output of that integrator enters the system in Equation  by its formula_34 input).
By letting
then the two-integrator system in Equation  becomes the single-integrator system
 = f_y(\mathbf{y}) + g_y(\mathbf{y}) z_2 &\quad \text{( where this } \mathbf{y} \text{ subsystem is stabilized by } z_2 = u_1(\mathbf{x},z_1) \text{ )}\\
\dot{z}_2 = u_2.
\end{cases}
By the single-integrator procedure, the control law formula_155 stabilizes the upper formula_22-to- subsystem using the Lyapunov function formula_145, and so Equation  is a new single-integrator system that is structurally equivalent to the single-integrator system in Equation . So a stabilizing control formula_41 can be found using the same single-integrator procedure that was used to find formula_34.
Many-integrator backstepping.
In the two-integrator case, the upper single-integrator subsystem was stabilized yielding a new single-integrator system that can be similarly stabilized. This recursive procedure can be extended to handle any finite number of integrators. This claim can be formally proved with mathematical induction. Here, a stabilized multiple-integrator system is built up from subsystems of already-stabilized multiple-integrator subsystems.
Hence, any system in this special many-integrator strict-feedback form can be feedback stabilized using a straightforward procedure that can even be automated (e.g., as part of an adaptive control algorithm).
Generic Backstepping.
Systems in the special strict-feedback form have a recursive structure similar to the many-integrator system structure. Likewise, they are stabilized by stabilizing the smallest cascaded system and then "backstepping" to the next cascaded system and repeating the procedure. So it is critical to develop a single-step procedure; that procedure can be recursively applied to cover the many-step case. Fortunately, due to the requirements on the functions in the strict-feedback form, each single-step system can be rendered by feedback to a single-integrator system, and that single-integrator system can be stabilized using methods discussed above.
Single-step Procedure.
Consider the simple strict-feedback system
 = f_x(\mathbf{x}) + g_x(\mathbf{x}) z_1\\
\dot{z}_1 = f_1(\mathbf{x}, z_1) + g_1(\mathbf{x}, z_1) u_1
\end{cases}
where
Rather than designing feedback-stabilizing control formula_34 directly, introduce a new control formula_252 (to be designed "later") and use control law
which is possible because formula_254. So the system in Equation  is
which simplifies to
This new formula_252-to- system matches the "single-integrator cascade system" in Equation . Assuming that a feedback-stabilizing control law formula_12 and Lyapunov function formula_87 for the upper subsystem is known, the feedback-stabilizing control law from Equation  is
with gain formula_123. So the final feedback-stabilizing control law is
g_x(\mathbf{x})-k_1(z_1-u_x(\mathbf{x})) + \frac{\partial u_x}{\partial \mathbf{x}}(f_x(\mathbf{x})+g_x(\mathbf{x})z_1)}^{u_{a1}(\mathbf{x},z_1)} \, - \, f_1(\mathbf{x}, z_1) \right)
with gain formula_123. The corresponding Lyapunov function from Equation  is
Because this "strict-feedback system" has a feedback-stabilizing control and a corresponding Lyapunov function, it can be cascaded as part of a larger strict-feedback system, and this procedure can be repeated to find the surrounding feedback-stabilizing control.
Many-step Procedure.
As in many-integrator backstepping, the single-step procedure can be completed iteratively to stabilize an entire strict-feedback system. In each step,
That is, any "strict-feedback system"
has the recursive structure
and can be feedback stabilized by finding the feedback-stabilizing control and Lyapunov function for the single-integrator formula_98 subsystem (i.e., with input formula_22 and output ) and iterating out from that inner subsystem until the ultimate feedback-stabilizing control is known. At iteration , the equivalent system is
By Equation , the corresponding feedback-stabilizing control law is
with gain formula_242. By Equation , the corresponding Lyapunov function is
By this construction, the ultimate control formula_244 (i.e., ultimate control is found at final iteration formula_245).
Hence, any strict-feedback system can be feedback stabilized using a straightforward procedure that can even be automated (e.g., as part of an adaptive control algorithm).


Small-gain theorem

In nonlinear systems, the formalism of input-output stability is an important tool in studying the stability of interconnected systems since the gain of a system directly relates to how the norm of a signal increases or decreases as it passes through the system. The small-gain theorem gives a sufficient condition for finite-gain formula_1 stability of the feedback connection. The small gain theorem was proved by George Zames in 1966. It can be seen as a generalization of the Nyquist criterion to non-linear time-varying MIMO systems (systems with multiple inputs and multiple outputs).
"Theorem". Assume two stable systems formula_2 and formula_3 are connected in a feedback loop, then the closed loop system is input-output stable if formula_4 and both formula_2 and formula_3 are stable by themselves. (This norm is typically the formula_7-norm, the size of the largest singular value of the transfer function over all frequencies. Any induced Norm will also lead to the same results).


Lyapunov redesign

In nonlinear control, the technique of "Lyapunov redesign" refers to the design where a stabilizing state feedback controller can be constructed with knowledge of the Lyapunov function formula_1. Consider the system
where formula_3 is the state vector and formula_4 is the vector of inputs. The functions formula_5, formula_6, and formula_7 are defined for formula_8, where formula_9 is a domain that contains the origin. A nominal model for this system can be written as
and the control law
stabilizes the system. The design of formula_12 is called "Lyapunov redesign".