Marginal stability

In the theory of dynamical systems and control theory, a linear time-invariant system is marginally stable if it is neither asymptotically stable nor unstable. Roughly speaking, a system is stable if it always returns to and stays near a particular state (called the steady state), and is unstable if it goes farther and farther away from any state, without being bounded. A marginal system, sometimes referred to as having neutral stability, is between these two types: when displaced, it does not return to near a common steady state, nor does it go away from where it started without limit.
Marginal stability, like instability, is a feature that control theory seeks to avoid; we wish that, when perturbed by some external force, a system will return to a desired state. This necessitates the use of appropriately designed control algorithms.
In econometrics, the presence of a unit root in observed time series, rendering them marginally stable, can lead to invalid regression results regarding effects of the independent variables upon a dependent variable, unless appropriate techniques are used to convert the system to a stable system.
Continuous time.
A homogeneous continuous linear time-invariant system is marginally stable if and only if the real part of every pole (eigenvalue) in the system's transfer-function is non-positive, one or more poles have zero real part, and all poles with zero real part are simple roots (i.e. the poles on the imaginary axis are all distinct from one another). In contrast, if all the poles have strictly negative real parts, the system is instead asymptotically stable. If the system is neither stable nor marginally stable, it is unstable.
If the system is in state space representation, marginal stability can be analyzed by deriving the Jordan normal form: if and only if the Jordan blocks corresponding to poles with zero real part are scalar is the system marginally stable.
Discrete time.
A homogeneous discrete time linear time-invariant system is marginally stable if and only if the greatest magnitude of any of the poles (eigenvalues) of the transfer function is 1, and the poles with magnitude equal to 1 are all distinct. That is, the transfer function's spectral radius is 1. If the spectral radius is less than 1, the system is instead asymptotically stable.
A simple example involves a single first-order linear difference equation: Suppose a state variable "x" evolves according to
with parameter "a" > 0. If the system is perturbed to the value formula_2 its subsequent sequence of values is formula_3 If "a" 1 the numbers get larger and larger without bound. But if "a" = 1, the numbers do neither of these: instead, all future values of "x" equal the value formula_5 Thus the case "a" = 1 exhibits marginal stability.
System response.
A marginally stable system is one that, if given an impulse of finite magnitude as input, will not "blow up" and give an unbounded output, but neither will the output return to zero. A bounded offset or oscillations in the output will persist indefinitely, and so there will in general be no final steady-state output. If a continuous system is given an input at a frequency equal to the frequency of a pole with zero real part, the system's output will increase indefinitely (this is known as pure resonance). This explains why for a system to be BIBO stable, the real parts of the poles have to be strictly negative (and not just non-positive).
A continuous system having imaginary poles, i.e. having zero real part in the pole(s), will produce sustained oscillations in the output. For example, an undamped second-order system such as the suspension system in an automobile (a mass–spring–damper system), from which the damper has been removed and spring is ideal, i.e. no friction is there, will in theory oscillate forever once disturbed. Another example is a frictionless pendulum. A system with a pole at the origin is also marginally stable but in this case there will be no oscillation in the response as the imaginary part is also zero ("jw" = 0 means "w" = 0 rad/sec). An example of such a system is a mass on a surface with friction. When a sidewards impulse is applied, the mass will move and never returns to zero. The mass will come to rest due to friction however, and the sidewards movement will remain bounded.
Since the locations of the marginal poles must be "exactly" on the imaginary axis or unit circle (for continuous time and discrete time systems respectively) for a system to be marginally stable, this situation is unlikely to occur in practice unless marginal stability is an inherent theoretical feature of the system.
Stochastic dynamics.
Marginal stability is also an important concept in the context of stochastic dynamics. For example, some processes may follow a random walk, given in discrete time as
where formula_7 is an i.i.d. error term. This equation has a unit root (a value of 1 for the eigenvalue of its characteristic equation), and hence exhibits marginal stability, so special time series techniques must be used in empirically modeling a system containing such an equation.
Marginally stable Markov processes are those that possess null recurrent classes.


Sumitomo Heavy Industries

 (SHI) is an integrated manufacturer of industrial machinery, automatic weaponry, ships, bridges and steel structure, equipment for environmental protection, including recycling, power transmission equipment, plastic molding machines, laser processing systems, particle accelerators, material handling systems, cancer diagnostic and treatment equipment and others.
History.
In 1888, a company was formed to provide equipment repair services to the Besshi copper mine. Almost 50 years later, in 1934, the company incorporated as Sumitomo Machinery Co., Ltd. to manufacture machinery for the steel and transportation industries in support of that period of rapid economic growth.
In 1969, Sumitomo Machinery Co., Ltd. merged with Uraga Heavy Industries Co., Ltd. to create Sumitomo Heavy Industries, Ltd. The company continues to innovate and expand to meet the demands of the new market frontiers. Today, Sumitomo Heavy Industries manufactures injection molding machines, laser systems, semiconductor machinery and liquid crystal production machinery.
In 1979, the company famously built the "Seawise Giant", an Ultra Large Crude Carrier (ULCC) supertanker; the longest ship ever built.
As of 2021, it's reported that SHI has ceased making light machine guns for the JSDF due to bleak economic prospects.


Evolutionary acquisition of neural topologies (EANT/EANT2) is an evolutionary reinforcement learning method that evolves both the topology and weights of artificial neural networks. It is closely related to the works of Angeline et al. and Stanley and Miikkulainen. Like the work of Angeline et al., the method uses a type of parametric mutation that comes from evolution strategies and evolutionary programming (now using the most advanced form of the evolution strategies CMA-ES in EANT2), in which adaptive step sizes are used for optimizing the weights of the neural networks. Similar to the work of Stanley (NEAT), the method starts with minimal structures which gain complexity along the evolution path.
Contribution of EANT to neuroevolution.
Despite sharing these two properties, the method has the following important features which distinguish it from previous works in neuroevolution.
It introduces a genetic encoding called common genetic encoding (CGE) that handles both direct and indirect encoding of neural networks within the same theoretical framework. The encoding has important properties that makes it suitable for evolving neural networks:
These properties have been formally proven.
For evolving the structure and weights of neural networks, an evolutionary process is used, where the "exploration" of structures is executed at a larger timescale (structural exploration), and the "exploitation" of existing structures is done at a smaller timescale (structural exploitation). In the structural exploration phase, new neural structures are developed by gradually adding new structures to an initially minimal network that is used as a starting point. In the structural exploitation phase, the weights of the currently available structures are optimized using an evolution strategy.
Performance.
EANT has been tested on some benchmark problems such as the double-pole balancing problem, and the RoboCup keepaway benchmark. In all the tests, EANT was found to perform very well. Moreover, a newer version of EANT, called EANT2, was tested on a visual servoing task and found to outperform NEAT and the traditional iterative Gauss–Newton method. Further experiments include results on a classification problem


Exploration problem

In robotics, the exploration problem deals with the use of a robot to maximize the knowledge over a particular area. The exploration problem arises in robotic mapping and search & rescue situations, where an environment might be dangerous or inaccessible to humans.
Overview.
The exploration problem naturally arises in situations in which a robot is utilized to survey an area that is dangerous or inaccessible for humans. The field of robotic explorations draws from various fields of information gathering and decision theory, and have been studied as far back as the 1950s.
The earliest work in robotic exploration was done in the context of simple finite state automata known as bandits, where algorithms were designed to distinguish and map different states in a finite state automaton. Since then, the primary emphasis has been shifted to the robotics system development domain, where exploration-algorithms guided robot have been used to survey volcanos, search and rescue, and abandoned mines mapping. Current state of the art system include advanced techniques on active localization, simultaneous localization and mapping (SLAM) based exploration, and multi-agent cooperative exploration.
Information gain.
The key concept in the exploration problem is the notion of information gain, that is, the amount of knowledge acquired while pushing the frontiers. A probabilistic measure of information gain is defined by the entropy
The function formula_2 is maximized if "p" is a uniform distribution and minimized when "p" is a point mass distribution. By minimizing the expected entropy of belief, information gain is maximized as


Proportional–integral–derivative controller

A proportional–integral–derivative controller (PID controller or three-term controller) is a control loop mechanism employing feedback that is widely used in industrial control systems and a variety of other applications requiring continuously modulated control. A PID controller continuously calculates an "error value" formula_1 as the difference between a desired setpoint (SP) and a measured process variable (PV) and applies a correction based on proportional, integral, and derivative terms (denoted "P", "I", and "D" respectively), hence the name.
PID systems automatically apply accurate and responsive correction to a control function. An everyday example is the cruise control on a car, where ascending a hill would lower speed if constant engine power were applied. The controller's PID algorithm restores the measured speed to the desired speed with minimal delay and overshoot by increasing the power output of the engine in a controlled manner.
The first theoretical analysis and practical application of PID was in the field of automatic steering systems for ships, developed from the early 1920s onwards. It was then used for automatic process control in the manufacturing industry, where it was widely implemented in pneumatic and then electronic controllers. Today the PID concept is used universally in applications requiring accurate and optimized automatic control.
Fundamental operation.
The distinguishing feature of the PID controller is the ability to use the three "control terms" of proportional, integral and derivative influence on the controller output to apply accurate and optimal control. The block diagram on the right shows the principles of how these terms are generated and applied. It shows a PID controller, which continuously calculates an "error value" formula_1 as the difference between a desired setpoint formula_3 and a measured process variable formula_4: formula_5, and applies a correction based on proportional, integral, and derivative terms. The controller attempts to minimize the error over time by adjustment of a "control variable" formula_6, such as the opening of a control valve, to a new value determined by a weighted sum of the control terms.
In this model:
Tuning – The balance of these effects is achieved by loop tuning to produce the optimal control function. The tuning constants are shown below as "K" and must be derived for each control application, as they depend on the response characteristics of the complete loop external to the controller. These are dependent on the behavior of the measuring sensor, the final control element (such as a control valve), any control signal delays, and the process itself. Approximate values of constants can usually be initially entered knowing the type of application, but they are normally refined, or tuned, by "bumping" the process in practice by introducing a setpoint change and observing the system response.
Control action – The mathematical model and practical loop above both use a "direct" control action for all the terms, which means an increasing positive error results in an increasing positive control output correction. The system is called "reverse" acting if it is necessary to apply negative corrective action. For instance, if the valve in the flow loop was 100–0% valve opening for 0–100% control output – meaning that the controller action has to be reversed. Some process control schemes and final control elements require this reverse action. An example would be a valve for cooling water, where the fail-safe mode, in the case of signal loss, would be 100% opening of the valve; therefore 0% controller output needs to cause 100% valve opening.
Mathematical form.
The overall control function
formula_8
where formula_9, formula_10, and formula_11, all non-negative, denote the coefficients for the proportional, integral, and derivative terms respectively (sometimes denoted "P", "I", and "D").
In the "standard form" of the equation (see later in article), formula_10 and formula_11 are respectively replaced by formula_14 and formula_15; the advantage of this being that formula_16 and formula_17 have some understandable physical meaning, as they represent an integration time and a derivative time respectively. formula_15 is the time constant with which the controller will attempt to approach the set point. formula_14 determines how long the controller will tolerate the output being consistently above or below the set point.
Selective use of control terms.
Although a PID controller has three control terms, some applications need only one or two terms to provide appropriate control. This is achieved by setting the unused parameters to zero and is called a PI, PD, P, or I controller in the absence of the other control actions. PI controllers are fairly common in applications where derivative action would be sensitive to measurement noise, but the integral term is often needed for the system to reach its target value.
Applicability.
The use of the PID algorithm does not guarantee optimal control of the system or its control stability . Situations may occur where there are excessive delays: the measurement of the process value is delayed, or the control action does not apply quickly enough. In these cases lead–lag compensation is required to be effective. The response of the controller can be described in terms of its responsiveness to an error, the degree to which the system overshoots a setpoint, and the degree of any system oscillation. But the PID controller is broadly applicable since it relies only on the response of the measured process variable, not on knowledge or a model of the underlying process.
History.
Origins.
Continuous control, before PID controllers were fully understood and implemented, has one of its origins in the centrifugal governor, which uses rotating weights to control a process. This was invented by Christiaan Huygens in the 17th century to regulate the gap between millstones in windmills depending on the speed of rotation, and thereby compensate for the variable speed of grain feed.
With the invention of the low-pressure stationary steam engine there was a need for automatic speed control, and James Watt’s self-designed "conical pendulum" governor, a set of revolving steel balls attached to a vertical spindle by link arms, came to be an industry standard. This was based on the millstone-gap control concept.
Rotating-governor speed control, however, was still variable under conditions of varying load, where the shortcoming of what is now known as proportional control alone was evident. The error between the desired speed and the actual speed would increase with increasing load. In the 19th century, the theoretical basis for the operation of governors was first described by James Clerk Maxwell in 1868 in his now-famous paper "On Governors". He explored the mathematical basis for control stability, and progressed a good way towards a solution, but made an appeal for mathematicians to examine the problem. The problem was examined further in 1874 by Edward Routh, Charles Sturm, and in 1895, Adolf Hurwitz, all of whom contributed to the establishment of control stability criteria.
In subsequent applications, speed governors were further refined, notably by American scientist Willard Gibbs, who in 1872 theoretically analyzed Watt's conical pendulum governor.
About this time, the invention of the Whitehead torpedo posed a control problem that required accurate control of the running depth. Use of a depth pressure sensor alone proved inadequate, and a pendulum that measured the fore and aft pitch of the torpedo was combined with depth measurement to become the pendulum-and-hydrostat control. Pressure control provided only a proportional control that, if the control gain was too high, would become unstable and go into overshoot with considerable instability of depth-holding. The pendulum added what is now known as derivative control, which damped the oscillations by detecting the torpedo dive/climb angle and thereby the rate-of-change of depth. This development (named by Whitehead as "The Secret" to give no clue to its action) was around 1868.
Another early example of a PID-type controller was developed by Elmer Sperry in 1911 for ship steering, though his work was intuitive rather than mathematically-based.
It was not until 1922, however, that a formal control law for what we now call PID or three-term control was first developed using theoretical analysis, by Russian American engineer Nicolas Minorsky. Minorsky was researching and designing automatic ship steering for the US Navy and based his analysis on observations of a helmsman. He noted the helmsman steered the ship based not only on the current course error but also on past error, as well as the current rate of change; this was then given a mathematical treatment by Minorsky.
His goal was stability, not general control, which simplified the problem significantly. While proportional control provided stability against small disturbances, it was insufficient for dealing with a steady disturbance, notably a stiff gale (due to steady-state error), which required adding the integral term. Finally, the derivative term was added to improve stability and control.
Trials were carried out on the USS "New Mexico", with the controllers controlling the "angular velocity" (not the angle) of the rudder. PI control yielded sustained yaw (angular error) of ±2°. Adding the D element yielded a yaw error of ±1/6°, better than most helmsmen could achieve.
The Navy ultimately did not adopt the system due to resistance by personnel. Similar work was carried out and published by several others in the 1930s.
Industrial control.
The wide use of feedback controllers did not become feasible until the development of wideband high-gain amplifiers to use the concept of negative feedback. This had been developed in telephone engineering electronics by Harold Black in the late 1920s, but not published until 1934. Independently, Clesson E Mason of the Foxboro Company in 1930 invented a wide-band pneumatic controller by combining the nozzle and flapper high-gain pneumatic amplifier, which had been invented in 1914, with negative feedback from the controller output. This dramatically increased the linear range of operation of the nozzle and flapper amplifier, and integral control could also be added by the use of a precision bleed valve and a bellows generating the integral term. The result was the "Stabilog" controller which gave both proportional and integral functions using feedback bellows. The integral term was called "Reset". Later the derivative term was added by a further bellows and adjustable orifice.
From about 1932 onwards, the use of wideband pneumatic controllers increased rapidly in a variety of control applications. Air pressure was used for generating the controller output, and also for powering process modulating devices such as diaphragm-operated control valves. They were simple low maintenance devices that operated well in harsh industrial environments and did not present explosion risks in hazardous locations. They were the industry standard for many decades until the advent of discrete electronic controllers and distributed control systems (DCSs).
With these controllers, a pneumatic industry signaling standard of was established, which had an elevated zero to ensure devices were working within their linear characteristic and represented the control range of 0-100%.
In the 1950s, when high gain electronic amplifiers became cheap and reliable, electronic PID controllers became popular, and the pneumatic standard was emulated by 10-50 mA and 4–20 mA current loop signals (the latter became the industry standard). Pneumatic field actuators are still widely used because of the advantages of pneumatic energy for control valves in process plant environments.
Most modern PID controls in industry are implemented as computer software in DCSs, programmable logic controllers (PLCs), or discrete compact controllers.
Electronic analog controllers.
Electronic analog PID control loops were often found within more complex electronic systems, for example, the head positioning of a disk drive, the power conditioning of a power supply, or even the movement-detection circuit of a modern seismometer. Discrete electronic analog controllers have been largely replaced by digital controllers using microcontrollers or FPGAs to implement PID algorithms. However, discrete analog PID controllers are still used in niche applications requiring high-bandwidth and low-noise performance, such as laser-diode controllers.
Control loop example.
Consider a robotic arm that can be moved and positioned by a control loop. An electric motor may lift or lower the arm, depending on forward or reverse power applied, but power cannot be a simple function of position because of the inertial mass of the arm, forces due to gravity, external forces on the arm such as a load to lift or work to be done on an external object.
By measuring the position (PV), and subtracting it from the setpoint (SP), the error (e) is found, and from it the controller calculates how much electric current to supply to the motor (MV).
Proportional.
The obvious method is proportional control: the motor current is set in proportion to the existing error. However, this method fails if, for instance, the arm has to lift different weights: a greater weight needs a greater force applied for the same error on the down side, but a smaller force if the error is low on the upside. That's where the integral and derivative terms play their part.
Integral.
An integral term increases action in relation not only to the error but also the time for which it has persisted. So, if the applied force is not enough to bring the error to zero, this force will be increased as time passes. A pure "I" controller could bring the error to zero, but it would be both slow reacting at the start (because the action would be small at the beginning, depending on time to get significant) and brutal at the end (the action increases as long as the error is positive, even if the error has started to approach zero).
Applying too much integral when the error is small and decreasing will lead to overshoot. After overshooting, if the controller were to apply a large correction in the opposite direction and repeatedly overshoot the desired position, the output would oscillate around the setpoint in either a constant, growing, or decaying sinusoid. If the amplitude of the oscillations increases with time, the system is unstable. If they decrease, the system is stable. If the oscillations remain at a constant magnitude, the system is marginally stable.
Derivative.
A derivative term does not consider the magnitude of the error (meaning it cannot bring it to zero: a pure D controller cannot bring the system to its setpoint), but the rate of change of error, trying to bring this rate to zero. It aims at flattening the error trajectory into a horizontal line, damping the force applied, and so reduces overshoot (error on the other side because of too great applied force).
Control damping.
In the interest of achieving a controlled arrival at the desired position (SP) in a timely and accurate way, the controlled system needs to be critically damped. A well-tuned position control system will also apply the necessary currents to the controlled motor so that the arm pushes and pulls as necessary to resist external forces trying to move it away from the required position. The setpoint itself may be generated by an external system, such as a PLC or other computer system, so that it continuously varies depending on the work that the robotic arm is expected to do. A well-tuned PID control system will enable the arm to meet these changing requirements to the best of its capabilities.
Response to disturbances.
If a controller starts from a stable state with zero error (PV = SP), then further changes by the controller will be in response to changes in other measured or unmeasured inputs to the process that affect the process, and hence the PV. Variables that affect the process other than the MV are known as disturbances. Generally, controllers are used to reject disturbances and to implement setpoint changes. A change in load on the arm constitutes a disturbance to the robot arm control process.
Applications.
In theory, a controller can be used to control any process that has a measurable output (PV), a known ideal value for that output (SP), and an input to the process (MV) that will affect the relevant PV. Controllers are used in industry to regulate temperature, pressure, force, feed rate, flow rate, chemical composition (component concentrations), weight, position, speed, and practically every other variable for which a measurement exists.
Controller theory.
The PID control scheme is named after its three correcting terms, whose sum constitutes the manipulated variable (MV). The proportional, integral, and derivative terms are summed to calculate the output of the PID controller. Defining formula_6 as the controller output, the final form of the PID algorithm is
where
Equivalently, the transfer function in the Laplace domain of the PID controller is
where formula_31 is the complex frequency.
Proportional term.
The proportional term produces an output value that is proportional to the current error value. The proportional response can be adjusted by multiplying the error by a constant "K"p, called the proportional gain constant.
The proportional term is given by
A high proportional gain results in a large change in the output for a given change in the error. If the proportional gain is too high, the system can become unstable (see the section on loop tuning). In contrast, a small gain results in a small output response to a large input error, and a less responsive or less sensitive controller. If the proportional gain is too low, the control action may be too small when responding to system disturbances. Tuning theory and industrial practice indicate that the proportional term should contribute the bulk of the output change.
Steady-state error.
The steady-state error is the difference between the desired final output and the actual one. Because a non-zero error is required to drive it, a proportional controller generally operates with a steady-state error. Steady-state error (SSE) is proportional to the process gain and inversely proportional to proportional gain. SSE may be mitigated by adding a compensating bias term to the setpoint AND output or corrected dynamically by adding an integral term.
Integral term.
The contribution from the integral term is proportional to both the magnitude of the error and the duration of the error. The integral in a PID controller is the sum of the instantaneous error over time and gives the accumulated offset that should have been corrected previously. The accumulated error is then multiplied by the integral gain ("K"i) and added to the controller output.
The integral term is given by
The integral term accelerates the movement of the process towards setpoint and eliminates the residual steady-state error that occurs with a pure proportional controller. However, since the integral term responds to accumulated errors from the past, it can cause the present value to overshoot the setpoint value (see the section on loop tuning).
Derivative term.
The derivative of the process error is calculated by determining the slope of the error over time and multiplying this rate of change by the derivative gain "K"d. The magnitude of the contribution of the derivative term to the overall control action is termed the derivative gain, "K"d.
The derivative term is given by
Derivative action predicts system behavior and thus improves settling time and stability of the system. An ideal derivative is not causal, so that implementations of PID controllers include an additional low-pass filtering for the derivative term to limit the high-frequency gain and noise. Derivative action is seldom used in practice though – by one estimate in only 25% of deployed controllers – because of its variable impact on system stability in real-world applications.
Loop tuning.
"Tuning" a control loop is the adjustment of its control parameters (proportional band/gain, integral gain/reset, derivative gain/rate) to the optimum values for the desired control response. Stability (no unbounded oscillation) is a basic requirement, but beyond that, different systems have different behavior, different applications have different requirements, and requirements may conflict with one another.
Even though there are only three parameters and it is simple to describe in principle, PID tuning is a difficult problem because it must satisfy complex criteria within the limitations of PID control. Accordingly, there are various methods for loop tuning, and more sophisticated techniques are the subject of patents; this section describes some traditional, manual methods for loop tuning.
Designing and tuning a PID controller appears to be conceptually intuitive, but can be hard in practice, if multiple (and often conflicting) objectives, such as short transient and high stability, are to be achieved. PID controllers often provide acceptable control using default tunings, but performance can generally be improved by careful tuning, and performance may be unacceptable with poor tuning. Usually, initial designs need to be adjusted repeatedly through computer simulations until the closed-loop system performs or compromises as desired.
Some processes have a degree of nonlinearity, so parameters that work well at full-load conditions do not work when the process is starting up from no load. This can be corrected by gain scheduling (using different parameters in different operating regions).
Stability.
If the PID controller parameters (the gains of the proportional, integral and derivative terms) are chosen incorrectly, the controlled process input can be unstable; i.e., its output diverges, with or without oscillation, and is limited only by saturation or mechanical breakage. Instability is caused by "excess" gain, particularly in the presence of significant lag.
Generally, stabilization of response is required and the process must not oscillate for any combination of process conditions and setpoints, though sometimes marginal stability (bounded oscillation) is acceptable or desired.
Mathematically, the origins of instability can be seen in the Laplace domain.
The closed-loop transfer function is:
where formula_36 is the PID transfer function and formula_37 is the plant transfer function. A system is "unstable" where the closed-loop transfer function diverges for some formula_31. This happens in situations where formula_39. Typically, this happens when formula_40 with a 180-degree phase shift. Stability is guaranteed when formula_41 for frequencies that suffer high phase shifts. A more general formalism of this effect is known as the Nyquist stability criterion.
Optimal behavior.
The optimal behavior on a process change or setpoint change varies depending on the application.
Two basic requirements are "regulation" (disturbance rejection – staying at a given setpoint) and "command tracking" (implementing setpoint changes). These terms refer to how well the controlled variable tracks the desired value. Specific criteria for command tracking include rise time and settling time. Some processes must not allow an overshoot of the process variable beyond the setpoint if, for example, this would be unsafe. Other processes must minimize the energy expended in reaching a new setpoint.
Overview of tuning methods.
There are several methods for tuning a PID loop. The most effective methods generally involve developing some form of process model and then choosing P, I, and D based on the dynamic model parameters. Manual tuning methods can be relatively time-consuming, particularly for systems with long loop times.
The choice of method depends largely on whether the loop can be taken offline for tuning, and on the response time of the system. If the system can be taken offline, the best tuning method often involves subjecting the system to a step change in input, measuring the output as a function of time, and using this response to determine the control parameters.
Manual tuning.
If the system must remain online, one tuning method is to first set formula_42 and formula_43 values to zero. Increase the formula_44 until the output of the loop oscillates; then set formula_44 to approximately half that value for a "quarter amplitude decay"-type response. Then increase formula_42 until any offset is corrected in sufficient time for the process, but not until too great a value causes instability. Finally, increase formula_43, if required, until the loop is acceptably quick to reach its reference after a load disturbance. Too much formula_43 causes excessive response and overshoot. A fast PID loop tuning usually overshoots slightly to reach the setpoint more quickly; however, some systems cannot accept overshoot, in which case an overdamped closed-loop system is required, which in turn requires a formula_44 setting significantly less than half that of the formula_44 setting that was causing oscillation.
Ziegler–Nichols method.
Another heuristic tuning method is known as the Ziegler–Nichols method, introduced by John G. Ziegler and Nathaniel B. Nichols in the 1940s. As in the method above, the formula_42 and formula_43 gains are first set to zero. The proportional gain is increased until it reaches the ultimate gain, formula_53, at which the output of the loop starts to oscillate constantly. formula_53 and the oscillation period formula_55 are used to set the gains as follows:
These gains apply to the ideal, parallel form of the PID controller. When applied to the standard PID form, only the integral and derivative gains formula_42 and formula_43 are dependent on the oscillation period formula_55.
Cohen–Coon parameters.
This method was developed in 1953 and is based on a first-order + time delay model. Similar to the Ziegler–Nichols method, a set of tuning parameters were developed to yield a closed-loop response with a decay ratio of formula_59. Arguably the biggest problem with these parameters is that a small change in the process parameters could potentially cause a closed-loop system to become unstable.
Relay (Åström–Hägglund) method.
Published in 1984 by Karl Johan Åström and Tore Hägglund, the relay method temporarily operates the process using bang-bang control and measures the resultant oscillations. The output is switched (as if by a relay, hence the name) between two values of the control variable. The values must be chosen so the process will cross the setpoint, but they need not be 0% and 100%; by choosing suitable values, dangerous oscillations can be avoided.
As long as the process variable is below the setpoint, the control output is set to the higher value. As soon as it rises above the setpoint, the control output is set to the lower value. Ideally, the output waveform is nearly square, spending equal time above and below the setpoint. The period and amplitude of the resultant oscillations are measured, and used to compute the ultimate gain and period, which are then fed into the Ziegler–Nichols method.
Specifically, the ultimate period formula_55 is assumed to be equal to the observed period, and the ultimate gain is computed as formula_61 where is the amplitude of the process variable oscillation, and is the amplitude of the control output change which caused it.
There are numerous variants on the relay method.
First order with dead time model.
The transfer function for a first-order process, with dead time, is:
formula_62
where kp is the process gain, τp is the time constant, θ is the dead time, and u(s) is a step change input. Converting this transfer function to the time domain results in:
formula_63
using the same parameters found above.
It is important when using this method to apply a large enough step change input that the output can be measured; however, too large of a step change can affect the process stability. Additionally, a larger step change ensures that the output does not change due to a disturbance (for best results, try to minimize disturbances when performing the step test).
One way to determine the parameters for the first-order process is using the 63.2% method. In this method, the process gain (kp) is equal to the change in output divided by the change in input. The dead time (θ) is the amount of time between when the step change occurred and when the output first changed. The time constant (τp) is the amount of time it takes for the output to reach 63.2% of the new steady-state value after the step change. One downside to using this method is that it can take a while to reach a new steady-state value if the process has large time constants.
Tuning software.
Most modern industrial facilities no longer tune loops using the manual calculation methods shown above. Instead, PID tuning and loop optimization software are used to ensure consistent results. These software packages gather data, develop process models, and suggest optimal tuning. Some software packages can even develop tuning by gathering data from reference changes.
Mathematical PID loop tuning induces an impulse in the system and then uses the controlled system's frequency response to design the PID loop values. In loops with response times of several minutes, mathematical loop tuning is recommended, because trial and error can take days just to find a stable set of loop values. Optimal values are harder to find. Some digital loop controllers offer a self-tuning feature in which very small setpoint changes are sent to the process, allowing the controller itself to calculate optimal tuning values.
Another approach calculates initial values via the Ziegler–Nichols method, and uses a numerical optimization technique to find better PID coefficients.
Other formulas are available to tune the loop according to different performance criteria. Many patented formulas are now embedded within PID tuning software and hardware modules.
Advances in automated PID loop tuning software also deliver algorithms for tuning PID Loops in a dynamic or non-steady state (NSS) scenario. The software models the dynamics of a process, through a disturbance, and calculate PID control parameters in response.
Limitations.
While PID controllers are applicable to many control problems, and often perform satisfactorily without any improvements or only coarse tuning, they can perform poorly in some applications and do not in general provide "optimal" control. The fundamental difficulty with PID control is that it is a feedback control system, with "constant" parameters, and no direct knowledge of the process, and thus overall performance is reactive and a compromise. While PID control is the best controller for an observer without a model of the process, better performance can be obtained by overtly modeling the actor of the process without resorting to an observer.
PID controllers, when used alone, can give poor performance when the PID loop gains must be reduced so that the control system does not overshoot, oscillate or hunt about the control setpoint value. They also have difficulties in the presence of non-linearities, may trade-off regulation versus response time, do not react to changing process behavior (say, the process changes after it has warmed up), and have lag in responding to large disturbances.
The most significant improvement is to incorporate feed-forward control with knowledge about the system, and using the PID only to control error. Alternatively, PIDs can be modified in more minor ways, such as by changing the parameters (either gain scheduling in different use cases or adaptively modifying them based on performance), improving measurement (higher sampling rate, precision, and accuracy, and low-pass filtering if necessary), or cascading multiple PID controllers.
Linearity and symmetry.
PID controllers work best when the loop to be controlled is linear and symmetric. Thus, their performance in non-linear and asymmetric systems is degraded.
A non-linear valve, for instance, in a flow control application, will result in variable loop sensitivity, requiring dampened action to prevent instability. One solution is the use of the valve's non-linear characteristic in the control algorithm to compensate for this.
An asymmetric application, for example, is temperature control in HVAC systems using only active heating (via a heating element), where there is only passive cooling available. When it is desired to lower the controlled temperature the heating output is off, but there is no active cooling due to control output. Any overshoot of rising temperature can therefore only be corrected slowly; it cannot be forced downward by the control output. In this case the PID controller could be tuned to be over-damped, to prevent or reduce overshoot, but this reduces performance by increasing the settling time of a rising temperature to the set point. The inherent degradation of control quality in this application could be solved by application of active cooling.
Noise in derivative term.
A problem with the derivative term is that it amplifies higher frequency measurement or process noise that can cause large amounts of change in the output. It is often helpful to filter the measurements with a low-pass filter in order to remove higher-frequency noise components. As low-pass filtering and derivative control can cancel each other out, the amount of filtering is limited. Therefore, low noise instrumentation can be important. A nonlinear median filter may be used, which improves the filtering efficiency and practical performance. In some cases, the differential band can be turned off with little loss of control. This is equivalent to using the PID controller as a PI controller.
Modifications to the algorithm.
The basic PID algorithm presents some challenges in control applications that have been addressed by minor modifications to the PID form.
Integral windup.
One common problem resulting from the ideal PID implementations is integral windup. Following a large change in setpoint the integral term can accumulate an error larger than the maximal value for the regulation variable (windup), thus the system overshoots and continues to increase until this accumulated error is unwound. This problem can be addressed by:
Overshooting from known disturbances.
For example, a PID loop is used to control the temperature of an electric resistance furnace where the system has stabilized. Now when the door is opened and something cold is put into the furnace the temperature drops below the setpoint. The integral function of the controller tends to compensate for error by introducing another error in the positive direction. This overshoot can be avoided by freezing of the integral function after the opening of the door for the time the control loop typically needs to reheat the furnace.
PI controller.
A PI controller (proportional-integral controller) is a special case of the PID controller in which the derivative (D) of the error is not used.
The controller output is given by
where formula_65 is the error or deviation of actual measured value (PV) from the setpoint (SP).
A PI controller can be modelled easily in software such as Simulink or Xcos using a "flow chart" box involving Laplace operators:
where
Setting a value for formula_70 is often a trade off between decreasing overshoot and increasing settling time.
The lack of derivative action may make the system more steady in the steady state in the case of noisy data. This is because derivative action is more sensitive to higher-frequency terms in the inputs.
Without derivative action, a PI-controlled system is less responsive to real (non-noise) and relatively fast alterations in state and so the system will be slower to reach setpoint and slower to respond to perturbations than a well-tuned PID system may be.
Deadband.
Many PID loops control a mechanical device (for example, a valve). Mechanical maintenance can be a major cost and wear leads to control degradation in the form of either stiction or backlash in the mechanical response to an input signal. The rate of mechanical wear is mainly a function of how often a device is activated to make a change. Where wear is a significant concern, the PID loop may have an output deadband to reduce the frequency of activation of the output (valve). This is accomplished by modifying the controller to hold its output steady if the change would be small (within the defined deadband range). The calculated output must leave the deadband before the actual output will change.
Setpoint step change.
The proportional and derivative terms can produce excessive movement in the output when a system is subjected to an instantaneous step increase in the error, such as a large setpoint change. In the case of the derivative term, this is due to taking the derivative of the error, which is very large in the case of an instantaneous step change. As a result, some PID algorithms incorporate some of the following modifications:
Feed-forward.
The control system performance can be improved by combining the feedback (or closed-loop) control of a PID controller with feed-forward (or open-loop) control. Knowledge about the system (such as the desired acceleration and inertia) can be fed forward and combined with the PID output to improve the overall system performance. The feed-forward value alone can often provide the major portion of the controller output. The PID controller primarily has to compensate for whatever difference or "error" remains between the setpoint (SP) and the system response to the open-loop control. Since the feed-forward output is not affected by the process feedback, it can never cause the control system to oscillate, thus improving the system response without affecting stability. Feed forward can be based on the setpoint and on extra measured disturbances. Setpoint weighting is a simple form of feed forward.
For example, in most motion control systems, in order to accelerate a mechanical load under control, more force is required from the actuator. If a velocity loop PID controller is being used to control the speed of the load and command the force being applied by the actuator, then it is beneficial to take the desired instantaneous acceleration, scale that value appropriately and add it to the output of the PID velocity loop controller. This means that whenever the load is being accelerated or decelerated, a proportional amount of force is commanded from the actuator regardless of the feedback value. The PID loop in this situation uses the feedback information to change the combined output to reduce the remaining difference between the process setpoint and the feedback value. Working together, the combined open-loop feed-forward controller and closed-loop PID controller can provide a more responsive control system.
Bumpless operation.
PID controllers are often implemented with a "bumpless" initialization feature that recalculates the integral accumulator term to maintain a consistent process output through parameter changes. A partial implementation is to store the integral gain times the error rather than storing the error and postmultiplying by the integral gain, which prevents discontinuous output when the I gain is changed, but not the P or D gains.
Other improvements.
In addition to feed-forward, PID controllers are often enhanced through methods such as PID gain scheduling (changing parameters in different operating conditions), fuzzy logic, or computational verb logic. Further practical application issues can arise from instrumentation connected to the controller. A high enough sampling rate, measurement precision, and measurement accuracy are required to achieve adequate control performance. Another new method for improvement of PID controller is to increase the degree of freedom by using fractional order. The order of the integrator and differentiator add increased flexibility to the controller.
Cascade control.
One distinctive advantage of PID controllers is that two PID controllers can be used together to yield better dynamic performance. This is called cascaded PID control. Two controllers are in cascade when they are arranged so that one regulates the set point of the other. A PID controller acts as outer loop controller, which controls the primary physical parameter, such as fluid level or velocity. The other controller acts as inner loop controller, which reads the output of outer loop controller as setpoint, usually controlling a more rapid changing parameter, flowrate or acceleration. It can be mathematically proven that the working frequency of the controller is increased and the time constant of the object is reduced by using cascaded PID controllers..
For example, a temperature-controlled circulating bath has two PID controllers in cascade, each with its own thermocouple temperature sensor. The outer controller controls the temperature of the water using a thermocouple located far from the heater, where it accurately reads the temperature of the bulk of the water. The error term of this PID controller is the difference between the desired bath temperature and measured temperature. Instead of controlling the heater directly, the outer PID controller sets a heater temperature goal for the inner PID controller. The inner PID controller controls the temperature of the heater using a thermocouple attached to the heater. The inner controller's error term is the difference between this heater temperature setpoint and the measured temperature of the heater. Its output controls the actual heater to stay near this setpoint.
The proportional, integral, and differential terms of the two controllers will be very different. The outer PID controller has a long time constant – all the water in the tank needs to heat up or cool down. The inner loop responds much more quickly. Each controller can be tuned to match the physics of the system "it" controls – heat transfer and thermal mass of the whole tank or of just the heater – giving better total response.
Alternative nomenclature and forms.
Standard versus parallel (ideal) form.
The form of the PID controller most often encountered in industry, and the one most relevant to tuning algorithms is the "standard form". In this form the formula_44 gain is applied to the formula_72, and formula_73 terms, yielding:
where
In this standard form, the parameters have a clear physical meaning. In particular, the inner summation produces a new single error value which is compensated for future and past errors. The proportional error term is the current error. The derivative components term attempts to predict the error value at formula_76 seconds (or samples) in the future, assuming that the loop control remains unchanged. The integral component adjusts the error value to compensate for the sum of all past errors, with the intention of completely eliminating them in formula_75 seconds (or samples). The resulting compensated single error value is then scaled by the single gain formula_44 to compute the control variable.
In the parallel form, shown in the controller theory section
the gain parameters are related to the parameters of the standard form through formula_81 and formula_82. This parallel form, where the parameters are treated as simple gains, is the most general and flexible form. However, it is also the form where the parameters have the weakest relationship to physical behaviors and is generally reserved for theoretical treatment of the PID controller. The standard form, despite being slightly more complex mathematically, is more common in industry.
Reciprocal gain, a.k.a. proportional band.
In many cases, the manipulated variable output by the PID controller is a dimensionless fraction between 0 and 100% of some maximum possible value, and the translation into real units (such as pumping rate or watts of heater power) is outside the PID controller. The process variable, however, is in dimensioned units such as temperature. It is common in this case to express the gain formula_44 not as "output per degree", but rather in the reciprocal form of a "proportional band" formula_84, which is "degrees per full output": the range over which the output changes from 0 to 1 (0% to 100%). Beyond this range, the output is saturated, full-off or full-on. The narrower this band, the higher the proportional gain.
Basing derivative action on PV.
In most commercial control systems, derivative action is based on process variable rather than error. That is, a change in the setpoint does not affect the derivative action. This is because the digitized version of the algorithm produces a large unwanted spike when the setpoint is changed. If the setpoint is constant then changes in the PV will be the same as changes in error. Therefore, this modification makes no difference to the way the controller responds to process disturbances.
Basing proportional action on PV.
Most commercial control systems offer the "option" of also basing the proportional action solely on the process variable. This means that only the integral action responds to changes in the setpoint. The modification to the algorithm does not affect the way the controller responds to process disturbances.
Basing proportional action on PV eliminates the instant and possibly very large change in output caused by a sudden change to the setpoint. Depending on the process and tuning this may be beneficial to the response to a setpoint step.
King describes an effective chart-based method.
Laplace form.
Sometimes it is useful to write the PID regulator in Laplace transform form:
Having the PID controller written in Laplace form and having the transfer function of the controlled system makes it easy to determine the closed-loop transfer function of the system.
Series/interacting form.
Another representation of the PID controller is the series, or "interacting" form
where the parameters are related to the parameters of the standard form through
with
This form essentially consists of a PD and PI controller in series. As the integral is required to calculate the controller's bias this form provides the ability to track an external bias value which is required to be used for proper implementation of multi-controller advanced control schemes.
Discrete implementation.
The analysis for designing a digital implementation of a PID controller in a microcontroller (MCU) or FPGA device requires the standard form of the PID controller to be "discretized". Approximations for first-order derivatives are made by backward finite differences. formula_6 and formula_1 are discretized with a sampling period formula_94, k is the sample index.
Differentiating both sides of PID equation using Newton's notation gives:
formula_95
Derivative terms are approximated as,
So,
Applying backward difference again gives,
By simplifying and regrouping terms of the above equation, an algorithm for an implementation of the discretized PID controller in a MCU is finally obtained:
or:
s.t. formula_101
Note: This method solves in fact formula_102 where formula_103 is a constant independent of t. This constant is useful when you want to have a start and stop control on the regulation loop. For instance, setting Kp,Ki and Kd to 0 will keep u(t) constant. Likewise, when you want to start a regulation on a system where the error is already close to 0 with u(t) non null, it prevents from sending the output to 0.
Pseudocode.
Here is a very simple and explicit group of pseudocode that can be easily understood by the layman:
 previous_error := 0
 integral := 0
 loop:
 error := setpoint − measured_value
 proportional := error;
 integral := integral + error × dt
 derivative := (error − previous_error) / dt
 output := Kp × proportional + Ki × integral + Kd × derivative
 previous_error := error
 wait(dt)
 goto loop
Here is a more complicated and much less explicit software loop that implements a PID algorithm:
 A0 := Kp + Ki*dt + Kd/dt
 A1 := -Kp - 2*Kd/dt
 A2 := Kd/dt
 error[2] := 0 // e(t-2)
 error[1] := 0 // e(t-1)
 error[0] := 0 // e(t)
 output := u0 // Usually the current value of the actuator
 loop:
 error[2] := error[1]
 error[1] := error[0]
 error[0] := setpoint − measured_value
 output := output + A0 * error[0] + A1 * error[1] + A2 * error[2]
 wait(dt)
 goto loop
Here, Kp is a dimensionless number, Ki is expressed in formula_104 and Kd is expressed in s. When doing a regulation where the actuator and the measured value are not in the same unit (ex. temperature regulation using a motor controlling a valve), Kp, Ki and Kd may be corrected by a unit conversion factor. It may also be interesting to use Ki in its reciprocal form (integration time). The above implementation allows to perform a I-only controller which may be useful in some cases.
In the real world, this is D-to-A converted and passed into the process under control as the manipulated variable (MV). The current error is stored elsewhere for re-use in the next differentiation, the program then waits until dt seconds have passed since start, and the loop begins again, reading in new values for the PV and the setpoint and calculating a new value for the error.
Note that for real code, the use of "wait(dt)" might be inappropriate because it doesn't account for time taken by the algorithm itself during the loop, or more importantly, any preemption delaying the algorithm.
A common issue when using formula_43 is the response to the derivative of a rising or falling edge of the setpoint as shown below:
A typical workaround is to filter the derivative action using a low pass filter of time constant formula_106 where formula_107:
A variant of the above algorithm using an infinite impulse response (IIR) filter for the derivative:
 A0 := Kp + Ki*dt
 A1 := -Kp
 error[2] := 0 // e(t-2)
 error[1] := 0 // e(t-1)
 error[0] := 0 // e(t)
 output := u0 // Usually the current value of the actuator
 A0d = Kd/dt
 A1d = - 2.0*Kd/dt
 A2d = Kd/dt
 N := 5
 tau := Kd / (Kp*N) // IIR filter time constant
 alpha = dt / (2*tau)
 d0 := 0
 d1 := 0
 fd0 := 0
 fd1 := 0
 loop:
 error[2] := error[1]
 error[1] := error[0]
 error[0] := setpoint − measured_value
 // PI
 output := output + A0 * error[0] + A1 * error[1]
 // Filtered D
 d1 = d0
 d0 = A0d * error[0] + A1d * error[1] + A2d * error[2]
 fd1 = fd0
 fd0 = ((alpha) / (alpha + 1)) * (d0 + d1) - ((alpha - 1) / (alpha + 1)) * fd1
 output := output + fd0
 wait(dt)
 goto loop


Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning.
Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected. Instead the focus is on finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge).
The environment is typically stated in the form of a Markov decision process (MDP), because many reinforcement learning algorithms for this context use dynamic programming techniques. The main difference between the classical dynamic programming methods and reinforcement learning algorithms is that the latter do not assume knowledge of an exact mathematical model of the MDP and they target large MDPs where exact methods become infeasible.
Introduction.
Due to its generality, reinforcement learning is studied in many disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, and statistics. In the operations research and control literature, reinforcement learning is called "approximate dynamic programming," or "neuro-dynamic programming." The problems of interest in reinforcement learning have also been studied in the theory of optimal control, which is concerned mostly with the existence and characterization of optimal solutions, and algorithms for their exact computation, and less with learning or approximation, particularly in the absence of a mathematical model of the environment. In economics and game theory, reinforcement learning may be used to explain how equilibrium may arise under bounded rationality.
Basic reinforcement learning is modeled as a Markov decision process (MDP):
The purpose of reinforcement learning is for the agent to learn an optimal, or nearly-optimal, policy that maximizes the "reward function" or other user-provided reinforcement signal that accumulates from the immediate rewards. This is similar to processes that appear to occur in animal psychology. For example, biological brains are hardwired to interpret signals such as pain and hunger as negative reinforcements, and interpret pleasure and food intake as positive reinforcements. In some circumstances, animals can learn to engage in behaviors that optimize these rewards. This suggests that animals are capable of reinforcement learning.
A basic reinforcement learning agent AI interacts with its environment in discrete time steps. At each time , the agent receives the current state formula_10 and reward formula_11. It then chooses an action formula_12 from the set of available actions, which is subsequently sent to the environment. The environment moves to a new state formula_13 and the reward formula_14 associated with the "transition" formula_15 is determined. The goal of a reinforcement learning agent is to learn a "policy": formula_16, formula_17 which maximizes the expected cumulative reward.
Formulating the problem as an MDP assumes the agent directly observes the current environmental state; in this case the problem is said to have "full observability". If the agent only has access to a subset of states, or if the observed states are corrupted by noise, the agent is said to have "partial observability", and formally the problem must be formulated as a Partially observable Markov decision process. In both cases, the set of actions available to the agent can be restricted. For example, the state of an account balance could be restricted to be positive; if the current value of the state is 3 and the state transition attempts to reduce the value by 4, the transition will not be allowed.
When the agent's performance is compared to that of an agent that acts optimally, the difference in performance gives rise to the notion of "regret". In order to act near optimally, the agent must reason about the long-term consequences of its actions (i.e., maximize future income), although the immediate reward associated with this might be negative.
Thus, reinforcement learning is particularly well-suited to problems that include a long-term versus short-term reward trade-off. It has been applied successfully to various problems, including energy storage operation, robot control, elevator scheduling, telecommunications, photovoltaic generators dispatch, backgammon, checkers and Go (AlphaGo).
Two elements make reinforcement learning powerful: the use of samples to optimize performance and the use of function approximation to deal with large environments. Thanks to these two key components, reinforcement learning can be used in large environments in the following situations:
The first two of these problems could be considered planning problems (since some form of model is available), while the last one could be considered to be a genuine learning problem. However, reinforcement learning converts both planning problems to machine learning problems.
Exploration.
The exploration vs. exploitation trade-off has been most thoroughly studied through the multi-armed bandit problem and for finite state space MDPs in Burnetas and Katehakis (1997).
Reinforcement learning requires clever exploration mechanisms; randomly selecting actions, without reference to an estimated probability distribution, shows poor performance. The case of (small) finite MDPs is relatively well understood. However, due to the lack of algorithms that scale well with the number of states (or scale to problems with infinite state spaces), simple exploration methods are the most practical.
One such method is formula_18-greedy, where formula_19 is a parameter controlling the amount of exploration vs. exploitation. With probability formula_20, exploitation is chosen, and the agent chooses the action that it believes has the best long-term effect (ties between actions are broken uniformly at random). Alternatively, with probability formula_18, exploration is chosen, and the action is chosen uniformly at random. formula_18 is usually a fixed parameter but can be adjusted either according to a schedule (making the agent explore progressively less), or adaptively based on heuristics.
Algorithms for control learning.
Even if the issue of exploration is disregarded and even if the state was observable (assumed hereafter), the problem remains to use past experience to find out which actions lead to higher cumulative rewards.
Criterion of optimality.
Policy.
The agent's action selection is modeled as a map called "policy":
The policy map gives the probability of taking action formula_5 when in state formula_3. There are also deterministic policies.
State-value function.
The value function formula_27 is defined as, "expected return" starting with state formula_3, i.e. formula_29, and successively following policy formula_30. Hence, roughly speaking, the value function estimates "how good" it is to be in a given state.
where the random variable formula_32 denotes the return, and is defined as the sum of future discounted rewards:
where formula_11 is the reward at step formula_2, formula_36 is the discount-rate. Gamma is less than 1, so events in the distant future are weighted less than events in the immediate future.
The algorithm must find a policy with maximum expected return. From the theory of MDPs it is known that, without loss of generality, the search can be restricted to the set of so-called "stationary" policies. A policy is "stationary" if the action-distribution returned by it depends only on the last state visited (from the observation agent's history). The search can be further restricted to "deterministic" stationary policies. A "deterministic stationary" policy deterministically selects actions based on the current state. Since any such policy can be identified with a mapping from the set of states to the set of actions, these policies can be identified with such mappings with no loss of generality.
Brute force.
The brute force approach entails two steps:
One problem with this is that the number of policies can be large, or even infinite. Another is that the variance of the returns may be large, which requires many samples to accurately estimate the return of each policy.
These problems can be ameliorated if we assume some structure and allow samples generated from one policy to influence the estimates made for others. The two main approaches for achieving this are value function estimation and direct policy search.
Value function.
Value function approaches attempt to find a policy that maximizes the return by maintaining a set of estimates of expected returns for some policy (usually either the "current" [on-policy] or the optimal [off-policy] one).
These methods rely on the theory of Markov decision processes, where optimality is defined in a sense that is stronger than the above one: A policy is called optimal if it achieves the best-expected return from "any" initial state (i.e., initial distributions play no role in this definition). Again, an optimal policy can always be found amongst stationary policies.
To define optimality in a formal manner, define the value of a policy formula_30 by
where formula_32 stands for the return associated with following formula_30 from the initial state formula_3. Defining formula_42 as the maximum possible value of formula_43, where formula_30 is allowed to change,
A policy that achieves these optimal values in each state is called "optimal". Clearly, a policy that is optimal in this strong sense is also optimal in the sense that it maximizes the expected return formula_46, since formula_47, where formula_48 is a state randomly sampled from the distribution formula_49 of initial states (so formula_50).
Although state-values suffice to define optimality, it is useful to define action-values. Given a state formula_3, an action formula_5 and a policy formula_30, the action-value of the pair formula_54 under formula_30 is defined by
where formula_32 now stands for the random return associated with first taking action formula_5 in state formula_3 and following formula_30, thereafter.
The theory of MDPs states that if formula_61 is an optimal policy, we act optimally (take the optimal action) by choosing the action from formula_62 with the highest value at each state, formula_3. The "action-value function" of such an optimal policy (formula_64) is called the "optimal action-value function" and is commonly denoted by formula_65. In summary, the knowledge of the optimal action-value function alone suffices to know how to act optimally.
Assuming full knowledge of the MDP, the two basic approaches to compute the optimal action-value function are value iteration and policy iteration. Both algorithms compute a sequence of functions formula_66 (formula_67) that converge to formula_65. Computing these functions involves computing expectations over the whole state-space, which is impractical for all but the smallest (finite) MDPs. In reinforcement learning methods, expectations are approximated by averaging over samples and using function approximation techniques to cope with the need to represent value functions over large state-action spaces.
Monte Carlo methods.
Monte Carlo methods can be used in an algorithm that mimics policy iteration. Policy iteration consists of two steps: "policy evaluation" and "policy improvement".
Monte Carlo is used in the policy evaluation step. In this step, given a stationary, deterministic policy formula_30, the goal is to compute the function values formula_70 (or a good approximation to them) for all state-action pairs formula_54. Assume (for simplicity) that the MDP is finite, that sufficient memory is available to accommodate the action-values and that the problem is episodic and after each episode a new one starts from some random initial state. Then, the estimate of the value of a given state-action pair formula_54 can be computed by averaging the sampled returns that originated from formula_54 over time. Given sufficient time, this procedure can thus construct a precise estimate formula_74 of the action-value function formula_75. This finishes the description of the policy evaluation step.
In the policy improvement step, the next policy is obtained by computing a "greedy" policy with respect to formula_74: Given a state formula_3, this new policy returns an action that maximizes formula_78. In practice lazy evaluation can defer the computation of the maximizing actions to when they are needed.
Problems with this procedure include:
1. The procedure may spend too much time evaluating a suboptimal policy.
2. It uses samples inefficiently in that a long trajectory improves the estimate only of the "single" state-action pair that started the trajectory.
3. When the returns along the trajectories have "high variance", convergence is slow.
4. It works in episodic problems only.
5. It works in small, finite MDPs only.
Temporal difference methods.
The first problem is corrected by allowing the procedure to change the policy (at some or all states) before the values settle. This too may be problematic as it might prevent convergence. Most current algorithms do this, giving rise to the class of "generalized policy iteration" algorithms. Many "actor-critic" methods belong to this category.
The second issue can be corrected by allowing trajectories to contribute to any state-action pair in them. This may also help to some extent with the third problem, although a better solution when returns have high variance is Sutton's temporal difference (TD) methods that are based on the recursive Bellman equation. The computation in TD methods can be incremental (when after each transition the memory is changed and the transition is thrown away), or batch (when the transitions are batched and the estimates are computed once based on the batch). Batch methods, such as the least-squares temporal difference method, may use the information in the samples better, while incremental methods are the only choice when batch methods are infeasible due to their high computational or memory complexity. Some methods try to combine the two approaches. Methods based on temporal differences also overcome the fourth issue.
Another problem specific to TD comes from their reliance on the recursive Bellman equation. Most TD methods have a so-called formula_79 parameter formula_80 that can continuously interpolate between Monte Carlo methods that do not rely on the Bellman equations and the basic TD methods that rely entirely on the Bellman equations. This can be effective in palliating this issue.
Function approximation methods.
In order to address the fifth issue, "function approximation methods" are used. "Linear function approximation" starts with a mapping formula_81 that assigns a finite-dimensional vector to each state-action pair. Then, the action values of a state-action pair formula_54 are obtained by linearly combining the components of formula_83 with some "weights" formula_84:
The algorithms then adjust the weights, instead of adjusting the values associated with the individual state-action pairs. Methods based on ideas from nonparametric statistics (which can be seen to construct their own features) have been explored.
Value iteration can also be used as a starting point, giving rise to the Q-learning algorithm and its many variants. Including Deep Q-learning methods when a neural network is used to represent Q, with various applications in stochastic search problems.
The problem with using action-values is that they may need highly precise estimates of the competing action values that can be hard to obtain when the returns are noisy, though this problem is mitigated to some extent by temporal difference methods. Using the so-called compatible function approximation method compromises generality and efficiency.
Direct policy search.
An alternative method is to search directly in (some subset of) the policy space, in which case the problem becomes a case of stochastic optimization. The two approaches available are gradient-based and gradient-free methods.
Gradient-based methods ("policy gradient methods") start with a mapping from a finite-dimensional (parameter) space to the space of policies: given the parameter vector formula_84, let formula_87 denote the policy associated to formula_84. Defining the performance function by
under mild conditions this function will be differentiable as a function of the parameter vector formula_84. If the gradient of formula_91 was known, one could use gradient ascent. Since an analytic expression for the gradient is not available, only a noisy estimate is available. Such an estimate can be constructed in many ways, giving rise to algorithms such as Williams' REINFORCE method (which is known as the likelihood ratio method in the simulation-based optimization literature). Policy search methods have been used in the robotics context. Many policy search methods may get stuck in local optima (as they are based on local search).
A large class of methods avoids relying on gradient information. These include simulated annealing, cross-entropy search or methods of evolutionary computation. Many gradient-free methods can achieve (in theory and in the limit) a global optimum.
Policy search methods may converge slowly given noisy data. For example, this happens in episodic problems when the trajectories are long and the variance of the returns is large. Value-function based methods that rely on temporal differences might help in this case. In recent years, "actor–critic methods" have been proposed and performed well on various problems.
Model-based algorithms.
Finally, all of the above methods can be combined with algorithms that first learn a model. For instance, the Dyna algorithm learns a model from experience, and uses that to provide more modelled transitions for a value function, in addition to the real transitions. Such methods can sometimes be extended to use of non-parametric models, such as when the transitions are simply stored and 'replayed' to the learning algorithm.
There are other ways to use models than to update a value function. For instance, in model predictive control the model is used to update the behavior directly.
Theory.
Both the asymptotic and finite-sample behaviors of most algorithms are well understood. Algorithms with provably good online performance (addressing the exploration issue) are known.
Efficient exploration of MDPs is given in Burnetas and Katehakis (1997). Finite-time performance bounds have also appeared for many algorithms, but these bounds are expected to be rather loose and thus more work is needed to better understand the relative advantages and limitations.
For incremental algorithms, asymptotic convergence issues have been settled. Temporal-difference-based algorithms converge under a wider set of conditions than was previously possible (for example, when used with arbitrary, smooth function approximation).
Research.
Research topics include:
Comparison of reinforcement learning algorithms.
Associative reinforcement learning.
Associative reinforcement learning tasks combine facets of stochastic learning automata tasks and supervised learning pattern classification tasks. In associative reinforcement learning tasks, the learning system interacts in a closed loop with its environment.
Deep reinforcement learning.
This approach extends reinforcement learning by using a deep neural network and without explicitly designing the state space. The work on learning ATARI games by Google DeepMind increased attention to deep reinforcement learning or end-to-end reinforcement learning.
Adversarial deep reinforcement learning.
Adversarial deep reinforcement learning is an active area of research in reinforcement learning focusing on vulnerabilities of learned policies. In this research area some studies initially showed that reinforcement learning policies are susceptible to imperceptible adversarial manipulations. While some methods have been proposed to overcome these susceptibilities, in the most recent studies it has been shown that these proposed solutions are far from providing an accurate representation of current vulnerabilities of deep reinforcement learning policies.
Fuzzy reinforcement learning.
By introducing fuzzy inference in RL, approximating the state-action value function with fuzzy rules in continuous space becomes possible. The IF - THEN form of fuzzy rules make this approach suitable for expressing the results in a form close to natural language. Extending FRL with Fuzzy Rule Interpolation allows the use of reduced size sparse fuzzy rule-bases to emphasize cardinal rules (most important state-action values).
Inverse reinforcement learning.
In inverse reinforcement learning (IRL), no reward function is given. Instead, the reward function is inferred given an observed behavior from an expert. The idea is to mimic observed behavior, which is often optimal or close to optimal.
Safe reinforcement learning.
Safe reinforcement learning (SRL) can be defined as the process of learning policies that maximize the expectation of the return in problems in which it is important to ensure reasonable system performance and/or respect safety constraints during the learning and/or deployment processes.


Yaw bearing

The yaw bearing is the most crucial and cost intensive component of a yaw system found on modern horizontal axis wind turbines. The yaw bearing must cope with enormous static and dynamic loads and moments during the wind turbine operation, and provide smooth rotation characteristics for the orientation of the nacelle under all weather conditions. It has also to be corrosion and wear resistant and extremely long lasting. It should last for the service life of the wind turbine) while being cost effective.
History.
Windmills of the 18th century began implementing rotatable nacelles to capture wind coming from different directions. The yaw systems of these "primitive" windmills were surprisingly similar to the ones on modern wind turbines. The nacelles rotated by means of wind driven yaw drives known as fantails, or by animal power, and were mounted on the windmill towers by means of an axial gliding bearing.
These gliding bearings consisted of multiple gliding blocks fixed on the windmill tower structure. These blocks maintained sliding contact with a gliding ring on the nacelle. The gliding blocks were wooden cube-like pieces with convex gliding surface covered with animal fat, or even lined with copper (or brass) sheet as a friction reduction means. These wooden blocks were fixed in wooden slots, carved in the wooden bearing substructure, by means of nails or wedges and were carefully leveled to create a flat surface where the nacelle gliding ring could glide. The gliding blocks, despite the lubrication would wear quite often and would have to be exchanged. This operation was relatively simple due to the wedge-based connection between substructure and gliding blocks. The gliding blocks were further locked via movable locking devices which, in a different form, remain as a technical solution in modern gliding yaw bearings.
The gliding ring of the windmill nacelle was made from multiple wooden parts and, despite the old construction techniques, was usually quite level, allowing the nacelle to rotate smoothly around the tower axis.
The "hybrid yaw bearing system" combines the solutions old windmills used. This system comprises multiple removable radial gliding pads in combination with an axial roller bearing.
Types.
The main categories of yaw bearings are:
Roller yaw bearing.
The roller yaw bearing is a common technical yaw bearing solution followed by many wind turbine manufacturers as it offers low turning friction and smooth rotation of the nacelle. The low turning friction permits the implementation of slightly smaller yaw drives (compared to the gliding bearing solution), but on the other hand requires a yaw braking system.
Some manufacturers use a plurality of smaller yaw drives (usually six) to facilitate easy replacement. Such a configuration with plurality of yaw drives often offers the possibility of active yaw braking using differential torque from the yaw drives. In this case half of the yaw drives apply a small amount of torque for clockwise rotation and the other half apply torque in the opposite direction and then activate the internal magnetic brakes of the electric motor. In this way the pinion-gear rim backlash is eliminated and the nacelle is fixed in place.
Gliding yaw bearing.
The gliding yaw bearing is a combined axial and radial bearing, which serves as a rotatable connection of the wind turbine nacelle and the tower. Contrary to the old windmill concept, the modern yaw bearings support the nacelle also from the to thus restraining the nacelle from being rotated by the Y-axis due to the moments induced by the upper half of the rotor sweep disk and the X-axis due to the torque of the drive train (i.e. rotor, shaft, generator, etc. ).
Principally, the simplest way to accomplish the yaw bearing tasks with gliding elements is with two gliding planes for the axial loads (top and bottom) and a radial gliding surface for the radial loads. Consequently, the gliding yaw bearing comprises three general surfaces covered with multiple gliding pads. These gliding pads come in sliding contact with a steel disk, which is usually equipped with gear teeth to form a gliding-disk/gear-rim. The teeth may be located at the inner or the outer cylindrical face of the disk, while the arrangement of the gliding pads and their exact number and location vary strongly among the existing designs. To assemble the gliding yaw bearings, their cages split in several segments that are assembled together during wind turbine installation or manufacturing.
In its simplest form, the gliding yaw bearing uses pads (usually made out of polymers) distributed around the three contact surfaces to provide a proper guiding system for the radial and axial movement with relatively low friction coefficient. Such systems are economical and very robust but do not allow individual adjustment of the axial and radial gliding elements. This function importantly minimizes the axial and radial "play" of the gliding bearing due to manufacturing tolerances as well as due to wear of the gliding pads during operation.
To solve this problem, yaw systems incorporate pre-tensioned gliding bearings. These bearings have gliding pads that are pressed via pressure elements against the gliding disk to stabilize the nacelle against undesirable movement. The pressure elements can be simple steel springs, pneumatic, or hydraulic pre-tension elements, etc. The use of pneumatic or hydraulic pre-tension elements allows active control of the yaw bearing pre-tension, which provides yaw brake function.
Wear and lubrication.
In all gliding bearings wear is an issue of concern, as well as lubrication. Conventional gliding yaw bearings incorporate gliding elements manufactured out of polymer plastics such as polyoxymethylene plastic (POM) or polyamide (PA). To reduce friction, wear, and avoid stick-slip effects (often present in such high friction slow moving systems), lubrication is often introduced. This solution generally solves the gliding issues, but introduces more components to the systems and increases the general complication (e.g., difficult maintenance procedures for removal of used lubricant). Some wind turbine manufacturers now use self lubricating gliding elements instead of a central lubrication system. These gliding elements are manufactured from low friction materials or composites (e.t.g polytetrafluoroethylene (Teflon)) that allow reliable operation of dry (non-lubricated) gliding yaw systems.
Maintenance and repair.
Despite the fact that the gliding yaw bearings and their components are designed and constructed to last the service life of the wind turbine, it should be possible to replace worn out yaw bearing gliding elements or other components of the yaw system. To allow for replace-ability of worn out components, the yaw systems are designed in segments. Usually one or more gliding planes comprise several sub-elements that contain a number of gliding elements (radial or axial or a combination). These sub-elements can be individually removed and repaired, re-fit or replaced. In this way the yaw bearing can be serviced without the need of dis-assembly of the whole gliding yaw bearing (e.g., in case of a roller yaw bearing, dis-assembly of the whole wind turbine). This rep-arability offered by the segmented design of the gliding yaw bearing is one of the most important advantages of this system against the roller yaw bearing solution.
The only remaining issue is the replacement of the gliding elements of the gliding yaw bearing surface, which is not segmented. This is usually the top axial surface of the gliding bearing, which constantly supports the weight of the whole nacelle-rotor assembly. For the gliding elements of this gliding surface to be replaced, the nacelle-rotor assembly must be lifted by an external crane. An alternative solution to this problem is the use of mechanical or hydraulic jacks able to partially or fully lift the nacelle-rotor assembly while the gliding yaw bearing is still in place. In this way and by providing a small clearance between the gliding elements and the gliding disk, it is possible to exchange the sliding elements without dismantling the gliding yaw bearing.
Bearing Adjustment.
When the wind turbine nacelle is positioned on the tower and the yaw bearing assembly is completed it is necessary to adjust the pressure on the individual gliding pads of the bearing. This is necessary in order to avoid un-even wear of the gliding pads and excessive loading on some sectors of the yaw bearing. In order to achieve that, an adjustment mechanism is necessary, which enables the technicians to adjust the contact pressure of each individual gliding element in a controllable and secure way. The most common solution is the utilization of bottom bearing plates equipped with large opening, which accommodate the adjustable gliding bearing systems. These adjustable gliding bearings comprise a gliding unit (i.e. gliding pad) and an adjustable pressure distribution plate. In between the gliding pad and the pressure plate several spring (pre-tension) elements are located. The vertical position of the pressure plates is usually controlled by an adjustment screw. This adjustment screw presses against the pressure plate while being retained by a counter-pressure support plate, fixed on the bearing assembly with strong bolts. In this way it is possible to apply various levels of contact pressure among the different gliding pads and therefore to ensure that each gliding component of the yaw bearing arrangement is performing as anticipated.