Linear quadratic control

In control engineering and systems and control theory, linear quadratic control or LQ control refers to controller design for a deterministic (meaning that there are no elements of randomness involved) linear plant based on the minimization of a quadratic cost functional (a functional is a term for a real or complex valued function). The method is founded on the state space formalism and is a fundamental concept in linear systems and control theory.

There are two main versions of the method, depending on the setting of the control problem:

1. Discrete time linear quadratic control
2. Continuous time linear quadratic control

LQ control aims to find a control signal that minimizes a prescribed quadratic cost functional. In the so-called optimal regulator problem, this functional can be viewed as an abstraction of the "energy" of the overall control system and minimization of the functional corresponds to minimization of that energy.

Discrete time linear quadratic control

Plant model

In discrete time, the plant (the system to be controlled) is assumed to be linear with input $u_{k}$ and state $x_{k}$, and evolves in discrete time k=0,1,... according to the following dynamics:

$x_{k+1}=Ax_{k}+Bu_{k};\,x_{0}=a,\quad (1)$

where $x_{k}\in \mathbb {R} ^{n}$ and $u_{k}\in \mathbb {R} ^{m}$ for all $k\geq 0$ and $A,\,B$ are real matrices of the corresponding sizes (i.e., for consistency, $A$ should be of the dimension $n\times n$ while $B$ should be of dimension $n\times m$). Here $a$ is the initial state of the plant.

Cost functional

For a finite integer $K>0$, called the control horizon or horizon, a real valued cost functional $J$ of a and the control sequence $u_{0},u_{1},\ldots ,u_{K-1}$ up to time $K-1$ is defined as follows:

$J^{K}(a,u_{0},u_{1},\ldots ,u_{K-1})=\sum _{k=0}^{K-1}(x_{k}^{T}Qx_{k}+u_{k}^{T}Ru_{k})+x_{K}^{T}\Gamma x_{K};\,\,x_{0}=a,\quad (2)$

where $Q,\Gamma$ are given symmetric matrices (of the corresponding sizes) satisfying $Q,\Gamma \geq 0$ and $R$ is a given symmetric matrix (of the corresponding size) satisfying $R>0$. In this setting, control is executed for a finite time and the horizon $K$ represents the terminal time of the control action. However, depending on some technical assumptions on the plant, it may also be possible to allow $K\uparrow \infty$ in which case one speaks of an infinite horizon.

Note that each term on the right hand side of (2) are non-negative definite quadratic terms and may be interpreted as abstract "energy" terms (e.g., $x_{k}^{T}Qx_{k}$ as the "energy" of $x_{K}$). The term $u_{k}^{T}Ru_{k}$ accounts for penalization of the control effort. This term is necessary because overly large control signals are not desirable in general; in practice this could mean that the resulting controller cannot be implemented. The final term $x_{K}^{T}\Gamma x_{K}$ is called the terminal cost and it penalizes the energy of the plant at the final state $x_{K}$ .

The LQ regulator problem in discrete time

The objective of LQ control is to solve the optimal regulator problem:

(Optimal regulator problem) For a given horizon K and initial state a, find a control sequence ${\tilde {u}}_{0},{\tilde {u}}_{1},\ldots ,{\tilde {u}}_{K-1}\in \mathbb {R} ^{m}$ that minimizes the cost functional $J^{K}$, that is,

$J^{K}(a,{\tilde {u}}_{0},{\tilde {u}}_{1},\ldots ,{\tilde {u}}_{K-1})\leq J^{K}(a,u_{0},u_{1},\ldots ,u_{K-1});\,\,x_{0}=a,\quad (3)$

over all possible control sequences $u_{0},u_{1},\ldots ,u_{K-1}\in \mathbb {R} ^{m}$.

Thus the optimal regulator problem is a type of optimal control problem and the control sequence ${\tilde {u}}_{0},{\tilde {u}}_{1},\ldots ,{\tilde {u}}_{K-1}\in \mathbb {R} ^{m}$ is called an optimal control sequence.

Solution of optimal regulator problem

A standard approach to solving the discrete time optimal regulator problem is to use dynamic programming. In this case, a special role is played by the so-called cost-to-go functional $J_{k}^{K}$, defined as the cost from step k to step K starting at $x_{k}=b$:

$J_{k}^{K}(b,u_{k},\ldots ,u_{K-1})=\sum _{j=k}^{K-1}(x_{j}^{T}Qx_{j}+u_{j}^{T}Ru_{j})+x_{K}^{T}\Gamma x_{K};\,\,x_{k}=b.$

Another important notion is that of the value function at step k, $V_{k}$, defined as:

$V_{k}(b)=\mathop {\inf } _{u_{k},\ldots ,u_{K-1}\in \mathbb {R} }J_{k}^{K}(b,u_{k},\ldots ,u_{K-1}).$

For the optimal regulator problem it can be shown that $V_{k}$ is also a quadratic function (of the variable b). It is given by:

$V_{k}(b)\,=\,b^{T}P_{k}b,$

where $P_{k}$ is a real non-negative definite symmetric matrix satisfying the backward recursion:

$P_{k}=Q+A^{T}(P_{k+1}-P_{k+1}B(R+B^{T}P_{k+1}B)^{-1}B^{T}P_{k+1})A,\,\,k=0,\ldots ,K-1,$

with the final condition:

$P_{K}\,=\,\Gamma .$

Moreover, the infimum of $J_{k}^{K}$ for a fixed b is attainable at the optimal sequence ${\tilde {u}}_{k},\ldots ,{\tilde {u}}_{K-1}$ given by:

${\tilde {u}}_{j}=-(R+B^{T}P_{j+1}B)^{-1}B^{T}P_{j+1}A{\tilde {x}}_{j},$

where ${\tilde {x}}_{j}$, $j=k,...,K-1$, satisfies the plant dynamics with ${\tilde {x}}_{k}=b$ (it should be noted that in optimization problems a minimum/maximum is not guaranteed in general to be attained at some point in the domain of the functional to be minimized/maximized, this needs to be verified). It then follows that the minimum value of the cost functional (2) is simply:

$\mathop {\inf } _{u_{0},\ldots ,u_{K-1}\in \mathbb {R} }J^{K}(a,u_{0},u_{1},\ldots ,u_{K-1})=V_{0}(a).$

The validity of the above results depends crucially on the assumption that $R>0$, this prevents the problem from becoming "singular". It also guarantees the uniqueness of the optimal sequence (i.e., there is exactly one sequence minimizing the cost functional (2)).

Variations of discrete-time LQ control

It is possible to formulate variations of the optimal regulator problem, but which can be solved with a similar technique. For example, suppose that the dynamics (1) is augmented with an output signal $y_{k}$ given by:

$y_{k}\,=\,Cx_{k},$

where $y_{k}\in \mathbb {R} ^{p}$ and $C\in \mathbb {R} ^{p\times n}$, and the cost functional $J^{K}$ in (2) is modified to be:

${\begin{aligned}J^{K}(a,u_{0},u_{1},\ldots ,u_{K-1})&=\sum _{k=1}^{K-1}((y_{k}^{d}-y_{k})^{T}Q(y_{k}^{d}-y_{k})+u_{k}^{T}Ru_{k})+(y_{K}^{d}-y_{K})^{T}\Gamma (y_{K}^{d}-y_{K});\,\,x_{0}=a,\quad (4)\end{aligned}}$

where $y_{1}^{d},y_{1}^{d},\ldots ,y_{K}^{d}\in \mathbb {R} ^{p}$ is a given sequence of desired output signals. Then a variation of the optimal regulator problem can be formulated as finding a control sequence ${\tilde {u}}_{0},{\tilde {u}}_{1},\ldots ,{\tilde {u}}_{K-1}$ satisfying (3), but with the cost functional now being defined as in (4).

The modified problem is referred to as the output tracking LQ problem because the objective is to force the output $y_{k}$ of the plant at step k to be close to (i.e., to "track") the desired output $y_{k}^{d}$. If C is the identity matrix and $y_{k}^{d}=x_{k}^{d}\in \mathbb {R} ^{n}$ then the tracking problem can be interpreted as forcing the state of the plant to try to track the desired sequence of states $x_{1}^{d},\ldots ,x_{K}^{d}\in \mathbb {R} ^{n}$ at every step.

Continuous time linear quadratic control

An analogous formulation of LQ control as described in the above can be given in continuous time. Most of the ideas involved and techniques used in the discrete time version can be adapted and extended to the continuous time version, but since it is necessary to deal with functions instead of sequences, the latter is more technical and additional care needs to be taken. Nonetheless, it turns out that the main results are analogous to the discrete time results.

Plant model

In the continuous time setting, the plant is again assumed to be linear with input $u$ and state $x$, and evolves in $t\geq 0$ according to the following ordinary differential equation (ODE):

${\frac {dx}{dt}}(t)=Ax(t)+Bu(t);\,x(0)=a,\quad (5)$

where $x(t)\in \mathbb {R} ^{n}$ and $u(t)\in \mathbb {R} ^{m}$ for all $t\geq 0$ and $A,\,B$ are real matrices of the corresponding sizes. As before, $a$ is the initial state of the plant.

Cost functional

Let $C([a,b];\mathbb {R} ^{m})$ (a<b) be the set of all continuous $\mathbb {R} ^{m}$-valued functions on the interval [a,b]. For a positive number $T>0$, the control horizon, a real valued cost functional $J$ of a and the control signal $u\in C([0,T];\mathbb {R} ^{m})$ is defined as follows:

$J(a,u;T)=\int _{0}^{T}(x(s)^{T}Qx(s)+u(s)^{T}Ru(s))ds+x(T)^{T}\Gamma x(T);\,\,x(0)=a,\quad (6)$

where $Q,\Gamma$ are given symmetric matrices (of the corresponding sizes) satisfying $Q,\Gamma \geq 0$ and $R$ is a given symmetric matrix (of the corresponding size) satisfying $R>0$. Control is executed for a finite time and the horizon $T$ represents the terminal time of the control action. However, depending on some technical assumptions on the plant, it may also be possible to consider the infinite horizon case by letting $T\uparrow \infty$.

The LQ regulator problem in continuous time

The objective of continuous time LQ control is to solve the continuous time optimal regulator problem:

(Optimal regulator problem) For a given horizon T and initial state a, find a control signal ${\tilde {u}}\in C([0,T];\mathbb {R} ^{m})$ that minimizes the cost functional $J$, that is,

$J(a,{\tilde {u}};T)\leq J(a,u;T);\,\,x(0)=a,\quad (7)$

over all possible control signals $u\in C([0,T];\mathbb {R} ^{m})$.

The signal ${\tilde {u}}$ is then said to be an optimal control signal.

Solution of optimal regulator problem

The continuous time optimal regulator problem may also be solved by dynamic programming. In this case, a special role is played by the so-called cost-to-go functional $J(t,b,u;T)$, defined as the cost from time t to time T starting at $x(t)=b$:

$J(t,b,u;T)=\int _{t}^{T}(x(s)^{T}Qx(s)+u(s)^{T}Ru(s))ds+x(T)^{T}\Gamma x(T);\,\,x(t)=b,$

Another important notion is that of the value function at time t, $V(t,b;T)$, defined as:

$V(t,b;T)=\mathop {\inf } _{u\in C([0,T];\mathbb {R} ^{m})}J(t,b,u;T).$

For the optimal regulator problem it can be shown that $V(t,b;T)$ is also a quadratic function (of the variable b). It is given by:

$V(t,b;T)\,=\,b^{T}P(T-t)b,$

where $P(t)$ is a real non-negative definite symmetric matrix-valued function satisfying the so-called continuous Riccati differential equation:

${\frac {dP}{dt}}(t)=Q+{\frac {1}{2}}A^{T}P(t)+{\frac {1}{2}}P(t)A-P(t)BR^{-1}B^{T}P(t),\,\,0\leq t\leq T,$

with the initial condition:

$P(0)\,=\,\Gamma$.

Moreover, the infimum of $J(t,b,u;T)$ for a fixed b is attainable at the unique (due to the condition R>0) optimal control signal ${\tilde {u}}$ given by:

${\tilde {u}}(t)=-R^{-1}B^{T}P(t){\tilde {x}}(t),$

where ${\tilde {x}}\in C([t,T];\mathbb {R} ^{m})$ satisfies the plant dynamics with ${\tilde {x}}(t)=b$. It then follows that the minimum value of the cost functional (2) is simply:

$\mathop {\inf } _{u\in C([0,T];\mathbb {R} ^{m})}J(a,u;T)=V(0,a;T).$

Variations of continuous time LQ control

As in the discrete time case, an output tracking variation of the optimal regulator problem can be formulated, but which can be solved with a similar technique. For example, suppose that the dynamics (5) is augmented with an output signal $y\in C([0,T];\mathbb {R} ^{p})$ given by:

$y(t)\,=\,Cx(t),$

where $C\in \mathbb {R} ^{p\times n}$, and the cost functional $J$ in (5) is modified to be:

${\begin{aligned}J(a,u;T)&=\int _{0}^{T}((y^{d}(t)-y(t))^{T}Q(y^{d}(t)-y(t))+u(t)^{T}Ru(t))dt+(y^{d}(T)-y(T))^{T}\Gamma (y^{d}(T)-y(T));\,\,x(0)=a,\quad (8)\end{aligned}}$

where $y^{d}\in C([0,T];\mathbb {R} ^{p})$ is a given sequence of desired output signals. Then the output tracking variation of the optimal regulator problem can be formulated as finding a control sequence ${\tilde {u}}\in C([0,T];\mathbb {R} ^{m})$ satisfying (7), but with the cost functional now being defined as in (8).

Related topics

Linear quadratic Gaussian control