1 Foundations of Optimal Control
1.1 Dynamical systems and state evolution
An optimal control problem begins with a dynamical system whose evolution is described by differential or difference equations. The system’s condition at any time is summarized by a state variable, often denoted \(x(t)\) in continuous time or \(x_k\) in discrete time. Control inputs, denoted \(u(t)\) or \(u_k\), influence how the state changes through a known model such as \[ \dot{x}(t)=f(x(t),u(t),t) \quad \text{or} \quad x_{k+1}=f(x_k,u_k,k). \] This state evolution couples decisions across time: choosing \(u\) at one instant affects future states, and therefore the final outcome.
1.2 Control inputs and admissible controls
Control inputs represent what the decision-maker can manipulate—forces, actuator commands, steering rates, or allocation rates. Not all inputs are permissible. An admissible control set encodes limits like actuator bounds, rate limits, or discrete choices. Mathematically, admissibility is often expressed as \(u(t)\in \mathcal{U}\) or \(u_k\in \mathcal{U}_k\) for a specified feasible region \(\mathcal{U}\).
When the control set is time-varying or depends on the state, admissibility reflects both hardware limits and operational rules, which later become central in constrained optimization.
1.3 Objective function and performance metrics
The objective is typically written as minimizing (or maximizing) a cost functional that aggregates “undesired behavior” over time and possibly at the terminal moment. A common form is \[ J=\int_{t_0}^{t_f} \ell(x(t),u(t),t)\,dt + m(x(t_f)), \] or in discrete time, \[ J=\sum_{k=0}^{N-1} \ell(x_k,u_k,k) + m(x_N). \] The running cost \(\ell\) can penalize deviation from a desired trajectory, control effort, or constraint violations, while \(m\) can represent terminal accuracy or safety margins. Selecting a cost function is a modeling choice: it translates engineering goals into quantities that the optimizer can evaluate.
1.4 Constraints and feasibility
Constraints ensure the computed plan is physically realizable and meets requirements. They include:
- Control constraints, restricting \(u\).
- State constraints, restricting \(x\) such as avoiding obstacles or staying within allowable operating regions.
- Dynamics constraints, ensuring consistency with the model.
- Additional algebraic constraints, such as energy budgets or coupling conditions.
Feasibility refers to whether there exists at least one admissible control that satisfies all constraints. In practice, even when the formulation is mathematically valid, numerical solvers may find solutions that violate constraints unless the formulation and algorithms are designed to enforce them.
1.5 Time horizon, terminal conditions, and receding horizon ideas
Optimal control problems specify a time horizon \(t_0\) to \(t_f\) (finite horizon) or consider long-run behavior (infinite horizon). Terminal conditions may be fixed (e.g., final state must match a target) or softened through terminal costs. For systems used in real-time, a receding horizon strategy repeatedly solves an optimization over a shorter look-ahead window. After applying only the first portion of the computed control, the optimization is updated using new measurements, producing an online feedback-like behavior.
This approach is closely related to model predictive control, where the horizon controls a trade-off between solution quality and computational burden.
2 Problem Formulation
2.1 Open-loop optimal control problems
Open-loop problems seek a complete control sequence (or function) in advance, assuming a fixed initial state and full knowledge of model parameters. The optimization then yields a plan \(\{u(t)\}_{t_0\le t\le t_f}\) or \(\{u_k\}_{k=0}^{N-1}\).
2.1.1 Finite-horizon formulations
Finite-horizon settings use a specified final time \(t_f\). The cost typically includes both integral/sum terms and a terminal term. Such formulations are common when a mission ends at a known time or when a controller is designed for a specific maneuver duration.
2.1.2 Infinite-horizon formulations
Infinite-horizon problems minimize accumulated cost over an unbounded time interval. Because direct infinite sums/integrals can diverge, the formulation often uses discounted costs or assumptions ensuring convergence. Infinite-horizon models help characterize steady behavior, long-run optimality, and control laws that resemble stationary feedback.
2.2 State-constraint and control-constraint handling
Constraints can be embedded into the formulation in multiple ways:
- Hard constraints, enforced exactly by requiring \(x(t)\in \mathcal{X}\) and \(u(t)\in \mathcal{U}\).
- Soft constraints, enforced through penalty terms in the cost when strict satisfaction is undesirable or numerically difficult.
A key modeling decision is whether constraint boundaries reflect physical impossibility or merely performance degradation. Soft constraints can improve solver robustness but may lead to solutions that temporarily violate limits unless penalties are strong enough.
2.3 Linear vs. nonlinear systems in formulation
Whether the dynamics are linear or nonlinear affects both theory and computation. In linear systems, the state equation may take the form \[ \dot{x}=Ax+Bu \quad \text{(continuous)} \quad \text{or} \quad x_{k+1}=Ax_k+Bu_k \quad \text{(discrete)}. \] Nonlinear systems use general \(f(x,u,t)\). Nonlinear dynamics and nonlinear constraints create nonconvex optimization landscapes, which complicates algorithm design and can yield multiple local minima.
2.4 Deterministic vs. stochastic optimal control
Deterministic optimal control assumes the model and disturbances are known exactly. Stochastic optimal control introduces uncertainty, such as noise in dynamics or measurement. The cost then depends on random trajectories. Typical approaches optimize an expected cost or risk-sensitive criteria, with the resulting control law often requiring belief updates or probabilistic reasoning.
Even high-level differences matter: deterministic formulations can often be solved via trajectory optimization, while stochastic versions may require scenario generation, belief states, or dynamic programming over distributions.
3 Classical Solution Approaches
3.1 Variational methods and calculus of variations
Variational methods derive necessary conditions for optimality by considering small perturbations of a candidate control and examining how the cost changes. In continuous time, this leads to Euler–Lagrange-type equations or related optimality systems. The approach is foundational for understanding why costates and Hamiltonians appear in later principles.
It is most effective when the system dynamics and cost allow smooth differentiability and when boundary conditions are compatible with the variational framework.
3.2 Dynamic programming and Bellman equations
Dynamic programming rewrites the global optimization problem into smaller subproblems. Instead of optimizing over entire trajectories directly, it builds a function representing the best achievable future cost from any given state and time. This function is often called the value function \(V(x,t)\). The key recursion is captured by the Bellman principle of optimality: \[ V(x,t) = \min_{u\in \mathcal{U}} \left[\ell(x,u,t)\,dt + V(x',t+dt)\right] \] in continuous time (schematically), or its discrete-time counterpart. Solving the resulting Bellman equations or their continuous-time limits yields an optimal policy, but computation can be challenging due to the curse of dimensionality.
3.3 Pontryagin’s Maximum Principle (PMP)
PMP provides necessary conditions for optimality in continuous-time problems. Rather than directly computing the optimal control, it characterizes it through stationarity conditions derived from a Hamiltonian.
3.3.1 Hamiltonian construction
The PMP defines a Hamiltonian function that combines the running cost and the dynamics: \[ H(x,u,\lambda,t)=\ell(x,u,t)+\lambda^\top f(x,u,t), \] where \(\lambda(t)\) is a costate (adjoint) variable. The optimal control at each time maximizes or minimizes the Hamiltonian with respect to \(u\), subject to admissibility constraints.
3.3.2 Costate dynamics and optimality conditions
Along an optimal trajectory, the costate evolves according to an adjoint differential equation derived from partial derivatives of the Hamiltonian, typically of the form \[ \dot{\lambda}(t) = -\frac{\partial H}{\partial x}(x(t),u(t),\lambda(t),t), \] with terminal conditions determined by the terminal cost or constraints. Together with the state dynamics and the Hamiltonian optimality condition, these form a two-point boundary value problem. In many problems, solving this boundary value problem yields an optimal control, though practical computation often uses numerical continuation or shooting methods.
3.4 Hamilton–Jacobi–Bellman (HJB) viewpoint
The HJB approach connects dynamic programming to a partial differential equation for the value function. In continuous time, the value function \(V(x,t)\) satisfies an HJB equation that equates the time evolution of value with the best instantaneous improvement given dynamics and control choice. When solved exactly, the HJB equation provides both the value function and the optimal feedback policy.
In practice, HJB equations are difficult to solve in high dimensions, motivating approximations, numerical schemes, and alternative optimality-based approaches.
4 Optimization Methods and Numerical Techniques
4.1 Discretization strategies (time and state/control)
Numerical solution typically requires converting the continuous-time problem into a finite-dimensional optimization. Time discretization replaces the integral/sum with a finite sum, while state/control discretization represents state trajectories at grid points and control values as piecewise constants, piecewise polynomials, or other parameterizations. This transforms the original optimal control into a nonlinear program (NLP) or quadratic program (QP) depending on structure.
The fidelity of the discretization affects accuracy: finer grids increase computational cost but can reduce discretization error.
4.2 Direct methods for trajectory optimization
Direct methods discretize first and then solve for the decision variables. They typically avoid explicitly solving the costate equations by treating the entire discretized trajectory as unknowns.
4.2.1 Direct transcription (collocation and shooting variants)
Two prominent families are:
- Shooting methods, where the control is parameterized and the state is obtained by integrating the dynamics forward. The optimization adjusts control parameters so that boundary conditions and constraints are satisfied.
- Collocation methods, where the state and control are approximated on the grid, and the dynamics are enforced at selected points via algebraic constraints.
Collocation is often favored for handling path constraints and producing sparse optimization structures, while shooting can be simpler for problems where integration is stable and constraints are mild.
4.2.2 Handling constraints in nonlinear programs
When constraints are included in an NLP, solvers may use techniques such as interior-point methods, sequential quadratic programming (SQP), or active-set strategies. Inequality constraints can be enforced exactly by the solver iterations or relaxed temporarily with penalty or barrier terms. A well-posed discretization ensures that constraints on the continuous-time trajectory are represented accurately in the discrete model.
4.3 Indirect methods using necessary optimality conditions
Indirect methods solve the optimality conditions (such as those from PMP) directly. The unknowns include both states and costates, leading to a boundary value problem. Numerical solvers then search for initial costate guesses or parameters that satisfy terminal requirements.
These methods can yield high precision when the problem is well behaved, but they may be sensitive to initial guesses and less robust for strongly constrained or highly nonlinear cases.
4.4 Gradient computation and sensitivity analysis
Optimization algorithms often rely on gradients of the cost and constraints with respect to decision variables. Gradients can be obtained through:
- Analytic differentiation when expressions are manageable,
- Automatic differentiation within software frameworks,
- Adjoint sensitivity analysis, which parallels the structure of costate equations.
Accurate derivatives improve convergence speed and reliability. Sensitivity analysis also helps interpret how changes in parameters or initial conditions affect the optimal plan.
4.5 Regularization, convexification, and stabilization tactics
Nonlinear optimal control can be difficult due to nonconvexity and numerical instability. Common remedies include:
- Regularization, adding terms that improve conditioning or discourage extreme control variations.
- Convexification, approximating nonconvex dynamics or cost by convex surrogates within trust regions.
- Stabilization strategies, such as adding penalties that encourage convergence to a desired equilibrium, or using constraints/tightening that reduce solver oscillations.
These techniques aim to produce solutions that are not only locally optimal with respect to the discrete model but also stable and physically reasonable when applied.
5 Model Predictive Control (MPC)
5.1 MPC overview and online optimization loop
Model predictive control repeatedly solves an optimization problem using the current state estimate as the initial condition. At each control interval, MPC optimizes over a finite horizon, produces a sequence of predicted controls, and applies only the first input to the plant. On the next interval, the state is updated and the optimization is resolved.
This “optimize-and-apply” loop provides responsiveness to disturbances and modeling errors, making MPC a practical bridge between offline optimal control theory and online control.
5.2 Constraint satisfaction and safety considerations
Because MPC formulations typically include constraints explicitly, it can enforce actuator limits and safety-related state bounds directly. Constraint satisfaction is influenced by the prediction model and discretization. In safety-critical contexts, designers often incorporate robust margins or conservative constraint handling to account for uncertainty.
A typical objective is to maintain feasibility at all times; if feasibility cannot be guaranteed, fallback strategies or soft constraints may be introduced.
5.3 Terminal costs/constraints for stability
To improve closed-loop stability, MPC designs often include a terminal cost and/or terminal constraint. These components encourage trajectories to enter a region where local control can keep the system near the desired equilibrium. Terminal ingredients can be derived from local control design methods or stability proofs, and they reduce the risk of “end-of-horizon artifacts,” where the controller neglects what happens after the horizon ends.
5.4 Tuning trade-offs: horizon length vs. computation
The horizon length affects both performance and computational load:
- A longer horizon can capture more of the system’s future behavior and improve constraint handling.
- A shorter horizon reduces computation time, supporting higher sampling rates.
Tuning also depends on system speed, solver efficiency, and how quickly the optimization can converge within each sampling period.
5.5 Robust and stochastic MPC extensions (high-level)
Robust MPC accounts for bounded disturbances and model uncertainty by optimizing against worst-case or set-based predictions. Stochastic MPC incorporates probabilistic models of uncertainty, often optimizing expected performance or meeting chance constraints. These extensions raise computational demands but can improve reliability in uncertain environments.
High-level designs commonly use scenario sampling, tube-based approximations, or probabilistic bounds to manage uncertainty during prediction.
6 Special Cases and Benchmark Problems
6.1 Linear Quadratic Regulator (LQR)
The linear quadratic regulator addresses linear dynamics with a quadratic cost on state deviation and control effort. For continuous-time systems, \[ \dot{x}=Ax+Bu, \quad J=\int_0^{t_f}\left[(x^\top Q x)+(u^\top R u)\right]dt, \] possibly with terminal penalties. For infinite horizons under standard assumptions, the optimal control often becomes a linear state-feedback law \(u=-Kx\), where \(K\) is computed from a matrix Riccati equation.
LQR is a benchmark because it is solvable analytically or semi-analytically and provides insight into how weighting matrices shape behavior.
6.2 Linear Quadratic Gaussian (LQG) (overview)
LQG combines LQR control with a Kalman filter state estimator for linear systems with Gaussian noise. The “separation principle” implies that optimal estimation and optimal control can be designed independently: the estimator provides the best linear estimate of the state, and the LQR controller uses that estimate. LQG is widely used in benchmarks because it offers a clean structure for integrating noise filtering with regulation.
6.3 Tracking problems and reference trajectories
Tracking formulations seek to follow a desired reference trajectory rather than regulate to a fixed point. The cost penalizes deviations \(x-x_{\text{ref}}\) and \(u-u_{\text{ref}}\) (sometimes using state augmentation to handle time-varying references). Tracking introduces additional modeling considerations, such as reference generation, feedforward terms, and how to align horizons with the reference timing.
6.4 Bang-bang and time-optimal control concepts (overview)
Time-optimal control aims to reach a target state as quickly as possible, typically with control constraints. For certain systems with bounded control, the optimal policy can switch between extreme input values—hence “bang-bang” behavior. These problems are often used to study how constraint saturation influences solution structure and how nonsmoothness affects numerical methods.
Because bang-bang solutions can be sensitive to discretization, specialized algorithms and careful parameterization are sometimes needed.
6.5 Underactuated and constrained control examples
Underactuated systems have fewer independent control inputs than degrees of freedom. Examples include certain robotic mechanisms or vehicles with limited steering/force directions. Solving such problems often requires exploiting dynamics to generate motion indirectly and coordinating constraints carefully. In constrained settings, the optimization must manage feasibility while steering through reachable regions that respect both dynamics and limits.
These examples serve as benchmarks for the ability of methods to handle nontrivial controllability and constraint interactions.
7 Practical Engineering Considerations
7.1 Modeling accuracy and system identification
Optimal control performance depends strongly on model fidelity. If system parameters, friction terms, actuator dynamics, or sensor delays are inaccurate, the computed plan may deviate from reality. System identification techniques estimate model parameters from data, while model reduction and augmentation can help capture dominant behaviors without excessive complexity. Practical modeling often balances accuracy with computational tractability.
7.2 Numerical stability and solver performance
Nonlinear optimal control problems can be ill-conditioned, especially with poorly scaled variables or tight tolerances. Solver performance is influenced by scaling, constraint regularity, and the quality of initial guesses. Techniques such as variable scaling, constraint smoothing, and choosing appropriate solver settings can improve robustness and reduce iteration counts.
7.3 Initialization strategies for nonlinear optimization
Initialization strongly affects convergence in nonconvex problems. Common strategies include warm-starting from the previous MPC solution, using simplified linearized controllers, or seeding trajectories with feasible guesses produced by heuristics. A good initial trajectory reduces the chance of stagnation and helps solvers satisfy constraints early in the process.
7.4 Computation limits and real-time implementation
Real-time control requires that optimization completes within a fixed time budget. This motivates efficient discretization, exploiting sparsity in NLPs, using faster solvers, and sometimes lowering model complexity. For hardware-in-the-loop systems, computation delays can be compensated using prediction or by adjusting sampling and horizon sizes.
Implementation also involves handling measurement noise, state estimation, and actuator command constraints under timing jitter.
7.5 Validation via simulation and benchmarking
Before deployment, controllers are validated through simulation with realistic noise models, parameter variations, and constraint checks. Benchmarks compare performance metrics such as tracking error, constraint violation rates, robustness to disturbances, and computational timing. Validation ensures that the theoretical guarantees (when available) and the empirical behavior align with the intended design.
8 Applications and Use Cases (Non-Controversial Examples)
8.1 Robotics trajectory optimization (overview)
Robotics uses optimal control to compute feasible motions that avoid collisions, respect joint limits, and reach targets efficiently. In many cases, the formulation includes smoothness objectives and obstacle avoidance constraints. The output is a trajectory and control sequence that can be executed directly or tracked by a feedback controller.
Trajectory optimization also supports tasks like manipulation and legged locomotion, where dynamic feasibility is essential.
8.2 Vehicle control and motion planning (overview)
For vehicles, optimal control helps generate steering and acceleration profiles consistent with dynamics and comfort/safety constraints. Motion planning often couples path geometry with speed control, resulting in trajectories that are not only collision-free but also physically realizable. In constrained environments such as indoor navigation or constrained road segments, MPC-based schemes are common because they adapt to real-time changes.
8.3 Aerospace guidance and control (overview)
Aerospace applications use optimal control to guide trajectories under thrust limits, aerodynamic constraints, and reference tracking goals. Guidance laws frequently appear as finite-horizon planning problems with terminal conditions that align with mission phases, such as approach or intercept. Numerical optimization can incorporate constraints related to attitude, actuator capabilities, and energy usage.
8.4 Process control and automation (overview)
In chemical, thermal, and industrial processes, optimal control supports regulation of temperatures, flows, concentrations, or mixing ratios. Costs often penalize deviation from setpoints and control effort, while constraints represent safety limits or physical operating bounds. Model predictive strategies are particularly relevant where multivariable dynamics and delays matter.
8.5 Resource management and scheduling-inspired control problems
Some optimal control formulations reflect allocation decisions that evolve over time—such as distributing power, managing energy storage, or coordinating timed tasks in automated systems. While not always identical to classical continuous-time control, these problems use similar principles: dynamical evolution, objective trade-offs, and constraints. This connection broadens the reach of optimal control methods to scheduling-inspired engineering tasks.