Jensen's inequality

1 Statement of Jensen’s inequality

Jensen’s inequality links the nonlinear behavior of convex (and concave) functions with averaging operations. In probability, the “averaging” is expectation: applying a convex function after averaging yields an upper bound, while applying a concave function after averaging yields a lower bound.

1.1 Convex-function version

Let \(X\) be a random variable and \(f:\mathbb{R}\to\mathbb{R}\) be convex. Under conditions ensuring the expectations exist, Jensen’s inequality states \[ f(\mathbb{E}[X])\le \mathbb{E}[f(X)]. \] The inequality reverses direction if \(f\) is concave.

1.2 Concave-function (reversed) version

If \(f\) is concave and expectations are well-defined, then \[ f(\mathbb{E}[X])\ge \mathbb{E}[f(X)]. \] Thus, concavity provides lower bounds for expected nonlinear transformations.

1.3 Equality conditions

Equality holds in the convex case under typical circumstances:

• If \(X\) is almost surely constant, then \(\mathbb{E}[f(X)]=f(\mathbb{E}[X])\).
• More generally, equality occurs when \(X\) concentrates on points where the convex function agrees with an affine supporting function at \(\mathbb{E}[X]\). For strictly convex \(f\), this reduces to \(X\) being almost surely constant (whenever the needed expectations are finite).

1.4 General expectation/integral form

A common formulation avoids probability language. Let \((\Omega,\mathcal{F},\mu)\) be a measure space with \(\mu(\Omega)=1\). For an integrable random variable \(X\) and convex \(f\), \[ f\!\left(\int_\Omega X\,d\mu\right)\le \int_\Omega f(X)\,d\mu. \] In this form, Jensen’s inequality applies to averages with respect to any probability measure.

2 Assumptions and technical conditions

Jensen’s inequality is conceptually simple, but technical hypotheses ensure expressions are meaningful. In practice, one verifies measurability and integrability, and handles functions defined on extended real values when necessary.

2.1 Measurability requirements

To form \(\mathbb{E}[X]\) and \(\mathbb{E}[f(X)]\), the mapping \(X:\Omega\to\mathbb{R}\) must be measurable, and \(f\circ X\) must be measurable. If \(f\) is defined on an interval containing the values of \(X\) almost surely, measurability is inherited from \(X\).

2.2 Integrability and moment existence

A standard sufficient condition for the basic inequality \(f(\mathbb{E}[X])\le \mathbb{E}[f(X)]\) with real-valued \(f\) is:

- \(\mathbb{E}[	X	]<\infty\) (so \(\mathbb{E}[X]\) exists as a finite real number), and
- \(\mathbb{E}[	f(X)	]<\infty\) or at least \(\mathbb{E}[f(X)]\) is not of the indeterminate form \(\infty-\infty\).

Some applications only require the inequality in an “extended” sense; those cases are handled using monotone convergence and the extended-real formulation below.

2.3 Random variables with discrete versus continuous distributions

The inequality holds uniformly across distribution types. For discrete \(X\), it becomes a weighted convexity statement: \[ f\Big(\sum_i p_i x_i\Big)\le \sum_i p_i f(x_i). \] For continuous \(X\), it is expressed through integrals with respect to the distribution measure. The same convexity mechanism underlies both cases.

2.4 Handling extended real values

If \(f\) may take values \(+\infty\) (common for convex functions defined via barriers or constraints), one works with extended-real expectations. A typical statement is that if \(f\) is convex and \(f(X)\) is measurable, then—when the left-hand side is well-defined—Jensen’s inequality holds in the extended sense: \[ f(\mathbb{E}[X])\le \mathbb{E}[f(X)], \] where \(\mathbb{E}[f(X)]\) may be \(+\infty\). Care is taken to avoid situations where both sides are undefined or where the expectation becomes indeterminate.

3 Connections to convex analysis

Jensen’s inequality can be viewed as a manifestation of fundamental geometry of convex functions. Its probabilistic statement is, at heart, a convex-analytic fact about how convex functions behave under averaging.

3.1 Convexity definitions and characterizations

A function \(f\) is convex if for all \(x,y\) and \(t\in[0,1]\), \[ f(tx+(1-t)y)\le t f(x)+(1-t)f(y). \] This definition extends from finite convex combinations to integrals by approximation. Conversely, Jensen-type inequalities can be used to characterize convexity in certain settings: if \(f\) satisfies Jensen for all suitable random variables, it must be convex.

3.2 Supporting lines and subgradients

For convex functions, at points in the interior of the effective domain there exist supporting affine functions. In one dimension, a subgradient \(g\) at \(m\) yields \[ f(x)\ge f(m)+g(x-m). \] This is the backbone of many proofs: take expectation of the supporting inequality and then substitute \(m=\mathbb{E}[X]\).

3.3 Alternative proofs via tangents

Because convex functions lie above their tangents (supporting lines), one can prove Jensen’s inequality by comparing \(f(X)\) to an affine function determined at \(f(\mathbb{E}[X])\). The “tangent” approach generalizes to higher dimensions via supporting hyperplanes and subgradients.

3.4 Relation to epigraphs and geometric interpretation

The epigraph of a function, \[ \mathrm{epi}(f)=\{(x,r): r\ge f(x)\}, \] is a convex set precisely when \(f\) is convex. Averaging points corresponds to taking convex combinations, so the convexity of the epigraph translates directly into Jensen’s inequality: the averaged point remains in the epigraph, leading to the desired inequality.

4 Proof techniques

Multiple proof strategies exist, each highlighting a different aspect of convexity or probability. The most direct proofs reduce the inequality to convexity for finite mixtures and then pass to limits.

4.1 Proof using the supporting hyperplane (tangent) method

Assume \(f\) is convex and finite at \(m=\mathbb{E}[X]\). Choose a subgradient \(g\in\partial f(m)\) so that \[ f(x)\ge f(m)+g(x-m)\quad \text{for all }x. \] Substituting \(x=X\) gives \[ f(X)\ge f(m)+g(X-m). \] Taking expectations yields \[ \mathbb{E}[f(X)]\ge f(m)+g(\mathbb{E}[X]-m)=f(m)=f(\mathbb{E}[X]). \] This establishes the convex-function version.

4.2 Proof via conditional expectation (tower property)

Jensen’s inequality can be applied “conditionally,” then averaged. If \(f\) is convex and \(X\) is integrable, one can show \[ f(\mathbb{E}[X\mid\mathcal{G}])\le \mathbb{E}[f(X)\mid\mathcal{G}]. \] Taking expectations over \(\mathcal{G}\) gives \[ \mathbb{E}\big[f(\mathbb{E}[X\mid\mathcal{G}])\big]\le \mathbb{E}[f(X)]. \] Using \(\mathbb{E}[\mathbb{E}[X\mid\mathcal{G}]]=\mathbb{E}[X]\) then recovers the unconditional inequality as a special case.

4.3 Proof using the secant-line/finite Jensen argument

First verify Jensen for discrete distributions where \(X\) takes values \(x_i\) with probabilities \(p_i\). Convexity gives \[ f\Big(\sum_i p_i x_i\Big)\le \sum_i p_i f(x_i). \] For general distributions, approximate \(X\) by simple functions (finite-valued random variables) and use the fact that convex functions behave well under pointwise limits in combination with dominated or monotone convergence, as permitted by integrability conditions.

4.4 Proof via approximation (simple functions, limiting arguments)

Let \(X\) be integrable and \(f\) convex. Construct a sequence of simple random variables \(X_n\) such that \(X_n\to X\) almost surely and \(\mathbb{E}[X_n]\to\mathbb{E}[X]\). Apply Jensen to each \(X_n\): \[ f(\mathbb{E}[X_n])\le \mathbb{E}[f(X_n)]. \] Then pass to the limit using continuity properties of convex functions on the relevant domain, together with dominated convergence or appropriate monotone convergence arguments for the expectations of \(f(X_n)\).

5 Multivariate Jensen’s inequality

Jensen’s inequality extends beyond real-valued variables. In multivariate settings, convexity is defined on \(\mathbb{R}^n\) (or more general vector spaces), and expectations are applied componentwise or via vector-valued convex functions.

5.1 Vector-valued random variables

Let \(X\) be a random vector in \(\mathbb{R}^n\) with \(\mathbb{E}[\|X\|]<\infty\). For a convex function \(f:\mathbb{R}^n\to\mathbb{R}\), the inequality takes the form

\[ f(\mathbb{E}[X])\le \mathbb{E}[f(X)]. \] Here \(\mathbb{E}[X]\) is the vector of coordinate-wise expectations.

5.2 Convex functions on \(\mathbb{R}^n\)

Multivariate convexity means that for all \(x,y\in\mathbb{R}^n\) and \(t\in[0,1]\), \[ f(tx+(1-t)y)\le t f(x)+(1-t)f(y). \] The geometric supporting hyperplane method generalizes: a supporting hyperplane to \(\mathrm{epi}(f)\) at \(\mathbb{E}[X]\) provides a linear lower bound whose expectation yields the inequality.

5.3 Weighted Jensen inequality

A weighted variant appears when averaging with respect to probability weights \(p_i\) that sum to one, or more generally with respect to a measure whose total mass is one. For convex \(f\) and points \(x_i\), \[ f\Big(\sum_i p_i x_i\Big)\le \sum_i p_i f(x_i). \] This form is often used in mixture models and in proofs where one “mixes” distributions or scenarios.

5.4 Matrix/tensor variants (conceptual overview)

In matrix analysis, Jensen-type statements hold for convex functions acting on eigenvalues or on positive-semidefinite matrices, typically expressed through operator convexity. The core idea remains similar: a convexity notion ensures that applying the function after averaging (in an operator sense) yields a bound by the average of the function. Tensor variants follow analogous principles, though their precise formulation depends on the adopted notion of convexity and the averaging operation.

6 Conditional Jensen’s inequality

Conditional versions allow one to bound nonlinear transformations under partial information. They are central in martingale theory, stochastic control, and iterative algorithms.

6.1 Statement and basic use

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space and \(\mathcal{G}\subseteq\mathcal{F}\) a sub-\(\sigma\)-algebra. If \(f\) is convex and expectations are finite enough, then \[ f(\mathbb{E}[X\mid\mathcal{G}])\le \mathbb{E}[f(X)\mid\mathcal{G}]\quad \text{a.s.} \] This inequality becomes a tool for producing bounds that hold pointwise with respect to the information in \(\mathcal{G}\).

6.2 Iterated conditioning and the tower property

Using \(\mathbb{E}[\mathbb{E}[Y\mid\mathcal{H}]\mid\mathcal{G}]=\mathbb{E}[Y\mid\mathcal{G}]\) for \(\mathcal{H}\supseteq\mathcal{G}\), conditional Jensen can be iterated. In practice, one applies Jensen conditionally at one stage, then conditions again, leading to progressively tighter bounds or to recursive inequalities that are compatible with dynamic systems.

6.3 Applications to random transformations

Conditional Jensen is especially useful when \(X\) is a nonlinear function of underlying randomness that becomes partially revealed by \(\mathcal{G}\). For instance, if \(X=g(Z)\) for some random input \(Z\), conditioning on \(\mathcal{G}\) can reduce uncertainty, and convexity then bounds the transformed conditional mean by the conditional mean of the transformed quantity.

6.4 Implications for martingales and supermartingales

In martingale contexts, conditional Jensen provides a convexity-based way to generate sub- or supermartingales. If \((X_t)\) is a martingale and \(f\) is convex, then under suitable integrability assumptions, the process \(f(X_t)\) behaves like a submartingale: roughly, conditioning does not decrease the convex transform’s expectation. Analogous statements hold for concave functions and for supermartingales, forming a standard bridge between convex analysis and stochastic processes.

7 Common applications in probability

Jensen’s inequality is widely used because it turns a difficult expectation of a nonlinear function into a bound involving a simpler expression. Typical applications involve nonlinear moments, logarithms, exponentials, and convex penalties.

7.1 Moment bounds and inequalities

Convex functions generate bounds on moments. For example, choosing \(f(x)=x^p\) for appropriate regimes (where convexity holds) yields inequalities that relate \(\mathbb{E}[	X	^p]\) to lower moments or to functions of \(\mathbb{E}[	X	]\). While Jensen alone does not give sharp tail bounds, it often provides foundational estimates used alongside other tools.

7.2 Risk and loss modeling (general, non-political framing)

In statistical learning and decision theory, loss functions are often convex, and risk is expressed as an expectation of loss. Jensen’s inequality then implies that evaluating the loss at the mean prediction underestimates or overestimates the average loss depending on whether the loss is convex or concave. This supports the rationale behind risk minimization and encourages the use of convex surrogate objectives.

7.3 Estimation and statistical inference bounds

In estimation problems, Jensen’s inequality appears when parameters enter models through nonlinear transformations. For instance, expectations of functions of estimators or likelihood components can be bounded by applying convexity to the relevant transform. Such bounds help compare estimators, justify approximations, and obtain inequalities needed in proofs of consistency or convergence.

7.4 Lyapunov-type inequalities and stability arguments

In stochastic stability analyses, one often studies a nonnegative “Lyapunov” function \(V(X_t)\) whose dynamics involve expectations. If \(V\) is convex and the update has a conditional mean structure, conditional Jensen can bound \(\mathbb{E}[V(X_{t+1})\mid \mathcal{F}_t]\) by \(V(\mathbb{E}[X_{t+1}\mid\mathcal{F}_t])\) or related quantities. This can yield drift conditions supporting stability or boundedness of trajectories in distribution.

8 Worked examples

Concrete applications clarify how to select a convex function and interpret the inequality. The following examples illustrate common convex choices and demonstrate how Jensen produces tractable bounds.

8.1 Applying Jensen to common convex functions (e.g., \(e^x\), \(-\log x\))

• For \(f(x)=e^x\), which is convex on \(\mathbb{R}\),

\[ e^{\mathbb{E}[X]}\le \mathbb{E}[e^{X}]. \] This inequality is frequently used to relate moment generating functions to exponentials of means.

- For \(f(x)=-\log x\), convex on \((0,\infty)\), and \(X>0\) almost surely with \(\mathbb{E}[	\log X	]<\infty\),

\[ -\log(\mathbb{E}[X])\le \mathbb{E}[-\log X], \] equivalently \[ \log(\mathbb{E}[X])\ge -\mathbb{E}[\log X]. \] This compares the log of an average with the average of logs, a recurring theme in information-theoretic and multiplicative settings.

8.2 Bounding expectations of nonlinear transforms

If \(f\) is convex and integrable, Jensen converts \(\mathbb{E}[f(X)]\) into a bound based on \(f(\mathbb{E}[X])\). For example, with \(f(x)=x^2\) (convex), \[ (\mathbb{E}[X])^2\le \mathbb{E}[X^2]. \] This is a simple but fundamental illustration: the inequality implies that mean-square dominates the square of the mean.

8.3 Comparing distributions via convex ordering

When comparing two random variables \(X\) and \(Y\) with the same mean, convex order can be characterized using Jensen-like inequalities. If \[ \mathbb{E}[\phi(X)]\le \mathbb{E}[\phi(Y)] \] for all convex functions \(\phi\) in a class, then \(X\) is “smaller” than \(Y\) in convex order. Jensen’s inequality provides the intuition: averaging under convex functions increases with spread, so distributions with greater variability tend to yield larger expectations of convex transforms.

8.4 Example calculations with discrete distributions

Let \(X\) take values \(\{0,2\}\) with probabilities \(\{1/2,1/2\}\). Then \(\mathbb{E}[X]=1\). With \(f(x)=e^x\), \[ e^{\mathbb{E}[X]}=e^1=e,\qquad \mathbb{E}[e^X]=\tfrac12 e^0+\tfrac12 e^2=\tfrac12(1+e^2). \] Since \(e\le \tfrac12(1+e^2)\), Jensen is verified. This example highlights how convexity makes \(\mathbb{E}[f(X)]\) exceed the function evaluated at the averaged value.

9 Related results and comparisons

Jensen’s inequality connects to several classic inequalities in analysis and probability. Many of these can be derived from convexity principles or share the same underlying geometric intuition.

9.1 Relation to the inequality of arithmetic and geometric means

The arithmetic–geometric mean (AM–GM) inequality is a special case of Jensen using the concave logarithm. Since \(\log x\) is concave, \[ \log(\mathbb{E}[X]) \ge \mathbb{E}[\log X], \] which rearranges (for discrete or uniform weighting) into AM–GM: \[ \frac{x_1+\cdots+x_n}{n}\ge (x_1\cdots x_n)^{1/n}. \]

9.2 Connection to Kullback–Leibler divergence and log-sum inequality

The log-sum inequality is closely tied to convexity of \(-\log\) and can be viewed through Jensen-type arguments. In information theory, nonnegativity of Kullback–Leibler divergence is typically proved using convexity (often via log-sum) rather than through Jensen directly, but the conceptual mechanism is the same: a convex transform of ratios yields an expectation bounded below by zero.

9.3 Relationship to Hölder’s and Minkowski’s inequalities

Hölder’s and Minkowski’s inequalities concern norms and products of functions. While their most standard proofs use measure theory and integral manipulations, convexity plays a role in their conceptual foundation. Jensen’s inequality can appear as a component in broader derivations, especially when comparing averages and transforming integrals using convexity of power functions.

9.4 Convex order and dominance concepts

Convex ordering between random variables is an example of how Jensen extends beyond single inequalities to comparison frameworks. If one random variable dominates another in convex order, then expectations of all convex functions respect that dominance. Jensen’s inequality provides local evidence of this principle by showing how convex functions respond to averaging and by quantifying how equality depends on degeneracy.

10 Practical checklist for using Jensen

Successful use of Jensen’s inequality is less about memorizing the statement and more about checking the conditions and choosing a function that matches the desired bound.

10.1 Choosing the convex function correctly

Identify the target nonlinear quantity you want to bound and select \(f\) so that:

• \(f\) is convex (for an upper bound \(f(\mathbb{E}[X])\le \mathbb{E}[f(X)]\)), or concave (for the reverse), and
• \(f\) is applied to a domain compatible with the random variable’s values (e.g., \(-\log x\) requires positivity).

10.2 Verifying integrability

Confirm that \(\mathbb{E}[X]\) is finite whenever you need \(f(\mathbb{E}[X])\) as a real number. Also ensure \(\mathbb{E}[f(X)]\) exists (not an indeterminate infinite difference). If the model naturally yields extended values, verify the inequality in the extended-real framework.

10.3 Interpreting equality cases

When the inequality becomes equality, interpret it in terms of degeneracy or alignment with supporting affine functions. For strictly convex \(f\), equality usually forces \(X\) to be almost surely constant, which can be a useful diagnostic in proofs and model checking.

10.4 Avoiding common pitfalls in application

Common mistakes include:

• using a function that is not actually convex on the relevant range;
• applying Jensen when expectations are undefined (for example, ignoring heavy tails that make \(\mathbb{E}[f(X)]\) infinite);
• confusing conditional and unconditional forms (ensuring the conditioning \(\sigma\)-algebra is specified correctly);
• overlooking domain constraints (such as applying \(-\log X\) when \(X\) can be nonpositive).