Total variation distance

1 Definition and Core Properties

1.1 Formal definition for distributions on a measurable space

Let \((\Omega,\mathcal{F})\) be a measurable space, and let \(P\) and \(Q\) be probability measures on it. The total variation distance between \(P\) and \(Q\) is \[

\\|P-Q\\|_{\mathrm{TV}} \;=\; \sup_{A\in\mathcal{F}}	P(A)-Q(A)	.

\] Equivalently, it measures the maximum discrepancy over all measurable events.

1.2 Discrete case: relationship to L1 distance

1.2.1 Expression using the maximum over events

If \(\Omega\) is finite or countable and \(P(\omega)\), \(Q(\omega)\) are probability mass functions, then for any event \(A\subseteq\Omega\), \[ P(A)-Q(A) = \sum_{\omega\in A}\big(P(\omega)-Q(\omega)\big). \] Taking the supremum over all subsets \(A\) yields the same definition as above, now over all subsets of the support.

1.2.2 Expression using absolute differences of probabilities

For discrete distributions, total variation admits the familiar closed form: \[

\\|P-Q\\|_{\mathrm{TV}} \;=\; \frac12 \sum_{\omega\in\Omega}	P(\omega)-Q(\omega)	.

\] The factor \(1/2\) arises because the positive and negative parts of \(\{P(\omega)-Q(\omega)\}\) sum to the same magnitude.

1.3 Basic metric properties

1.3.1 Non-negativity and identity of indiscernibles

For all \(P,Q\), \(\\|P-Q\\|_{\mathrm{TV}}\ge 0\). Moreover, \(\\|P-Q\\|_{\mathrm{TV}}=0\) holds if and only if \(P(A)=Q(A)\) for every measurable event \(A\), meaning the measures are equal.

1.3.2 Symmetry

Total variation is symmetric: \[

\\|P-Q\\|_{\mathrm{TV}} = \\|Q-P\\|_{\mathrm{TV}}.

\] This follows directly from the absolute value in the eventwise difference.

1.3.3 Triangle inequality

Total variation satisfies the triangle inequality. For probability measures \(P,Q,R\), \[

\\|P-R\\|_{\mathrm{TV}} \le \\|P-Q\\|_{\mathrm{TV}} + \\|Q-R\\|_{\mathrm{TV}}.

One proof route uses the supremum-over-events definition: for each event \(A\), \(	P(A)-R(A)	\le	P(A)-Q(A)	+	Q(A)-R(A)	\), then take the supremum over \(A\).

2 Interpretation as Distinguishability

2.1 Hypothesis testing viewpoint

Total variation can be interpreted as the maximum advantage obtainable when deciding whether data arose from \(P\) or from \(Q\). When the two hypotheses are equiprobable and one uses an optimal decision rule, the best achievable success probability depends on \(\\|P-Q\\|_{\mathrm{TV}}\).

2.2 Maximum event discrepancy (“worst-case” event)

2.2.1 Connection to decision rules

Consider any measurable decision rule that maps outcomes to one of the two hypotheses. Such a rule effectively partitions the space into “accept \(P\)” versus “accept \(Q\)” regions. Under suitable balancing assumptions, the difference between the two hypotheses’ acceptance probabilities is maximized by choosing the event that attains (or nearly attains) \(\sup_{A}	P(A)-Q(A)	\). Hence total variation represents the most challenging event to distinguish.

2.2.2 Connection to decision rules

More concretely, many optimal or near-optimal tests can be written using an event \(A\) of the form “declare \(P\)” when the observation falls in \(A\). The resulting performance gap is exactly \(	P(A)-Q(A)	\), so the best possible gap corresponds to the supremum over all \(A\).

2.3 Coupling interpretation

2.3.1 Infimum over coupling mismatch probabilities

A coupling of \((X,Y)\) with laws \(P\) and \(Q\) is a joint distribution on \(\Omega\times\Omega\) whose marginals are \(P\) and \(Q\). Total variation admits the representation \[

\\|P-Q\\|_{\mathrm{TV}} \;=\; \inf_{\text{couplings }(X,Y)} \Pr[X\neq Y].

\] Thus, the distance equals the smallest mismatch probability achievable by correlating the two random variables as favorably as possible.

3.1 Relation to Lp norms (where applicable)

On a finite space, total variation is closely tied to the \(L^1\) norm. Writing \(p(\omega)=P(\omega)\) and \(q(\omega)=Q(\omega)\), the formula \[

\\|P-Q\\|_{\mathrm{TV}}=\tfrac12\\|p-q\\|_1

\] shows that, up to scaling, it is the \(L^1\) distance between the probability vectors. On general measurable spaces, analogous relationships appear via norms of Radon–Nikodym derivatives where they exist.

3.2 Comparison with other metrics/divergences

3.2.1 Total variation vs. Kullback–Leibler divergence

The Kullback–Leibler (KL) divergence \[

D_{\mathrm{KL}}(P\\|Q)=\int \log\!\left(\frac{dP}{dQ}\right)dP

\] is an information-theoretic quantity sensitive to relative densities and requires absolute continuity (or yields \(+\infty\)). Total variation is bounded between \(0\) and \(1\) and does not require densities. In practice, KL can be large even when total variation is moderate, and conversely small total variation does not necessarily control KL without additional assumptions.

3.2.2 Total variation vs. Hellinger distance

The Hellinger distance (often defined via Hellinger affinity) uses square-root density behavior: \[ H^2(P,Q)=\frac12\int\left(\sqrt{dP}-\sqrt{dQ}\right)^2. \] Like total variation, it yields a measure of discrepancy, but it weights discrepancies differently, especially in regions where densities are small. Various inequalities relate Hellinger distance and total variation, providing cross-compatibility between probabilistic and statistical bounds.

3.3 Pinsker-type inequalities (high-level overview)

3.3.1 Upper/lower bounds involving divergences

Pinsker-type results connect total variation to divergences such as KL. Typically, they provide an inequality of the form “total variation is at most (or at least) a function of KL,” showing that small KL implies small total variation. Such statements are useful for converting information-theoretic estimates into probabilistic distinguishability guarantees.

4 Convergence and Limits

4.1 Convergence in total variation

A sequence \(P_n\) converges to \(P\) in total variation if \[

\\|P_n-P\\|_{\mathrm{TV}} \to 0.

\] This is a strong notion: it forces \(P_n(A)\to P(A)\) uniformly over all measurable events \(A\).

4.2 Implications for convergence in distribution

Total variation convergence implies convergence in distribution (weak convergence) in common settings, because it controls differences on all events. In contrast, convergence in distribution alone does not guarantee total variation convergence, especially when mass can “move” in ways detectable only through finer event classes.

4.3 Stability under transformations and marginalization

4.3.1 Contraction under measurable mappings

Let \(T:\Omega\to\Omega'\) be measurable. If \(P^T\) and \(Q^T\) denote the pushforwards (laws of \(T(X)\) when \(X\sim P\) and similarly for \(Q\)), then total variation cannot increase: \[

\\|P^T-Q^T\\|_{\mathrm{TV}} \le \\|P-Q\\|_{\mathrm{TV}}.

\] This contraction property expresses that applying a measurable mapping can only lose information about whether the original distributions differed.

5 Computing and Bounding Total Variation

5.1 Exact computation for common distributions (overview)

5.1.1 Finite state spaces

For distributions supported on a finite set, computing total variation reduces to evaluating \[

\tfrac12\sum_{\omega}	P(\omega)-Q(\omega)	.

\] In many applied problems, this summation can be carried out directly; in others, symmetry or sparsity simplifies the computation.

5.1.2 Bernoulli/Binomial-type comparisons

For Bernoulli distributions with parameters \(p\) and \(q\), \[

\\| \mathrm{Bern}(p)-\mathrm{Bern}(q)\\|_{\mathrm{TV}} =	p-q	.

\] For binomial and related families, exact expressions often involve summing absolute differences over the discrete support; however, one can frequently obtain sharp bounds using tail probabilities and monotonicity with respect to parameters.

5.2 Bounding techniques

5.2.1 Union bounds and event decomposition

Since total variation is a supremum over event discrepancies, it is often bounded by analyzing a collection of events whose probabilities under \(P\) and \(Q\) can be controlled. Decomposing the sample space into “good” and “bad” regions is a typical strategy: total variation is then bounded by the maximum mismatch on the good set plus the probability mass of the bad set under both measures.

5.2.2 Moment/coupling-based bounds (conceptual)

Bounding via couplings leverages the identity \[

\\|P-Q\\|_{\mathrm{TV}} = \inf \Pr[X\neq Y].

\] Instead of optimizing over all couplings, one proposes a natural joint construction and bounds the mismatch probability. This approach is central in probabilistic proofs where the coupling is designed to make the two processes track each other for long times.

5.3 TV distance for mixtures and conditional distributions

5.3.1 Bounding differences via conditioning

When distributions are formed as mixtures, total variation bounds can be obtained by conditioning on the mixing variable. A typical idea is that discrepancy is reduced after conditioning on information that explains differences between the component laws; one then averages or bounds the resulting conditional total variation distances, often using convexity-type properties.

6 Applications

6.1 Markov chains and mixing time intuition

6.1 Distance to stationarity

For an ergodic Markov chain with stationary distribution \(\pi\), the law of the chain after \(t\) steps, denoted \(P^t(x,\cdot)\), can be compared to \(\pi\) using total variation: \[

\\|P^t(x,\cdot)-\pi\\|_{\mathrm{TV}}.

\] This quantity indicates how well the chain has “forgotten” its starting state and provides a direct way to quantify mixing.

6.2 Approximation and randomized algorithms

Many randomized algorithms approximate a target distribution by simulating a process whose output law is close in total variation. If \(\\|P_{\text{alg}}-P_{\text{target}}\\|_{\mathrm{TV}}\) is small, then any downstream probabilistic event has nearly the same probability under both, yielding robust error control for Monte Carlo-style methods.

6.3 Statistical inference and confidence in model separation

6.3.1 Risk of misclassification under distribution shift

In inference problems where the true generating mechanism may differ from the assumed one, total variation helps assess the worst-case change in event probabilities. When a classifier or estimator is trained under one distribution and evaluated under another, the misclassification risk can be related to \(\|P-Q\|_{\mathrm{TV}}\), since it governs the largest possible deviation in probabilities of the events that drive the decision.