Channel capacity is a fundamental concept in information theory, introduced by Claude Shannon in 1948. It defines the maximum rate at which information can be reliably transmitted over a communication channel, given a specified noise or interference level. Mathematically, channel capacity is the supremum of mutual information between input and output over all possible input probability distributions, expressed in bits per channel use. It serves as an upper bound for error‑free data transmission and underpins the design of modern coding and modulation schemes.
1 Introduction
1.1 Historical background
The concept of channel capacity arose from Shannon’s 1948 paper *A Mathematical Theory of Communication*, which laid the foundations of information theory. Prior to Shannon, engineers had only empirical rules for telegraph and telephone line capacity. Shannon provided a rigorous mathematical framework, showing that every communication channel has a definite maximum transmission rate beyond which reliable communication is impossible. This result was revolutionary and spurred the development of digital communications.
1.2 Relation to Shannon's noisy-channel coding theorem
Shannon’s noisy‑channel coding theorem states that for any channel with capacity *C* and any transmission rate *R* < *C*, there exists a sequence of codes that makes the probability of decoding error arbitrarily small. Conversely, if *R* > *C*, reliable communication is impossible. This theorem establishes channel capacity as the fundamental limit on the rate of error‑free data transmission and motivates the search for practical codes that approach this bound.
2 Mathematical definition
2.1 Discrete memoryless channels
| A discrete memoryless channel (DMC) is characterized by a finite input alphabet *𝒳*, a finite output alphabet *𝒴*, and a set of conditional probabilities *p(y | x)* that are independent of previous inputs and outputs. The channel is “memoryless” because the probability of an output depends only on the current input. |
|---|
2.1.1 Formal expression using mutual information
The mutual information *I(X;Y)* between the input random variable *X* and the output *Y* is given by:
\[ I(X;Y) = \sum_{x\in\mathcal{X}}\sum_{y\in\mathcal{Y}} p(x,y) \log_2 \frac{p(x,y)}{p(x)p(y)} \]
where *p(x,y)* is the joint probability distribution and *p(x)*, *p(y)* are the marginal distributions. The logarithm base 2 expresses information in bits.
2.1.2 Capacity as maximization over input distributions
Channel capacity *C* for a DMC is defined as the maximum mutual information over all possible input distributions *p(x)*:
\[ C = \max_{p(x)} I(X;Y) \]
This optimization captures the best way to assign probabilities to input symbols to maximize the reliable transmission rate.
2.2 Continuous channels
For channels with continuous alphabets (e.g., real‑valued signals), the same definition applies using differential entropy. The input and output are real random variables, and mutual information becomes an integral over probability densities.
2.2.1 Additive white Gaussian noise (AWGN) channel
In the AWGN channel, the output is the sum of the input and a Gaussian noise sample: *Y = X + N*, where *N* is zero‑mean Gaussian with variance *σ²*. This is a fundamental model for many communication systems (e.g., satellite links, wired transmission).
2.2.1.1 Shannon–Hartley theorem
For an AWGN channel with bandwidth *B* (Hz) and signal‑to‑noise ratio *SNR = P/N₀B* (where *P* is average signal power and *N₀* is noise power spectral density), the capacity is:
\[ C = B \log_2\left(1 + \mathrm{SNR}\right) \quad \text{bits/second} \]
This is the Shannon–Hartley theorem. It shows that capacity grows logarithmically with power and linearly with bandwidth.
2.2.2 Bandlimited channels
Bandlimited channels restrict the input to a finite frequency range. The Shannon–Hartley theorem applies directly when the channel is ideal (flat gain within the band and zero outside). Practical channels may have non‑ideal frequency responses, requiring more complex capacity analysis.
2.3 Channels with memory
When channel noise or interference depends on previous inputs or outputs, the channel has memory. The capacity is generally harder to compute and may require consideration of sequences rather than individual symbols.
2.3.1 Finite-state channels
Finite‑state channels (FSCs) model channels with a finite number of internal states (e.g., fading channels with a finite number of fading levels). The capacity is given by the supremum over all input sequences of the limit of the mutual information per symbol, often requiring techniques such as the “blahut‑arimoto algorithm” adapted for FSCs.
2.3.2 Feedback capacity
When the transmitter knows the channel output (feedback), the capacity can sometimes be increased. For memoryless channels, feedback does not increase capacity, but for channels with memory, it can. The feedback capacity is the maximum rate achievable with feedback, which may be larger than the feed‑forward capacity.
3 Types of channel models
3.1 Binary symmetric channel (BSC)
The BSC is a discrete memoryless channel with binary input and output (0 or 1). With probability *p* (the crossover probability) the output is the opposite of the input; otherwise it is correct.
3.1.1 Capacity formula and crossover probability
The capacity of the BSC is:
\[ C = 1 - H(p) = 1 - \left[ p \log_2 \frac{1}{p} + (1-p) \log_2 \frac{1}{1-p} \right] \quad \text{bits per channel use} \]
where *H(p)* is the binary entropy function. When *p=0.5*, the capacity becomes zero; when *p=0*, *C=1* bit.
3.2 Binary erasure channel (BEC)
The BEC has binary inputs and three possible outputs: 0, 1, or *e* (erasure). With erasure probability *ε*, the input symbol is erased; otherwise it is received correctly.
3.2.1 Erasure probability and capacity
The capacity of the BEC is:
\[ C = 1 - \varepsilon \quad \text{bits per channel use} \]
This simple linear form makes the BEC a popular model for analyzing modern codes like LDPC and polar codes.
3.3 M-ary channels
In M‑ary channels, the input alphabet has size *M* (>2). Common examples are M‑PSK or M‑QAM in digital modulation.
3.3.1 M-ary symmetric channel
The M‑ary symmetric channel (MSC) generalizes the BSC: each input symbol is correctly received with probability *1‑p*, and with probability *p* it is equally likely turned into any of the other *M‑1* symbols. Its capacity is:
\[ C = \log_2 M - H(p) - p \log_2(M-1) \]
where *H(p)* is the binary entropy.
3.3.2 M-ary orthogonal signaling
In orthogonal signaling, *M* distinct waveforms are used (e.g., frequency‑shift keying). The channel is usually modeled as a Rayleigh fading or AWGN channel. The capacity can be expressed in terms of the signal‑to‑noise ratio and number of orthogonal dimensions.
3.4 Multiple access channels
A multiple access channel (MAC) has multiple transmitters and a single receiver. The goal is to find the set of achievable rates for each user.
3.4.1 Capacity region for two users
For a two‑user discrete memoryless MAC, the capacity region is the set of rate pairs (*R₁,R₂*) satisfying:
\[
| R_1 \leq I(X_1;Y | X_2), \quad R_2 \leq I(X_2;Y | X_1), \quad R_1+R_2 \leq I(X_1,X_2;Y) |
|---|
\]
for some input distribution *p(x₁)p(x₂)*. This region is a convex pentagon.
3.5 Broadcast channels
A broadcast channel has one transmitter and multiple receivers. The capacity region defines the set of simultaneously achievable rates for all receivers.
3.5.1 Degraded broadcast channel
A degraded broadcast channel is one where the output of one receiver is a “worse” version of another’s. For two receivers (a better one and a worse one), the capacity region is given by the superposition coding region. Achievable rate pairs satisfy:
\[
| R_2 \leq I(U;Y_2), \quad R_1 \leq I(X;Y_1 | U) |
|---|
\]
for some auxiliary random variable *U* and input *X*, with *U—X—Y₁Y₂* forming a Markov chain.
4 Relationship to coding
4.1 Error-correcting codes
Channel capacity provides the theoretical limit; practical error‑correcting codes aim to approach it. While early codes (e.g., Hamming codes) operated far from capacity, the development of capacity‑approaching codes revolutionized communications.
4.1.1 Capacity-approaching codes (LDPC, turbo, polar)
- Low‑density parity‑check (LDPC) codes achieve near‑capacity performance with iterative belief‑propagation decoding. They are used in DVB‑S2, Wi‑Fi, and 5G.
- Turbo codes inspired the field of iterative decoding and can operate within 0.5 dB of capacity.
- Polar codes (invented by Erdal Arıkan in 2009) are the first constructive codes proven to achieve capacity for binary memoryless symmetric channels with low encoding/decoding complexity.
4.2 Rate–distortion theory
Rate–distortion theory deals with lossy compression, where some distortion is allowed. It provides a lower bound on the compression rate needed to achieve a given distortion level.
4.2.1 Lossy compression and channel capacity duality
There is a well‑known duality between lossy compression and channel transmission: the rate–distortion function *R(D)* for a source with distortion measure *D* has a mathematical form analogous to channel capacity *C*. In particular, the Blahut–Arimoto algorithm can compute both. This duality was established by Shannon in his 1959 paper on coding for noisy channels.
5 Computing channel capacity
5.1 Blahut–Arimoto algorithm
The Blahut–Arimoto algorithm is an iterative method for computing the capacity of a DMC. Starting from a random input distribution, it alternates between updating the conditional probabilities of a “tilted” distribution and the input distribution until convergence to the maximum mutual information.
5.2 Numerical methods for non-Gaussian channels
For channels that are not Gaussian or have complex constraints (e.g., peak‑power limits, interference), capacity may not have a closed form. Numerical methods include convex optimization (since mutual information is concave in the input distribution), Monte Carlo integration, and gradient‑based approaches.
5.3 Capacity per unit cost
In many practical scenarios, energy is limited, and the cost per transmission is proportional to the input power. The capacity per unit cost (in bits per joule) is defined as the supremum over input distributions of the ratio *I(X;Y) / E*, where *E* is the average energy. This measure is especially relevant for low‑power sensor networks and deep‑space communications.
6 Applications
6.1 Digital communications
Channel capacity guides the selection of modulation schemes, code rates, and power allocation in wired and wireless systems. Modern standards (e.g., LTE, 5G NR) use adaptive modulation and coding that attempt to operate close to the channel capacity given the current channel conditions.
6.2 Wireless and cellular systems
Capacity analysis of fading channels (see Section 7.1) is critical for cellular network design. Concepts like ergodic capacity and outage capacity help engineers balance coverage, throughput, and reliability in mobile environments.
6.3 Data storage (magnetic, optical, flash)
In magnetic and optical storage, the readback signal is corrupted by noise and interference (e.g., intersymbol interference). Channel capacity informs the design of partial‑response channels, modulation codes, and error‑correcting codes used in hard drives and Blu‑ray discs. Flash memories are modeled as binary erasure or symmetric channels, and capacity provides an upper bound on storage density.
6.4 Quantum channel capacity (brief overview)
Information can also be transmitted over quantum channels (e.g., optical fibers with quantum effects). Quantum channel capacity generalizes classical capacity and includes phenomena like superdense coding and quantum teleportation. Key results include the Holevo bound and the Lloyd‑Shor‑Devetak theorem for the capacity of a quantum channel to transmit classical information.
7 Extensions and advanced topics
7.1 Capacity of fading channels
In wireless communications, the channel gain varies over time due to multipath fading. The capacity depends on whether the fading is known at the transmitter and on whether the receiver can track the channel.
7.1.1 Ergodic capacity vs. outage capacity
- Ergodic capacity is the long‑term average of the instantaneous capacity, assuming the fading process is ergodic and the transmitter can adapt its rate. It is achievable when the code length spans many fading realizations.
- Outage capacity is the maximum rate that guarantees a given outage probability (i.e., the probability that the instantaneous capacity falls below that rate). It is used in delay‑sensitive applications where one cannot wait for long‑term averaging.
7.2 MIMO channels
Multiple‑input multiple‑output (MIMO) channels employ multiple antennas at both transmitter and receiver. They offer a significant capacity increase without additional bandwidth or power.
7.2.1 Spatial multiplexing gain
For a MIMO channel with *Nₜ* transmit and *Nᵣ* receive antennas and independent Rayleigh fading, the ergodic capacity grows as *min(Nₜ, Nᵣ) log₂(SNR)* at high SNR. The factor *min(Nₜ, Nᵣ)* is the spatial multiplexing gain, representing the number of independent data streams that can be transmitted simultaneously.
7.3 Network information theory
Network information theory studies the fundamental limits of communication networks with multiple nodes, interference, and cooperation.
7.3.1 Relay channels
A relay channel consists of a source, a destination, and a relay node that helps forward the message. The capacity is not fully known in general, but known results include the decode‑and‑forward and compress‑and‑forward strategies. The capacity often lies between the cut‑set bounds.
7.3.2 Interference channels
In an interference channel, multiple transmitter‑receiver pairs share the same medium and interfere with each other. The capacity region is an open problem for more than two users. Notable results include the Han‑Kobayashi region for the two‑user Gaussian interference channel, which is known to be within one bit of capacity.