1 Definition and origins

1.1 Historical context: Claude Shannon and the birth of information theory

The bit was introduced by Claude Shannon in his 1948 paper *A Mathematical Theory of Communication*, which laid the foundations of information theory. Shannon needed a unit to quantify the amount of information produced by a source or transmitted through a channel. He chose the term *bit* (a contraction of *binary digit*) to emphasize the role of binary choices as the fundamental measure of uncertainty reduction.

1.2 Formal definition as a logarithmic measure of uncertainty

In information theory, the bit is defined as the logarithm base 2 of the number of equally likely outcomes. If a random variable has two equally probable states (e.g., a fair coin flip), learning the outcome reduces uncertainty by exactly one bit. More generally, if there are *N* equally likely possibilities, the information gained when one is specified is log₂(N) bits.

1.3 Alternative names and early uses (binary digit, binit)

Before Shannon’s coinage, the term *binary digit* was used in computing and telegraphy. John Tukey, a colleague of Shannon, is often credited with suggesting the contraction *bit*. An earlier term, *binit*, appeared in some engineering contexts but never gained widespread acceptance. Shannon’s 1948 paper firmly established *bit* as the standard unit.

2 Mathematical properties

2.1 Bit as the logarithm base 2 of the number of equally likely outcomes

For a set of *n* equally probable symbols, the information content of any one symbol is log₂(n) bits. This logarithmic relationship ensures additivity: the information from two independent events (with *n₁* and *n₂* outcomes) is log₂(n₁) + log₂(n₂) = log₂(n₁·n₂) bits.

2.2 Relation to entropy: average information per symbol

The entropy *H* of a discrete random variable *X* with probability distribution *p(x)* is the average number of bits needed to describe its value: *H(X)* = −∑ p(x) log₂ p(x). Entropy thus measures the expected information content, or uncertainty, in bits per symbol.

2.3 Binary representation and the bit string

A bit string is a sequence of bits (e.g., 10110). Any piece of data can be encoded as a bit string. The length of the string (in bits) determines the number of possible distinct values: a string of *k* bits can represent 2^k distinct messages.

2.3.1 Bit length and data compression limits

The entropy of a source sets a lower bound on the average bit length needed to represent its symbols without loss. Shannon’s source coding theorem states that no lossless compression scheme can achieve an average bit length less than the entropy, making entropy the theoretical limit of data compression.

2.4 Bitwise operations (AND, OR, XOR, NOT) in information theory

Bitwise operations treat individual bits as Boolean variables. AND, OR, XOR, and NOT are fundamental to both computer arithmetic and information-theoretic constructions such as parity checks and coding theory. For example, XOR corresponds to addition modulo 2, which is essential in linear error-correcting codes.

3 Physical and practical implementations

3.1 Representing bits in electronic circuits (voltage levels)

In digital electronics, a bit is typically represented by a voltage level: a high voltage (e.g., 3.3 V) denotes 1, and a low voltage (e.g., 0 V) denotes 0. Circuits called flip-flops store a single bit. The use of two distinct, stable states makes binary logic robust against noise.

3.2 Bits in optical and magnetic media

In optical discs (CDs, DVDs), bits are encoded as microscopic pits and lands that reflect laser light differently. Magnetic storage (hard drives, tape) uses the direction of magnetization in tiny regions to represent 0 and 1. In both cases, a single bit corresponds to a small physical area or volume.

3.3 Quantum bits (qubits) as a generalization

A qubit (quantum bit) extends the classical bit by exploiting quantum superposition. Whereas a classical bit is either 0 or 1, a qubit can be in a linear combination of both states simultaneously. Qubits are the fundamental units of quantum computing and quantum information theory, enabling algorithms with exponential speedups for certain problems.

4 Applications in information theory

4.1 Data compression (source coding)

Source coding aims to represent data using as few bits as possible while allowing perfect reconstruction (lossless) or acceptable fidelity (lossy).

4.1.1 Shannon–Fano encoding

Shannon–Fano coding assigns shorter code words to more probable symbols. It constructs a binary tree by recursively splitting the symbol set into two parts with roughly equal total probability. While not optimal, it was one of the first practical compression methods.

4.1.2 Huffman codes

Huffman coding is an optimal prefix code for a given symbol probabilities. It builds a binary tree from the bottom up, merging the two least probable symbols at each step. Huffman codes achieve an average code length within one bit of the entropy.

4.2 Error detection and correction (channel coding)

Channel coding adds redundant bits to data so that errors introduced during transmission can be detected or corrected.

4.2.1 Parity bits and Hamming distance

A parity bit is a single extra bit appended to a group of bits to make the total number of 1s even (even parity) or odd (odd parity). It detects an odd number of bit errors. The Hamming distance between two bit strings is the number of positions at which they differ; codes with minimum Hamming distance *d* can detect up to *d*−1 errors.

4.2.2 Error-correcting codes (Reed–Solomon, LDPC)

Reed–Solomon codes treat data as polynomial coefficients over finite fields and can correct burst errors. Low-density parity-check (LDPC) codes use sparse parity-check matrices and approach the Shannon capacity. Both are widely used in storage and communication systems.

4.3 Cryptography and key entropy

In cryptography, the security of a symmetric key depends on its entropy measured in bits. A key of *k* bits with uniform randomness has *k* bits of entropy. The number of possible keys is 2^k, so larger key lengths exponentially increase the effort for brute‑force attacks. Shannon’s work on information-theoretic security also defines perfect secrecy when the key is at least as long as the message.

5.1 Byte (8 bits) and multiples (kilobit, megabit, etc.)

A byte is a group of 8 bits, historically the smallest addressable unit in many computer architectures. Larger units are:

  • 1 kilobit (kb) = 1000 bits (or 1024 in some contexts, but the binary prefix *kibibit* is used for 1024 bits)
  • 1 megabit (Mb) = 10⁶ bits
  • 1 gigabit (Gb) = 10⁹ bits

Decimal prefixes (kilo, mega, giga) are standard for data transmission speeds, while binary prefixes (kibi, mebi, gibi) are used for memory sizes.

5.2 Natural units: nat (natural unit) and Hartley (ban)

The nat (or natural unit) uses the natural logarithm instead of base 2: 1 nat = log₂(e) bits ≈ 1.4427 bits. The hartley (also called the ban) uses base 10: 1 hartley = log₂(10) bits ≈ 3.3219 bits. Both are used in theoretical work where algebraic convenience suggests a different logarithmic base.

5.3 Relationship between bits and shannons (Sh) as a unit of information

The unit *shannon*, denoted Sh, is a synonym for the bit when used strictly in an information‑theoretic context. Shannon himself often used *shannon* as the unit name to avoid confusion with the physical binary digit. Today, *bit* is more common, but *shannon* appears in standards to emphasize the informational aspect.

6 Advanced topics

6.1 Bits in algorithmic information theory (Kolmogorov complexity)

Algorithmic information theory defines the Kolmogorov complexity of a string as the length in bits of the shortest program that outputs it on a universal Turing machine. This captures the notion of “inherent information content” independent of probability distributions. Strings with high Kolmogorov complexity are effectively random.

6.2 Bits in communication systems: capacity and the Shannon–Hartley theorem

The Shannon–Hartley theorem determines the maximum rate (in bits per second) at which information can be transmitted over a communication channel of bandwidth *B* (in hertz) in the presence of additive white Gaussian noise with signal‑to‑noise ratio *S/N*: *C* = *B* log₂(1 + S/N). This fundamental limit is expressed in bits per second.

6.3 Bits in neural information theory

In neuroscience, the bit quantifies how much information a neuron’s firing pattern conveys about a stimulus. For example, the mutual information between a stimulus and the spike count of a neuron is measured in bits per spike. This approach helps model sensory processing and neural encoding.