1 Overview of Percent-Encoding
Percent-encoding is a mechanism used in URLs and other text-based protocols to represent characters that cannot be safely transmitted as-is. It represents selected bytes as a percent sign followed by two hexadecimal digits (for example, %2F).
1.1 Percent-encoding basics
In practice, the percent sign (%) introduces a byte escape. Each escape corresponds to a single underlying byte value. For canonicalization or normalization, implementations typically treat each escape atomically—either decoding it to its byte and then applying rules, or validating it as already well-formed before transforming its formatting.
1.2 Relation to character encoding (e.g., UTF-8)
Because percent-encoding operates on bytes rather than abstract characters, the meaning depends on the character encoding used before encoding. In the common case of UTF-8, multibyte characters (such as many non-ASCII glyphs) are first converted to their UTF-8 byte sequence, and then each byte is percent-escaped as needed.
1.3 Where percent-encoding appears (URLs, headers, form data)
Percent-encoded sequences commonly occur in:
- URL components such as path and query
- Application/x-www-form-urlencoded bodies (often using
+for spaces alongside percent-escapes) - Certain header values and configuration fields where URL-like escaping is reused
Although usage varies by system, normalization rules are typically defined with a specific layer or component in mind.
1.4 Canonicalization vs. normalization
Canonicalization is an aggressive form of normalization aiming for a single “best” representation. Normalization is a broader category: it includes case normalization, boundary correction, and other transformations that yield a consistent output. Percent-encoding normalization often targets a canonical representation while still preserving semantics and protocol requirements.
2 Normalization Goals and Invariants
Normalization aims to reduce representational variance so that equivalent inputs map to the same output, improving interoperability across producers and consumers.
2.1 Meaning-preserving transformations
A core goal is to transform encodings without changing the underlying decoded byte sequence and without altering how the target protocol interprets delimiters. For safe interoperability, normalization should preserve semantics defined by the relevant URL parsing rules.
2.2 Idempotence and canonical output
An effective normalization procedure is idempotent: applying it multiple times yields the same result as applying it once. This property is valuable for caching, signing, and comparisons, because it avoids oscillation between equivalent forms.
2.3 Minimizing differences across producers
Different clients may generate different but equivalent percent-escaped strings due to choices like hex digit case or whether they escape certain characters. Normalization reduces these differences to allow stable behavior across heterogeneous implementations.
2.4 Safety constraints (avoiding semantic changes)
Not all string rewrites are safe. Normalization must avoid changes that could:
- reinterpret reserved delimiters (such as
?,#,&,=) in a different component - change the byte sequence produced by decoding
- convert between fundamentally different encoding schemes in ambiguous contexts
3 Hex Case and Formatting Rules
Many normalization regimes begin with formatting-level changes because they do not affect decoded bytes.
3.1 Uppercase vs. lowercase hex digits
The two hexadecimal digits after % may appear in either uppercase or lowercase form. Normalization typically selects one convention (commonly uppercase or lowercase) and rewrites every percent escape to match it, ensuring uniformity for equality checks.
3.2 Consistent percent sign usage
Normalization also ensures that percent escapes consistently use the percent sign prefix and that each escape has exactly two valid hex digits. Sequences that do not match the expected pattern are generally handled according to defined policy (see later sections on malformed input).
3.3 Handling leading zeros in hex bytes
Bytes are represented by two hex digits; a value like decimal 10 must appear as %0A (or %0a) rather than %A to be a valid two-digit escape. When producers omit leading zeros in nonstandard ways, normalizers typically reject or define explicit rewrite rules rather than inferring bytes silently.
3.4 Normalizing mixed-escape sequences
Inputs may contain a mix of already-escaped bytes and literal characters that are equivalent after decoding in some contexts. Normalization rules decide which portions are treated as raw text and which are percent-decoded, then re-encoded consistently, preventing partial canonicalization from producing inconsistent outputs.
4 Decoding and Re-encoding Strategy
Normalization often follows a decode-transform-reencode pattern, but whether decoding is permitted and how re-encoding is performed depends on context.
4.1 When decoding is allowed
Decoding is allowed when the percent escapes are validated and when the target layer’s semantics are byte-based (or explicitly defined in terms of decoded bytes). Some systems avoid decoding in certain components to prevent accidental delimiter changes.
4.2 Re-encoding after transformations
After transformations (such as hex case normalization or byte-level remapping), bytes are re-encoded according to the permitted character set for the target component. This step ensures that the output string conforms to the same semantics used for parsing.
4.3 Preserving already-safe characters
Normalization should preserve characters that are already safe and allowed in the relevant component to avoid needless expansion. A common rule is to decode percent escapes only when necessary, then re-encode only those bytes that must be escaped for compliance or safety.
4.4 Avoiding double-encoding pitfalls
Double-encoding occurs when percent-encoded sequences are encoded again after decoding, leading to different bytes on re-decoding. Robust normalizers track whether a % sequence was originally produced as an escape versus being a literal percent sign, ensuring that the same decoded byte sequence results.
4.5 Context-aware re-encoding (path vs. query)
URL components differ in delimiter meanings. A byte representing ? in a path may need different treatment than the same byte in a query, because ? separates the URL’s path from its query. Normalization therefore applies re-encoding rules that are specific to the component boundaries.
5 Treatment of Reserved and Unreserved Characters
Percent-encoding interacts with the classification of characters into unreserved and reserved sets, as defined by URL specifications.
5.1 Unreserved character passthrough
Unreserved characters are those that can appear in a URL without special escaping. Normalization often leaves them as literal characters to keep the string readable and short while still maintaining consistent decoding results.
5.2 Reserved characters and their correct interpretation
Reserved characters may have structural meanings (for instance, delimiters between components or parameters). When normalization decodes and re-encodes these characters, it must preserve the structural role they play, rather than converting them into forms that could shift parsing behavior.
5.3 Component-specific interpretation (path, query, fragment)
The same byte value may behave differently depending on whether it appears in the path, query, or fragment. Normalization rules thus use component-specific “allowed vs. escaped” character sets so that the resulting string is interpreted consistently by parsers.
5.4 Preventing normalization from altering delimiters
A key invariant is that delimiter characters separating components should remain structurally correct. For example, normalization should avoid turning a delimiter into its encoded form in a way that would prevent a parser from recognizing the boundary (or vice versa).
6 Character Set and Byte Semantics
Because percent-encoding represents bytes, normalization must account for how bytes map to characters and how errors are handled.
6.1 UTF-8 byte sequences and normalization
When UTF-8 is the expected character encoding, percent escapes corresponding to UTF-8 sequences can be decoded into Unicode code points. Normalization may then decide whether to preserve normalized Unicode forms or simply re-encode bytes without applying Unicode normalization (since percent-encoding typically targets byte fidelity rather than canonical Unicode text forms).
6.2 Handling invalid byte sequences
If decoded bytes do not form valid UTF-8 (when UTF-8 is expected), normalization must follow a policy. Common choices include preserving the original byte sequence by re-encoding bytes individually, replacing invalid sequences with a placeholder, or rejecting the input as malformed.
6.3 Overlong or non-canonical byte representations
Unicode encodings like UTF-8 have restrictions that disallow overlong representations. Systems that normalize based on decoding into characters may treat overlong sequences as invalid. Systems that operate at the byte level can preserve them, but interoperability goals vary: some normalizers choose a strict mode to reduce ambiguity.
6.4 Policy choices for malformed encodings
Normalization implementations typically expose or internally select a policy:
- strict: reject malformed percent escapes and invalid byte sequences
- tolerant: preserve bytes where possible, normalizing only safe formatting aspects
- lossy: substitute invalid sequences to produce a usable output
The chosen policy affects both security posture and interoperability.
7 Normalization Scope and Layering
Normalization is rarely uniform across the whole input string; it is applied at specific layers with awareness of parsing boundaries.
7.1 Normalizing per URL component
A URL is parsed into components (scheme, authority, path, query, fragment). Normalization rules applied per component help prevent accidental changes to separators that exist between components.
7.2 Percent-encoding in query parameters
Query strings often use & to separate parameters and = to separate keys and values. Normalization must preserve the parameter structure while deciding which bytes within keys and values should be escaped or unescaped. Some systems also define how spaces and + are treated in query strings.
7.3 Percent-encoding in form submission bodies
In application/x-www-form-urlencoded bodies, + commonly represents a space, and percent escapes represent other bytes. Normalization may need to convert between representations carefully, especially when both + and %20 appear and should be treated equivalently or distinctly depending on the system’s rules.
7.4 Interactions with higher-level parsing
Higher-level parsers (framework routers, form handlers, signature libraries) may decode percent escapes before applying their own logic. If normalization occurs at the wrong layer, the output may not round-trip correctly. For interoperability, systems often specify whether normalization runs before or after parsing and decoding.
8 Interoperability and Standards Considerations
Interoperability depends on alignment with widely used standards and on understanding differences among parsers.
8.1 Common standards and guidance sources
Percent-encoding behavior is commonly guided by URL standards, URL-processing libraries, and protocol documentation that define component boundaries and allowed character sets. Normalization rules that cite these sources tend to be more predictable across ecosystems.
8.2 Differences among parsers and servers
Parsers may differ in tolerance for malformed encodings, in whether they decode before splitting on delimiters, and in how they treat invalid UTF-8. These differences can yield divergent “normalized” outputs unless the normalization procedure is clearly defined.
8.3 Compatibility testing across systems
A practical approach is to test normalization output against known-good round-trip behavior in target systems. Compatibility testing typically includes:
- matrix tests over component types (path vs query vs fragment)
- inputs containing mixed valid and malformed escapes
- comparisons of normalized strings and their interpreted byte sequences
8.4 Versioning normalization rules
Because behavior can change over time (for example, tightening validation), normalization libraries often version their rules. Versioning allows systems to reproduce prior normalized outputs for backward compatibility, especially when outputs are used for caching keys or cryptographic signatures.
9 Algorithms and Implementation Patterns
Normalization is usually implemented as a deterministic transformation from an input string to a canonical form.
9.1 Step-by-step normalization workflow
A common workflow is:
- Parse the input into relevant components (or otherwise define boundaries).
- Validate percent-escape patterns within the scoped component.
- Decode validated escapes into bytes (optionally preserving literal
%behavior). - Apply formatting normalization rules (hex case, escape decisions).
- Re-encode bytes using the component’s allowed set and the selected escape policy.
- Assemble the normalized component back into the full output.
9.2 Data structures for tokenizing percent sequences
Tokenization often uses a scanner that identifies three kinds of segments: literal characters, valid percent escapes, and invalid/partial escape fragments. Implementations may represent these as tokens in a small buffer before re-encoding, or they may process in a single pass while writing to an output buffer.
9.3 Performance considerations for large inputs
For large strings, normalization should minimize allocations and avoid repeated scans. Efficient implementations typically:
- pre-size output buffers
- use streaming write patterns
- avoid converting entire strings to intermediate Unicode representations unless required by policy
9.4 Streaming vs. whole-string normalization
Streaming normalization processes input incrementally, which is beneficial for memory usage and low-latency scenarios. However, correct handling of boundary rules may require buffering around component separators or partial escape sequences that span chunk boundaries. Whole-string normalization simplifies correctness but can be heavier for very large payloads.
10 Verification and Edge Cases
Correctness is evaluated using systematic tests and property checks, because edge cases are frequent in URL escaping.
10.1 Property-based testing for normalization
Property-based tests generate diverse inputs and verify invariants such as:
- normalization output contains only valid percent escapes where required
- decoded byte sequence of the normalized output equals the intended decoded bytes of the original (under the selected policy)
- component delimiter structure remains unchanged
10.2 Idempotence tests
A straightforward verification is to apply normalization twice and confirm that the second result matches the first. This catches issues such as inconsistent re-encoding choices or partial decoding mistakes.
10.3 Round-trip tests (where applicable)
Where the system specifies a reversible mapping, tests can decode a normalized output and re-encode it, verifying that the normalized string remains stable. Round-trip tests are especially useful when normalization is tied to canonical output requirements.
10.4 Tricky cases: adjacent escapes, empty escapes, boundary conditions
Notable edge cases include:
- adjacent percent escapes with no intervening characters (e.g.,
%41%42) - incomplete escapes at the end of a string (e.g.,
%4) - literal percent signs that are not part of a valid escape sequence
- percent escapes that cross the boundary between component separators when parsing is incorrect
Handling these cases consistently is essential for deterministic behavior.
11 Security and Robustness (Non-political, engineering-focused)
Percent-encoding normalization can affect security properties by influencing how systems interpret and compare inputs.
11.1 Mitigating ambiguous encoding interpretations
Ambiguities arise when different systems decode percent escapes differently, especially around reserved delimiters and malformed sequences. Normalization can reduce discrepancies by enforcing a consistent decoding/encoding policy.
11.2 Preventing normalization bypass scenarios
Some security problems occur when a component is validated in one form but used in another. Normalizing at a clearly defined stage, then validating the normalized representation (or validating bytes after decoding), helps prevent attackers from exploiting alternate representations that pass checks but behave differently later.
11.3 Logging and observability of normalized vs. raw inputs
For debugging and incident response, systems often record both raw and normalized values. Logging raw inputs aids for forensics, while logging normalized outputs helps confirm what the system actually processed.
11.4 Safe failure modes for malformed input
Robust designs define predictable behavior on malformed data. Options include rejecting the request, producing a conservative normalized output that preserves bytes without introducing new semantics, or triggering a controlled error path rather than continuing with partially interpreted data.
12 Examples and Test Vectors
Examples illustrate how formatting and policy choices manifest in normalized outputs.
12.1 Simple hex-case normalization examples
An input like abc%2fdef may normalize to abc%2Fdef (if uppercase hex is selected), while leaving surrounding literal characters untouched.
12.2 Context-specific examples (path vs. query)
Consider a byte sequence that represents a character with reserved meaning. Normalization rules differ by component: the same percent-encoded byte may be emitted as an escape in one component and as a literal (if allowed) in another, ensuring the output remains structurally correct.
12.3 Mixed encodings and expected canonical outputs
A string containing both %7e and ~ (where ~ is unreserved) may normalize to a fully canonical mix such as consistently using either literal ~ or consistently escaping it, depending on the chosen policy. Test vectors capture this decision so the behavior is deterministic.
12.4 Malformed encodings with defined outcomes
Inputs with invalid patterns (such as %ZZ or a truncated %1) yield outcomes based on policy: strict mode rejects, tolerant mode preserves literal characters as-is where possible, and lossy mode replaces invalid sequences with a defined placeholder. Test vectors specify which path is used so that normalization does not vary across implementations.