Two-sample comparison

1 Introduction to ECDF-based two-sample comparison

Two-sample comparison using empirical cumulative distribution functions (ECDFs) is a nonparametric technique for contrasting two sets of observations by examining how their full distribution functions differ. Rather than focusing on a single parameter such as a mean or variance, ECDF-based approaches use the cumulative proportion of sample values below a threshold, producing a curve that reflects location, spread, skewness, and tail behavior.

1.1 Motivation and intuition

Suppose two samples are thought to come from similar generating processes. If that is true, then for every candidate threshold \(x\), the fraction of observations less than or equal to \(x\) should be comparable between the samples. ECDFs formalize this idea by turning each sample into a stepwise estimate of its cumulative distribution. When the underlying distributions differ, the ECDF curves typically separate in consistent ways: one curve may lie above the other (suggesting a shift), they may diverge mainly in the middle (indicating different spread or shape), or they may separate in the extremes (indicating tail differences).

1.2 ECDF definition and notation

Let sample \(A = \{X_1,\dots,X_n\}\) and sample \(B = \{Y_1,\dots,Y_m\}\). Their ECDFs are

\[ F_n(x) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}\{X_i \le x\}, \qquad G_m(x) = \frac{1}{m}\sum_{j=1}^m \mathbf{1}\{Y_j \le x\}. \]

These functions are nondecreasing, right-continuous, and take values in \(\{0,1/n,2/n,\dots,1\}\) for \(F_n\) and \(\{0,1/m,2/m,\dots,1\}\) for \(G_m\). They approximate the unknown cumulative distribution functions that generated the samples.

1.3 Visual comparison of two ECDF curves

A basic exploratory workflow is to plot both ECDFs on the same axes. Visual comparison can quickly indicate:

Horizontal shifts: one curve reaches higher cumulative probability at smaller values.
Spread or variability changes: one curve rises more gradually or more sharply.
Tail behavior differences: curves separate near the upper or lower extremes.
Shape differences: nonparallel separation patterns suggest changes beyond location and scale.

ECDF comparison is often most informative when interpreted together with summary distance measures or formal tests.

2 Two ECDF constructions from data

Constructing ECDFs is straightforward, but details matter for accurate comparison—especially with ties, discreteness, and choice of evaluation points.

2.1 Computing the ECDF for each sample

The standard computation is to sort the sample values and compute the cumulative fraction at each distinct observed value.

2.1.1 Handling ties and repeated values

If a sample contains repeated values, multiple observations contribute at the same threshold. The ECDF’s step height at a value reflects the multiplicity: if a value \(v\) occurs \(k\) times in a sample of size \(n\), the ECDF increases by \(k/n\) at \(v\). Correctly grouping tied values ensures that the cumulative probability increments match the empirical frequencies.

2.1.2 Step-function representation and plotting choices

ECDFs are inherently step functions. A common representation uses horizontal steps between consecutive sorted values and vertical jumps at observed points. Plotting conventions vary (e.g., whether the jump is drawn at the left edge or right-continuous form), but the underlying mathematical definition remains consistent.

2.2 Domain alignment and evaluation points

When comparing two ECDFs, the question becomes: at which values \(x\) should the difference \(F_n(x)-G_m(x)\) be evaluated?

2.2.1 Interpolation vs. direct step comparison

There are two practical approaches:

Direct step comparison: evaluate the difference at ECDF jump locations (or just after jumps), respecting the stepwise nature.
Interpolation-based comparisons: compute ECDF values on a fine grid using step logic; this can appear smooth but should not imply extra information beyond the step function.

For many ECDF statistics, the relevant extremum occurs at or near jump points, so direct step evaluation is typically appropriate.

2.2.2 Scaling and binning considerations (avoidance)

ECDF methods are often sensitive to unnecessary discretization. Because ECDFs already summarize data cumulatively, adding additional binning can blur distinctions and alter statistics. A reliable practice is to avoid histogram-like binning and instead compare ECDF values directly at the data-derived thresholds.

2.3 Small-sample behavior and resolution limits

With small \(n\) or \(m\), ECDFs have coarse steps because the set of attainable cumulative probabilities is limited (e.g., increments of \(1/n\)). This affects both visual interpretation and the stability of distance measures. Statistics that rely on maxima of differences can be especially sensitive when few jumps drive large gaps by chance.

3 Distance measures between two ECDFs

Distances and discrepancy summaries translate curve differences into numerical quantities, enabling both interpretation and inference.

3.1 Pointwise differences

A basic discrepancy function is

\[ D(x) = F_n(x) - G_m(x), \]

which indicates whether sample \(A\) tends to have more mass at or below \(x\) relative to sample \(B\). Plotting \(D(x)\) or examining its sign across the range highlights where the separation occurs.

3.2 Cumulative discrepancy summaries

Instead of focusing only on pointwise separation, one can aggregate discrepancies across \(x\). Common choices include integrated absolute differences or integrated squared differences, which capture both magnitude and breadth of separation.

3.3 Maximum separation concepts

Many classic ECDF comparisons are driven by the largest vertical distance between the curves.

3.3.1 Interpreting the largest vertical gap

If the maximum absolute difference

\sup_x	F_n(x) - G_m(x)

is large, then at some threshold \(x\) one sample has substantially higher cumulative probability than the other. This does not specify whether the difference is due to a global shift, local divergence, or tail effects, but it provides a clear measure of the strongest mismatch between cumulative proportions.

4 Hypothesis testing with two-sample ECDF methods

ECDF-based tests often target the question: “Are these two samples plausibly drawn from the same underlying distribution?”

4.1 Null hypothesis and alternative perspectives

A typical null hypothesis is that both samples come from a common distribution \(H\). If true, then for every \(x\), the true cumulative probabilities match; the observed ECDF differences then arise only from sampling variability. Alternatives include any kind of distributional discrepancy—shifts, scale changes, or changes in tail behavior—since ECDFs are distribution-shape sensitive.

4.2 Kolmogorov–Smirnov type comparisons

The Kolmogorov–Smirnov (KS) framework measures the maximum ECDF separation.

4.2.1 Test statistic based on maximum ECDF difference

For two samples, a KS-type statistic is

D_{n,m} = \sup_x	F_n(x) - G_m(x)	.

Under the null hypothesis and appropriate assumptions (notably independent observations), the distribution of \(D_{n,m}\) can be used to compute a p-value, either via asymptotic approximations or exact/permutation calculations depending on sample sizes and implementation.

4.2.2 One-sample vs two-sample distinctions

In a one-sample KS test, ECDF from one sample is compared to a specified theoretical CDF. In the two-sample KS test, neither distribution is specified; both are estimated from data, which changes the null distribution and the interpretation as an empirical-to-empirical discrepancy.

4.3 Alternative discrepancy-based tests

Other tests replace “maximum gap” with different discrepancy notions that may be more sensitive to certain types of differences.

4.3.1 Integrated squared/absolute discrepancies

Tests based on integrated discrepancies compare the ECDF difference across the domain rather than only the peak gap. Integrated squared differences emphasize broader, sustained separation, while integrated absolute differences provide a more robust aggregate of deviations.

4.3.2 Tail-weighted variants (conceptual overview)

Some methods place more weight on differences in the tails, reflecting scenarios where tail behavior is particularly relevant. Conceptually, these tests adjust the discrepancy measure so that deviations at extreme quantiles contribute more to the statistic than mid-range disagreements.

4.4 Permutation and resampling strategies

Resampling methods can generate reference distributions for test statistics without relying solely on analytic approximations.

4.4.1 Permutation distribution construction

A permutation test pools both samples, randomly reassigning observations into two groups of sizes \(n\) and \(m\). For each permutation, the ECDF-based statistic (e.g., maximum separation or integrated discrepancy) is recomputed. The resulting distribution of the statistic under random labeling provides an empirical p-value.

4.4.2 Multiple testing and p-value interpretation

When multiple comparisons are performed (e.g., testing many pairs of groups), p-values can be misleading if interpreted in isolation. Adjustments or careful reporting of effect sizes alongside significance are typically recommended. A small p-value indicates inconsistency with the null model, but it does not automatically imply that the difference is practically important.

5 Effect size and practical significance

Statistical significance alone does not quantify how large or useful a distributional difference is. ECDF-based distances provide a path to effect size characterization.

5.1 From “significant” to “meaningful”

A dataset may yield a statistically significant ECDF difference even when the curves are close, especially with large sample sizes. Conversely, small samples may fail to detect moderate discrepancies. Practical significance focuses on the magnitude, the region where differences occur, and how those differences relate to the context of the measurement.

5.2 Reporting distances between ECDFs

Reporting the chosen discrepancy measure—such as the maximum gap \(D_{n,m}\) or an integrated discrepancy—helps others compare results across studies. Clear reporting also includes which statistic was used, since different measures can rank differences differently (e.g., maximum-gap versus integrated distance).

5.3 Confidence intervals for distributional differences

Confidence intervals can quantify uncertainty about the discrepancy magnitude or about differences at specific thresholds.

5.3.1 Bootstrap-based interval estimation

Bootstrap procedures resample with replacement within each group, recompute the ECDFs and discrepancy measure, and use the resulting bootstrap distribution to form percentile or bias-corrected intervals. This provides an empirical estimate of variability without relying on closed-form variance formulas for complex statistics.

5.4 Linking results to distributional features

Effect-size reporting becomes more interpretable when paired with qualitative curve summaries. For example, one might note whether the largest ECDF separation occurs near the upper tail or whether the curves differ mostly in the center, aligning the distance metric with a substantive interpretation of distributional features.

6 Assumptions, robustness, and limitations

ECDF-based two-sample methods are designed to be broadly applicable, but they still depend on sampling conditions and data characteristics.

6.1 Independence and identically distributed sampling

Most standard ECDF test theory assumes that observations within each sample are independent and identically distributed (i.i.d.) and that the two samples are independent of each other. Dependence can distort the null distribution of test statistics, potentially changing error rates.

6.2 Sensitivity to outliers and heavy tails

Although ECDFs are nonparametric, they are not immune to extreme observations. Heavy-tailed data can create abrupt ECDF behavior near extremes, and distance measures—especially those emphasizing maxima—may be driven by a small number of tail points. Interpreting results alongside distribution summaries can help identify when extremes dominate the discrepancy.

6.3 Discrete data and mass points

When data are discrete, the cumulative distribution has jumps even for the true underlying distribution. ECDFs will then reflect mass points as additional structure.

6.3.1 Stepwise ECDF behavior in discrete settings

In discrete contexts, ECDF curves can only change at a finite set of values, producing tied jump locations. As a result, maximum-gap statistics may concentrate on those discrete thresholds. Visual and numerical comparisons remain valid, but expectations about smoothness and the meaning of “crossing” should be adjusted.

6.4 Dependence between observations (when ECDF methods may fail)

If observations are correlated (for example, repeated measures from the same unit), ECDF-based inference that assumes i.i.d. sampling can be unreliable. In such cases, methods designed for dependent data, block resampling, or other correlation-aware strategies may be needed.

7 Implementation details and reproducibility

Reliable ECDF comparisons depend on careful implementation choices and transparent reporting.

7.1 Algorithmic computation of ECDFs

Computing an ECDF typically requires sorting the sample and then computing cumulative fractions along the sorted values.

7.1.1 Complexity and efficiency notes

Sorting dominates runtime. For sample sizes \(n\) and \(m\), complexity is generally \(O(n\log n + m\log m)\) plus the cost of producing evaluation grids or computing the statistic over combined jump locations.

7.2 Numerical stability for large samples

With very large samples, storing and comparing floating-point values can introduce rounding issues, especially when values are close. Using stable sorting, careful equality handling for ties, and consistent comparison thresholds helps avoid accidental artifacts.

7.3 Choosing plotting and comparison grids

To compare curves accurately, evaluation points should align with the ECDF jump locations from both samples. For distance calculations, using the union of observed values (or the specific points relevant to the statistic) reduces approximation error.

7.4 Reproducible workflow (data → ECDF → statistic → inference)

A reproducible analysis typically records:

The exact samples used and their preprocessing rules (e.g., handling missing values).
How ECDFs were computed (treatment of ties, definition of right-continuity).
The statistic used for discrepancy (maximum gap, integrated distance, etc.).
The inference method (analytic approximation or permutation/resampling), including random seeds.
The reporting format for both p-values and effect sizes.

8 Worked examples and interpretation guides

The following scenarios illustrate how ECDF comparisons map to distributional features.

8.1 Comparing shifted distributions

If distribution \(B\) is mostly a shifted version of \(A\), the ECDF curve for one sample will tend to be consistently higher than the other over a broad range of \(x\). The largest separation often appears near the central region where cumulative probabilities change most rapidly.

8.2 Comparing changes in spread (scale)

When spread differs but central tendency is similar, ECDFs may cross near the center and separate in the tails. One curve may rise faster (narrower distribution), while the other rises more slowly (wider distribution), producing a distinctive nonparallel shape.

8.3 Comparing differing tail behavior

Tail differences typically manifest as strong divergence near the extremes with limited separation in the middle. For example, if \(B\) has heavier upper tail, then \(G_m(x)\) may be smaller than \(F_n(x)\) for moderate-to-high thresholds, then eventually catch up or exceed depending on how the tail mass is reallocated.

8.4 Interpreting non-crossing vs crossing ECDFs

Non-crossing (one curve stays above the other): suggests a consistent dominance in cumulative mass ordering, often consistent with a monotone type of distribution shift.
Crossing curves: indicates that the ordering changes across thresholds, implying differences in shape that cannot be summarized by a single-direction shift alone.

Crossing does not invalidate ECDF methods; it simply guides interpretation toward more complex distributional differences.

ECDF-based two-sample comparisons connect to other nonparametric tools that also compare distributions with minimal parametric assumptions.

9.1 Rank-based and distance-based nonparametric tests

Many alternatives use ranks or other distribution-free summaries. Rank-based tests can be sensitive to particular distributional features such as shifts in central tendency, while distance-based tests may emphasize global or local discrepancies depending on their metric.

9.2 Multivariate analogs and distribution-free generalizations (high-level)

Extending ECDF logic to multivariate settings is challenging because cumulative probability ordering is not one-dimensional. High-level approaches include using depth functions, projection strategies, or other distributional metrics designed to mimic the role of the ECDF in higher dimensions.

9.3 ECDF methods for time series or censored data (overview-level)

When data are time-dependent or subject to censoring, naive ECDFs may not represent the intended cumulative distribution. Extensions exist that adapt the cumulative estimation idea to these settings, often requiring specialized assumptions or modified estimators so that inference remains coherent.