1. Motivation and Core Idea

Estimating functions provide a flexible route to estimation when the target parameter can be characterized by equations derived from the data’s probabilistic structure. Rather than directly optimizing an objective such as a likelihood, one specifies a function whose expected value vanishes at the true parameter. Solving the resulting estimating equation yields an estimator, often with desirable large-sample properties.

1.1 What makes an equation an estimating function

An estimating function is typically written as a measurable mapping \( \psi(\theta; X) \) (or \( \psi(\theta) \) when the data are understood) such that, under the data-generating mechanism, \[ \mathbb{E}\big[\psi(\theta_0; X)\big] = 0 \] for the true parameter value \( \theta_0 \). In practice, an empirical version replaces expectation with sample averages, producing \[ \hat{\psi}(\theta) = \frac{1}{n}\sum_{i=1}^n \psi(\theta; X_i). \] The equation \(\hat{\psi}(\theta)=0\) is then used to compute an estimator.

1.2 Estimators as solutions to estimating equations

A common estimator in this framework is defined implicitly as a root of the sample estimating equation. Specifically, an estimator \(\hat{\theta}\) satisfies \[ \hat{\psi}(\hat{\theta})=0, \] possibly up to numerical tolerance or with additional constraints. When multiple roots exist, selection rules (e.g., choosing the root with certain stability properties) may be required.

1.3 Relationship to likelihood-based estimation

Likelihood-based methods can be cast in the estimating-function language. For a parametric model with log-likelihood \(\ell(\theta)\), the score function \(U(\theta)=\nabla_\theta \ell(\theta)\) often satisfies an expectation-null property at the true parameter. Thus, score equations can be interpreted as estimating equations. The estimating-function approach generalizes this idea by allowing moment conditions not necessarily tied to a particular likelihood.

1.4 When estimating functions are preferred

Estimating functions are particularly attractive when:

  • a full likelihood is complicated or misspecified, while key moment conditions remain trustworthy;
  • robust or computationally convenient procedures are desired;
  • models involve nuisance components where likelihood construction is difficult;
  • one seeks methods that remain valid under weaker assumptions (e.g., only requiring certain moments to exist).

2. Formal Setup

2.1 Notation and data structure

Consider data \(X_1,\dots,X_n\) drawn from a distribution indexed by an unknown parameter \(\theta \in \Theta \subseteq \mathbb{R}^p\). In independent and identically distributed settings, the estimating function is built from each observation and aggregated across \(n\). For dependent data, the construction may use cluster-wise contributions or additional structure, but the core idea remains the same: an estimating equation targets \(\theta_0\).

2.2 Defining the estimating function \( \psi(\theta) \)

An estimating function is usually vector-valued: \[ \psi(\theta; X) \in \mathbb{R}^p, \] so that solving a system of \(p\) equations identifies the \(p\)-dimensional parameter. In many cases, \(\psi(\theta; X)\) is derived so that its population mean equals zero at \(\theta_0\), making \(\theta_0\) the solution of the population equation \[ \mathbb{E}[\psi(\theta; X)] = 0. \]

2.2.1 Root-finding vs. optimization viewpoints

Solving \(\hat{\psi}(\theta)=0\) is the root-finding perspective. An alternative is to optimize a criterion built from \(\hat{\psi}\), for example minimizing a quadratic form \[ Q(\theta) = \hat{\psi}(\theta)^\top W \hat{\psi}(\theta), \] with \(W\) a positive definite weight matrix. Under suitable conditions, the minimizer coincides with a solution of the estimating equations, linking optimization and equation-based formulations.

2.3 Constructing estimating equations from moments

A common recipe starts with moment conditions such as \[ \mathbb{E}[m(X,\theta)] = 0. \] One then sets \(\psi(\theta; X)=m(X,\theta)\) or transforms moments into a more stable form (e.g., using bounded functions, scaling, or incorporating auxiliary weights). The resulting empirical moment equations become the estimating equations used for inference.

3. Key Assumptions and Design Criteria

Designing estimating functions involves ensuring that the target parameter is recoverable and that the resulting estimator behaves well as \(n\) grows.

3.1 Consistency conditions

Consistency typically requires that:

  1. the population estimating equation has a unique solution at \(\theta_0\);
  2. the empirical estimating equation converges uniformly to its population counterpart;
  3. the solution of the empirical equation converges to the population solution.

These conditions are often formalized through continuity, compactness or coercivity arguments, and stochastic convergence results.

The most direct property is “population unbiasedness,” meaning \(\mathbb{E}[\psi(\theta_0; X)]=0\). In some settings, unbiasedness is replaced by weaker calibrated conditions, such as requiring the expectation to be close to zero under a working model, or relying on “estimating-function unbiasedness” at the true parameter even when the model is not fully correct.

3.3 Identifiability and uniqueness of solutions

Identifiability ensures that \(\theta_0\) is the only parameter satisfying the population equation. Without uniqueness, estimators can drift among multiple roots or depend sensitively on numerical starting points. For vector parameters, identifiability can fail if the moment conditions are linearly dependent or insufficient in dimension.

3.4 Sensitivity and robustness considerations

Sensitivity refers to how much \(\hat{\psi}(\theta)\) changes with \(\theta\) and to how disturbances or outliers affect the estimation. Robust design may involve:

  • using bounded influence functions or redescending estimating functions;
  • centering moments in a way that reduces leverage from extreme observations;
  • adopting weights that downweight observations with high variability.

In general, robustness is not automatic; it is engineered through the choice of \(\psi\).

3.5 Regularity conditions for asymptotics

Asymptotic results require conditions such as differentiability of \(\psi\) in \(\theta\), finite moments of certain order, and non-singularity of relevant matrices. These assumptions support Taylor expansions around \(\theta_0\) and allow central-limit-type approximations for the empirical estimating equation.

4. Asymptotic Theory

4.1 Asymptotic distribution of estimating equation solutions

Under regularity conditions, solutions \(\hat{\theta}\) to \(\hat{\psi}(\theta)=0\) are typically asymptotically normal: \[ \sqrt{n}\,(\hat{\theta}-\theta_0) \;\xrightarrow{d}\; \mathcal{N}(0,\, \Sigma), \] where \(\Sigma\) depends on the variability of \(\psi\) and on how rapidly its expectation changes with \(\theta\).

4.2 Asymptotic variance and the sandwich form

Let \[ A = \mathbb{E}\left[\nabla_\theta \psi(\theta_0; X)\right], \quad B = \mathbb{E}\left[\psi(\theta_0; X)\psi(\theta_0; X)^\top\right]. \] A widely used asymptotic variance is the “sandwich” form: \[ \Sigma = A^{-1} B (A^{-1})^\top. \] This structure separates sensitivity (through \(A\)) from randomness or dispersion (through \(B\)).

4.3 Godambe information and efficiency

In this framework, “information” is generalized via the Godambe quantity, which links \(A\) and \(B\) to describe achievable efficiency under the chosen estimating function. Different choices of \(\psi\) can lead to different asymptotic variances; selecting an estimating function with smaller \(\Sigma\) yields more efficient estimators in the asymptotic sense.

4.4 Consistency of variance estimates

In practice, \(A\) and \(B\) are unknown and are replaced by empirical analogues evaluated at \(\hat{\theta}\). Under suitable conditions, these plug-in estimates converge to their population counterparts, making standard errors based on the sandwich variance consistent.

5. Common Construction Techniques

5.1 Method of moments estimating functions

Method-of-moments (MoM) techniques build estimating functions directly from theoretical moments. If certain moments of \(X\) can be expressed as functions of \(\theta\), one equates empirical moments to their theoretical counterparts. This yields estimating equations that are often intuitive and easy to compute, though efficiency may be limited compared with likelihood-based or optimally weighted methods.

5.2 Score-type (likelihood-inspired) estimating functions

Score-type estimating functions arise from differentiating a log-likelihood or pseudo-likelihood. Even when a full likelihood is unavailable or replaced by a working model, score-like estimating equations can be used to target parameters associated with the assumed structure. These methods can be efficient when the working model is correct, while still providing a coherent estimating framework when it is not fully accurate (especially with robust variance calculations).

5.3 GMM-style moment conditions

Generalized method of moments (GMM) uses moment conditions \(\mathbb{E}[m(X,\theta)]=0\) and estimates \(\theta\) by minimizing a weighted quadratic form of empirical moments. A GMM estimator can be viewed as an estimator determined by the estimating-function machinery, where \(\psi(\theta)\) can incorporate weighting choices. Proper selection of the weight matrix aims to reduce asymptotic variance.

5.4 Estimating functions for dependent data

For dependent observations, the naive independent-data aggregation may fail to capture the correct variability. A common remedy is to define estimating functions using cluster sums or to incorporate a working dependence structure. While the form of \(\psi\) remains the same conceptually, the variance component \(B\) changes to reflect dependence, and consistent sandwich-type adjustments are often used.

5.5 Resampling-based alternatives (bootstrap/jackknife)

When asymptotic approximations are unreliable—such as in small samples—resampling methods may be used to approximate the sampling distribution of \(\hat{\theta}\). Bootstrap variants can handle many estimating-equation settings by repeatedly drawing samples (with or without resampling structure) and solving the estimating equations each time. The jackknife can also be used to estimate bias and variability, though its performance depends on regularity and the sensitivity of the estimator to individual observations.

6. Implementation Details

6.1 Numerical solution strategies for estimating equations

Solving \(\hat{\psi}(\theta)=0\) often requires numerical methods, especially for nonlinear \(\psi\). Common approaches include:

  • Newton or quasi-Newton iterations based on approximations to derivatives;
  • fixed-point iterations when \(\psi\) can be rearranged;
  • root-finding routines that use Jacobian estimates or line searches.

Convergence can depend strongly on initialization and the conditioning of the derivative matrix.

6.2 Handling constraints on parameters

If \(\theta\) must satisfy constraints (e.g., positivity, bounds, or manifold constraints), implementation may use reparameterizations that map an unconstrained space into the constrained one. Alternatively, constrained optimization or constrained root-finding methods can be used. The constraint handling must preserve the validity of asymptotic approximations, typically requiring regularity at the constrained solution.

6.3 Convergence diagnostics

Because estimating equations can be sensitive to numerical issues, diagnostics help assess whether the computed solution is meaningful. Typical checks include:

- verifying that \(\|\hat{\psi}(\hat{\theta})\|\) is sufficiently small;
  • monitoring iteration progress and step sizes;
  • inspecting whether the estimated Jacobian is near-singular;
  • repeating with multiple starting points to ensure consistent convergence.

6.4 Regularization and stabilization

When estimating equations are ill-conditioned, regularization can improve stability. Examples include adding damping to Newton updates, using ridge-type adjustments to the Jacobian inversion, or adopting robust weighting to limit the effect of extreme observations. Regularization must be chosen carefully, as overly strong stabilization can distort the estimator and affect inference.

6.5 Practical choice of tuning parameters

Some estimating procedures include tuning parameters such as bandwidths, trimming levels, robustification constants, or weight-matrix choices. Practical selection often relies on cross-validation, information criteria, or theory-guided heuristics. Regardless of strategy, tuning choices should be accompanied by sensitivity checks to assess how inference changes across reasonable alternatives.

7. Variance Estimation and Inference

7.1 Standard errors via asymptotic approximations

Given an asymptotic variance \(\Sigma(\theta)\), standard errors are obtained by evaluating a consistent estimator \(\hat{\Sigma}=\hat{A}^{-1}\hat{B}(\hat{A}^{-1})^\top\) at \(\hat{\theta}\). This yields approximate standard errors for each component of \(\hat{\theta}\), enabling routine reporting and downstream inference.

7.2 Confidence intervals for parameter estimates

Using asymptotic normality, approximate confidence intervals take the form \[ \hat{\theta}_j \pm z_{\alpha/2}\,\mathrm{se}(\hat{\theta}_j). \] For nonlinear or constrained settings, alternative interval constructions (e.g., transforming parameters or using profile-type approaches) may provide more reliable coverage.

7.3 Hypothesis testing based on estimating functions

Hypothesis tests can be built from estimating equations using Wald-type, score-type, or likelihood-ratio analogues designed for moment conditions. A common Wald framework compares a function of the estimator to a hypothesized value using the estimated covariance. For testing multiple restrictions, quadratic forms in \(\hat{\psi}\) or in restricted and unrestricted estimates are often used.

7.4 Small-sample adjustments

When sample sizes are limited or when higher-order terms matter, asymptotic inference may be inaccurate. Adjustments include:

  • using improved sandwich estimators with finite-sample corrections;
  • applying bootstrap calibration for test statistics and confidence intervals;
  • employing bias-correction techniques for estimators derived from implicit equations.

The choice depends on computational budget and how close the data are to the asymptotic regime.

8. Diagnostics and Model Checking

8.1 Residuals and empirical process checks

Although estimating functions do not rely on likelihood residuals in the same way as purely likelihood-based models, residual-like diagnostics can still be informative. For moment-based setups, one can examine the empirical moment contributions \(\psi(\hat{\theta}; X_i)\) and look for systematic patterns. More advanced checks compare the observed behavior to what would be expected if the moment conditions were well satisfied.

8.2 Moment condition diagnostics

A direct diagnostic is to evaluate the magnitude of \(\hat{\psi}(\hat{\theta})\) and to inspect each component if \(\psi\) is vector-valued. If certain moments are persistently far from zero, the model’s implied moment restrictions may be violated, suggesting misspecification or the need for an alternative estimating function.

8.3 Sensitivity analysis under misspecification

Estimating functions often rely on a “working” assumption that moments are correct even if other parts of the model are not. Sensitivity analysis varies the estimating function or the weighting scheme to see whether conclusions change materially. Large shifts in estimates or standard errors can indicate that the chosen moment conditions are not robust to plausible deviations.

8.4 Simulation-based validation

Simulation studies can be used to assess finite-sample performance, including bias, coverage, and type-I error under relevant data-generating mechanisms. This approach is particularly useful when the estimating equations are complex or when asymptotic approximations are questionable.

9. Applications and Examples

9.1 Estimating mean and regression parameters

Estimating equations appear in straightforward problems such as estimating a mean or fitting regression models. For example, setting \(\psi(\theta; X)\) equal to score-like residual moment conditions yields estimators closely related to least squares or generalized linear model estimators, depending on the structure. In these settings, the estimation equation approach clarifies which moments are being enforced and how they translate into parameter updates.

9.2 Robust estimation in simple models

In simple distributional models, robustness can be achieved by replacing fragile moments (e.g., those heavily influenced by outliers) with more stable estimating functions. For instance, using bounded transformations of residuals can reduce the effect of extreme observations on both the estimator and its variance. The sandwich variance provides a way to maintain valid inference even when the model’s full structure is not perfectly satisfied.

9.3 Survival and time-to-event contexts (high-level overview)

Time-to-event analyses commonly involve hazard-related models and moment conditions that incorporate censoring. Estimating-function methods can be adapted to handle censoring mechanisms through appropriate contributions to \(\psi\), yielding estimators that solve censoring-adjusted moment equations. Inference then uses variance estimators consistent with the dependence induced by censoring and the estimating equation structure.

9.4 Generalized estimating equations in practice (overview)

Generalized estimating equations (GEEs) are a widely used framework for correlated outcomes. The core idea is to choose \(\psi\) so that it targets marginal effects under a working model for the mean structure, while allowing robust variance adjustments to accommodate within-cluster correlation. In implementation, GEEs solve estimating equations iteratively and report sandwich-based standard errors to reflect potential misspecification of the correlation working structure.

9.5 Teaching examples and worked toy problems

Toy problems often illustrate the full pipeline: define a moment condition, form \(\hat{\psi}(\theta)\), solve for \(\hat{\theta}\), and compute asymptotic standard errors using the sandwich formula. These examples help build intuition about uniqueness of solutions, the role of weighting, and the effect of misspecifying the assumed moment conditions.

10. Variants and Extensions

10.1 Penalized estimating functions

Penalized versions add regularization terms to stabilize estimation, especially in high-dimensional or ill-posed settings. One approach is to alter the estimating function by including gradients of a penalty, turning the root-finding problem into one that balances moment matching with complexity control.

Empirical likelihood methods combine estimating-equation constraints with likelihood-inspired weighting. They treat the moment conditions as constraints on a nonparametric distribution and can yield confidence regions with appealing calibration. These hybrids connect estimation-equation thinking with likelihood-like inference.

10.3 Two-stage and composite estimating approaches

Some problems benefit from separating nuisance estimation from the main parameter estimation. Two-stage methods first estimate nuisance quantities, plug them into the estimating function for \(\theta\), and then solve the resulting equation. Composite estimating approaches combine multiple sets of moment restrictions or submodels to improve efficiency or capture different aspects of the data.

10.4 Multivariate parameter cases

When \(\theta\) is multivariate, estimating functions become vector-valued systems. Implementation and inference then depend on the conditioning of Jacobian matrices and on how the chosen moments span the parameter space. Identifiability and uniqueness become central considerations, and variance estimation requires careful handling of cross-covariances.

10.5 Estimating functions for model selection criteria (overview)

Beyond estimating a parameter, estimating-function machinery can be adapted to assist model choice by evaluating how well moment conditions fit across candidate models. Criteria may be constructed from the estimated discrepancy of moments or from sensitivity-variance tradeoffs. These tools provide an alternative to purely likelihood-based selection, particularly when likelihoods are unavailable or unreliable.