The sigmoid kernel is a kernel function used in machine learning, particularly in support vector machines (SVMs) and other kernel-based methods. It is defined as \( K(\mathbf{x}, \mathbf{y}) = \tanh(\alpha \mathbf{x}^\top \mathbf{y} + c) \), where \(\alpha\) is a scaling parameter and \(c\) is a constant offset. This kernel is derived from the activation function of neural networks and allows SVMs to mimic the behavior of a two-layer perceptron. However, it is not positive semidefinite for all parameter choices, which can limit its theoretical guarantees in kernel methods.
1 Definition and Mathematical Formulation
1.1 Standard form: \( K(x, y) = \tanh(\alpha x^\top y + c) \)
The sigmoid kernel is expressed as \[ K(\mathbf{x}, \mathbf{y}) = \tanh(\alpha \mathbf{x}^\top \mathbf{y} + c), \] where \(\mathbf{x}, \mathbf{y} \in \mathbb{R}^n\) are input vectors, \(\alpha > 0\) is a scaling parameter, and \(c \in \mathbb{R}\) is a constant offset. The hyperbolic tangent function \(\tanh\) squashes the real-valued inner product into the range \((-1, 1)\).
1.2 Parameter roles: slope parameter \(\alpha\) and intercept constant \(c\)
The parameter \(\alpha\) controls the slope of the hyperbolic tangent at the origin. Larger \(\alpha\) values make the kernel more sensitive to differences between \(\mathbf{x}\) and \(\mathbf{y}\), while smaller values produce a flatter response. The intercept \(c\) shifts the argument of \(\tanh\), effectively adjusting the decision boundary's offset in the kernel-induced feature space. When \(c\) is positive, the kernel tends to yield values near 1 for strongly correlated inputs; when negative, it approaches \(-1\) for dissimilar inputs.
1.3 Relationship to hyperbolic tangent (tanh) function
The hyperbolic tangent function, \(\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}\), is a sigmoid-shaped function that maps any real number to the interval \((-1, 1)\). Its use as a kernel function directly borrows from the activation function of artificial neurons in multi-layer perceptrons (MLPs). For the sigmoid kernel, the argument \(z = \alpha \mathbf{x}^\top \mathbf{y} + c\) makes it a nonlinear similarity measure that can be negative for sufficiently dissimilar pairs, unlike positive-valued kernels such as the Gaussian RBF.
2 Properties and Theoretical Considerations
2.1 Positive semidefiniteness condition
A kernel function must be positive semidefinite (PSD) for most kernel-method guarantees (such as convexity of the optimization problem in SVMs). For the sigmoid kernel, the Gram matrix \(\mathbf{K}\) with entries \(K_{ij} = \tanh(\alpha \mathbf{x}_i^\top \mathbf{x}_j + c)\) is not always PSD. The condition for PSD depends on both \(\alpha\) and \(c\). In practice, for many datasets and certain parameter choices, the Gram matrix may be indefinite, meaning it has both positive and negative eigenvalues.
2.2 Mercer’s theorem and validity as a kernel
Mercer’s theorem requires the kernel to be continuous, symmetric, and positive semidefinite to define a reproducing kernel Hilbert space (RKHS). The sigmoid kernel often fails the PSD requirement, and therefore it does not satisfy Mercer's condition for all parameter values. Nevertheless, it is still used empirically because the resulting SVM optimization may converge even with an indefinite kernel, though theoretical convergence guarantees are weakened.
2.2.1 Cases where the kernel is conditionally positive definite
Under certain restrictions, the sigmoid kernel can be conditionally positive definite (CPD). For example, when \(c = 0\) and \(\alpha\) is sufficiently small, the kernel \(\tanh(\alpha \mathbf{x}^\top \mathbf{y})\) can be shown to be PSD for data with unit norm. More generally, specific choices of \(\alpha\) and \(c\) can produce a kernel that is CPD, meaning it is positive semidefinite on the subspace orthogonal to the constant vector.
2.2.2 Lack of universal kernel property
A universal kernel (such as the Gaussian RBF) is capable of approximating any continuous function uniformly on compact sets. The sigmoid kernel does not possess this property, partly because its feature map is bounded (outputs lie in \((-1, 1)\)) and because it is not characteristic. This limits its expressiveness in tasks like density estimation or two-sample testing.
2.3 Symmetry and stationarity
The sigmoid kernel is symmetric: \(K(\mathbf{x}, \mathbf{y}) = K(\mathbf{y}, \mathbf{x})\) because the dot product is symmetric. However, it is not stationary: it depends on the absolute positions of the vectors via the dot product rather than on the difference \(\mathbf{x} - \mathbf{y}\). This means the kernel is not invariant under translation, a property that distinguishes it from the Gaussian RBF and the Laplace kernel.
3 Applications in Machine Learning
3.1 Use in Support Vector Machines (SVMs)
In SVMs, the sigmoid kernel is often employed to create nonlinear decision boundaries. Despite its non-PSD behavior in some parameter regimes, practitioners have successfully used it for binary and multiclass classification tasks. The kernel function replaces the standard linear inner product, allowing the SVM to learn separating hyperplanes in a feature space induced by the hyperbolic tangent mapping.
3.1.1 Multiclass classification via sigmoid kernel
For multiclass problems, the sigmoid kernel can be combined with one-vs-one or one-vs-rest strategies. The resulting SVM classifiers produce decision functions that are sums of weighted sigmoid kernel evaluations. Because the kernel output can be negative, the decision values may not directly correspond to probabilities, but they still suffice for class assignment.
3.1.2 Parameter tuning for SVMs
Choosing \(\alpha\) and \(c\) is critical for SVM performance. A common approach is to perform grid search with cross-validation, evaluating metrics such as accuracy or F1-score. Values of \(\alpha\) near 1 and \(c\) between 0 and 1 are typical starting points. Some implementations (e.g., LIBSVM) use default values of \(\alpha = 1/n\) (where \(n\) is the number of features) and \(c = 0\).
3.2 Use in Kernel Principal Component Analysis (KPCA)
Kernel PCA uses eigendecomposition of the kernel matrix to find nonlinear principal components. The sigmoid kernel can be applied in KPCA, though the resulting kernel matrix may not be PSD. In practice, many KPCA implementations ignore the PSD requirement and still produce meaningful projections; negative eigenvalues may be treated as noise or discarded.
3.3 Use in Kernel Ridge Regression
Kernel ridge regression (KRR) assumes the kernel matrix is PSD to ensure a unique solution. Using a sigmoid kernel that yields indefinite matrices can cause numerical instability or non-convex loss surfaces. Nonetheless, some studies have applied the sigmoid kernel in KRR with regularization, where the regularized inverse \((\mathbf{K} + \lambda \mathbf{I})^{-1}\) can stabilize the problem even for indefinite \(\mathbf{K}\).
3.4 Connection to multi-layer perceptrons (MLPs)
The sigmoid kernel is directly inspired by the activation function of a two-layer perceptron. In fact, an SVM with a sigmoid kernel is equivalent to a neural network with one hidden layer and one output neuron, where the hidden layer uses the hyperbolic tangent activation and the output layer is a linear SVM. This bridges kernel methods and neural networks, though the kernel version does not train the hidden layer weights (they are fixed by the support vectors).
4 Comparison with Other Kernel Functions
4.1 Sigmoid vs. Gaussian RBF kernel
4.1.1 Shape and locality differences
| The Gaussian RBF kernel \(K(\mathbf{x}, \mathbf{y}) = \exp(-\gamma \|\mathbf{x} - \mathbf{y}\|^2)\) is local: it decays to zero for distant points. The sigmoid kernel is global: it can yield large negative values for very dissimilar vectors (due to the tanh function) and positive values for similar ones. This global nature can cause the sigmoid kernel to produce a decision boundary influenced by all training points, whereas the RBF kernel yields more local boundaries. |
|---|
4.1.2 Impact on decision boundaries
In SVMs, the sigmoid kernel often produces decision boundaries that resemble those of a neural network, sometimes with multiple disconnected regions. The RBF kernel, by contrast, yields smooth, connected decision boundaries with a single region per class (unless combined with other kernels). The sigmoid kernel can also create "holes" in class regions because its negative values introduce repulsion effects.
4.2 Sigmoid vs. Polynomial kernel
4.2.1 Similarities in global behavior
Both the sigmoid and polynomial kernels are global: they depend on the dot product \(\mathbf{x}^\top \mathbf{y}\) rather than on the Euclidean distance. For the polynomial kernel \(K(\mathbf{x}, \mathbf{y}) = (\gamma \mathbf{x}^\top \mathbf{y} + r)^d\), the decision boundary is a polynomial surface. The sigmoid kernel's tanh function provides a similar global effect but with a bounded output, which can help control the influence of outliers.
4.2.2 Differences in hyperparameter sensitivity
The polynomial kernel has a degree parameter \(d\) that controls the complexity of the feature space, while the sigmoid kernel lacks such an explicit degree. However, the sigmoid kernel is more sensitive to the scaling of input features: because \(\tanh\) saturates for large arguments, features must be normalized (e.g., to unit length) to avoid numerical issues. The polynomial kernel, on the other hand, can handle unscaled data if the intercept is set appropriately.
5 Training and Optimization Issues
5.1 Instability due to non-positive definite kernel matrices
Indefinite kernel matrices introduce eigenvalues that are negative or zero. In SVM training, the dual optimization problem becomes non-convex, and solvers relying on sequential minimal optimization (SMO) may fail to converge or produce suboptimal solutions. Some implementations incorporate a "soft" correction by adding a small multiple of the identity matrix to force positive definiteness.
5.2 Choice of \(\alpha\) and \(c\) via cross-validation
Because the theoretical conditions for PSD are often unknown for a given dataset, practitioners typically rely on empirical validation. Grid search over \(\alpha\) (e.g., \(\{0.01, 0.1, 1, 10\}\)) and \(c\) (e.g., \(\{-2, -1, 0, 1, 2\}\)) is common. Cross-validation accuracy is used to select the pair that yields the best generalization. It is also advisable to check whether the resulting kernel matrix is nearly PSD (e.g., by examining its smallest eigenvalue) to avoid severe instability.
5.3 Practical recommendations for using sigmoid kernel
- Normalize input features to have unit variance or unit norm to prevent saturation of the \(\tanh\) function.
- Use small values of \(\alpha\) (e.g., \(1/\text{number of features}\)) and \(c\) near 0 as a baseline.
- Monitor training convergence: if the dual objective oscillates or does not decrease monotonically, consider switching to a PSD kernel (e.g., RBF).
- For large datasets, the sigmoid kernel can be computationally cheaper than the RBF kernel because it avoids computing Euclidean distances (only dot products are needed).
6 Extensions and Variants
6.1 Scaled and shifted sigmoid kernels
Variants such as \(K(\mathbf{x}, \mathbf{y}) = \tanh(\alpha \mathbf{x}^\top \mathbf{y} + c) + \beta\) introduce an additive constant \(\beta\) to adjust the range. Another extension is the "sigmoid-like" kernel \(K(\mathbf{x}, \mathbf{y}) = \sigma(a \mathbf{x}^\top \mathbf{y} + b)\) where \(\sigma\) is the logistic sigmoid instead of \(\tanh\). The logistic version outputs values in \((0,1)\) and is sometimes used in binary classification tasks to mimic probability outputs.
6.2 Hyperbolic tangent kernels with different base functions
| Instead of the standard dot product, some formulations replace \(\mathbf{x}^\top \mathbf{y}\) with a similarity measure such as \(\|\mathbf{x} - \mathbf{y}\|^2\) or a distance-based term. For example, \(K(\mathbf{x}, \mathbf{y}) = \tanh(-\gamma \|\mathbf{x} - \mathbf{y}\|^2 + c)\) yields a localized variant. However, such forms lose symmetry (if the argument is not symmetric) or become trivial. |
|---|
6.3 Kernel with bias term in neural network context
In neural networks, the hyperbolic tangent function is often applied to \(\mathbf{w}^\top \mathbf{x} + b\) (bias). The sigmoid kernel mirrors this by incorporating the "bias" \(c\) in the kernel expression. Some researchers have proposed learning the parameters \(\alpha\) and \(c\) directly from data using gradient-based optimization, treating the kernel as a differentiable function in meta-learning frameworks.
7 Historical Context
7.1 Origin in neural network community
The hyperbolic tangent function has been used as an activation function in neural networks since the 1980s, particularly in multi-layer perceptrons. Its use as a kernel function was first proposed in the 1990s when researchers sought to connect neural networks and SVMs. The idea was to replace the hidden layer of a neural network with a kernel–the sigmoid kernel–so that the SVM would effectively learn a neural network architecture.
7.2 Adoption in kernel methods literature
The sigmoid kernel became popular with the advent of SVM toolkits such as LIBSVM (2000) and was included as a standard kernel option. Early works by Vapnik, Schölkopf, and others noted its empirical effectiveness despite theoretical limitations. The kernel's non-PSD nature sparked research into indefinite kernels and methods for handling them (e.g., the "Krein" SVM approach).
7.3 Relation to the “neural” kernel concept
The term "neural kernel" has been used to describe kernels that mimic neural network computations. The sigmoid kernel is a prime example, alongside the "arcsin" kernel and other derived forms. In 2019, the discovery of the "neural tangent kernel" (NTK) revived interest in connections between infinite-width neural networks and kernel methods, where the NTK uses a different formulation but shares the spirit of the sigmoid kernel as a bridge between the two fields.