A feedforward neural network (FNN) is an artificial neural network in which connections between nodes do not form cycles. Information flows strictly in one direction—from input nodes, through one or more hidden layers, to output nodes—making it the simplest and most fundamental architecture in deep learning. FNNs are used for tasks such as classification, regression, and pattern recognition, and are the basis for more complex network designs.
1 Architecture
The architecture of a feedforward neural network consists of multiple layers of nodes (neurons), each fully connected to the next layer. Information passes from the input layer through successive hidden layers to the output layer, with no feedback connections.
1.1 Input layer
The input layer is the first layer of the network. It receives the raw data, with each neuron representing a single feature of the input. No computation is performed here; the input values are simply passed to the first hidden layer. The number of neurons in the input layer equals the dimensionality of the input data.
1.2 Hidden layers
Hidden layers are the intermediate layers between the input and output layers. Each hidden layer applies a weighted sum of its inputs followed by a nonlinear activation function. The term "hidden" refers to the fact that these layers are not directly observable from outside the network.
1.2.1 Depth and width
Depth refers to the number of hidden layers in the network; a network with many hidden layers is called a deep network. Width refers to the number of neurons in a given hidden layer. Increasing depth allows the network to learn hierarchical representations, while increasing width provides more capacity per layer. Both depth and width must be balanced to avoid overfitting or underfitting.
1.3 Output layer
The output layer produces the final result of the network. Its structure and activation function depend on the task: regression tasks typically use a linear activation, binary classification uses a single neuron with a sigmoid activation, and multi-class classification uses one neuron per class with a softmax activation.
1.4 Activation functions
Activation functions introduce nonlinearity into the network, allowing it to learn complex patterns. Without them, the entire network would be equivalent to a linear transformation.
1.4.1 Sigmoid
The sigmoid function maps any real-valued number to a value between 0 and 1, defined as σ(x) = 1 / (1 + e⁻ˣ). It is historically popular for binary classification output layers but suffers from saturation, which can cause vanishing gradients during training.
1.4.2 Rectified linear unit (ReLU)
The ReLU function outputs the input directly if it is positive, and zero otherwise: ReLU(x) = max(0, x). It is computationally efficient and helps mitigate the vanishing gradient problem, making it the default activation for most hidden layers in modern FNNs.
1.4.3 Softmax
The softmax function converts a vector of raw scores (logits) into a probability distribution over multiple classes. For an output vector z, the softmax for class i is e^(z_i) / Σ_j e^(z_j). It is the standard activation for the output layer in multi-class classification.
2 Training
Training a feedforward neural network involves adjusting its weights and biases to minimize a loss function. This is typically done using gradient-based optimization and the backpropagation algorithm.
2.1 Loss functions
The loss function quantifies the difference between the network’s predictions and the true targets. The choice of loss function depends on the task.
2.1.1 Mean squared error
Mean squared error (MSE) is the average of the squared differences between predicted and actual values: MSE = (1/n) Σ (ŷ – y)². It is commonly used for regression problems.
2.1.2 Cross-entropy loss
Cross-entropy loss measures the dissimilarity between the predicted probability distribution and the true distribution. For binary classification it is often called binary cross-entropy; for multi-class, categorical cross-entropy. It is preferred over MSE for classification tasks because it penalizes confident wrong predictions more heavily.
2.2 Backpropagation
Backpropagation is the algorithm used to compute the gradient of the loss function with respect to each weight in the network. It applies the chain rule of calculus to propagate errors backward from the output layer to the input layer.
2.2.1 Chain rule of calculus
The chain rule allows the derivative of a composite function to be expressed as the product of derivatives of its constituent functions. In a neural network, the loss is a composition of many layers; the chain rule lets us compute how changes in early-layer weights affect the final loss through intermediate derivatives.
2.2.2 Gradient computation
During backpropagation, the gradient for each weight is computed as the product of the local gradient (derivative of the activation and weighted sum) and the upstream gradient from the next layer. This process is repeated layer by layer, from output to input, resulting in efficient gradient calculations for all weights in the network.
2.3 Optimization algorithms
Optimization algorithms use the computed gradients to update the network’s weights and minimize the loss. The most fundamental of these is gradient descent, but various enhancements have been developed.
2.3.1 Stochastic gradient descent (SGD)
Stochastic gradient descent updates weights using the gradient computed from a single training example (or a small mini-batch) rather than the full dataset. This introduces noise that can help escape local minima and accelerates training on large datasets. The update rule is w ← w – η ∇L, where η is the learning rate.
2.3.2 Adam optimizer
Adam (Adaptive Moment Estimation) combines ideas from SGD with momentum and adaptive learning rates. It maintains per-parameter learning rates that are adjusted based on the first and second moments of the gradients. Adam is widely used due to its robust performance across many architectures and tasks.
3 Variations and extensions
While the basic feedforward architecture is simple, several variations and extensions have been developed to improve performance or adapt to specific problems.
3.1 Multi-layer perceptron (MLP)
The term "multi-layer perceptron" is often used synonymously with a feedforward neural network that has at least one hidden layer. Historically, the perceptron was a single-layer model; the addition of hidden layers and nonlinear activations turned it into the MLP, which can learn nonlinear decision boundaries.
3.2 Deep feedforward networks
A deep feedforward network is simply an FNN with many hidden layers (typically three or more). The depth allows the network to learn hierarchical feature representations, from simple edges in early layers to complex concepts in later layers. Deep FNNs are the foundation of modern deep learning but can be challenging to train due to issues like vanishing gradients.
3.3 Relationship to convolutional and recurrent architectures
Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are specialized variants of feedforward networks with additional structural constraints. CNNs replace fully connected layers with convolutional and pooling layers to exploit spatial locality, while RNNs introduce feedback connections to process sequential data. Despite these modifications, both can be seen as extensions of the feedforward concept, and many modern hybrid architectures combine feedforward components with these specialized layers.
4 Applications
Feedforward neural networks are versatile and have been applied to a wide range of domains. The following are typical use cases.
4.1 Pattern recognition
FNNs can be trained to recognize patterns in data, such as handwritten digits, facial features, or voice commands. The input layer receives raw pixel values or preprocessed features, and the output layer predicts the class label. Despite the rise of CNNs for image tasks, FNNs remain useful for smaller or fully connected feature problems.
4.2 Regression analysis
In regression, an FNN predicts a continuous value, such as housing prices, temperature, or stock market indices. The network learns a nonlinear mapping from input features to the output, often outperforming traditional linear regression when the relationship is complex.
4.3 Time series forecasting
Although recurrent networks are better suited for sequences, FNNs can be applied to time series forecasting using a sliding window approach: past values are fed as inputs to predict future values. This works well for short-term predictions with moderate temporal dependencies, such as energy load forecasting or sales prediction.
5 Limitations and challenges
Despite their simplicity and effectiveness, feedforward neural networks face several inherent limitations that practitioners must address.
5.1 Vanishing gradient problem
During backpropagation, gradients can become extremely small as they are multiplied through many layers, especially when using saturating activation functions like sigmoid or tanh. This slows or halts learning in early layers. ReLU activations and techniques like batch normalization help mitigate this issue, but very deep FNNs still require careful initialization and architectural choices.
5.2 Overfitting
Because FNNs have many parameters, they are prone to overfitting—learning noise in the training data instead of generalizable patterns. Overfitting is especially problematic when the dataset is small relative to the network size.
5.2.1 Regularization techniques (dropout, L2 regularization)
Regularization methods reduce overfitting by imposing constraints on the network. Dropout randomly disables a fraction of neurons during training, forcing the network to learn redundant representations. L2 regularization (weight decay) adds a penalty proportional to the squared magnitude of weights to the loss function, encouraging smaller weights and simpler models. Both are standard tools for training FNNs.