1 Architecture
EfficientNet's architecture is built upon a carefully designed baseline network discovered through neural architecture search, combined with a novel compound scaling method that uniformly scales multiple dimensions.
1.1 Baseline Design (EfficientNet-B0)
The baseline EfficientNet-B0 network uses a series of mobile inverted bottleneck convolutional (MBConv) layers with squeeze-and-excitation (SE) blocks, arranged in a certain depth pattern. The network consists of an initial 3×3 convolutional layer, followed by seven stages of MBConv blocks, and ends with a 1×1 convolutional layer, a global average pooling layer, and a fully connected classification layer. Each stage’s feature map size reduces and channel count increases according to the design discovered by neural architecture search.
1.1.1 Mobile Inverted Bottleneck (MBConv)
The MBConv block is the core building block of EfficientNet. It follows an inverted bottleneck structure: first a 1×1 convolution expands the number of channels by a factor (expansion ratio), then a depthwise convolution (e.g., 3×3 or 5×5) performs spatial filtering, and finally a 1×1 convolution projects the channels back to a lower dimension (the output channels). This design reduces computational cost compared to standard convolutions while preserving representational power.
1.1.2 Squeeze-and-Excitation (SE) Blocks
Each MBConv block in EfficientNet incorporates a squeeze-and-excitation block. The SE operation performs global average pooling on the feature map (the “squeeze”), passes the resulting vector through two fully connected layers with a reduction ratio, and then multiplies the output as a set of channel-wise scaling factors (the “excitation”). This mechanism adaptively recalibrates channel importance, improving performance with minimal extra parameters.
1.2 Compound Scaling Method
The key innovation of EfficientNet is compound scaling, which uniformly scales network depth, width, and resolution using a single coefficient ϕ. Prior methods scaled only one dimension (e.g., depth or width), leading to suboptimal efficiency.
1.2.1 Scaling Dimensions: Depth, Width, Resolution
- Depth (number of layers): deeper networks can capture more complex features.
- Width (number of channels per layer): wider networks capture finer features.
- Resolution (input image size): higher resolution provides more detail.
Compound scaling recognizes that these dimensions are interdependent. For instance, higher resolution requires deeper or wider networks to effectively use the extra pixels. Scaling them together yields better accuracy per FLOP.
1.2.2 Scaling Coefficient Formula
Given a base network (EfficientNet-B0) with dimensions depth d₀, width w₀, and resolution r₀, the scaled networks (EfficientNet-B1 through B7) use:
- Depth: d = d₀ · α^ϕ
- Width: w = w₀ · β^ϕ
- Resolution: r = r₀ · γ^ϕ
where α, β, γ are constants determined by a small grid search (α·β²·γ² ≈ 2 for FLOPs scaling by 2^ϕ). ϕ is the compound coefficient (e.g., ϕ=0 for B0, ϕ=1 for B1, up to ϕ=7 for B7). The constants α=1.2, β=1.1, γ=1.15 were chosen from the search.
1.3 Variant Progression (B1–B7)
Starting from EfficientNet-B0, applying the compound scaling with increasing ϕ yields a family of models: B1 (ϕ=1) through B7 (ϕ=7). Each variant increases the number of layers, channels, and input resolution. For example, B0 uses 224×224 input; B7 uses 600×600. The scaling rule ensures that FLOPs roughly double for each increment in ϕ. All variants maintain the same base architecture shape (i.e., stage structure) but with scaled dimensions.
2 Training and Optimization
EfficientNet models require careful training procedures to achieve their reported accuracy. The original paper uses a combination of hyperparameter tuning, data augmentation, and regularization.
2.1 Training Hyperparameters
All EfficientNet variants are trained on ImageNet (1.28 million images, 1000 classes) with consistent hyperparameters.
2.1.1 Learning Rate and Scheduler
A momentum optimizer (Nesterov) with an initial learning rate of 0.256 (for batch size 4096) is used, decayed using a cosine learning rate scheduler. The learning rate is scaled linearly with batch size. Training runs for 350 epochs.
2.1.2 Data Augmentation (AutoAugment, RandAugment)
The original EfficientNet uses AutoAugment, a search-based augmentation policy, to improve generalization. Later work introduced RandAugment, a simpler approach with two parameters (number of augmentations and magnitude), which also works well with EfficientNet. These augmentations include geometric and color transformations.
2.2 Regularization Techniques
To prevent overfitting on the large ImageNet dataset, several regularization methods are employed.
2.2.1 Dropout and Stochastic Depth
Dropout is applied before the final classification layer with a dropout rate that increases with model size (e.g., 0.2 for B0, 0.5 for B7). Stochastic depth (a variant of drop-connect) is used: during training, some MBConv blocks are randomly dropped with a survival probability that decreases linearly from 1.0 (first block) to 0.8 (last block).
2.2.2 Label Smoothing
Label smoothing is applied during training, setting the target label distribution to a mixture of one-hot labels and a uniform distribution (ε=0.1). This reduces overconfidence and improves generalization.
2.3 Comparison with Prior Architectures
EfficientNet models are compared against prior state-of-the-art networks such as ResNet, DenseNet, and AmoebaNet.
2.3.1 Parameter Efficiency
EfficientNet-B7 achieves 84.3% top-1 accuracy on ImageNet with 66 million parameters, while ResNet-152 (a much deeper network) achieves 77.8% with 60 million parameters. Smaller variants like B0 (5.3M parameters) outperform ResNet-50 (25M parameters) by 1.6% absolute accuracy.
2.3.2 Accuracy vs. FLOPs Trade-off
The compound scaling method yields a better accuracy-FLOPs Pareto frontier compared to prior scaling approaches. For example, EfficientNet-B3 uses about 1.8B FLOPs and achieves 81.1% top-1 accuracy, while AmoebaNet-C (similar FLOPs) achieves 80.5%. Larger models (B7, 37B FLOPs) maintain the advantage.
3 Applications and Transfer Learning
EfficientNet serves as a strong backbone for many computer vision tasks beyond classification.
3.1 Image Classification
EfficientNet was designed primarily for image classification and excels on ImageNet and other datasets.
3.1.1 ImageNet Performance
On ImageNet, EfficientNet-B7 achieved a top-1 accuracy of 84.3% (with extra training data), and B8 (a variant not part of the standard B1–B7 progression) reached 85.0%. These results were state-of-the-art at the time of publication.
3.1.2 Beyond ImageNet: Fine-tuning on Custom Datasets
Pre-trained EfficientNet models are widely used for transfer learning. They achieve strong performance on smaller datasets (e.g., CIFAR-100, Flowers, or medical imaging) after fine-tuning, often with fewer parameters than other backbones.
3.2 Object Detection
EfficientNet can be used as a feature extractor in object detection frameworks like RetinaNet or Mask R-CNN.
3.2.1 EfficientDet Integration
The EfficientDet family, also from Google Research, uses EfficientNet as a backbone combined with a weighted bi-directional feature pyramid network. EfficientDet variants achieve superior accuracy and speed trade-offs on the COCO dataset.
3.3 Semantic Segmentation
When used as an encoder in segmentation models (e.g., DeepLabV3+, PSPNet), EfficientNet provides a high-quality feature representation. Pre-trained weights on ImageNet improve convergence and accuracy on segmentation tasks like Cityscapes or PASCAL VOC.
3.4 Edge and Mobile Deployment
EfficientNet’s efficiency makes it attractive for resource-constrained environments.
3.4.1 EfficientNet-Lite Variants
EfficientNet-Lite is a set of variants optimized for mobile and on-device inference using TensorFlow Lite. They use ReLU instead of Swish activation, remove SE blocks in early stages, and adjust scaling to reduce latency while maintaining reasonable accuracy.
4 Extensions and Variants
The success of EfficientNet has spurred numerous extensions and improvements.
4.1 EfficientNetV2
EfficientNetV2, introduced in 2021, improves training speed and efficiency.
4.1.1 Fused-MBConv Blocks
EfficientNetV2 replaces early MBConv blocks with Fused-MBConv, which uses a single 3×3 standard convolution followed by a 1×1 projection. This reduces the number of depthwise convolutions in early stages, improving training speed on modern hardware.
4.1.2 Progressive Training Strategy
EfficientNetV2 uses progressive training: it gradually increases image size and regularization strength during training, leading to faster convergence and higher final accuracy.
4.2 EfficientNet with Attention Mechanisms
Several works have integrated additional attention mechanisms (e.g., coordinate attention, convolutional block attention) into EfficientNet blocks to further improve performance, particularly for fine-grained classification tasks.
4.3 Noisy Student Semi-Supervised Training
Google Research applied a semi-supervised learning method called “Noisy Student” to EfficientNet. This technique trains a student model using pseudo-labels produced by a teacher model and adds noise (dropout, stochastic depth, and data augmentation). EfficientNet-L2 (an extra-large variant not in the B series) achieved 88.5% top-1 accuracy on ImageNet using this approach.
5 Evaluation Benchmarks
EfficientNet models are evaluated on standard image classification benchmarks and latency measurements.
5.1 Standard Benchmarks
The primary benchmark is ImageNet ILSVRC 2012. Models are also tested on CIFAR-10/100, Flowers, and other transfer learning datasets. EfficientNet consistently sets new state-of-the-art accuracy on these benchmarks at the time of release.
5.2 Model Size and Inference Latency
Model size (number of parameters) and inference latency (on TPUs and GPUs) are reported. For instance, EfficientNet-B0 has 5.3M parameters and takes about 0.4 seconds per 100 images on a single GPU (at 224×224), while B7 has 66M parameters and about 2.0 seconds per 100 images (at 600×600).
6 Limitations and Considerations
Despite its advantages, EfficientNet has some limitations.
6.1 Sensitivity to Input Resolution
The compound scaling method fixes the input resolution for each variant. Using a resolution different from the intended one may degrade performance. Additionally, high-resolution variants (e.g., B7 at 600×600) require significant memory and may not be feasible on small hardware.
6.2 Memory Constraints on Small Devices
On mobile devices or microcontrollers, even the Lite variants may be too large or slow. The depthwise convolutions in MBConv blocks are not always efficient on all hardware platforms (e.g., CPU without dedicated depthwise instructions).
7 Future Directions
Several exciting research avenues build upon EfficientNet.
7.1 Integration with Modern Architectures (e.g., Transformers)
Vision Transformers (ViTs) and hybrid architectures (e.g., ConvNeXt) have emerged. Future work may combine EfficientNet’s compound scaling principle with transformer blocks or use EfficientNet as a tokenizer for vision-language models.
7.2 Automated Machine Learning (AutoML) for Compound Scaling
The compound scaling coefficients (α, β, γ) were determined by a small grid search. Future AutoML methods could automatically discover optimal scaling strategies for specific tasks, hardware constraints, or data distributions, further improving efficiency.