OpenImages is a large-scale, publicly available dataset of images with multiple annotation types, developed and released by Google. It contains approximately 9 million images annotated with image-level labels, object bounding boxes, visual relationships, and in later versions, segmentation masks. Spanning over 600 object classes and thousands of attributes, the dataset is widely used for training and evaluating computer vision models in tasks such as object detection, classification, and visual relationship detection. First released in 2016, OpenImages has undergone several major updates, each expanding the number of images, annotations, and label granularity.

1 History and Development

The development of OpenImages was motivated by the need for large, diverse, and richly annotated image datasets that could support a wide range of computer vision tasks. Google initiated the project to provide a resource that complemented existing datasets such as ImageNet and COCO, with a particular emphasis on multi-label classification and visual relationship understanding.

1.1 Initial release (V1)

The first version of OpenImages was released in July 2016. It included approximately 9 million images, each annotated with a single image-level label drawn from a set of 6,000 categories. These labels were automatically generated using human reviewers to verify machine-generated predictions. V1 provided a solid foundation for large-scale image classification but lacked spatial annotations.

1.2 Major version updates (V2–V6)

* V2 (2017): Introduced bounding box annotations for millions of objects across 600 object classes, along with visual relationship annotations. This version significantly expanded the dataset’s utility for object detection and scene understanding. * V3 (2018): Added more bounding boxes and improved the coverage of occluded and truncated objects. The class hierarchy was refined, and new attributes were introduced. * V4 (2019): Expanded the number of annotated images to over 1.9 million for bounding boxes, and added new relationship triplets. The dataset also included segmentation masks for a subset of images. * V5 (2020): Introduced full-image segmentation masks for over 350 classes, covering 2.8 million instances. This version enabled instance segmentation model training. * V6 (2021): The latest major update added more images and annotations for rare and long-tail classes, improved annotation consistency, and provided enhanced tools for data filtering.

1.3 Contributors and institutional backing

OpenImages was developed and maintained by the Google Research team. External contributors from the computer vision community provided feedback and contributed to annotation efforts through crowdsourcing platforms. The dataset was supported by Google’s infrastructure for storage, computation, and quality assurance.

2 Dataset Structure

OpenImages is structured as a collection of images and associated annotation files. Each image is stored as a JPEG file with a unique identifier. Annotations are provided in separate text files, organized by annotation type.

2.1 Image sources

2.1.1 Public domain and user-uploaded content

Images in OpenImages are sourced from Flickr, a photo-sharing platform. Only images with Creative Commons licenses or those explicitly in the public domain are included. This ensures legal usability for research and commercial applications, though users must verify license terms for individual images.

2.1.2 Filtering and quality control

Google applied automatic and manual filters to remove inappropriate content, duplicates, and low-resolution or corrupted images. Additionally, images were screened for privacy concerns (e.g., faces were not explicitly removed but were not used as annotation targets). A human-verification pipeline ensured that only images with discernible, representative content were retained.

2.2 Annotation types

OpenImages supports multiple annotation formats, enabling diverse use cases.

2.2.1 Image-level labels

Every image in the dataset has one or more image-level labels indicating the presence of certain objects, scenes, or attributes. These labels are based on a fixed ontology of over 19,000 classes, though not all classes appear in every version. Labels are provided as a list of class names per image, with confidence scores (often binary or human-verified).

2.2.2 Bounding boxes

Bounding box annotations define the spatial extent of object instances. They cover approximately 600 object classes across the dataset versions, with over 15 million boxes in V6.

2.2.2.1 Annotation format (X, Y, width, height)

Each bounding box is specified by four floating-point values: the normalized coordinates of the top-left corner (X, Y) and the normalized width and height. All values are relative to the image dimensions, ranging from 0.0 to 1.0. For example, a box with (0.2, 0.3, 0.4, 0.5) indicates that the top-left corner is at 20% of the image width and 30% of the image height, with a width of 40% and a height of 50%.

2.2.2.2 Occlusion and truncation flags

Each bounding box annotation includes two boolean flags: IsOccluded (whether the object is partially hidden by another object) and IsTruncated (whether the object extends beyond the image boundary). These flags help models handle challenging cases and are used in evaluation metrics such as average precision with mismatch tolerance.

2.2.3 Visual relationships

Visual relationship annotations describe interactions between objects. They were introduced in V2 and cover over 1,600 relationship types.

2.2.3.1 Relationship triplets (subject, predicate, object)

A relationship is represented as a triplet: (subject_object_id, predicate, object_object_id). For example, "a person holding a cup" would be represented as (person_id, "holding", cup_id). Both subject and object are identified by their bounding box annotation IDs. Predicates are drawn from a fixed set of phrases such as "on", "wearing", "playing", and "next to".

2.2.3.2 Attribute annotations

Attributes are properties associated with objects, such as color, material, or shape. They are represented as object-attribute pairs (e.g., "car – red"). Attributes are annotated for selected classes and are often used in conjunction with bounding boxes to provide fine-grained descriptions.

2.2.4 Segmentation masks (V5 onward)

Starting with V5, OpenImages includes instance segmentation masks for a large subset of images. These masks are polygon annotations that outline the exact pixels belonging to each object instance. They cover over 350 object classes and are stored in COCO-compatible format. The masks enable training of instance segmentation models and provide ground truth for pixel-level evaluation.

2.3 Class hierarchy

OpenImages uses a hierarchical ontology to organize its object classes.

2.3.1 Top-level categories

The top-level categories include major semantic groups such as "Animals", "Vehicles", "Furniture", "Food", "Clothing", and "Sports equipment". These broad categories simplify navigation and allow multi-grained learning, where models can predict both coarse and fine labels.

2.3.2 Fine-grained subclasses

Under each top-level category, fine-grained subclasses are defined. For example, under "Animals" one finds subclasses like "Dog" and "Cat", and under "Dog" further subclasses such as "Golden Retriever" and "Poodle". The hierarchy supports multi-label classification at various levels of specificity, with over 600 leaf-node classes in the bounding box ontology.

2.4 Dataset splits

OpenImages is divided into three standard splits for reproducible benchmarking.

2.4.1 Training set

The training set contains the largest portion of images – over 1.7 million images with bounding box annotations, and up to 9 million images with image-level labels. All annotations are provided publicly.

2.4.2 Validation set

The validation set consists of approximately 25,000 images with full annotations, including bounding boxes, relationships, and segmentation masks. It is used for model tuning and early evaluation.

2.4.3 Test set (without public labels)

The test set contains roughly 100,000 images. Annotations for the test set are withheld from public release to prevent overfitting. Researchers can submit predictions to a private evaluation server to obtain performance metrics on the test set.

3 Statistics

3.1 Number of images per version

  • V1: 9 million images (image-level labels only)
  • V2: 9 million image-level + 1.5 million with bounding boxes
  • V3: 9 million image-level + 1.9 million with bounding boxes
  • V4: 9 million image-level + 1.9 million with bounding boxes
  • V5: 9 million image-level + 1.9 million with bounding boxes + 2.8 million mask instances
  • V6: 9 million image-level + 2 million with bounding boxes + 3 million mask instances

3.2 Annotation counts per type

  • Image-level labels: Approximately 200 million label instances across 19,000 classes.
  • Bounding boxes: Over 15 million boxes covering 600 classes (V6).
  • Visual relationships: Over 400,000 relationship triplets with 316 object classes and 17 predicate types.
  • Segmentation masks: Over 3 million mask instances across 350 classes (V6).

3.3 Class distribution and long-tail analysis

OpenImages exhibits a long-tailed distribution: a few common classes (e.g., "Person", "Car", "Dog") have thousands of annotations, while many rare classes have fewer than 100 instances. This imbalance reflects real-world object frequencies and poses challenges for model generalization. The dataset includes explicit long-tail class splits for benchmarking.

3.4 Image resolution and diversity metrics

Images in OpenImages vary in resolution, with a median width of 640 pixels and median height of 480 pixels. Approximately 20% of images have a resolution above 1024 pixels on the longer side. The dataset covers a wide range of scenes, lighting conditions, and geographic locations, making it a diverse resource for general-purpose vision training.

4 Usage and Applications

OpenImages is extensively used in computer vision research and industry.

4.1 Object detection benchmarks

The bounding box annotations support standard object detection tasks. Researchers often evaluate models using mean Average Precision (mAP) over the 600 classes. The dataset’s large scale and class diversity make it a common benchmark for comparing detection architectures such as Faster R-CNN, YOLO, and DETR.

4.2 Visual relationship detection

The visual relationship annotations enable training models to predict interactions between objects. This task is evaluated using metrics that account for correct localization and predicate prediction.

4.2.1 Evaluation metrics (Recall@K, weighted AP)

  • Recall@K: Measures the fraction of correct relationship triplets among the top K predictions. Common values are K=50 and K=100.
  • Weighted Average Precision (wAP): Computes average precision over relationship detections, weighting by the inverse frequency of predicates to account for class imbalance.

4.3 Image classification and multi-label recognition

The image-level labels support multi-label classification tasks. OpenImages is frequently used to benchmark models for recognizing multiple objects and attributes in a single image, using metrics like mean average precision (mAP) across all labels.

4.4 Transfer learning and pretraining

Due to its size and diversity, OpenImages is used as a pretraining dataset for a variety of downstream tasks, including fine-grained classification, detection, and segmentation. Models pretrained on OpenImages often achieve competitive performance when fine-tuned on smaller datasets.

5 Limitations and Challenges

Despite its strengths, OpenImages has several known limitations.

5.1 Label noise and ambiguity

Image-level labels and bounding boxes may contain errors from the automated or crowdsourced annotation pipeline. For example, objects may be mislabeled or missed entirely. This noise can affect model training, though it also reflects real-world annotation imperfections.

5.2 Class imbalance

The long-tail distribution leads to poor performance for rare classes, especially in detection and segmentation tasks. Models may overfit to common classes and fail to generalize to rare ones.

5.3 Annotation consistency across versions

As new versions were released, annotation guidelines and class ontologies changed. This can cause compatibility issues when comparing results across versions or when using models trained on one version with another.

5.4 Bias in image sources

Because images are sourced from Flickr, the dataset may overrepresent Western, urban, and leisure-oriented content. This geographic and cultural bias can limit the generalization of models to other regions or contexts.

OpenImages shares similarities and complements several other widely used computer vision datasets.

6.1 COCO (Common Objects in Context)

COCO is a smaller dataset (330K images) with dense annotations for 80 object categories, including segmentation masks and captions. Unlike OpenImages, COCO has a more balanced class distribution and focuses on non-iconic images.

6.2 ImageNet

ImageNet contains over 14 million images with single-label classification for 22,000 categories. It lacks spatial annotations. OpenImages provides multiple labels and bounding boxes, making it more suitable for detection tasks.

6.3 Visual Genome

Visual Genome provides dense annotations for images, including objects, attributes, and relationships, but with a smaller scale (108K images). OpenImages shares a similar goal but at a much larger scale and with more structured annotations.

6.4 LVIS (Large Vocabulary Instance Segmentation)

LVIS is a dataset for instance segmentation with over 1,200 object classes, focusing on long-tail categories. It uses a class hierarchy similar to OpenImages and provides frequent, common, and rare class splits. OpenImages and LVIS are often used together for benchmarking large-vocabulary detection.

7 License and Access

7.1 License terms (CC-BY 4.0 for images; custom for annotations)

Individual images in OpenImages are made available under Creative Commons licenses (mostly CC-BY 4.0), while the annotations are released under a custom permissive license that allows research, commercial use, and redistribution, provided that the dataset source is acknowledged.

7.2 Download methods

7.2.1 Official website and CVDF mirrors

The dataset can be downloaded from the official OpenImages website (hosted by Google) or from mirrors provided by the Computer Vision Data Foundation (CVDF). The downloads are available as single-file archives or per-class subsets.

7.2.2 Tools for streaming and subset selection

Google provides open-source tools (e.g., the openimages Python library and TensorFlow Datasets integration) that allow users to stream images and annotations on demand, select specific classes or splits, and filter by additional attributes. These tools facilitate large-scale experimentation without requiring full local storage.