1 Pub/Sub Fundamentals

Pub/Sub, short for publish–subscribe, is a messaging model where senders emit messages to named logical destinations rather than directly addressing individual recipients. Interested receivers create subscriptions for specific destinations or categories and are notified only when relevant messages are published.

The model is widely used in distributed and event-driven environments because it separates concerns: producing a message is independent of consuming it. This separation supports scalable distribution to many consumers, asynchronous workflows, and flexible integration patterns.

1.1 Core Concepts: Publisher, Subscriber, Broker

A publisher creates and publishes messages. It does not typically need to know which consumers will receive those messages.

A subscriber expresses interest in a set of messages. Subscribers may receive data through continuous listening, polling, or callback mechanisms, depending on the system implementation.

A broker (also called an event bus in some platforms) is the intermediary that manages routing and delivery. It accepts published messages, determines which subscriptions match, and forwards messages to the appropriate subscribers.

1.2 Topics, Channels, and Message Routing

Routing in Pub/Sub is usually performed using a logical destination such as a topic. Publishers send messages to a topic; subscribers subscribe to that topic directly or to a related grouping.

Some systems introduce additional notions such as channels or streams. In practice, these concepts often represent different levels of organization, such as grouping by domain, enabling multiple data flows, or supporting partitioned delivery.

Routing rules define which subscriptions match a published message. Common approaches include exact topic matches, wildcard patterns, and attribute-based routing using message metadata.

1.3 Decoupling and Asynchronous Communication

Decoupling reduces dependencies between parts of a system. Publishers can emit data without waiting for each consumer, and consumers can process messages at their own pace.

Asynchronous communication improves throughput and resilience: bursts of publishing can be absorbed by the broker, while slow consumers do not necessarily block publishers or other consumers. This decoupling is a central reason Pub/Sub is used for event-driven design.

2 System Architecture

Pub/Sub architectures typically consist of clients (publishers and subscribers) connected to a broker that performs routing, buffering, and delivery management. While implementations differ, most share a similar conceptual pipeline.

2.1 Message Flow Lifecycle

A message lifecycle describes what happens from the moment a publisher sends data until the subscriber receives it.

2.1.1 Publish Operation

Publishing includes steps such as selecting the destination and delivering the message to the broker.

2.1.1.1 Topic selection and routing rules

The publisher chooses a topic (or channel) for the message. The broker then applies its routing logic to determine which subscriptions should receive it. If routing is attribute-based, the broker evaluates message fields—such as type, category, or user-defined tags—to find matching subscriptions.

2.1.2 Subscribe Operation

Subscriptions define which messages a subscriber is eligible to receive.

2.1.2.1 Subscription filters and patterns

A subscriber registers a filter that may be as simple as “all messages on a topic” or as complex as “messages where metadata matches specific criteria.” Pattern matching with wildcards is common when topics follow hierarchical naming conventions.

2.2 Roles of a Broker or Event Bus

The broker acts as the control point for distribution. It typically handles:

  • Maintaining subscription registrations and their matching rules
  • Accepting incoming publishes
  • Storing messages when required by delivery semantics
  • Tracking delivery attempts and acknowledgements
  • Applying flow-control mechanisms such as limits or backpressure signals

In larger deployments, the broker layer may be distributed across nodes, with internal components for partition management, persistence, and failover.

2.3 Client Connections and Session Handling

Clients communicate with the broker over network connections. Depending on the system, delivery may be push-based (the broker streams messages to subscribers) or pull-based (subscribers request new messages).

Long-lived sessions are often used for streaming delivery. To manage disconnects, brokers may use session state such as cursors, delivery checkpoints, or acknowledgements that allow resuming without restarting from the beginning.

3 Delivery Semantics

Delivery semantics define what guarantees a Pub/Sub system provides. These guarantees heavily influence application design, especially around retries, duplicates, and consistency.

3.1 At-Least-Once Delivery

In at-least-once delivery, the system strives to deliver every message but may deliver duplicates. Duplicates can occur due to retries, timeouts, or partial failures where acknowledgements are not recorded reliably.

Applications typically compensate using idempotent processing or deduplication.

3.2 At-Most-Once Delivery

In at-most-once delivery, the system attempts to deliver messages but may drop them under failure conditions. This can happen when delivery buffers are lost or retries are not performed.

At-most-once is often chosen when low latency is more important than strict completeness, such as certain telemetry scenarios where occasional loss is acceptable.

3.3 Exactly-Once and Idempotency Approaches

Exactly-once delivery is the strongest guarantee, where each logical message is processed once and only once. Achieving it usually requires coordinated mechanisms such as transactional processing, strict ordering with acknowledgements, and careful state management.

Because “exactly-once” is complex in distributed systems, many platforms provide effectively-once behavior through idempotency: the system or the application ensures that repeated deliveries do not change the end result.

3.4 Ordering Guarantees and Trade-offs

Ordering guarantees specify whether messages are delivered in the order they were published. Some systems guarantee ordering only within a partition or topic key, not globally.

Stronger ordering requirements can limit parallelism and reduce throughput. Trade-offs commonly involve balancing latency, scale, and the complexity of coordination needed to preserve strict sequence.

4 Scalability and Performance

Pub/Sub scalability addresses how systems handle growth in message volume, number of topics, and subscriber count without collapsing under load.

4.1 Fan-Out and Throughput Scaling

One major advantage of Pub/Sub is fan-out: a single published message can be delivered to many subscribers. Performance depends on how efficiently the broker evaluates routing rules and manages concurrent deliveries.

Throughput scales when the broker can distribute work across nodes and when subscribers can process messages without excessive waiting.

4.2 Backpressure Handling

Backpressure occurs when downstream components cannot keep up with incoming message rates. Pub/Sub systems commonly handle it by:

  • Limiting in-flight messages per subscriber
  • Applying buffering and flow-control windows
  • Slowing or throttling pulls from consumers
  • Temporarily rejecting publishes or requiring retry, depending on configuration

Effective backpressure prevents memory overload and reduces cascading failures.

4.3 Batching and Latency Optimization

Brokers can improve throughput by sending messages in batches, reducing per-message overhead such as network round trips and acknowledgements.

Batching can increase latency because messages may wait to accumulate. Systems often tune batch size and flush intervals to balance throughput with near-real-time delivery needs.

4.4 Partitioning Strategies for Topics

Partitioning divides a topic into segments that can be processed in parallel. Messages are assigned to partitions using a key, such as an identifier or hashing strategy.

Partitioning improves scalability and throughput, while sometimes constraining ordering to within each partition. Subscriber consumption may also parallelize by partition assignment.

5 Reliability and Fault Tolerance

Reliability engineering in Pub/Sub focuses on preserving message flow despite network interruptions, broker failures, and client crashes.

5.1 Retry Policies

Retries are used when delivery does not complete. Retry strategies consider factors such as maximum attempts, exponential backoff, and retry classification (e.g., transient vs. permanent errors).

Well-designed retry policies avoid duplicate storms and help ensure that transient failures do not permanently block delivery.

5.2 Persistence, Durability, and Replay

Some Pub/Sub systems persist messages to durable storage so that they can survive broker restarts. Durability relates to whether a message remains available after failures.

Replay enables subscribers to reprocess messages from a point in time or from a specific offset, which is helpful for recovering from consumer outages or for rebuilding derived views.

5.3 Failure Modes and Recovery

Common failure modes include broker node crashes, network timeouts, partial persistence, and client disconnects. Recovery typically involves:

  • Re-establishing connections
  • Re-sending or re-requesting unacknowledged messages
  • Restoring subscription state from persisted metadata
  • Resuming from checkpoints or cursors

The exact behavior depends on the delivery semantics and the broker’s internal bookkeeping.

5.4 Dead-Letter Queues (DLQ) and Error Paths

Dead-letter queues (DLQs) capture messages that cannot be processed successfully after repeated attempts. A DLQ acts as a quarantine path for error analysis and manual or automated remediation.

DLQs often record error details such as the failure type, the processing stage, and a timestamp, enabling operators to identify systemic issues.

6 Subscription Management

Subscription management concerns how subscriptions are created, maintained, scaled, and tuned for correct message selection.

6.1 Durable vs. Non-Durable Subscriptions

A durable subscription preserves delivery state so that messages published while a subscriber is offline can still be available for later consumption, depending on retention policies.

A non-durable subscription may lose messages when the subscriber is disconnected. This distinction helps systems choose between stronger delivery continuity and lower storage/management cost.

6.2 Consumer Groups and Load Balancing

When multiple subscribers handle the same subscription, consumer groups distribute messages among group members. This enables horizontal scaling and prevents every consumer from receiving every message.

Load balancing within a group typically uses partition assignment or message allocation rules, balancing both work distribution and ordering constraints.

6.3 Offset Tracking and Checkpointing

To resume processing, systems track where consumption left off using offsets or cursors. Checkpointing records progress at intervals, usually after successful processing or after acknowledgements.

The frequency of checkpointing affects performance and recovery behavior: frequent checkpoints reduce reprocessing after failures but add overhead.

6.4 Filter Expressions and Interest Matching

Filter expressions define which messages a subscriber receives. Filters may evaluate topic name patterns, message headers, payload-derived fields, or event attributes.

Because filtering can influence broker load, designs often favor efficient matching strategies and encourage publishers to include routing-relevant metadata.

7 Integration and Interoperability

Integration topics cover how Pub/Sub systems connect with other software layers, evolve over time, and interoperate across different platforms.

7.1 API Styles: REST vs. Messaging APIs

Pub/Sub can be exposed through APIs using various paradigms. Some systems present publishers through REST-like endpoints that translate requests into published events. Subscribers may use streaming connections (e.g., persistent sockets) or messaging-native clients.

Messaging APIs often provide better semantics for asynchronous delivery, acknowledgements, and backpressure compared with request/response models.

7.2 Schema and Contract Management

To ensure interoperability, publishers and subscribers typically rely on a shared schema or contract that describes message structure, required fields, and data types.

Schema management may include validation at publish time, validation at consumption time, or both. Contract enforcement helps reduce runtime errors caused by mismatched fields or unexpected formats.

7.3 Versioning Message Formats

As systems evolve, message formats change. Versioning strategies include:

  • Including an explicit version field in message metadata
  • Using backward-compatible additions (e.g., optional fields)
  • Maintaining multiple schema versions concurrently

Subscribers can then interpret messages according to the version they support.

7.4 Bridging Between Systems and Brokers

Bridging allows events to flow across different messaging platforms, protocols, or organizational boundaries. A bridge component typically:

  • Subscribes to messages from a source broker
  • Transforms and maps them into the destination format
  • Publishes them to the target broker

Bridges must handle differences in delivery semantics, ordering, and schema evolution to avoid unintended duplicates or data loss.

8 Security Considerations (High-Level)

Security in Pub/Sub generally focuses on controlling access, protecting data in transit, and monitoring usage patterns.

8.1 Authentication and Authorization

Authentication confirms a client’s identity, while authorization determines what actions the client can perform, such as publishing to certain topics or subscribing to specific channels.

Role-based or policy-based models are common. Effective authorization prevents unauthorized data exposure and limits the impact of compromised credentials.

8.2 Transport Encryption

Encryption protects messages while they move across the network. Systems typically use TLS or equivalent secure transport protocols for client-broker communication.

Transport encryption helps mitigate eavesdropping and tampering, especially in multi-tenant or cross-network environments.

8.3 Topic-Level Access Control

Topic-level access control restricts which publishers can write and which subscribers can read specific destinations. Some systems also provide granular controls for namespaces, patterns, or resource identifiers.

Fine-grained policies reduce blast radius by limiting exposure to only the necessary event streams.

8.4 Auditing and Monitoring of Access

Auditing records events such as authentication attempts, authorization decisions, and subscription changes. Monitoring can detect unusual patterns, including spikes in publishing, repeated authorization failures, or access outside expected schedules.

These measures support incident response and compliance requirements without deeply embedding application logic into security checks.

9 Common Use Cases

Pub/Sub appears in a variety of application patterns where events represent state changes, user actions, or flowing data.

9.1 Event-Driven Microservices

In microservices architectures, Pub/Sub connects loosely coupled services. One service publishes events like “order placed” or “user updated,” while other services subscribe to react with independent logic, such as billing, analytics, or notifications.

This promotes modular development and reduces hard dependencies between services.

9.2 Real-Time Notifications

Pub/Sub supports real-time updates by distributing events as they occur. Examples include chat message delivery, status updates, and alerting systems that notify clients when new events are generated.

Message fan-out is particularly useful when many users or devices must receive the same kind of update.

9.3 Streaming Data Pipelines

Streaming pipelines use continuous event ingestion and transformation. Pub/Sub can act as the transport layer between ingestion sources, processing stages, and sinks.

When paired with stream processing components, it enables responsive data workflows and iterative transformations.

9.4 IoT Telemetry and Device Events

Internet of Things (IoT) devices often produce frequent telemetry and event signals. Pub/Sub helps route these messages to monitoring dashboards, alerting rules, and device management services.

Systems must address reliability, ordering constraints, and backpressure because devices may vary in connectivity and message rate.

10 Observability and Operations

Operational maturity for Pub/Sub includes visibility into message flow, processing health, and system capacity.

10.1 Metrics: Throughput, Lag, and Error Rates

Common metrics include:

  • Throughput: publish and delivery rates
  • Lag: how far behind consumers are relative to the newest available messages
  • Error rates: failures in publishing, routing, and processing

Tracking these metrics helps operators identify bottlenecks and predict scaling needs.

10.2 Logging and Tracing of Events

Structured logs can capture message identifiers, routing decisions, and processing outcomes. Distributed tracing connects publisher actions to subscriber processing across services, enabling root-cause analysis for failures or latency spikes.

Traces are especially useful in complex systems where a single business event triggers multiple downstream actions.

10.3 Health Checks and Broker Monitoring

Health checks verify broker readiness, connectivity, and internal component status. Monitoring may include resource usage such as CPU, memory, storage I/O, and network saturation.

Detecting early warnings—like increasing queue sizes or rising retry counts—helps prevent outages.

10.4 Capacity Planning and Rate Limits

Capacity planning estimates how many topics, messages, and concurrent consumers a system can sustain. Rate limits protect shared infrastructure by constraining publish rates or subscription fetch rates.

Well-tuned limits reduce overload risk and create more predictable performance under heavy workloads.

11 Design Guidance

Design guidance focuses on practical decisions that influence correctness, performance, and maintainability.

11.1 Choosing Topics and Message Granularity

Choosing topics affects routing efficiency and operational clarity. Topics should align with meaningful domains or event types rather than purely technical categories.

Message granularity involves deciding whether to emit many small events or fewer larger events. Smaller messages can improve selective consumption, while larger messages may reduce overhead but complicate evolution and filtering.

11.2 Handling Schema Evolution Safely

Safe schema evolution typically follows backward compatibility principles. Adding optional fields is usually less disruptive than changing existing field meanings.

When breaking changes are necessary, producers may publish new topics or new versions while consumers migrate at their own pace.

11.3 Idempotent Consumers Best Practices

Idempotent consumers ensure that duplicate deliveries do not cause duplicated side effects. Best practices include storing processed message identifiers, using deterministic update logic, or applying deduplication mechanisms.

This approach is essential in environments where at-least-once delivery is common.

11.4 Backward Compatibility Strategies

Backward compatibility strategies include maintaining multiple consumer versions, supporting versioned schemas, and designing processing logic that tolerates missing or additional fields.

Operationally, teams often roll out changes gradually by deploying new subscribers alongside older ones, minimizing disruption.

Related concepts clarify how Pub/Sub overlaps with or differs from other messaging and event-processing patterns.

12.1 Message Queues vs. Pub/Sub

Message queues often deliver messages to a single consumer (or a competing consumer set), emphasizing task distribution. Pub/Sub emphasizes broadcast-like distribution where multiple subscribers can independently receive matching messages.

However, many modern systems combine both ideas, offering queue-like consumption with topic-based routing.

Event sourcing represents application state as a sequence of events. Pub/Sub can support the transport of those events to projections, analytics systems, or downstream services.

While Pub/Sub is primarily a messaging pattern, event sourcing is a modeling approach; together they can form a cohesive architecture.

12.3 Streaming vs. Event Processing

Streaming describes continuous data flow, commonly involving time-ordered or unbounded datasets. Event processing emphasizes reacting to discrete occurrences represented as events.

Pub/Sub is often the connective tissue enabling both styles, depending on how messages are produced, interpreted, and processed.

12.4 Webhooks as a Pub/Sub-Like Pattern

Webhooks can resemble Pub/Sub when an application “subscribes” to callbacks triggered by external events. In contrast to broker-mediated delivery, webhooks typically rely on HTTP callbacks from the event source to the receiver.

Despite differences in mechanics and guarantees, the user experience—automatic notification of relevant events—shares a conceptual similarity to Pub/Sub.