Building microservices is relatively straightforward. Building reliable microservices is where the real challenge begins. Modern microservices rarely work in isolation. A single business operation often involves multiple services communicating through message brokers such as Kafka, RabbitMQ, or Azure Service Bus.

Consider a simple Order Management System:

• Order Service saves an order.
• Inventory Service reserves stock.
• Payment Service processes payment.
• Notification Service sends an email.

The challenge is that these services communicate asynchronously.

Imagine a customer placing an order:

• They click Pay Now
• Their internet connection is unstable
• The browser automatically retries
• Kafka is temporarily unavailable
• A consumer crashes while processing a message

Without proper safeguards ( design patterns ), these failures can lead to:

• Duplicate payments
• Lost events
• Duplicate inventory updates
• Inconsistent data across services
• Duplicate message processing

Fortunately, three proven patterns solve these problems:

• Idempotency
• Outbox Pattern
• Inbox Pattern

Let’s understand how each pattern fits into the request lifecycle.

Idempotency

What is Idempotency?

Idempotency is the ability to perform the same operation multiple times while producing the same result. In simple terms, if the client sends the same request more than once, the server should process it only once and return the same response for every subsequent retry.

This is especially important for operations like creating orders, processing payments, or booking tickets, where duplicate execution can have serious business consequences.

Why Do We Need It?

In distributed systems, duplicate requests are more common than you might think. They can occur due to:

• Network timeouts
• Browser or mobile app retries
• Users clicking the submit button multiple times
• API Gateway or load balancer retries
• Temporary network interruptions

Without idempotency, these retries can create duplicate records or trigger the same business operation multiple times.

The Problem

Consider a payment API:

POST /payments
↓
Payment Created
↓
Response Timeout
↓
Client Retries
↓
Payment Created Again

Although the user intended to make a single payment, the application processes the request twice, resulting in a duplicate charge.

The Solution: Idempotency Key

The client includes a unique Idempotency-Key with every request.

POST /payments

Idempotency-Key: 8f3c9d2a

The server stores this key along with the generated response. If another request arrives with the same key, it simply returns the previously stored response instead of executing the business logic again.

Request Processing Flow

Client Request
      │
      ▼
Read Idempotency-Key
      │
      ▼
Key Already Exists?
   │          │
  Yes         No
   │           │
Return      Execute
Stored      Business Logic
Response        │
                ▼
         Store Response

This ensures that the business operation is executed only once, regardless of how many times the client retries.

Why Implement It as Middleware?

Implementing idempotency in middleware keeps the logic centralized and reusable.

Instead of adding duplicate-checking code in every controller or endpoint, the middleware intercepts incoming requests, validates the idempotency key, and decides whether to execute the request or return a cached response.

This approach keeps controllers focused solely on business logic.

Where Should Idempotency Data Be Stored?

The idempotency key and its corresponding response can be stored in:

• Redis – Fast, scalable, and ideal for high-traffic APIs.
• SQL Server – Suitable when persistence and transactional consistency are required.
• PostgreSQL – A good choice for relational applications.

Redis is the most common choice because it supports high throughput and automatic expiration (TTL).

Best Practices

• Generate a unique UUID/GUID for each idempotent request.
• Store both the response and HTTP status code along with the key.
• Configure a suitable TTL to automatically remove expired keys.
• Scope keys to the authenticated user or tenant to avoid conflicts.
• Apply idempotency primarily to POST endpoints that create or modify resources.

Key Takeaway

Idempotency protects your APIs from duplicate client requests caused by retries or network failures. By implementing it as middleware, you can ensure that critical operations—such as payments, order creation, and bookings—are executed exactly once while keeping your application code clean and maintainable.

Outbox Pattern

What Problem Does It Solve?

The Outbox Pattern guarantees that database changes and event creation happen atomically. Modern microservices often need to save business data and publish an event at the same time.

Instead of publishing directly to Kafka after saving data, the application stores the event in an Outbox table within the same database transaction.

If the transaction commits successfully, both the business data and the event are saved.

If the transaction rolls back, neither is saved.

The Problem

Imagine this sequence:

Save Order to Database
        │
        ▼
Publish OrderCreated Event

Now consider this scenario:

• ✅ The order is successfully saved in SQL Server.
• ❌ Kafka (or RabbitMQ) is temporarily unavailable.

The result?

• The order exists in the database.
• Inventory Service never reserves stock.
• Payment Service never starts.
• Notification Service never sends a confirmation.

Your system is now in an inconsistent state.

Why Not Publish the Event First?

You might think of reversing the order.

Publish Event
      │
      ▼
Save Order

Unfortunately, this introduces another problem.

If the event is published successfully but the database transaction fails, other services will assume the order exists—even though it doesn’t.

This creates an even bigger inconsistency.

The Outbox Pattern Solution

The Outbox Pattern solves this problem by storing the event inside the same database transaction as the business data.

Instead of publishing directly to Kafka, the application writes the event to an Outbox table.

If the transaction succeeds, both the order and the event are committed together.

If the transaction fails, neither is saved.

This guarantees that your business data and event remain consistent.

How the Outbox Pattern Works

Begin Transaction
        │
        ▼
Save Order
        │
        ▼
Insert Event into Outbox Table
        │
        ▼
Commit Transaction
        │
        ▼
Background Worker Reads Pending Events
        │
        ▼
Publish Event to Message Broker
        │
        ▼
Mark Event as Processed

The database becomes the single source of truth. Events are only published after they have been safely stored.

Sample Outbox Table –

A typical Outbox table contains:

A background worker periodically checks this table, publishes pending events to the message broker, and updates their status once successful.

Benefits of the Outbox Pattern

Things to Keep in Mind

The Outbox Pattern introduces a few additional responsibilities:

• A background worker to publish pending events.
• Retry logic for failed publications.
• Monitoring for stuck or failed messages.
• Periodic cleanup of processed Outbox records.

Although it adds a little complexity, the reliability benefits make it a standard pattern for production-grade microservices.

Inbox Pattern

Understanding the Inbox Pattern

Once an event has been successfully published to a message broker, the next challenge is ensuring that it is processed exactly once by the consuming service.

In distributed systems, this is more difficult than it sounds.

Most message brokers, such as Kafka, RabbitMQ, and Azure Service Bus, follow an At-Least-Once Delivery model. This guarantees that a message will eventually be delivered, but it also means the same message may be delivered more than once under certain failure conditions.

Without proper safeguards, duplicate message delivery can result in inconsistent business data and unintended side effects.

Why Is the Inbox Pattern Needed?

Message brokers consider a message successfully processed only after the consumer sends an acknowledgement (ACK).

If the consumer processes the message but crashes before sending the acknowledgement, the broker assumes the message was never processed and delivers it again.

This behavior is intentional—it prevents message loss—but it also introduces the possibility of duplicate processing.

The Problem

Consider an Inventory Service that consumes an OrderCreated event.

Receive OrderCreated Event
        │
        ▼
Reduce Inventory
        │
        ▼
Application Crashes
        │
        ▼
Acknowledgement Not Sent
        │
        ▼
Broker Redelivers the Same Event

When the service restarts, it receives the same event again.

Without any duplicate detection mechanism, the inventory will be updated twice.

Initial Stock = 10
        │
        ▼
Process Event
        │
        ▼
Stock = 8
        │
        ▼
Receive Duplicate Event
        │
        ▼
Process Again
        │
        ▼
Stock = 6

Although only one order was placed, the inventory was reduced twice, leading to incorrect business data.

The Inbox Pattern Solution

The Inbox Pattern prevents duplicate processing by maintaining a record of every successfully processed message.

Each event includes a unique identifier (typically a MessageId or EventId). Before executing any business logic, the consumer checks whether this identifier already exists in an Inbox table.

If the message has already been processed, it is ignored. Otherwise, the application processes the event and records its identifier within the same transaction.

How the Inbox Pattern Works

Receive Event
      │
      ▼
Check Inbox Table
      │
      ▼
Message Already Processed?
   │                 │
  Yes                No
   │                  │
Ignore Event     Execute Business Logic
                      │
                      ▼
              Save MessageId
                      │
                      ▼
                 Commit Transaction

This approach ensures that each event affects the system only once, even if the broker delivers it multiple times.

Sample Inbox Table –

A typical Inbox table stores the unique identifier of each processed message.

When a duplicate event with MessageId = A12345 arrives, the consumer immediately recognizes that it has already been processed and safely ignores it.

Benefits of the Inbox Pattern

The Inbox Pattern provides several important advantages:

• Prevents duplicate message processing, even when the broker redelivers events.
• Supports at-least-once delivery semantics without compromising data integrity.
• Enables safe retries after application failures or service restarts.
• Maintains data consistency across distributed services.
• Improves the overall reliability of event-driven microservices.

Best Practices

When implementing the Inbox Pattern, consider the following recommendations:

• Assign a globally unique MessageId to every event.
• Check the Inbox table before executing any business logic.
• Store the MessageId and business changes within the same database transaction.
• Create an index on the MessageId column for fast duplicate detection.
• Periodically archive or clean up old Inbox records based on your retention policy.

Final Thoughts

When combined, these patterns create resilient, production-ready microservices that can safely handle retries, temporary failures, and at-least-once message delivery without sacrificing data consistency.

Production-ready architecture:
Client → Idempotency Middleware → Business Transaction + Outbox → Message Broker → Consumer + Inbox

so the logical progression is:

1. How do I prevent duplicate API requests? → Idempotency
2. How do I reliably publish events? → Outbox Pattern
3. How do I prevent duplicate event processing? → Inbox Pattern

This combination is widely adopted in enterprise systems because it balances reliability, scalability, and simplicity without relying on expensive distributed transactions.