How to implement idempotency keys and request hashing in .NET and Azure

.NET and Azure developers often face the challenge of handling duplicate requests without breaking application logic. Whether it’s retried API calls, asynchronous messaging, or concurrent updates, ensuring idempotency is critical to maintaining data integrity. This guide walks you through implementing idempotency keys, request hashing, and ETags in .NET and Azure to prevent duplicate operations and race conditions.

Designing idempotency for HTTP APIs

Idempotency ensures that multiple identical requests produce the same result as a single request, which is especially useful for POST endpoints where retries might occur. The key is to require an Idempotency-Key header on non-idempotent methods like POST while relying on natural identifiers and concurrency controls for PUT and PATCH.

Core contract rules for idempotency

• Require the Idempotency-Key header on POST requests that create resources. This is optional for safe methods like GET.

• For PUT and PATCH, prefer using the resource’s natural URI and ETags for concurrency rather than custom idempotency keys.

• Respond with an Idempotency-Replayed: true header when a duplicate request is detected.

Storing idempotency data in Azure

Storing idempotency data correctly is essential for tracking processed requests and preventing duplicates. A well-designed schema includes the tenant identifier, the idempotency key, a request hash, response details, and timestamps. This ensures uniqueness and enables efficient lookups.

Azure SQL schema for idempotency

The following table structure works well for Azure SQL. It enforces uniqueness on the combination of TenantId and IdempotencyKey, stores a hash of the request for verification, and keeps track of response details and timestamps.

CREATE TABLE Idempotency (
    TenantId NVARCHAR(64) NOT NULL,
    IdempotencyKey NVARCHAR(128) NOT NULL,
    RequestHash VARBINARY(32) NOT NULL,  -- SHA-256 hash
    StatusCode INT NULL,
    ResponseBody VARBINARY(MAX) NULL,    -- or NVARCHAR(MAX) for JSON
    CreatedAtUtc DATETIME2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
    CompletedAtUtc DATIME2(3) NULL,
    CONSTRAINT PK_Idem PRIMARY KEY (TenantId, IdempotencyKey)
);

CREATE INDEX IX_Idem_Created ON Idempotency (CreatedAtUtc);

Cosmos DB can also be used with a container configured for a composite unique key on /tenantId/idempotencyKey and a time-to-live setting to automatically clean up old entries.

Implementing idempotency middleware in ASP.NET Core

A middleware component can intercept incoming requests, validate the idempotency key, and handle replayed requests efficiently. The middleware should reserve the key upfront to prevent race conditions and store the response details for future replays.

Minimal idempotency middleware example

public class IdempotencyMiddleware : IMiddleware
{
    private readonly IIdemStore _store;

    public IdempotencyMiddleware(IIdemStore store) => _store = store;

    public async Task InvokeAsync(HttpContext ctx, RequestDelegate next)
    {
        var tenantId = ctx.User.FindFirst("tenant_id")?.Value ?? "public";
        var key = ctx.Request.Headers["Idempotency-Key"].ToString();

        if (string.IsNullOrWhiteSpace(key))
        {
            await next(ctx);
            return;
        }

        var hash = await HashRequestAsync(ctx.Request);
        var existing = await _store.TryGetAsync(tenantId, key);

        if (existing is { Completed: true } && existing.RequestHash.SequenceEqual(hash))
        {
            ctx.Response.Headers["Idempotency-Replayed"] = "true";
            ctx.Response.StatusCode = existing.StatusCode;
            if (existing.ResponseBody is { Length: > 0 })
            {
                ctx.Response.ContentType = "application/json";
                await ctx.Response.Body.WriteAsync(existing.ResponseBody);
            }
            return;
        }

        var reserved = await _store.ReserveAsync(tenantId, key, hash);

        if (!reserved)
        {
            ctx.Response.StatusCode = StatusCodes.Status409Conflict;
            await ctx.Response.WriteAsync("{\"error\":\"Request in progress\"}");
            return;
        }

        var originalBody = ctx.Response.Body;
        await using var mem = new MemoryStream();
        ctx.Response.Body = mem;

        try
        {
            await next(ctx);
            await _store.CompleteAsync(
                tenantId,
                key,
                ctx.Response.StatusCode,
                mem.ToArray()
            );
        }
        finally
        {
            mem.Position = 0;
            await mem.CopyToAsync(originalBody);
            ctx.Response.Body = originalBody;
        }
    }

    static async Task<byte[]> HashRequestAsync(HttpRequest req)
    {
        req.EnableBuffering();
        using var sha = System.Security.Cryptography.SHA256.Create();

        string body = "";
        if (req.ContentLength > 0)
        {
            using var reader = new StreamReader(req.Body, leaveOpen: true);
            body = await reader.ReadToEndAsync();
            req.Body.Position = 0;
        }

        var canonical = $"{req.Method}\n{req.Path}\n{NormalizeJson(body)}";
        return sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(canonical));
    }

    static string NormalizeJson(string json) => 
        string.IsNullOrWhiteSpace(json) 
            ? ""
            : System.Text.Json.JsonDocument.Parse(json).RootElement.GetRawText();
}

This middleware captures the response body, status code, and headers for future replays, ensuring idempotency even if the original request is retried.

Using ETags for PUT semantics and concurrency control

PUT requests are inherently idempotent, but concurrent updates can still lead to lost changes. Using ETags or a rowversion column in Azure SQL helps prevent overwrites by ensuring the client sends the latest version of the resource.

Implementing ETags in ASP.NET Core controllers

[HttpPut("{id}")]
public async Task<IActionResult> Put(
    string id,
    [FromBody] Widget dto,
    [FromHeader(Name = "If-Match")] string etag
)
{
    var ok = await _repo.ReplaceAsync(id, dto, etag);
    return ok ? NoContent() : StatusCode(StatusCodes.Status412PreconditionFailed);
}

In Azure Cosmos DB and Azure Table Storage, ETags are natively supported. For Azure SQL, emulate ETags using a rowversion column to track changes and prevent concurrent modifications.

De-duplicating messages in Azure Service Bus and queues

Message queues like Azure Service Bus and Storage Queues can deliver messages more than once due to retries or failures. Implementing an inbox pattern ensures that each message is processed only once, even if it’s received multiple times.

Azure Service Bus with duplicate detection

Azure Service Bus offers built-in duplicate detection, which can be configured with a history window (e.g., 10 minutes to 7 days). Set the MessageId to a business-specific idempotency key, such as an order ID, to align with your idempotency strategy.

Producer example

var client = new ServiceBusClient(connectionString);
var sender = client.CreateSender("orders");
var msg = new ServiceBusMessage(BinaryData.FromObjectAsJson(order))
{
    MessageId = order.Id, // Business idempotency key
    Subject = "OrderCreated"
};
await sender.SendMessageAsync(msg);

Consumer with idempotent processing

Use an inbox store to track processed messages and prevent duplicates. Reserve the message ID before processing to avoid reprocessing the same message.

public class OrderHandler
{
    private readonly IInboxStore _inbox;

    public async Task HandleAsync(
        ServiceBusReceivedMessage m,
        CancellationToken ct
    )
    {
        if (!await _inbox.ReserveAsync(m.MessageId))
        {
            return; // Already processed
        }

        try
        {
            var order = m.Body.ToObjectFromJson<OrderCreated>();
            await ProcessAsync(order, ct);
            await _inbox.CompleteAsync(m.MessageId);
        }
        catch
        {
            await _inbox.ReleaseAsync(m.MessageId);
            throw;
        }
    }
}

For Azure Storage Queues, which lack built-in duplicate detection, always maintain an inbox table keyed by the message or business idempotency key. Event Hubs, being stream-oriented, should treat processing as idempotent or checkpoint and maintain an inbox.

Best practices and cleanup

Implementing idempotency is only half the battle—regular cleanup ensures your storage doesn’t bloat over time. Set a time-to-live (TTL) of 24 to 72 hours for completed entries to remove old records automatically. This balances performance with storage efficiency.

As applications grow, consider scaling your idempotency store and monitoring for race conditions. With these patterns in place, you can confidently handle retries, concurrent updates, and asynchronous messaging without compromising data integrity.