gRPC deadlines: spend one request budget across every downstream call

Operational decision

A receipt endpoint has a 430-millisecond budget for a ledger lookup. The Go function below derives a shorter context from its parent, so an earlier caller deadline still wins. The service's actual gRPC method should receive lookupContext and pass it to any further RPC instead of creating context.Background. Make the ledger response take 600 milliseconds in a test and verify that the client sees a deadline error, the server cancels its query, and no retry continues after cancellation. Then use a fast ledger response and measure how much time remains for serialization and the return trip. Count deadline expirations separately from server errors. A cancellation signal does not roll back a side effect that has already committed; pair retry policy with an idempotency rule for mutating methods.

package main

import (
    "context"
    "time"
)

func fetchReceiptWithinBudget(
    parentContext context.Context,
    ledgerLookup func(context.Context) (string, error),
) (string, error) {
    lookupContext, cancelLookup := context.WithTimeout(parentContext, 430*time.Millisecond)
    defer cancelLookup()
    return ledgerLookup(lookupContext)
}

Cost and verification

A shorter deadline frees resources earlier but raises false timeouts during normal tail latency. A longer one ties up workers and may allow retries to outlive the user request. Measure the percentile of successful calls, network transit, queue time, and cancellation lag before choosing a budget. The function adds only a timer and context object per call; the material cost comes from operations that ignore cancellation. Prove cancellation reaches the database or worker rather than counting client errors alone.

Common Mistakes

• Do not replace the incoming context with a new background context.
• Do not retry a mutation after its deadline without an idempotency check.
• Do not assume a cancelled RPC reverses a committed external effect.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• Retries and timeouts: bound the cost of a failed request
• Retry amplification: assign one owner for each failed operation
• Queue consumers: acknowledgement, idempotency, and backlog
• Idempotency keys: reconcile an accepted write before repeating it

Practice and check

• Project: release a service through connection and cache failures
• DevOps service traffic decisions