Model Gateway Identity and Request Budgets

Working case

A reviewer asks the case portal to summarize 47 maintenance notes for pump station 29. The request arrives from two tabs after one tab times out. Each tab carries the same case revision and operation key. The gateway checks the reviewer and tenant, caps the number and size of notes, and admits one operation. A changed case revision with the old key returns a conflict. The browser receives an operation ID and later reads the result through the same record permission check. It never receives the provider key, and the summary cannot expose a note from a different tenant merely because that note appears in a cached search result.

Implementation boundary

function admitSummary(request) {
  if (!request.authorized || request.inputBytes > 49152) return 'deny';
  if (request.activeTenantJobs >= 3) return 'defer';
  return 'admit';
}
console.log(admitSummary({ authorized: true, inputBytes: 32768, activeTenantJobs: 3 }));
// Output: defer

Define an input contract with case ID, revision, user intent, maximum output size, and a scoped idempotency key. Resolve the permitted note set on the server, then minimize what is sent downstream. Keep the provider call behind a short-lived service credential and a timeout. Apply per-user, per-tenant, and per-operation admission budgets before expensive retrieval or generation starts. Put a hard ceiling on input bytes and output bytes as well as an estimated cost ceiling. Store a small operation record with state, input revision, charge estimate, and actual usage category; keep private note bodies out of routine logs. If the caller disconnects, decide whether work should stop or continue as a job, and make that decision visible in the status contract. A provider outage should return a retryable state without pretending the answer exists.

Cost and boundaries

Input assembly is O(N + B) for N selected notes and B total text bytes, before the external operation cost. An indexed operation lookup is near O(log M) for M retained operations; a unique scoped key prevents duplicate admissions. Provider charges and latency often dominate local CPU, so concurrency must be bounded separately from ordinary case reads. A budget that counts only requests misses very large inputs and outputs. Record admitted units, actual usage, timeout rate, and abandoned work by tenant. Retain operation metadata for the retry window, then expire it under a stated policy. Cache only when tenant, permission, input revision, and operation parameters are part of the key.

Failure trace

Inspect the built browser assets for provider credentials. Replay the same operation before the first call completes and require one admission; reuse its key after editing a note and require a conflict. Replace the selected case ID with another tenant’s case and deny before retrieval. Return a provider timeout after the gateway accepted the request and show a failed or retryable operation state rather than an empty success. Open the status path after losing case permission and deny the result. Push an oversized note set and verify the byte cap applies before sending data outside the application. Saturate generation workers and confirm ordinary case pages still load.

Verification

• Credential material is absent from browser assets.
• Duplicate admission does not double downstream work.
• Result reads recheck current record permission.

Practice drill

Create a gateway contract for case 47, revision 29, and 63 selected notes. Set an input cap of 48 KiB, an output cap of 6 KiB, and a tenant concurrency cap of three operations. Issue two identical requests, then one request with the same key and revision 30. Simulate a provider delay and revoke the reviewer before result access. Inspect the browser bundle and routine logs. Report admitted work, blocked work, actual usage, and the status shown in each failure.

Decision note

The gateway owns identity, budgets, and result access; the model response is only data produced inside that contract.

Common Mistakes

• Calling the provider directly with a browser-visible key.
• Budgeting requests without input or output size.
• Caching across tenants or revisions.

Related lessons

Model-Backed Web Application Boundaries; Generated Response Streaming and Cancel State; Retrieved Content, Instructions, and Tool Permission; Model Output Evaluation and Release Control; Accepted Operations and Status Resources; Rate Limits and Request Budgets.

Connected practice

Build Project: permission-bound maintenance summary and review Web Development: model-backed application decisions quiz.