Inference logs: keep diagnostic joins without copying sensitive payloads

Last updated: 6 Oct 20265 min read

tutorial

Prediction logs need stable request and model identities, feature diagnostics and outcome joins under a retention and access policy.

Log the decision boundary

Record request ID, model version, feature or tokenizer version, decision time, route or score summary, abstention flag and latency. Store enough to join a delayed outcome, but avoid raw ticket text, receipt images and unbounded feature dumps in routine logs. A one-way identifier or governed mapping can preserve correlation without exposing a customer ID broadly. Text serving should follow the same contract.

Separate operational and audit needs

Fast operational metrics need counts and histograms; an incident investigation may require a tightly controlled sample of raw inputs. These have different retention, consent and access rules. Do not make every engineer’s observability dashboard a raw-data archive. A deletion request must be traceable across logs and derived datasets according to the product policy.

Make joins honest

Join predictions to outcomes using stable IDs and the correct decision-time window. Retries may produce multiple predictions for one request; choose whether the first, last or acted-on decision is evaluated. Otherwise duplicate rows inflate metric denominators. Dataset grain matters in production metrics too.

Test a redaction failure

Inject a request containing a sample account number and verify that ordinary logs, error traces and alerts do not include the raw payload. Then verify that authorized outcome joins still work. Inspect both success and exception paths; exception logging often bypasses the intended redaction helper.

Implementation

def inference_log_record(request_id, manifest, result, elapsed_ms):
    return {
        "request_id": request_id,
        "model_version": manifest["model_version"],
        "feature_version": manifest["feature_version"],
        "decision": result["decision"],
        "abstained": result["decision"] == "manual-review",
        "elapsed_ms": round(elapsed_ms, 2),
    }

Performance and operating cost

Writing one bounded record per request is O(1) per event; storage grows O(N) with request volume. Sampling and aggregate rollups reduce storage but must retain denominators for valid monitoring.

Common Mistakes

Do not log raw sensitive payloads by default.
Do not count retries as independent decisions without a grain policy.
Do not assume exception traces follow normal redaction.