Log pipelines: preserve incident evidence without ingesting secrets

Last updated: 5 Oct 20266 min read

tutorial

A log pipeline moves application events from processes through a collector to storage and search. It can fail at the producer, collector, transport, or index. Structured records help correlate a request across services, but logging every payload creates cost and privacy exposure. A log search result is incomplete unless the pipeline's loss and delay are understood.

Operational decision

A claims API writes one JSON event for a failed claim submission with request ID, stable error code, and deployment revision. It excludes claimant names, document contents, tokens, and payment data. The sample is one record to test parsing and redaction, not a logging-library configuration. Sample routine successes if volume demands it, but retain actionable errors under a clear policy. Alert on collector queue depth, dropped records, and end-to-end ingest delay. Test a storage outage: the application should have a bounded logging path rather than blocking every request indefinitely. Give incident responders enough retention to investigate an event, and restrict access to logs that still contain operational identifiers.

{"timestamp":"2026-10-01T08:17:23Z","service":"claims-api","revision":"claims-047","requestId":"req-6c49","event":"claim_submit_failed","errorCode":"DOCUMENT_TIMEOUT"}

Cost and verification

Ingest volume grows with event rate and fields, while indexing high-cardinality values increases storage and query cost. Retaining every debug event for months can crowd out the incidents that matter. Dropping logs during collector pressure may preserve application availability but removes evidence; count and alert on that loss. Apply deletion and access controls to archived copies as well as the active index. A trace can add context, but it does not replace a durable business event record.

Common Mistakes

Do not log credentials or full customer documents.
Do not treat a quiet log search as proof that no error occurred.
Do not let synchronous logging stall a critical request indefinitely.

Connected lessons

Advanced follow-up

Telemetry redaction: remove sensitive fields before an exporter or sampler sees them