Telemetry redaction: remove sensitive fields before an exporter or sampler sees them

Operational decision

A payout request accidentally records an authorization header and a customer email as trace attributes. Change instrumentation to emit a bounded route template and payment-state label instead of raw request data. In an isolated collector pipeline, remove the two named attributes before batching or export; the YAML fragment shows only the processor portion and needs receivers, exporters, and a pipeline to run. Send a synthetic canary token through a test request, then search exported spans, logs, metrics, and collector diagnostics for that exact token. Keep the test token non-privileged and short-lived. Verify that a failed export queue or debug logger does not retain a raw copy. If historical telemetry contains the field, handle its retention and access as an incident rather than declaring it fixed by a new processor. Review each new instrumentation library for defaults that add headers or user IDs.

processors:
  attributes/payout_scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: delete

Cost and verification

Allowing fewer attributes can lower storage and indexing cost, but broad deletion can also remove the join key an incident needs. Define safe correlation fields before changing the pipeline. Hashing a predictable identifier may still permit reidentification, and a regex scrub can miss an unexpected encoding. Sample a test payload through every telemetry signal and inspect the exported result, not only the collector configuration. Redaction should occur before durable queues whenever the Collector topology permits it.

Common Mistakes

• Do not rely on span-attribute deletion to scrub log bodies or metric labels.
• Do not use a hash of a small predictable ID as proof of anonymity.
• Do not keep a debug exporter enabled with raw payment telemetry in production.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• Log pipelines: preserve incident evidence without ingesting secrets
• Distributed traces: preserve context without leaking data
• Metric cardinality: keep observability usable during a surge
• Collector export queues: measure the failure budget before telemetry is dropped

Practice and check

• Project: recover an observability pipeline without hiding user impact
• DevOps observability pipeline decisions

Object storage follow-up

• Presigned object access: bound authority and expiry