AI Agent Trace Redaction: Keep Secrets Out of Spans

Short answer: AI agent trace redaction should happen twice: first where the agent creates spans, then again at the telemetry collector before export. Disable prompt and tool payload capture by default, keep only an explicit allowlist of operational attributes, and run a canary-secret test that fails the release if the canary appears anywhere in exported traces.

Agent traces are unusually useful and unusually dangerous. A single run can contain model input, model output, function arguments, function results, handoff context, exception strings, file paths, user identifiers, and raw audio. That is enough to debug a broken workflow—but also enough to copy a password, access token, private document fragment, or customer record into a second system with a different retention policy.

This guide builds a fail-closed trace path for the OpenAI Agents SDK and an OpenTelemetry Collector. It does not remove observability. It keeps the structure engineers need—operation name, model, latency, token counts, status, error type, and correlation IDs—while dropping content that is not required to operate the service.

Start with a trace data inventory

Before changing configuration, capture one disposable run with synthetic values and enumerate what the exporter receives. Do this in a non-production project with canary data, never real secrets. The inventory should include span attributes, events, status descriptions, resource attributes, links, logs, and any vendor-specific payload fields.

Step 1: minimize data at the Agents SDK

The OpenAI Agents SDK enables tracing by default. Its generation spans can record model inputs and outputs, while function spans can record tool inputs and outputs. The current documentation says sensitive-data capture is enabled by default. Reverse that default for production workers.

# Process-wide production default
export OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA=false

Also set the policy explicitly in code. An environment variable can be changed by a deployment template or shell; an explicit run configuration makes the intended behavior visible in review.

from agents import Agent, Runner, RunConfig

safe_trace_config = RunConfig(
    trace_include_sensitive_data=False,
)

result = await Runner.run(
    Agent(name="support-router", instructions="Route the request safely."),
    input=user_request,
    run_config=safe_trace_config,
)

For audio agents, apply the corresponding voice pipeline setting so raw PCM input and output are not added to audio spans. If a workflow is covered by Zero Data Retention requirements, follow the documented product behavior rather than assuming tracing remains available.

Step 2: make exception handling trace-safe

Exception strings frequently concatenate the failing URL, request body, command, database key, or upstream response. Treat str(error) as untrusted content. Keep a stable error class or code in telemetry and place detailed diagnostics in a separately controlled log sink only when policy permits it.

def trace_safe_error(error: Exception) -> dict[str, str]:
    return {
        "error.type": type(error).__name__,
        "error.code": getattr(error, "code", "unclassified"),
        "error.detail": "Error details are redacted.",
    }

The July 16 SDK change routes additional agent-run, input-filter, speech, and transcription error details through the sensitive-data setting. It landed after the latest release available at research time, so do not assume a package upgrade already contains it. Pin a released version that explicitly includes the fix when one is available, record that version in your deployment manifest, and run the leakage test below against the exact artifact you deploy. Until then, keep export disabled for affected paths or add an application-level sanitizer.

Step 3: enforce an OpenTelemetry Collector allowlist

Source-side controls reduce risk but should not be the only boundary. A future SDK, custom span, third-party integration, or application developer can add new attributes. Put a collector between the worker and the external backend, and allow only attributes with an explicit operational purpose.

processors:
  transform/agent_resource_allowlist:
    error_mode: propagate
    trace_statements:
      - keep_keys(resource.attributes, [
          "service.name",
          "deployment.environment.name"
        ])

  redaction/agent_span_allowlist:
    allow_all_keys: false
    allowed_keys:
      - gen_ai.operation.name
      - gen_ai.provider.name
      - gen_ai.request.model
      - gen_ai.response.model
      - gen_ai.usage.input_tokens
      - gen_ai.usage.output_tokens
      - error.type
      - error.code
      - agent.workflow.version
    blocked_key_patterns:
      - ".*(token|secret|password|authorization|api[_-]?key).*"
    blocked_values:
      - "(?i)bearer\\s+[a-z0-9._~-]+"
      - "gw-canary-[a-z0-9-]+"
    summary: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors:
        - memory_limiter
        - transform/agent_resource_allowlist
        - redaction/agent_span_allowlist
        - batch
      exporters: [otlphttp/approved_backend]

The transform processor constrains resource attributes, while the redaction processor removes span attributes that are not in allowed_keys and can mask values that match blocked patterns. This distinction matters because resource and span attributes occupy different parts of the OpenTelemetry data model. The policy is intentionally fail-closed: a newly introduced attribute disappears until someone documents why it is safe and adds it to the correct allowlist.

Step 4: prove the policy with a canary secret

A configuration review is not evidence that every export path is clean. Create a unique, harmless canary and place it in each risky surface: model input, model output fixture, tool argument, tool result, exception message, custom attribute, and log body. Run the agent against a local capture exporter, then scan the complete exported payload.

import json
from pathlib import Path

CANARY = "gw-canary-trace-redaction-9f3d"

def assert_canary_absent(capture: Path) -> None:
    raw = capture.read_text(encoding="utf-8")
    if CANARY in raw:
        raise AssertionError("Sensitive canary reached the trace exporter")

    payload = json.loads(raw)
    assert payload["summary"]["span_count"] > 0
    assert payload["summary"]["failed_span_count"] == 1
    assert payload["summary"]["input_tokens"] >= 0

The test must fail before the controls are enabled. That failure-first step proves the harness is observing the real export path rather than scanning the wrong file. After enabling SDK minimization and collector redaction, the same run should preserve safe counts and error classification while removing the canary everywhere.

Step 5: decide where detailed debugging is allowed

Some incidents require more context than the production trace policy permits. Do not solve that by permanently recording every prompt. Create a time-bounded diagnostic mode with a separate project, restricted operators, short retention, synthetic or explicitly approved data, and an audit event when the mode is enabled.

Never use a user ID, email address, or predictable account number as an “anonymous” correlation key. Plain hashing can be reversible in practice when the input space is small. Prefer an HMAC with a protected key, rotate it deliberately, and document whether cross-period correlation is required at all.

Common failure modes

Disabling prompt capture but leaving exception strings

A tool may raise an error containing the full request or response. Your canary matrix must include failures from model calls, tools, filters, voice paths, and background workers—not only successful generations.

Redacting attributes but ignoring events and logs

Telemetry has multiple containers. Scan span attributes, events, status descriptions, links, resource attributes, log records, and vendor extensions. If the collector policy covers only attributes, another signal may still carry the canary.

Using a denylist as the primary control

Secret formats change and personal data rarely has one reliable pattern. A denylist is useful as a second layer, but an allowlist better answers the operational question: which fields are intentionally exported?

Keeping redaction diagnostics in production forever

Verbose redaction summaries can disclose attribute names or create cardinality. Use detailed summaries during integration, then reduce them after the allowlist is stable.

Testing a different exporter path than production

Run the gate through the same SDK configuration, collector processors, and exporter serialization used by the deployed worker. A unit test of a sanitizer function is helpful but insufficient.

Production rollout sequence

1. Inventory: capture a synthetic trace and classify every exported field.
2. Minimize: disable sensitive data in the Agents SDK and voice pipeline.
3. Allowlist: place a pinned Collector policy before the external backend.
4. Probe: inject the canary into success and failure paths; prove the test fails without controls.
5. Canary deploy: enable the policy for one worker cohort and compare safe telemetry parity.
6. Enforce: block releases when the canary appears or required operational fields disappear.
7. Review: re-run the inventory when SDKs, tools, models, exporters, or retention rules change.

A dedicated agent worker makes this boundary easier to operate because runtime configuration, scoped secrets, logs, schedules, and deployment evidence can be reviewed together. The same worker should still use least-privilege tools and sandboxing; trace redaction is one layer, not a substitute for access control.

Reusable AI agent trace redaction checklist

• Record the exact Agents SDK and Collector versions deployed.
• Set trace_include_sensitive_data=False for production runs.
• Disable sensitive audio capture for voice workflows.
• Keep error type and stable code; redact exception details.
• Put a Collector allowlist before every external trace backend.
• Review span attributes, events, status, logs, links, and resource attributes.
• Use a unique harmless canary across prompts, tools, results, and errors.
• Prove the test fails without controls and passes with them.
• Verify safe operational fields remain available after redaction.
• Time-bound and audit any diagnostic mode that captures more context.
• Repeat the gate after SDK, tool, model, exporter, or policy changes.

Continue with the production stack

Use AI agent observability with OpenTelemetry for the underlying trace model, then add gVisor sandbox security to contain untrusted tool execution. If you want an isolated worker with browser, terminal, files, memory, schedules, and scoped secrets already available, create a GolemWorkers AI agent and apply this checklist before connecting a production trace backend.

Sources

• OpenAI Agents SDK tracing — default tracing, sensitive-data controls, processors, and flushing behavior.
• OpenAI Agents SDK error-redaction change — sensitive-data handling for additional exception paths.
• OpenTelemetry: handling sensitive data — data minimization and Collector processor guidance.
• OpenTelemetry Collector redaction processor — fail-closed allowed keys, blocked keys and values, hashing, and summaries.