2026-07-19

OpenClaw Logs: Read, Filter, and Troubleshoot Gateway Events

A fresh bounded Gateway sample shows how source metadata, record shape, subsystem gaps, correlation, and redaction turn noisy OpenClaw logs into defensible incident evidence.

OpenClaw Logs: Read, Filter, and Troubleshoot Gateway Events cover illustration

A noisy log is not the same thing as an unhealthy Gateway. The useful question is narrower: which source covered the failure, what time window was actually captured, which subsystem dominated it, and what evidence connects one event to the user-visible symptom?

OpenClaw logs are easiest to misuse when an operator starts by scrolling. A red line catches the eye, an earlier warning looks suspicious, and the investigation quietly loses its source boundary. The safer approach is to acquire a bounded JSON sample, preserve its metadata, reduce it to a few fields, and only then read messages around one incident.

A fresh read-only pull on 23 July from a running OpenClaw 2026.7.1-2 installation made the problem concrete. The command requested at most 400 lines and 500,000 bytes. It returned 320 parseable events covering just under two hours: 190 info and 128 warn records, with no error record in the slice. The Gateway was active with zero supervisor restarts. There was no red line to chase, yet the sample still contained useful operational evidence.

The warning count was the trap. Of the 177 log records with an explicit subsystem, 116 belonged to skills and 36 to gateway/ws. Another 141 log records had no subsystem field at all, while two used module. Severity without record shape, subsystem, and time context would have sent the investigation toward the loudest labeled component while silently dropping almost half the log records.

Measured result: what the bounded pull actually proved

320 records parsed

There were no malformed JSON lines in the captured CLI stream.

318 + 1 + 1

The stream contained 318 log records, one metadata record, and one notice—not one undifferentiated list.

<2h of coverage

The sample ran from 14:56:01 to 16:55:24 UTC. “Today’s log” would have overstated what the caps actually retained.

0 service restarts

The sample belongs to one active service interval that began at 09:51:23 UTC.

The command used for that inspection is intentionally boring:

openclaw logs \
  --json \
  --limit 400 \
  --max-bytes 500000 \
  --utc

--limit bounds records. --max-bytes bounds acquisition volume. --utc removes a timezone argument from the incident review. JSON mode keeps meta, log, and notice records distinguishable. Those are not cosmetic flags; together they define what the evidence can and cannot support.

Pin the source before filtering the content

openclaw logs normally tails the Gateway file through authenticated RPC. A one-shot local call can fall back to the configured file when the loopback Gateway is unavailable. A following Linux session is stricter: it may use the active user-systemd journal by PID, retry the Gateway, and later return to the file stream.

That recovery behavior is useful operationally and dangerous analytically. A follow session can change sourceKind. The cursor belongs to that source, not to an imaginary universal timeline. On reconnection, overlapping lines may appear as the file cursor is replayed. Deduplicate with a stable event identity or a conservative tuple such as source, timestamp, subsystem, level, and a hash of the sanitized message—not with line number.

Three transparent log conduits with one amber recovery path switching between sources under operator inspection
A resilient tail can move between the Gateway file and a local service journal. Preserve source metadata whenever the stream changes.

Keep metadata records in the incident packet. The observed stream identified its source as file and carried separate metadata and notice records. Discarding them would save almost no space while removing the facts needed to interpret cursor, size, truncation, rotation, or recovery.

An oversized incident slice fails before analysis

Start from the symptom timestamp and work outward. For a delayed Telegram reply, for example, the first slice might be two minutes before the inbound update through two minutes after the final send. Include the relevant channel, routing, agent-run, model, tool, and outbound subsystems only when the timeline reaches them.

FilterQuestion it answersWhat it cannot prove
Source kind and service intervalWhich producer and process lifetime emitted this line?Whether the event belongs to the failing user turn
UTC time windowWas the line close enough to be relevant?Causality when several turns overlap
SubsystemWhich boundary was speaking?Whether the boundary caused or merely reported the failure
LevelHow did the component classify the event?Business impact or uniqueness
Trace or session correlationWhich request chain or conversation context carried it?Safety of publishing the identifier
Sanitized messageWhat state transition or error was recorded?Completeness when debug fields were not enabled

The CLI JSON output exposes a compact surface suitable for an initial filter: time, level, subsystem, message, and the underlying raw record. Keep the raw object private. Produce a derived view for review:

openclaw logs --json --limit 400 --max-bytes 500000 --utc \
  > gateway-sample.jsonl

jq -c '
  select(.type == "log") |
  {time, level, subsystem, message}
' gateway-sample.jsonl

That projection is a convenience, not automatic redaction. Messages can still contain paths, target names, provider errors, or user text. Store the capture with restrictive permissions, produce a sanitized derivative, and share only the derivative.

The current sample exposed four log-record shapes. Only 177 of 318 log records carried subsystem; 141 omitted it and two used module. A pipeline that groups only .subsystem does not create an “other” bucket—it silently excludes those records. Count key shapes first, then decide whether an absent subsystem, a module field, or an unclassified record is relevant to the incident.

Testing a JSONL hypothesis

JSONL keeps each event independently parseable. Use that property. Count record types before searching messages. Check the earliest and latest timestamp after applying caps. Group by subsystem and level. Inspect notices. Only then open the narrow message set that could explain the incident.

jq -s '
  map(select(.type == "log")) |
  {
    events: length,
    by_level: group_by(.level) |
      map({level: .[0].level, count: length}),
    by_subsystem: group_by(.subsystem) |
      map({subsystem: .[0].subsystem, count: length}) |
      sort_by(-.count) | .[:10]
  }
' gateway-sample.jsonl

The new bounded sample is an even sharper warning against headline metrics. It contained 128 warnings and no error record. That does not make the warnings unimportant, nor does it prove the Gateway was healthy. It means the useful unit is the event chain around a named symptom, not the global count or the presence of a red line.

Correlate one failed turn across boundaries

When a valid diagnostic trace context exists, OpenClaw file logs can include traceId, spanId, parentSpanId, and traceFlags. Gateway HTTP requests and WebSocket frames establish internal trace scope, and agent runs or model calls can become children. That makes a trace ID a strong join key across logs and OpenTelemetry spans.

Trace fields will not appear on every line. Channel, agent, and session fields also depend on the context carried by the log call. Missing correlation data is a real finding: it limits what the sample can prove. Do not compensate by matching the first similar error string in a crowded window.

  1. Anchor the user-visible symptom in UTC.
  2. Find the nearest inbound or RPC boundary.
  3. Carry the trace ID when present; otherwise use the smallest safe context tuple.
  4. Follow state transitions through routing, agent execution, model or tool activity, and outbound delivery.
  5. Stop when the causal edge is proven. Unrelated warnings belong in a separate backlog.

Redaction belongs at acquisition time

OpenClaw's default sensitive-data redaction helps on console output, file logs, OTLP logs, transcript text, and several UI or tool-event surfaces. It is still best-effort. The documentation explicitly warns that not every identifier or binary payload field is covered. Custom patterns can add project-specific masks; replacing the default log pattern set carelessly can weaken coverage.

An operator passes a dense event stream through layered privacy filters into a small sealed evidence capsule
The shareable artifact should be smaller than the private capture: bounded fields, masked identities, and only the events needed to support the incident claim.

Raw stream debugging has a different risk class. It can include full prompts, tool output, and user data. Keep it local, use it for a short named question, and remove the capture after the investigation. A support ticket is not an acceptable storage backend for an unreviewed raw stream.

Failure modes of global verbosity

Global debug or trace logging is tempting because it avoids choosing a hypothesis. It also creates a larger, noisier, more sensitive record and can push the relevant window out of a bounded sample faster. --verbose changes console and WebSocket presentation; it does not raise the file log level. Those controls should not be treated as interchangeable.

Prefer a targeted diagnostic when the missing fact is already known. OpenClaw offers bounded model-transport, payload-summary, SSE, code-mode, and diagnostics-flag surfaces. A reasonable escalation note names four things before enabling one:

  • the precise unknown, such as “did the provider return headers before the stream stalled?”;
  • the subsystem and expected event;
  • the start and stop condition for extra logging;
  • the privacy impact and deletion plan for the capture.

If the extra output does not answer the named question, turn it off before forming the next hypothesis. Accumulating every debug surface at once destroys the clean baseline needed to compare attempts.

What another operator needs to reproduce the call

A useful handoff does not need the entire daily log. It needs enough context for another operator to reproduce the conclusion without receiving unrelated conversations or credentials.

Acquisition contract

OpenClaw version, command, UTC bounds, line and byte caps, source kind, service interval, and whether the stream changed source.

Symptom contract

One user-visible failure, its timestamp, affected channel or RPC surface, and expected versus observed result.

Causal slice

The smallest ordered set of sanitized events that connects the boundary to the failure, including correlation fields when safe.

Confidence boundary

What the sample proves, what caps or missing fields exclude, and which targeted diagnostic would close the remaining gap.

Approval rule: accept a log-based diagnosis only when it names the source, process interval, observed window, correlation method, sanitization step, and causal edge. “There were many warnings” is an observation. It is not a root cause.

For a broader runtime boundary, pair this workflow with the OpenClaw Gateway operator guide. For conversation-state evidence, use the separate OpenClaw sessions guide; session rows and log transport answer different questions. If maintaining the Gateway, log pipeline, and incident process is the work you want to avoid, deploy a managed AI agent on GolemWorkers.

Sources

  • OpenClaw Logging — file and console surfaces, JSONL fields, rotation, levels, targeted diagnostics, trace correlation, redaction, and privacy boundaries.
  • OpenClaw logs CLI — RPC tailing, line and byte caps, JSON record types, fallback behavior, reconnection, cursor scope, and source transitions.
  • OpenClaw configuration reference — live logging configuration surface and defaults.
  • OpenClaw debugging guide — targeted runtime debugging and the privacy risks of raw stream capture.