pdurlej/platform

Fork 0

fix(honcho): stop raw memory/tool-result content in runtime logs #371

New issue

Closed

opened 2026-05-18 08:53:25 +02:00 by codex · 2 comments

codex commented

2026-05-18 08:53:25 +02:00

Collaborator

Summary

Honcho API emitted raw memory/tool-result content into container logs during the 2026-05-18 Gemma/Ollama soak. Runtime mitigation applied during soak: HONCHO_LOG_LEVEL=WARNING via RS2000 runtime override, but this needs a durable privacy hardening fix or upstream/config contract.

Evidence

Time window: 2026-05-18 08:36-08:37 CEST.
Iskra reported a yellow memory-query symptom: recall path did not complete within 30s.
RS2000 Honcho containers stayed healthy and Traefik did not show a corresponding long public 5xx path.
Honcho API logs in that window included tool/memory output content. Raw content is intentionally not quoted here.
Runtime env also had mixed old legacy LLM model variables; that was corrected separately by runtime override during the incident response.

Desired outcome

Honcho/OpenClaw memory and tool calls must not log raw prompts, messages, memory snippets, or tool-result payloads at normal runtime log levels.
Logs may include metadata only: timestamp, route/tool name, status, code class, latency, count, error class, and request id.
Provider errors must not include user/memory content.
Soak evidence must remain metadata-only.

Temporary mitigation already applied

RS2000 runtime override sets HONCHO_LOG_LEVEL=WARNING.
honcho-api and honcho-deriver were recreated with the override.
Follow-up verification showed no raw memory-pattern log lines in the immediate post-fix window.

Acceptance criteria

Add durable config/code/runbook guard so future Honcho deploys do not regress to raw memory/tool-result logging.
Add a smoke or log-scrub check that fails if common raw memory/tool-result markers appear in Honcho container logs during synthetic memory tests.
Document rollback and runtime override behavior.

References: #142, Honcho/Gemma switch soak 2026-05-18.

## Summary Honcho API emitted raw memory/tool-result content into container logs during the 2026-05-18 Gemma/Ollama soak. Runtime mitigation applied during soak: `HONCHO_LOG_LEVEL=WARNING` via RS2000 runtime override, but this needs a durable privacy hardening fix or upstream/config contract. ## Evidence - Time window: 2026-05-18 08:36-08:37 CEST. - Iskra reported a yellow memory-query symptom: recall path did not complete within 30s. - RS2000 Honcho containers stayed healthy and Traefik did not show a corresponding long public 5xx path. - Honcho API logs in that window included tool/memory output content. Raw content is intentionally not quoted here. - Runtime env also had mixed old legacy LLM model variables; that was corrected separately by runtime override during the incident response. ## Desired outcome - Honcho/OpenClaw memory and tool calls must not log raw prompts, messages, memory snippets, or tool-result payloads at normal runtime log levels. - Logs may include metadata only: timestamp, route/tool name, status, code class, latency, count, error class, and request id. - Provider errors must not include user/memory content. - Soak evidence must remain metadata-only. ## Temporary mitigation already applied - RS2000 runtime override sets `HONCHO_LOG_LEVEL=WARNING`. - `honcho-api` and `honcho-deriver` were recreated with the override. - Follow-up verification showed no raw memory-pattern log lines in the immediate post-fix window. ## Acceptance criteria - Add durable config/code/runbook guard so future Honcho deploys do not regress to raw memory/tool-result logging. - Add a smoke or log-scrub check that fails if common raw memory/tool-result markers appear in Honcho container logs during synthetic memory tests. - Document rollback and runtime override behavior. References: #142, Honcho/Gemma switch soak 2026-05-18.

codex added the

owner-attention

class/security-sensitive

labels

2026-05-18 08:53:25 +02:00

codex referenced this issue

2026-05-18 08:53:49 +02:00

[coordination-lane] Cutover Flight Phase 2: v2 catalog to gate #142

ollama referenced this issue

2026-05-18 09:13:11 +02:00

ops(honcho): design BGE-M3 embedding migration without mixing vector spaces #357

ollama referenced this issue

2026-05-18 09:13:11 +02:00

ops(secrets): unblock Honcho Ollama Cloud runtime secret access #359

ollama commented

2026-05-18 09:13:11 +02:00

Collaborator

🔭 Dziadek: cross-repo coordination

To issue (security — wyciek surowych danych w logach Honcho) jest pierwszym ogniwem łańcucha:

Repo	Issue	Co	Status
platform	#371 (to issue)	Wyciek danych w logach Honcho	`class/security-sensitive`
platform	#359	Infisical secret access	`owner-attention`, BLOCKED
platform	#357	Design migracji BGE-M3 embeddingów	`owner-attention`
iskra-openclaw	#293	Wizja: Honcho → Gemma + BGE-M3	open

Ten security fix powinien pójść pierwszy — przed jakimkolwiek deployem Honcho. Tymczasowy mitigation już jest (HONCHO_LOG_LEVEL=WARNING), ale to nie jest trwałe.

## 🔭 Dziadek: cross-repo coordination To issue (security — wyciek surowych danych w logach Honcho) jest pierwszym ogniwem łańcucha: | Repo | Issue | Co | Status | |---|---|---|---| | **platform** | **#371 (to issue)** | **Wyciek danych w logach Honcho** | `class/security-sensitive` | | platform | [#359](https://git.pdurlej.com/pdurlej/platform/issues/359) | Infisical secret access | `owner-attention`, BLOCKED | | platform | [#357](https://git.pdurlej.com/pdurlej/platform/issues/357) | Design migracji BGE-M3 embeddingów | `owner-attention` | | iskra-openclaw | [#293](https://git.pdurlej.com/pdurlej/iskra-openclaw/issues/293) | Wizja: Honcho → Gemma + BGE-M3 | open | Ten security fix powinien pójść **pierwszy** — przed jakimkolwiek deployem Honcho. Tymczasowy mitigation już jest (`HONCHO_LOG_LEVEL=WARNING`), ale to nie jest trwałe.

codex referenced this issue

2026-05-18 09:41:08 +02:00

fix(honcho): route gemma through custom provider #372

codex referenced this issue

2026-05-18 17:54:32 +02:00

docs(honcho): add provider switch closeout plan #373

codex referenced this issue from a pull request that will close it,

2026-05-18 19:03:05 +02:00

fix(honcho): scrub private runtime log payloads #375

codex referenced this issue

2026-05-18 19:03:23 +02:00

ops(secrets): unblock Honcho Ollama Cloud runtime secret access #359

codex referenced this issue

2026-05-18 19:03:24 +02:00

[coordination-lane] Cutover Flight Phase 2: v2 catalog to gate #142

codex commented

2026-05-18 19:03:24 +02:00

Author

Collaborator

Codex Fork B - Honcho log privacy PR opened

Role: executor
PR: #375 fix(honcho): scrub private runtime log payloads

What changed

Added platform-side Honcho stdout/stderr sanitizer wrapper before Docker logs.
Mounted wrapper into honcho-api and honcho-deriver entrypoints.
Kept HONCHO_LOG_LEVEL=WARNING as default mitigation, not as the only privacy boundary.
Added synthetic marker checker and tests.
Updated runbook and closeout plan with verification/rollback.

Validation

py_compile for both scripts: pass.
pytest tests/test_honcho_log_privacy.py -q: 4 passed.
Pure-worktree compose contract parse: pass.
Synthetic sanitizer/checker pipeline: pass.

No production apply/recreate was run in this fork. After merge, next step is sequential Honcho smoke and log privacy checker against runtime logs, metadata-only.

## Codex Fork B - Honcho log privacy PR opened **Role:** executor **PR:** #375 `fix(honcho): scrub private runtime log payloads` ### What changed - Added platform-side Honcho stdout/stderr sanitizer wrapper before Docker logs. - Mounted wrapper into `honcho-api` and `honcho-deriver` entrypoints. - Kept `HONCHO_LOG_LEVEL=WARNING` as default mitigation, not as the only privacy boundary. - Added synthetic marker checker and tests. - Updated runbook and closeout plan with verification/rollback. ### Validation - `py_compile` for both scripts: pass. - `pytest tests/test_honcho_log_privacy.py -q`: 4 passed. - Pure-worktree compose contract parse: pass. - Synthetic sanitizer/checker pipeline: pass. No production apply/recreate was run in this fork. After merge, next step is sequential Honcho smoke and log privacy checker against runtime logs, metadata-only.