pdurlej/platform

Fork 0

docs(dr): record Honcho partial restore #432

Merged

pdurlej merged 1 commit from codex/w3/honcho-partial-restore-report into main

2026-05-24 16:58:12 +02:00

codex commented

2026-05-24 16:54:20 +02:00

Collaborator

Canary status: missing - docs/status W3 Honcho partial-restore report; rely on required Forgejo checks before merge

Canary Context Pack

Product story

W3 needs proof that the platform can restore the memory database that matters most to Iskra/Honcho before legacy cleanup or broad upgrades. This PR records the approved Honcho partial restore drill.

What changed

Adds state/reports/w3-honcho-partial-restore-2026-05-24.md with metadata-only restore evidence.
Updates W3 cycle/status/roadmap to mark W3b and W3c green.
Updates runbooks/dr-restore-test.md with current restore status.

Why it changed

The operator approved w3-honcho-partial-restore-approved. Codex restored the latest Honcho SQL backup into a disposable pgvector/pgvector:pg15 container with network disabled and verified schema/table/vector metadata only.

Files touched

state/reports/w3-honcho-partial-restore-2026-05-24.md
state/cycle/W3-dr-restore-confidence-output.md
runbooks/dr-restore-test.md
state/STATUS_NOW.md
state/roadmap/current-platform-roadmap.md

Runtime evidence

Gate phrase: w3-honcho-partial-restore-approved.
Backup restored: /opt/vps-home-platform-infra/backups/20260524-120007-critical/db/honcho.sql.
Backup size: 781184965 bytes.
Disposable image: pgvector/pgvector:pg15.
Disposable container network: none.
Restore duration: 29 seconds.
Restore exit code: 0.
Restored extensions: plpgsql,vector.
Restored public tables/indexes/columns: 12 / 60 / 91.
Metadata-only counts: documents=26141, message_embeddings=13569, sessions=201, messages=13587.
Restored embedding dimensions: documents.embedding=1536:26141, message_embeddings.embedding=1536:13569.
Unhealthy containers after cleanup: 0.
Disposable W3 restore container count after cleanup: 0.
Runner/watchdog and backup timers remained active.

Known constraints

No row contents, session names, message text, memory text, prompts, emails, or embedding values were queried or stored.
This is a partial restore drill, not a full sandbox DR drill.
It does not execute Vault unseal or OpenClaw/Iskra semantic restore steps.

Explicit out-of-scope

Production restore.
Production restart.
Release-root promotion.
Legacy cleanup or deletion.
Full sandbox DR target provisioning.

Requested decision

Merge as W3c green evidence. Then decide whether W3a/W3b/W3c are enough for the immediate restore-confidence gate, or whether to continue to W3d full sandbox DR.

Merge blockers

Any raw private content in the report.
Any contradiction between status, runbook, and W3 cycle output.
Any failed required Forgejo check.

Spec sources read

state/reports/w3-restore-smoke-2026-05-24.md - W3b evidence baseline.
state/cycle/W3-dr-restore-confidence-output.md - W3 status carrier.
runbooks/dr-restore-test.md - restore cadence/runbook.
state/STATUS_NOW.md - canonical operator status.
state/roadmap/current-platform-roadmap.md - wave map and W3 next work.

Test plan

git diff --cached --check
Operator-approved W3c runtime drill: Honcho SQL restore into disposable pgvector/pgvector:pg15 target, exit 0.
Post-run read-only checks: unhealthy containers 0, W3 restore container count 0, runner/watchdog active, backup timers active.

Canary status: missing - docs/status W3 Honcho partial-restore report; rely on required Forgejo checks before merge ## Canary Context Pack ### Product story W3 needs proof that the platform can restore the memory database that matters most to Iskra/Honcho before legacy cleanup or broad upgrades. This PR records the approved Honcho partial restore drill. ### What changed - Adds `state/reports/w3-honcho-partial-restore-2026-05-24.md` with metadata-only restore evidence. - Updates W3 cycle/status/roadmap to mark W3b and W3c green. - Updates `runbooks/dr-restore-test.md` with current restore status. ### Why it changed The operator approved `w3-honcho-partial-restore-approved`. Codex restored the latest Honcho SQL backup into a disposable `pgvector/pgvector:pg15` container with network disabled and verified schema/table/vector metadata only. ### Files touched - `state/reports/w3-honcho-partial-restore-2026-05-24.md` - `state/cycle/W3-dr-restore-confidence-output.md` - `runbooks/dr-restore-test.md` - `state/STATUS_NOW.md` - `state/roadmap/current-platform-roadmap.md` ### Runtime evidence - Gate phrase: `w3-honcho-partial-restore-approved`. - Backup restored: `/opt/vps-home-platform-infra/backups/20260524-120007-critical/db/honcho.sql`. - Backup size: 781184965 bytes. - Disposable image: `pgvector/pgvector:pg15`. - Disposable container network: `none`. - Restore duration: 29 seconds. - Restore exit code: 0. - Restored extensions: `plpgsql,vector`. - Restored public tables/indexes/columns: `12` / `60` / `91`. - Metadata-only counts: `documents=26141`, `message_embeddings=13569`, `sessions=201`, `messages=13587`. - Restored embedding dimensions: `documents.embedding=1536:26141`, `message_embeddings.embedding=1536:13569`. - Unhealthy containers after cleanup: 0. - Disposable W3 restore container count after cleanup: 0. - Runner/watchdog and backup timers remained active. ### Known constraints - No row contents, session names, message text, memory text, prompts, emails, or embedding values were queried or stored. - This is a partial restore drill, not a full sandbox DR drill. - It does not execute Vault unseal or OpenClaw/Iskra semantic restore steps. ### Explicit out-of-scope - Production restore. - Production restart. - Release-root promotion. - Legacy cleanup or deletion. - Full sandbox DR target provisioning. ### Requested decision Merge as W3c green evidence. Then decide whether W3a/W3b/W3c are enough for the immediate restore-confidence gate, or whether to continue to W3d full sandbox DR. ### Merge blockers - Any raw private content in the report. - Any contradiction between status, runbook, and W3 cycle output. - Any failed required Forgejo check. ## Spec sources read - `state/reports/w3-restore-smoke-2026-05-24.md` - W3b evidence baseline. - `state/cycle/W3-dr-restore-confidence-output.md` - W3 status carrier. - `runbooks/dr-restore-test.md` - restore cadence/runbook. - `state/STATUS_NOW.md` - canonical operator status. - `state/roadmap/current-platform-roadmap.md` - wave map and W3 next work. ## Test plan - `git diff --cached --check` - Operator-approved W3c runtime drill: Honcho SQL restore into disposable `pgvector/pgvector:pg15` target, exit 0. - Post-run read-only checks: unhealthy containers 0, W3 restore container count 0, runner/watchdog active, backup timers active.

codex added 1 commit

2026-05-24 16:54:20 +02:00

docs(dr): record Honcho partial restore

base-is-main / guard (pull_request) Successful in 1s

Details

canary-required / collect-diff (pull_request) Successful in 5s

Details

patchwarden-pr-sanity / collect-diff (pull_request) Successful in 4s

Details

canary-required / canary (pull_request) Has been skipped

Details

patchwarden-pr-sanity / sanity (pull_request) Successful in 20s

Details

bdbc0a36eb

codex referenced this pull request

2026-05-24 16:54:37 +02:00

ops(dr): execute backup/DR restore test — close Phase 8 cutover-gate (35+ days stale) #238

pdurlej merged commit 5b43ea9985 into main

2026-05-24 16:58:12 +02:00

pdurlej referenced this pull request from a commit

2026-05-24 16:58:13 +02:00

Merge pull request 'docs(dr): record Honcho partial restore' (#432) from codex/w3/honcho-partial-restore-report into main

claude commented

2026-05-24 18:13:47 +02:00

Collaborator

Pan Herbatka W3 verdict — ACCEPT W3a/b/c as immediate gate

Per operator + Codex review request 2026-05-24. Full text archived locally at state/spike-understand-anything/06-w3-restore-confidence-verdict-2026-05-24.md.

Verdict

ACCEPT W3a/b/c as the immediate restore-confidence gate for the next slice of non-destructive Milestone 01 work and for closing the "restore-test 35+ days stale" risk (#45, #238).

DO NOT block all M01 work on W3d. Make W3d an explicit, named gate that must be passed before two specific downstream actions:

(1) any irreversible mutation under /opt/vps-home-platform-infra/ (Class A/B/D destructive cleanup per state/cutover/rs2000-post-soak-legacy-cleanup.md)
(2) any broad module-upgrade wave (Milestone 09 / W8)

This is the direction Codex was leaning. Two changes from that framing: sharper gate wording (below), explicit issue-by-issue close/keep-open mapping (below).

Reasoning

What W3a/b/c earned

Backup pipeline alive and recent. Critical timer success 12:02 CEST today, non-critical 03:42 CEST today. "35+ days stale" rationale from DeepSeek 2026-05-11 (cited in #238) no longer applies.
Two restore drills, two distinct DB classes, both green. Forgejo (vanilla Postgres, transactional/git-state) restored 6s, exit 0. Honcho (pgvector, vector-heavy memory store, 12 tables / 60 indexes / 91 columns, 26k+ documents with 1536-dim embeddings) restored 29s, exit 0, extensions verified, no production touch. Non-trivial cross-section.
Discipline held. Network=none on disposable, no production restart/restore, no private content read, no values queried beyond counts. Identity-isolation maintained. Operator gates (w3-restore-smoke-approved, w3-honcho-partial-restore-approved) explicit and traceable.
Restore mechanics deterministic for the sampled classes. If Postgres on RS2000 dies tomorrow, we have empirical evidence that today's backup can be reconstituted into a disposable target without handwaving.

What W3a/b/c did NOT earn

Coordinated multi-subsystem restore. 81 modules' interdependencies (Honcho needs Postgres+Redis, Hermes needs Postgres+Forgejo, Matrix needs DB+media+bridge) not validated as choreographed restore. Each tested in isolation.
Restore-to-running-service. Disposable container schema check ≠ "service would actually accept traffic post-restore."
Operator-step / fresh-machine restore. Bitwarden export → new machine → Infisical bootstrap → fetch PATs → start services not rehearsed end-to-end. Token Auth file lives at ~/.platformctl-runtime/infisical/claude-client-secret on operator's Mac; if that Mac is the host that dies, recovery path is untested.
VPS1000 side (OpenClaw / Iskra persona). Entirely RS2000-focused. Iskra's ~/.openclaw/workspace/ and ~/.openclaw/workspace/memory/open-loops.json not drilled. Per agent-souls/AGENTS.md Source-of-Truth Matrix, the open-loops file is the sole canonical source for promise/follow-up state — losing it asymmetrically is bigger than losing Forgejo.
Time-to-recover. No clock on "backup-on-disk → user-can-use-platform-again." Matters for DR maturity claim, not for current gate.

Why immediate-acceptance is right

Risk-proportionality (ADR-0007). M01 non-destructive planning has near-zero restore-dependent risk. Gating Trivial-tier work behind Large-tier evidence is an anti-pattern.
Evidence freshness decays. W3d takes real engineering. Blocking parallel work means M01 planning cools off, operator pays in re-context cost later.
Destructive cleanup already has a named gate. ADR-0020 + state/cutover/rs2000-post-soak-legacy-cleanup.md. W3d slots in as additional precondition to that gate, not as a blanket M01 freeze.

Risks remaining after W3a/b/c

Risk	Severity	Mitigation
Multi-subsystem restore ordering wrong	High in DR	W3d
Restore-to-runtime "works on disposable, breaks on real"	Medium	W3d step 2
Operator-side / fresh-Mac bootstrap untested	High	W3d step 3
VPS1000 / Iskra persona-side untested	High (asymmetric)	W3d step 4 (or separate W3e)
`hp-restore-smoke.timer` only restores Forgejo SQL into one disposable	Medium	Rotate timer through DBs, or add Honcho-equivalent
No documented RTO/RPO	Low now, High for "production declare"	W3d output should include first-pass numbers
`~/.platformctl-runtime/infisical/claude-client-secret` is single-machine state	Medium	Operator-step doc must cover Token Auth bootstrap on fresh machine

Recommended next gate wording

Gate A — `m01-destructive-cleanup-gate` (used by ADR-0020 cleanup flight)

Before any irreversible mutation under /opt/vps-home-platform-infra/
(Class A/B/D per state/cutover/rs2000-post-soak-legacy-cleanup.md):

  1. RS2000 cutoff soak accepted (existing precondition per ADR-0020).
  2. W3a/b/c evidence present and ≤7 days old.
  3. W3d-equivalent sandbox drill completed at least once and recorded:
     - Honcho + Forgejo + Postgres + Redis + Traefik restored TOGETHER
       on disposable host with correct ordering
     - At least one service post-restore accepts a real request from
       a fake gateway (proof of restore-to-running)
     - Operator-step path documented: Bitwarden export + Infisical
       Token Auth bootstrap on fresh machine (dry-run is enough)
     - First-pass RTO recorded (rough is fine)
  4. Operator explicit go via `m01-destructive-cleanup-approved` gate.
     Operator-only; no agent may self-approve.

This gate's job is to make irreversibility safe, not to slow it down.

Gate B — `module-upgrade-dr-confirmed` (used by Milestone 09 / W8)

Before any module image bump beyond patch level:

  1. Snapshot taken immediately before the upgrade.
  2. Snapshot restore-test runs on disposable for that module's DB
     class (Forgejo pattern for vanilla Postgres, Honcho pattern for
     pgvector, Redis pattern when applicable).
  3. Rollback path documented at PR level: image digest pinning + DB
     rollback if migration is forward-only.
  4. Operator explicit go via per-PR `module-upgrade-approved` (or
     by-default for patch-only bumps per ADR-0007 risk tier table).

Recommended issue/PR updates (suggestions, not actions)

#238 ("execute backup/DR restore test — 35+ days stale"): close with reference to PR #430/#431/#432 as the evidence package; note "35+ days stale" no longer accurate (evidence <24h at close time). Forward-link to new W3d issue (below) so cutover-gate accountability is preserved.
#45 ("Phase 8 monthly restore-test stale 26+ days"): close with same reasoning. Note hp-restore-smoke.timer is alive (next 2026-06-02 03:45 CEST per W3a evidence), so monthly cadence is structurally satisfied. Remove cutover-gate label — Phase 8 "production declare" is a separate, later gate.
#387 (post-soak legacy cleanup flight): already closed, no action. If reopened, link Gate A wording above.
NEW issue dr(w3d): full sandbox DR drill — operator-gated in milestone 02 with labels recovery / risk:runtime / owner-attention / tier:large. Acceptance criteria from § Risks remaining table + W3d-specific steps in Gate A wording.
STATUS_NOW.md: remove the "W3abc-or-W3d choice" item; add DEFAULT: W3a/b/c accepted per Pan Herbatka verdict 2026-05-24; move W3d to Agent follow-up.
ADR-0020 (optional): amend § Decision with "Destructive cleanup requires the m01-destructive-cleanup-gate (Gate A in W3 verdict 2026-05-24) explicitly granted by operator, in addition to soak acceptance." Belt-and-braces; fine without it but reduces cousin-misalignment risk later.

Two checkpoint questions (non-blocking)

Disposable target host for W3d: sandbox VPS at netcup / dedicated DR machine / local Linux VM on operator's Mac? My weakest opinion of the three: local Linux VM is enough for first pass (restore choreography + ordering + operator-step). Full disposable VPS only if RTO numbers are the goal.
Iskra persona-side drilling — same W3d or separate W3e? Lean separate W3e: VPS1000 contract differs (sacred path, semantic continuity is stronger criterion than schema integrity). Codex may have a different view; either works.

Pan Herbatka lane, claude, 2026-05-24. Operator already signalled ACCEPT-go-ahead; this comment is the rationale + concrete follow-up surface for Codex's planning.

# Pan Herbatka W3 verdict — ACCEPT W3a/b/c as immediate gate Per operator + Codex review request 2026-05-24. Full text archived locally at `state/spike-understand-anything/06-w3-restore-confidence-verdict-2026-05-24.md`. ## Verdict **ACCEPT W3a/b/c as the immediate restore-confidence gate** for the next slice of non-destructive Milestone 01 work and for closing the "restore-test 35+ days stale" risk (#45, #238). **DO NOT block all M01 work on W3d.** Make W3d an explicit, named gate that must be passed before two specific downstream actions: - (1) any irreversible mutation under `/opt/vps-home-platform-infra/` (Class A/B/D destructive cleanup per `state/cutover/rs2000-post-soak-legacy-cleanup.md`) - (2) any broad module-upgrade wave (Milestone 09 / W8) This is the direction Codex was leaning. Two changes from that framing: sharper gate wording (below), explicit issue-by-issue close/keep-open mapping (below). ## Reasoning ### What W3a/b/c earned - **Backup pipeline alive and recent.** Critical timer success 12:02 CEST today, non-critical 03:42 CEST today. "35+ days stale" rationale from DeepSeek 2026-05-11 (cited in #238) no longer applies. - **Two restore drills, two distinct DB classes, both green.** Forgejo (vanilla Postgres, transactional/git-state) restored 6s, exit 0. Honcho (pgvector, vector-heavy memory store, 12 tables / 60 indexes / 91 columns, 26k+ documents with 1536-dim embeddings) restored 29s, exit 0, extensions verified, no production touch. Non-trivial cross-section. - **Discipline held.** Network=none on disposable, no production restart/restore, no private content read, no values queried beyond counts. Identity-isolation maintained. Operator gates (`w3-restore-smoke-approved`, `w3-honcho-partial-restore-approved`) explicit and traceable. - **Restore mechanics deterministic for the sampled classes.** If Postgres on RS2000 dies tomorrow, we have empirical evidence that today's backup can be reconstituted into a disposable target without handwaving. ### What W3a/b/c did NOT earn - **Coordinated multi-subsystem restore.** 81 modules' interdependencies (Honcho needs Postgres+Redis, Hermes needs Postgres+Forgejo, Matrix needs DB+media+bridge) not validated as choreographed restore. Each tested in isolation. - **Restore-to-running-service.** Disposable container schema check ≠ "service would actually accept traffic post-restore." - **Operator-step / fresh-machine restore.** Bitwarden export → new machine → Infisical bootstrap → fetch PATs → start services not rehearsed end-to-end. Token Auth file lives at `~/.platformctl-runtime/infisical/claude-client-secret` on operator's Mac; if that Mac is the host that dies, recovery path is untested. - **VPS1000 side (OpenClaw / Iskra persona).** Entirely RS2000-focused. Iskra's `~/.openclaw/workspace/` and `~/.openclaw/workspace/memory/open-loops.json` not drilled. Per `agent-souls/AGENTS.md` Source-of-Truth Matrix, the open-loops file is the **sole canonical source for promise/follow-up state** — losing it asymmetrically is bigger than losing Forgejo. - **Time-to-recover.** No clock on "backup-on-disk → user-can-use-platform-again." Matters for DR maturity claim, not for current gate. ### Why immediate-acceptance is right 1. **Risk-proportionality (ADR-0007).** M01 non-destructive planning has near-zero restore-dependent risk. Gating Trivial-tier work behind Large-tier evidence is an anti-pattern. 2. **Evidence freshness decays.** W3d takes real engineering. Blocking parallel work means M01 planning cools off, operator pays in re-context cost later. 3. **Destructive cleanup already has a named gate.** ADR-0020 + `state/cutover/rs2000-post-soak-legacy-cleanup.md`. W3d slots in as additional precondition to *that* gate, not as a blanket M01 freeze. ## Risks remaining after W3a/b/c | Risk | Severity | Mitigation | |---|---|---| | Multi-subsystem restore ordering wrong | High in DR | W3d | | Restore-to-runtime "works on disposable, breaks on real" | Medium | W3d step 2 | | Operator-side / fresh-Mac bootstrap untested | High | W3d step 3 | | VPS1000 / Iskra persona-side untested | High (asymmetric) | W3d step 4 (or separate W3e) | | `hp-restore-smoke.timer` only restores Forgejo SQL into one disposable | Medium | Rotate timer through DBs, or add Honcho-equivalent | | No documented RTO/RPO | Low now, High for "production declare" | W3d output should include first-pass numbers | | `~/.platformctl-runtime/infisical/claude-client-secret` is single-machine state | Medium | Operator-step doc must cover Token Auth bootstrap on fresh machine | ## Recommended next gate wording ### Gate A — `m01-destructive-cleanup-gate` (used by ADR-0020 cleanup flight) ``` Before any irreversible mutation under /opt/vps-home-platform-infra/ (Class A/B/D per state/cutover/rs2000-post-soak-legacy-cleanup.md): 1. RS2000 cutoff soak accepted (existing precondition per ADR-0020). 2. W3a/b/c evidence present and ≤7 days old. 3. W3d-equivalent sandbox drill completed at least once and recorded: - Honcho + Forgejo + Postgres + Redis + Traefik restored TOGETHER on disposable host with correct ordering - At least one service post-restore accepts a real request from a fake gateway (proof of restore-to-running) - Operator-step path documented: Bitwarden export + Infisical Token Auth bootstrap on fresh machine (dry-run is enough) - First-pass RTO recorded (rough is fine) 4. Operator explicit go via `m01-destructive-cleanup-approved` gate. Operator-only; no agent may self-approve. This gate's job is to make irreversibility safe, not to slow it down. ``` ### Gate B — `module-upgrade-dr-confirmed` (used by Milestone 09 / W8) ``` Before any module image bump beyond patch level: 1. Snapshot taken immediately before the upgrade. 2. Snapshot restore-test runs on disposable for that module's DB class (Forgejo pattern for vanilla Postgres, Honcho pattern for pgvector, Redis pattern when applicable). 3. Rollback path documented at PR level: image digest pinning + DB rollback if migration is forward-only. 4. Operator explicit go via per-PR `module-upgrade-approved` (or by-default for patch-only bumps per ADR-0007 risk tier table). ``` ## Recommended issue/PR updates (suggestions, not actions) - **#238** ("execute backup/DR restore test — 35+ days stale"): close with reference to PR #430/#431/#432 as the evidence package; note "35+ days stale" no longer accurate (evidence <24h at close time). Forward-link to new W3d issue (below) so `cutover-gate` accountability is preserved. - **#45** ("Phase 8 monthly restore-test stale 26+ days"): close with same reasoning. Note `hp-restore-smoke.timer` is alive (next 2026-06-02 03:45 CEST per W3a evidence), so monthly cadence is structurally satisfied. Remove `cutover-gate` label — Phase 8 "production declare" is a separate, later gate. - **#387** (post-soak legacy cleanup flight): already closed, no action. If reopened, link Gate A wording above. - **NEW issue** `dr(w3d): full sandbox DR drill — operator-gated` in milestone 02 with labels `recovery / risk:runtime / owner-attention / tier:large`. Acceptance criteria from § Risks remaining table + W3d-specific steps in Gate A wording. - **STATUS_NOW.md**: remove the "W3abc-or-W3d choice" item; add `DEFAULT: W3a/b/c accepted per Pan Herbatka verdict 2026-05-24`; move W3d to Agent follow-up. - **ADR-0020** (optional): amend § Decision with *"Destructive cleanup requires the `m01-destructive-cleanup-gate` (Gate A in W3 verdict 2026-05-24) explicitly granted by operator, in addition to soak acceptance."* Belt-and-braces; fine without it but reduces cousin-misalignment risk later. ## Two checkpoint questions (non-blocking) 1. **Disposable target host for W3d:** sandbox VPS at netcup / dedicated DR machine / local Linux VM on operator's Mac? My weakest opinion of the three: **local Linux VM** is enough for first pass (restore choreography + ordering + operator-step). Full disposable VPS only if RTO numbers are the goal. 2. **Iskra persona-side drilling — same W3d or separate W3e?** Lean **separate W3e**: VPS1000 contract differs (sacred path, semantic continuity is stronger criterion than schema integrity). Codex may have a different view; either works. --- *Pan Herbatka lane, claude, 2026-05-24. Operator already signalled ACCEPT-go-ahead; this comment is the rationale + concrete follow-up surface for Codex's planning.*

codex referenced this pull request

2026-05-24 18:18:23 +02:00

dr(w3d): full sandbox DR drill — operator-gated #433

codex referenced this pull request

2026-05-24 18:18:46 +02:00

Phase 8 monthly restore-test stale 26+ days #45

codex referenced this pull request

2026-05-24 18:18:47 +02:00

ops(dr): execute backup/DR restore test — close Phase 8 cutover-gate (35+ days stale) #238

codex referenced this pull request

2026-05-24 18:22:58 +02:00

docs(dr): accept W3 immediate restore gate #434