Add post-merge analytics feedback loop for autonomy and trust learning #147

New issue

Closed

opened 2026-06-23 22:29:42 +02:00 by codex · 0 comments

codex commented

2026-06-23 22:29:42 +02:00

Collaborator

Goal

Make Patchwarden learn from what happened after a PR was merged, not only decide whether a PR is merge-ready before merge.

Today the Swiss-cheese stack mainly answers: can this exact PR head enter? The missing analytics layer should answer: did the merged change create value, friction, incident risk, rework, or trust evidence?

Why

The operator explicitly called out analytics as part of the June 2026 Patchwarden vision. Without it, Patchwarden can gate incoming work but cannot improve trust tiers, agent autonomy, or follow-up creation based on real outcomes.

This maps to:

PW-G007 Analytics Feedback Loop
PW-G015 Vision gap intake ledger
downstream trust-tier decisions from PW-G018

Scope

Add a small, read-only Patchwarden contract for post-merge feedback intake. It should normalize merge outcome and optional human/system signals into a machine-readable artifact that an external issue/job writer can consume later.

Suggested command shape:

patchwarden post-merge-feedback-check \
  --contract-pr-result-file contract-pr-result.json \
  --merge-outcome-file merge-outcome.json \
  --feedback-signal-file optional-feedback.json

Suggested artifact:

{
  "schema_version": "patchwarden.post_merge_feedback.v1",
  "verdict": "recorded | follow_up_needed | trust_signal | blocked",
  "target": { "kind": "pull_request", "id": "..." },
  "target_sha": "...",
  "signals": [],
  "follow_up_candidate": null,
  "trust_signal": null,
  "external_write_allowed": false
}

Signals to consider

merge lead time / time-to-green
reverted or follow-up-fix-needed
incident or degraded operator workflow
high-value outcome / low-friction success
agent/model identity and PR class correlation
repeated clean merges for trust progression
repeated soft-fails or redrives for trust tightening

Acceptance criteria

New read-only CLI command emits a structured post-merge feedback artifact.
Artifact is exact-head / PR-target bound.
Artifact can emit at least: recorded, follow_up_needed, trust_signal, blocked.
No Forgejo issue creation, no job dispatch, no merge, no approval, no external writes.
Schema and example are added under spec/schemas/.
patchwarden status / docs/status.html mention the new analytics feedback contract or update PW-G007 next action.
Tests prove malformed/stale/mismatched inputs fail closed.

No-go

Do not make Patchwarden create issues directly in this slice.
Do not let analytics widen trust by itself.
Do not add a live writer/controller here; keep that as an external consumer of the artifact.

Product framing

This is the missing memory layer of the Swiss-cheese architecture: pre-merge gates decide whether a PR may enter, while post-merge analytics decides whether the autonomy system is actually getting safer, faster, and more valuable over time.

## Goal Make Patchwarden learn from what happened after a PR was merged, not only decide whether a PR is merge-ready before merge. Today the Swiss-cheese stack mainly answers: **can this exact PR head enter?** The missing analytics layer should answer: **did the merged change create value, friction, incident risk, rework, or trust evidence?** ## Why The operator explicitly called out analytics as part of the June 2026 Patchwarden vision. Without it, Patchwarden can gate incoming work but cannot improve trust tiers, agent autonomy, or follow-up creation based on real outcomes. This maps to: - `PW-G007 Analytics Feedback Loop` - `PW-G015 Vision gap intake ledger` - downstream trust-tier decisions from `PW-G018` ## Scope Add a small, read-only Patchwarden contract for post-merge feedback intake. It should normalize merge outcome and optional human/system signals into a machine-readable artifact that an external issue/job writer can consume later. Suggested command shape: ```bash patchwarden post-merge-feedback-check \ --contract-pr-result-file contract-pr-result.json \ --merge-outcome-file merge-outcome.json \ --feedback-signal-file optional-feedback.json ``` Suggested artifact: ```json { "schema_version": "patchwarden.post_merge_feedback.v1", "verdict": "recorded | follow_up_needed | trust_signal | blocked", "target": { "kind": "pull_request", "id": "..." }, "target_sha": "...", "signals": [], "follow_up_candidate": null, "trust_signal": null, "external_write_allowed": false } ``` ## Signals to consider - merge lead time / time-to-green - reverted or follow-up-fix-needed - incident or degraded operator workflow - high-value outcome / low-friction success - agent/model identity and PR class correlation - repeated clean merges for trust progression - repeated soft-fails or redrives for trust tightening ## Acceptance criteria - New read-only CLI command emits a structured post-merge feedback artifact. - Artifact is exact-head / PR-target bound. - Artifact can emit at least: `recorded`, `follow_up_needed`, `trust_signal`, `blocked`. - No Forgejo issue creation, no job dispatch, no merge, no approval, no external writes. - Schema and example are added under `spec/schemas/`. - `patchwarden status` / `docs/status.html` mention the new analytics feedback contract or update `PW-G007` next action. - Tests prove malformed/stale/mismatched inputs fail closed. ## No-go - Do not make Patchwarden create issues directly in this slice. - Do not let analytics widen trust by itself. - Do not add a live writer/controller here; keep that as an external consumer of the artifact. ## Product framing This is the missing memory layer of the Swiss-cheese architecture: pre-merge gates decide whether a PR may enter, while post-merge analytics decides whether the autonomy system is actually getting safer, faster, and more valuable over time.

Rows
Columns