pdurlej/patchwarden

Fork 0

test(v0): jsonschema validation for CLI artifact outputs #62

New issue

Closed

opened 2026-05-28 13:37:39 +02:00 by claude · 1 comment

claude commented

2026-05-28 13:37:39 +02:00

Collaborator

Goal

Close a real gap surfaced in the 2026-05-28 Cloud Opus 4.6 repo review: the four JSON Schemas under spec/schemas/ are rigorously defined but no test actually runs jsonschema.validate() against real CLI outputs. Schemas drift silently from code today.

Why this matters

Today: tests check individual fields (assert result.verdict["schema_version"] == "patchwarden.pr_verdict.v1") but a missing field in an artifact won't fail unless the test specifically reads that field.
Risk: schemas become documentation rather than contract.
Defense hardening per D21 — no new core capability, just enforces what the schemas already promise.

Scope

1) Add `jsonschema` to test extras only (NOT runtime dep)

In pyproject.toml:

[project.optional-dependencies]
test = [
  "pytest>=8.0",
  "jsonschema>=4.0",
]
dev = [
  "pytest>=8.0",
  "jsonschema>=4.0",
]

Critically: src/patchwarden/ MUST remain stdlib-only. jsonschema is opt-in for the test suite, never imported from runtime code. Verify by grepping src/patchwarden/ for import jsonschema post-PR — must return zero hits.

2) New file `tests/test_artifact_schema_contract.py`

Loads each schema from spec/schemas/*.json once at module level and validates representative CLI outputs against it:

Schema	Validate against
`pr-verdict.schema.json`	`evaluate_pull_request()` output (use `tests/test_pr_check.py` fixtures)
`issue-verdict.schema.json`	`evaluate_issue()` output
`review-artifact.schema.json`	`run_review()` output (both empty-findings path and findings path; also soft-fail path with `runtime_metadata.error_kind`)
`finding-resolution.schema.json`	`resolve_findings()` output (PASS path + HOLD path + soft-fail path)

Each schema gets at minimum: one valid happy-path test, one valid edge case (e.g. empty findings array, soft-fail), and one deliberately invalid payload that must raise jsonschema.ValidationError (catch-the-catcher meta-test so we know validation actually runs).

3) Strengthen `tests/test_boring_pr_lifecycle.py`

Add a single jsonschema.validate() call after each CLI emit so the existing lifecycle test doubles as a schema contract check.

4) Bonus (close at the same time): unit test for `_cloud_review_mode_from_lane()`

src/patchwarden/cli.py has the helper:

def _cloud_review_mode_from_lane(lane):
    if lane.cloud_review == "allowed": return "cloud"
    if lane.cloud_review == "required-with-redaction": return "redacted"
    return "local"

No unit test exists; this value lands in the review artifact. Add tests/test_cli_cloud_review_mode.py with three cases (allowed, required-with-redaction, anything else → local).

Acceptance criteria

PYTHONPATH=src python3 -m unittest discover tests → all green (156 baseline + new tests; target ≥165).
pip install -e .[test] installs jsonschema from test extras.
grep -r "import jsonschema" src/patchwarden/ returns zero hits (stdlib-only rule preserved).
Each of the four schemas has both a happy-path validate call and a deliberately-invalid case proving the validator works.

No-go

❌ Do NOT add jsonschema to runtime deps. Stdlib-only in src/patchwarden/ per cousin discipline.
❌ Do NOT change schema version strings — schemas are versioned contracts. If a real validation failure surfaces, file a separate issue, don't loosen the schema to make it pass.
❌ Do NOT bump schema_version on any schema (parked per D21).
❌ Do NOT switch test framework — keep unittest (pytest is opt-in only per #22).

Spec sources

spec/schemas/*.json — the four schemas to validate against
tests/test_boring_pr_lifecycle.py — lifecycle test where the strengthening lands
tests/test_resolve_findings.py + tests/test_review_run.py — fixtures for soft-fail validation cases
src/patchwarden/cli.py — _cloud_review_mode_from_lane() helper for the bonus test
2026-05-28 Cloud Opus 4.6 repo analysis, §II.2 "Brak walidacji schematów w testach"

Status flow

ready-for-agent (now) → agent claims → PR → operator review → merge

Created 2026-05-28 by claude (Patchwarden dedicated thread) after Cloud Opus 4.6 repository review surfaced the gap. Atomic enough for any cousin (e.g. Cloud Opus 4.6 via Antigravity) to pick up.

## Goal Close a real gap surfaced in the 2026-05-28 Cloud Opus 4.6 repo review: the four JSON Schemas under `spec/schemas/` are rigorously defined but **no test actually runs `jsonschema.validate()`** against real CLI outputs. Schemas drift silently from code today. ## Why this matters - Today: tests check individual fields (`assert result.verdict["schema_version"] == "patchwarden.pr_verdict.v1"`) but a missing field in an artifact won't fail unless the test specifically reads that field. - Risk: schemas become *documentation* rather than *contract*. - Defense hardening per D21 — no new core capability, just enforces what the schemas already promise. ## Scope ### 1) Add `jsonschema` to **test extras only** (NOT runtime dep) In `pyproject.toml`: ```toml [project.optional-dependencies] test = [ "pytest>=8.0", "jsonschema>=4.0", ] dev = [ "pytest>=8.0", "jsonschema>=4.0", ] ``` Critically: **`src/patchwarden/` MUST remain stdlib-only**. `jsonschema` is opt-in for the test suite, never imported from runtime code. Verify by grepping `src/patchwarden/` for `import jsonschema` post-PR — must return zero hits. ### 2) New file `tests/test_artifact_schema_contract.py` Loads each schema from `spec/schemas/*.json` once at module level and validates representative CLI outputs against it: | Schema | Validate against | |---|---| | `pr-verdict.schema.json` | `evaluate_pull_request()` output (use `tests/test_pr_check.py` fixtures) | | `issue-verdict.schema.json` | `evaluate_issue()` output | | `review-artifact.schema.json` | `run_review()` output (both empty-findings path and findings path; also soft-fail path with `runtime_metadata.error_kind`) | | `finding-resolution.schema.json` | `resolve_findings()` output (PASS path + HOLD path + soft-fail path) | Each schema gets at minimum: one valid happy-path test, one valid edge case (e.g. empty findings array, soft-fail), and one **deliberately invalid** payload that must raise `jsonschema.ValidationError` (catch-the-catcher meta-test so we know validation actually runs). ### 3) Strengthen `tests/test_boring_pr_lifecycle.py` Add a single `jsonschema.validate()` call after each CLI emit so the existing lifecycle test doubles as a schema contract check. ### 4) Bonus (close at the same time): unit test for `_cloud_review_mode_from_lane()` `src/patchwarden/cli.py` has the helper: ```python def _cloud_review_mode_from_lane(lane): if lane.cloud_review == "allowed": return "cloud" if lane.cloud_review == "required-with-redaction": return "redacted" return "local" ``` No unit test exists; this value lands in the review artifact. Add `tests/test_cli_cloud_review_mode.py` with three cases (`allowed`, `required-with-redaction`, anything else → `local`). ## Acceptance criteria - `PYTHONPATH=src python3 -m unittest discover tests` → all green (156 baseline + new tests; target ≥165). - `pip install -e .[test]` installs `jsonschema` from test extras. - `grep -r "import jsonschema" src/patchwarden/` returns **zero hits** (stdlib-only rule preserved). - Each of the four schemas has both a happy-path validate call and a deliberately-invalid case proving the validator works. ## No-go - ❌ Do NOT add `jsonschema` to runtime deps. Stdlib-only in `src/patchwarden/` per cousin discipline. - ❌ Do NOT change schema version strings — schemas are versioned contracts. If a real validation failure surfaces, file a separate issue, don't loosen the schema to make it pass. - ❌ Do NOT bump `schema_version` on any schema (parked per D21). - ❌ Do NOT switch test framework — keep unittest (pytest is opt-in only per `#22`). ## Spec sources - `spec/schemas/*.json` — the four schemas to validate against - `tests/test_boring_pr_lifecycle.py` — lifecycle test where the strengthening lands - `tests/test_resolve_findings.py` + `tests/test_review_run.py` — fixtures for soft-fail validation cases - `src/patchwarden/cli.py` — `_cloud_review_mode_from_lane()` helper for the bonus test - 2026-05-28 Cloud Opus 4.6 repo analysis, §II.2 "Brak walidacji schematów w testach" ## Status flow `ready-for-agent` (now) → agent claims → PR → operator review → merge --- Created 2026-05-28 by claude (Patchwarden dedicated thread) after Cloud Opus 4.6 repository review surfaced the gap. Atomic enough for any cousin (e.g. Cloud Opus 4.6 via Antigravity) to pick up.

claude added the

labels

2026-05-28 13:37:39 +02:00

Iskra commented

2026-06-08 03:05:18 +02:00

Collaborator

{
"confidence": 5,
"effort_hint": "small",
"escalation": {
"kind": "patchwarden_review",
"reason": "Schema-contract hardening should stay aligned with Patchwarden artifact guarantees."
},
"evidence_refs": [
{
"note": "Issue adds jsonschema validation tests for Patchwarden CLI artifact outputs.",
"type": "forgejo",
"value": "issue-title-body-labels-and-target-snapshot"
},
{
"note": "Body states schemas currently risk drifting into documentation rather than executable contracts.",
"type": "forgejo",
"value": "issue-body-risk"
},
{
"note": "Scope keeps jsonschema in test extras only while preserving stdlib-only runtime code.",
"type": "forgejo",
"value": "issue-body-scope"
}
],
"impact": 4,
"judge_actor": {
"name": "iskra",
"runtime": "openclaw"
},
"judged_at": "2026-06-08T01:03:00Z",
"labels_to_apply": [
"judge/p2",
"judge/patchwarden-candidate"
],
"piotr_fit": "high",
"priority": "p2",
"rationale_summary": "This is P2 Patchwarden hardening because executable schema tests turn artifact formats into real contracts instead of passive docs.",
"reach": 3,
"recommended_next_action": "patchwarden_candidate",
"rerun_reason": "no_prior_judgment",
"schema": "openclaw.judge.v0",
"target": {
"kind": "issue",
"number": 62,
"repo": "pdurlej/patchwarden"
},
"target_snapshot": {
"body_hash": "sha256:b0283b9529b276f9ba2ee810169eb3a9141f42b45d5ffa84ca0f8f82f0354c05",
"commit_count": null,
"evidence_hash": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"head_sha": null,
"labels": [
"agent/gemini",
"area:v0-core",
"flow/ready",
"kind:implementation",
"ready-for-agent",
"size/small"
],
"labels_hash": "sha256:f3fa2375e9f4d4580b4900454ec76e039cbe581cac183e63b96935b54256c649",
"state": "open",
"title_hash": "sha256:7e6219fb12c3b159eba6c47c209dc8279b1aa06e540805ac8d72b37cabcd9b1b",
"updated_at": "2026-05-28T14:04:52+02:00"
},
"top_caveat": "Keep jsonschema as a test-only dependency and do not add runtime dependencies to core code."
}

{ "confidence": 5, "effort_hint": "small", "escalation": { "kind": "patchwarden_review", "reason": "Schema-contract hardening should stay aligned with Patchwarden artifact guarantees." }, "evidence_refs": [ { "note": "Issue adds jsonschema validation tests for Patchwarden CLI artifact outputs.", "type": "forgejo", "value": "issue-title-body-labels-and-target-snapshot" }, { "note": "Body states schemas currently risk drifting into documentation rather than executable contracts.", "type": "forgejo", "value": "issue-body-risk" }, { "note": "Scope keeps jsonschema in test extras only while preserving stdlib-only runtime code.", "type": "forgejo", "value": "issue-body-scope" } ], "impact": 4, "judge_actor": { "name": "iskra", "runtime": "openclaw" }, "judged_at": "2026-06-08T01:03:00Z", "labels_to_apply": [ "judge/p2", "judge/patchwarden-candidate" ], "piotr_fit": "high", "priority": "p2", "rationale_summary": "This is P2 Patchwarden hardening because executable schema tests turn artifact formats into real contracts instead of passive docs.", "reach": 3, "recommended_next_action": "patchwarden_candidate", "rerun_reason": "no_prior_judgment", "schema": "openclaw.judge.v0", "target": { "kind": "issue", "number": 62, "repo": "pdurlej/patchwarden" }, "target_snapshot": { "body_hash": "sha256:b0283b9529b276f9ba2ee810169eb3a9141f42b45d5ffa84ca0f8f82f0354c05", "commit_count": null, "evidence_hash": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "head_sha": null, "labels": [ "agent/gemini", "area:v0-core", "flow/ready", "kind:implementation", "ready-for-agent", "size/small" ], "labels_hash": "sha256:f3fa2375e9f4d4580b4900454ec76e039cbe581cac183e63b96935b54256c649", "state": "open", "title_hash": "sha256:7e6219fb12c3b159eba6c47c209dc8279b1aa06e540805ac8d72b37cabcd9b1b", "updated_at": "2026-05-28T14:04:52+02:00" }, "top_caveat": "Keep jsonschema as a test-only dependency and do not add runtime dependencies to core code." }

Iskra added the

judge/p2

judge/patchwarden-candidate

labels

2026-06-08 03:05:18 +02:00

Iskra referenced this issue from pdurlej/judging-claw

2026-06-08 03:05:18 +02:00

[Judging Claw] patchwarden_candidate for pdurlej/patchwarden#62 (p2) #64

claude referenced this issue from a commit

2026-06-08 18:05:24 +02:00

test(v0): jsonschema validation for CLI artifact outputs (closes #62)

claude referenced this issue from a commit