pdurlej/platform

Fork 0

test(verify): add l4 deterministic suite #123

Merged

pdurlej merged 1 commit from codex/issues/66-l4-verify-suite into codex/issues/63-validate-jsonschema

2026-05-09 23:37:59 +02:00

codex commented

2026-05-09 00:56:33 +02:00

Collaborator

Canary status: missing — tests/ PR stacked on PR #119; fire canary 3+3 after #119 review or retarget to main after #119 merges.

Closes #66

Stack note

This PR is based on codex/issues/63-validate-jsonschema / PR #119 because Packet B depends on Packet A's platformctl validate --strict-v2 work and #119 is not merged. Review this PR as the delta from #119. Before final merge to main, retarget or rebase after #119 lands.

Canary Context Pack

Product story

The platform needs a deterministic local verification suite so agents can catch schema, prompt, Python, and YAML drift before creating more follow-up work.

What changed

Added tests/test_l4_verify.py with five check categories:
- module manifest schema validation through platformctl.manifest.load_manifest
- module/prompt token budget checks
- prompt literal relative cross-link checks
- py_compile over control-plane/**/*.py
- YAML parsing over .forgejo/, modules/, and schema/
Added tests/run-verify.sh as a convenience wrapper.
Filed #122 for current prompt token-budget and cross-link debt, surfaced as xfail baseline in the new suite.

Why it changed

Issue #66 asks for the L4 deterministic verification suite. Current repo state already has prompt budget and cross-link debt, so the suite establishes a baseline without silently fixing unrelated prompt files.

Files touched

tests/test_l4_verify.py
tests/run-verify.sh

Relevant context

PR #119 provides the schema-validation behavior this PR depends on.
Issue #122 tracks current prompt budget/cross-link debt discovered while implementing this suite.
state/agent-execution-template.md requires explicit verification output and spec-source disclosure.

Runtime evidence

N/A — tests-only change. No runtime reads/writes and no services touched.

Known constraints

This is a stacked PR on #119, not directly independent from current main.
Current prompt debt is represented as xfail and tracked in #122. New module schema failures, new module budget failures, Python compile failures, and YAML syntax failures fail normally.
The cross-link baseline currently xfails at prompt-file granularity when existing missing links are present; #122 is the cleanup path to tighten that gate.

Explicit out-of-scope

No fixes to prompts, modules, schemas, workflows, or control-plane source.
No CI integration.
No Makefile added; repo has none and the packet explicitly uses pytest invocation instead.
No actionlint dependency; YAML parsing is the fallback.

Requested decision

Review as stacked on #119. If #119 merges cleanly, retarget/rebase this PR to main, then request approve_merge after canary.

Merge blockers

#119 is rejected or rewritten in a way that removes the platformctl validation behavior this suite uses.
Reviewers reject the #122 xfail baseline strategy.
The suite fails outside the documented current xfails.

Spec sources read

prompts/codex-platform-final-dispatch-2026-05-09.md: dispatch order, stacked Packet B allowance, and no-auto-merge constraint.
prompts/codex-night-close-2026-05-09.md: Packet B scope and acceptance criteria.
schema/module.schema.json: v1 schema source.
schema/module.schema.v2.json: v2 schema source via PR #119 dependency.
modules/n8n-worker/module.yaml: sample manifest.
prompts/01-freeze.md: sample prompt for token/cross-link behavior.
.forgejo/workflows/canary-required.yml: workflow YAML sample.
control-plane/platformctl/: platformctl Python files for compile/schema behavior.
state/agent-execution-template.md: execution protocol.
state/STATUS_NOW.md: status context named by Packet B.
docs/forgejo-agent-operations.md: Forgejo identity and PR behavior.

Stale prompt source note

Packet B's handoff named prompts/codex-pr-53-rebase.md, but that file does not exist on current main. Issue #66's own spec sources are broader (prompts/*.md), so I did not fabricate a replacement; the suite scans current prompts/*.md instead.

Verification

$ PYTHONPATH=control-plane python3 -m pytest tests/test_l4_verify.py -q
312 passed, 31 xfailed in 120.85s (0:02:00)

$ bash -n tests/run-verify.sh
# passed

$ git diff --check
# passed

Test plan

Review as diff against PR #119 / codex/issues/63-validate-jsonschema.
Confirm #122 is an acceptable baseline issue for current prompt debt.
After #119 merges, retarget/rebase to main and re-run tests/run-verify.sh.

Canary status: missing — tests/ PR stacked on PR #119; fire canary 3+3 after #119 review or retarget to main after #119 merges. Closes #66 ## Stack note This PR is based on `codex/issues/63-validate-jsonschema` / PR #119 because Packet B depends on Packet A's `platformctl validate --strict-v2` work and #119 is not merged. Review this PR as the delta from #119. Before final merge to `main`, retarget or rebase after #119 lands. ## Canary Context Pack ### Product story The platform needs a deterministic local verification suite so agents can catch schema, prompt, Python, and YAML drift before creating more follow-up work. ### What changed - Added `tests/test_l4_verify.py` with five check categories: - module manifest schema validation through `platformctl.manifest.load_manifest` - module/prompt token budget checks - prompt literal relative cross-link checks - `py_compile` over `control-plane/**/*.py` - YAML parsing over `.forgejo/`, `modules/`, and `schema/` - Added `tests/run-verify.sh` as a convenience wrapper. - Filed #122 for current prompt token-budget and cross-link debt, surfaced as `xfail` baseline in the new suite. ### Why it changed Issue #66 asks for the L4 deterministic verification suite. Current repo state already has prompt budget and cross-link debt, so the suite establishes a baseline without silently fixing unrelated prompt files. ### Files touched - `tests/test_l4_verify.py` - `tests/run-verify.sh` ### Relevant context - PR #119 provides the schema-validation behavior this PR depends on. - Issue #122 tracks current prompt budget/cross-link debt discovered while implementing this suite. - `state/agent-execution-template.md` requires explicit verification output and spec-source disclosure. ### Runtime evidence N/A — tests-only change. No runtime reads/writes and no services touched. ### Known constraints - This is a stacked PR on #119, not directly independent from current `main`. - Current prompt debt is represented as `xfail` and tracked in #122. New module schema failures, new module budget failures, Python compile failures, and YAML syntax failures fail normally. - The cross-link baseline currently xfails at prompt-file granularity when existing missing links are present; #122 is the cleanup path to tighten that gate. ### Explicit out-of-scope - No fixes to prompts, modules, schemas, workflows, or control-plane source. - No CI integration. - No Makefile added; repo has none and the packet explicitly uses pytest invocation instead. - No actionlint dependency; YAML parsing is the fallback. ### Requested decision Review as stacked on #119. If #119 merges cleanly, retarget/rebase this PR to `main`, then request `approve_merge` after canary. ### Merge blockers - #119 is rejected or rewritten in a way that removes the platformctl validation behavior this suite uses. - Reviewers reject the #122 xfail baseline strategy. - The suite fails outside the documented current xfails. ## Spec sources read - `prompts/codex-platform-final-dispatch-2026-05-09.md`: dispatch order, stacked Packet B allowance, and no-auto-merge constraint. - `prompts/codex-night-close-2026-05-09.md`: Packet B scope and acceptance criteria. - `schema/module.schema.json`: v1 schema source. - `schema/module.schema.v2.json`: v2 schema source via PR #119 dependency. - `modules/n8n-worker/module.yaml`: sample manifest. - `prompts/01-freeze.md`: sample prompt for token/cross-link behavior. - `.forgejo/workflows/canary-required.yml`: workflow YAML sample. - `control-plane/platformctl/`: platformctl Python files for compile/schema behavior. - `state/agent-execution-template.md`: execution protocol. - `state/STATUS_NOW.md`: status context named by Packet B. - `docs/forgejo-agent-operations.md`: Forgejo identity and PR behavior. ## Stale prompt source note Packet B's handoff named `prompts/codex-pr-53-rebase.md`, but that file does not exist on current main. Issue #66's own spec sources are broader (`prompts/*.md`), so I did not fabricate a replacement; the suite scans current `prompts/*.md` instead. ## Verification ```text $ PYTHONPATH=control-plane python3 -m pytest tests/test_l4_verify.py -q 312 passed, 31 xfailed in 120.85s (0:02:00) $ bash -n tests/run-verify.sh # passed $ git diff --check # passed ``` ## Test plan - [ ] Review as diff against PR #119 / `codex/issues/63-validate-jsonschema`. - [ ] Confirm #122 is an acceptable baseline issue for current prompt debt. - [ ] After #119 merges, retarget/rebase to main and re-run `tests/run-verify.sh`.

codex added 1 commit

2026-05-09 00:56:33 +02:00

test(verify): add l4 deterministic suite

canary-required / collect-diff (pull_request) Successful in 4s

Details

python-ci / Python 3.11 (pull_request) Successful in 34s

Details

python-ci / Python 3.12 (pull_request) Successful in 34s

Details

python-ci / Python 3.13 (pull_request) Successful in 33s

Details

canary-required / canary (pull_request) Successful in 13s

Details

9b20eb3d1f

codex referenced this pull request

2026-05-09 01:06:16 +02:00

[meta] Phase 02 v2 cataloging — first 5-10 priority modules to v2 #70

claude referenced this pull request

2026-05-09 23:32:20 +02:00

feat(platformctl): validate manifests against jsonschema #119

claude commented

2026-05-09 23:32:20 +02:00

Collaborator

Orchestrator review (claude / Pan Herbata)

Verdict: MERGE_READY (merge AFTER #119)

This branch is built on top of #119 (codex/issues/63-validate-jsonschema). Merge #119 first, then this. The diff currently appears as 6 files / 355 lines but post-#119-merge only the 2 unique files (tests/test_l4_verify.py + tests/run-verify.sh, 130 lines) remain.

Codex correctly used the new suite to surface real repo violations (Issue #122 — token-budget overruns + missing cross-links). That issue is operator's call (waive vs fix); does NOT block this PR.

Identity OK.

## Orchestrator review (claude / Pan Herbata) **Verdict: MERGE_READY (merge AFTER #119)** This branch is built on top of #119 (codex/issues/63-validate-jsonschema). Merge #119 first, then this. The diff currently appears as 6 files / 355 lines but post-#119-merge only the 2 unique files (tests/test_l4_verify.py + tests/run-verify.sh, 130 lines) remain. Codex correctly used the new suite to surface real repo violations (Issue #122 — token-budget overruns + missing cross-links). That issue is operator's call (waive vs fix); does NOT block this PR. Identity OK.