pdurlej/platform

Fork 0

test(smoke): lock registry digest drift checks #651

Merged

pdurlej merged 1 commit from codex/m06-smoke-contract-closeout into main

2026-05-30 18:04:09 +02:00

codex commented

2026-05-30 17:49:45 +02:00

Collaborator

Canary status: missing - fire canary 3+3 manually before merge

Summary

Adds regression tests that lock the already-fixed tests/smoke.sh behavior for issue #46: registry digest comparison, runbook-derived container names, dependency-light runtime checks, dead-code removal, and safe JSON output.

Canary Context Pack

Product story

Smoke evidence should not create false drift alarms or hide runtime issues behind brittle shell output. The old #46 finding was about false-positive digest drift and unsafe assumptions; the current script behavior should stay protected.

What changed

Added tests/test_smoke_contract.py to freeze the smoke contract.
Verified n8n-worker live read-only smoke now reports image-digest-match:PASS.

Why it changed

Issue #46 remained open after the functional fixes had landed. This PR makes the fixed behavior explicit and prevents regression.

Files touched

tests/test_smoke_contract.py

Relevant context

#46
tests/smoke.sh
modules/n8n-worker/runbook.md

Runtime evidence

Read-only smoke only; no mutation:

{"module":"n8n-worker","passed":4,"failed":0,"skipped":3,"checks":["manifest-exists:PASS","schema-valid:SKIP-use-tests/validate-schema.sh","container-name:PASS-home-platform-n8n-worker-1","container-running:PASS","image-digest-match:PASS","health-http:SKIP-no-url","smoke-extra:SKIP-not-defined"]}

Known constraints

This closes the original false-positive drift finding by test/evidence. It does not redesign the whole smoke framework.

Explicit out-of-scope

Adding remediation mode
Rewriting smoke.sh
Runtime mutation
Changing module manifests

Requested decision

Merge if checks are green. The PR is a regression-test closeout for a previously fixed script.

Merge blockers

Smoke contract test failure
Platform validation failure

Spec sources read

#46 issue body: original smoke drift finding and acceptance criteria
tests/smoke.sh: current implementation
modules/n8n-worker/runbook.md: live container name evidence path

Validation

tests/smoke.sh --json n8n-worker - passed, image-digest-match:PASS
PYTHONPATH=control-plane control-plane/.venv/bin/python -m pytest tests/test_smoke_contract.py - 5 passed
PYTHONPATH=control-plane control-plane/.venv/bin/python -m platformctl.cli validate all --json - exitCode 0

Closes #46

Canary status: missing - fire canary 3+3 manually before merge ## Summary Adds regression tests that lock the already-fixed `tests/smoke.sh` behavior for issue #46: registry digest comparison, runbook-derived container names, dependency-light runtime checks, dead-code removal, and safe JSON output. ## Canary Context Pack ### Product story Smoke evidence should not create false drift alarms or hide runtime issues behind brittle shell output. The old #46 finding was about false-positive digest drift and unsafe assumptions; the current script behavior should stay protected. ### What changed - Added `tests/test_smoke_contract.py` to freeze the smoke contract. - Verified `n8n-worker` live read-only smoke now reports `image-digest-match:PASS`. ### Why it changed Issue #46 remained open after the functional fixes had landed. This PR makes the fixed behavior explicit and prevents regression. ### Files touched - `tests/test_smoke_contract.py` ### Relevant context - #46 - `tests/smoke.sh` - `modules/n8n-worker/runbook.md` ### Runtime evidence Read-only smoke only; no mutation: ```json {"module":"n8n-worker","passed":4,"failed":0,"skipped":3,"checks":["manifest-exists:PASS","schema-valid:SKIP-use-tests/validate-schema.sh","container-name:PASS-home-platform-n8n-worker-1","container-running:PASS","image-digest-match:PASS","health-http:SKIP-no-url","smoke-extra:SKIP-not-defined"]} ``` ### Known constraints This closes the original false-positive drift finding by test/evidence. It does not redesign the whole smoke framework. ### Explicit out-of-scope - Adding remediation mode - Rewriting `smoke.sh` - Runtime mutation - Changing module manifests ### Requested decision Merge if checks are green. The PR is a regression-test closeout for a previously fixed script. ### Merge blockers - Smoke contract test failure - Platform validation failure ## Spec sources read - #46 issue body: original smoke drift finding and acceptance criteria - `tests/smoke.sh`: current implementation - `modules/n8n-worker/runbook.md`: live container name evidence path ## Validation - `tests/smoke.sh --json n8n-worker` - passed, `image-digest-match:PASS` - `PYTHONPATH=control-plane control-plane/.venv/bin/python -m pytest tests/test_smoke_contract.py` - 5 passed - `PYTHONPATH=control-plane control-plane/.venv/bin/python -m platformctl.cli validate all --json` - exitCode 0 Closes #46

codex added 1 commit

2026-05-30 17:49:45 +02:00

test(smoke): lock registry digest drift checks

canary-required / collect-diff (pull_request) Successful in 5s

Details

python-ci / Python 3.11 (pull_request) Successful in 46s

Details

python-ci / Python 3.12 (pull_request) Successful in 46s

Details

python-ci / Python 3.13 (pull_request) Successful in 46s

Details

canary-required / canary (pull_request) Successful in 14s

Details

base-is-main / guard (pull_request) Successful in 1s

Details

patchwarden-client-dry-run / collect-diff (pull_request) Successful in 5s

Details

patchwarden-pr-sanity / collect-diff (pull_request) Successful in 5s

Details

patchwarden-client-dry-run / dry-run (pull_request) Successful in 28s

Details

patchwarden-pr-sanity / sanity (pull_request) Successful in 2m0s

Details

c84bfa8f17

codex added the

agent/codex

risk/runtime

tier/lite

labels

2026-05-30 17:50:10 +02:00

codex commented

2026-05-30 17:55:04 +02:00

Author

Collaborator

Patchwarden PR sanity

Status: advisory_findings
PR: 651
Commit: c84bfa8f179cfb32b4b0f0f3d9e4b7e2f2d47508
Security-sensitive label: missing
Authority: advisory model review plus deterministic blockers only
3+3 canary: still alive; this does not replace it

Deterministic findings

No deterministic findings.

Model reviewers

`global-glm` / `glm-5.1:cloud`

Status: ok
Verdict: NOT_OK
medium Fragile negative string assertions will break on comments or documentation
- Evidence: tests/test_smoke_contract.py lines 28, 34, 40: assert "home-platform-${compose_service}-1" not in text, assert "pyyaml" not in text.lower(), assert "read_field()" not in text — these will fail if anyone adds comments like '# Old pattern: ho
- Next: Use more specific patterns: check for function definitions (e.g., 'read_field()' only as 'function read_field' or 'read_field() {' patterns), check for 'pyyaml' only in import/require/pip contexts, check for the old container pattern only in assignment or usage contexts, not in comments

`global-deepseek` / `deepseek-v4-pro:cloud`

Status: ok
Verdict: OK
low Test fragility due to exact string matching
- Evidence: tests/test_smoke_contract.py contains assertions on exact strings from tests/smoke.sh (e.g., '{{if gt (len .RepoDigests) 0}}{{index .RepoDigests 0}}{{end}}'). Any refactoring of the smoke script will break these tests even if behavior is un
- Next: Consider adding a comment that these tests are intentionally brittle to catch unintended changes, or use a more semantic check (e.g., running the script and parsing output).

`redteam` / `kimi-k2.6:cloud`

Status: ok
Verdict: NOT_OK
high Source-substring contract tests bypassable and brittle
- Evidence: tests/test_smoke_contract.py:15-51 asserts literal strings like 'image-digest-match:PASS' and 'jq -cn' exist in tests/smoke.sh source text, never executing the script
- Next: Add an integration test that runs 'tests/smoke.sh --json n8n-worker' and asserts the returned JSON contains 'image-digest-match:PASS' and the expected container name, rather than grepping source code

Policy notes

GLM 5.1 + DeepSeek V4 Pro are the operator-required model mix for this bot.
Optional red-team model is enabled only when PLATFORMCTL_PR_SANITY_REDTEAM_MODEL is configured.
Auto-merge is not enabled here.

# Patchwarden PR sanity - Status: `advisory_findings` - PR: `651` - Commit: `c84bfa8f179cfb32b4b0f0f3d9e4b7e2f2d47508` - Security-sensitive label: `missing` - Authority: advisory model review plus deterministic blockers only - 3+3 canary: still alive; this does not replace it ## Deterministic findings No deterministic findings. ## Model reviewers ### `global-glm` / `glm-5.1:cloud` - Status: `ok` - Verdict: `NOT_OK` - **`medium`** Fragile negative string assertions will break on comments or documentation - Evidence: `tests/test_smoke_contract.py lines 28, 34, 40: assert "home-platform-${compose_service}-1" not in text, assert "pyyaml" not in text.lower(), assert "read_field()" not in text — these will fail if anyone adds comments like '# Old pattern: ho` - Next: Use more specific patterns: check for function definitions (e.g., 'read_field()' only as 'function read_field' or 'read_field() {' patterns), check for 'pyyaml' only in import/require/pip contexts, check for the old container pattern only in assignment or usage contexts, not in comments ### `global-deepseek` / `deepseek-v4-pro:cloud` - Status: `ok` - Verdict: `OK` - **`low`** Test fragility due to exact string matching - Evidence: `tests/test_smoke_contract.py contains assertions on exact strings from tests/smoke.sh (e.g., '{{if gt (len .RepoDigests) 0}}{{index .RepoDigests 0}}{{end}}'). Any refactoring of the smoke script will break these tests even if behavior is un` - Next: Consider adding a comment that these tests are intentionally brittle to catch unintended changes, or use a more semantic check (e.g., running the script and parsing output). ### `redteam` / `kimi-k2.6:cloud` - Status: `ok` - Verdict: `NOT_OK` - **`high`** Source-substring contract tests bypassable and brittle - Evidence: `tests/test_smoke_contract.py:15-51 asserts literal strings like 'image-digest-match:PASS' and 'jq -cn' exist in tests/smoke.sh source text, never executing the script` - Next: Add an integration test that runs 'tests/smoke.sh --json n8n-worker' and asserts the returned JSON contains 'image-digest-match:PASS' and the expected container name, rather than grepping source code ## Policy notes - GLM 5.1 + DeepSeek V4 Pro are the operator-required model mix for this bot. - Optional red-team model is enabled only when `PLATFORMCTL_PR_SANITY_REDTEAM_MODEL` is configured. - Auto-merge is not enabled here.