feat(tests): smoke.sh template — generic per-module smoke + drift detection #40

Merged

pdurlej merged 1 commit from claude/orders/tests-smoke-template into main

2026-05-03 20:56:43 +02:00

pdurlej commented

2026-05-03 11:00:28 +02:00

Owner

Summary

Cross-module open loop closure: every Phase 02 v2 runbook (PR #29, #30, #32, #33, #34) references tests/smoke.sh as declared-but-not-yet-written. This PR provides the template — a generic per-module smoke test that reads each module.yaml and runs 5+1 checks against runtime.

What it checks

manifest-exists — modules/<id>/module.yaml file present
schema-valid — module.yaml validates against schema/module.schema.json
container-running — docker container running on declared spec.runtime.host
image-digest-match — live container image digest matches manifest's spec.runtime.image_observed (drift detection)
health-http — curl spec.health.http.url returns expected_status
(optional) smoke-extra — modules/<id>/tests/smoke-extra.sh hook for module-specific checks (e.g., honcho-postgres can verify pgvector extension installed; karakeep can verify search returns results)

Real drift caught during validation

Running the new script on n8n-worker immediately surfaced drift (separate from this PR's scope, just noting):

Manifest image_observed: sha256:72e2242d5d3b… (audit 2026-05-02)
Live container: sha256:74660bbc672e…

Tag-only pinning (UPTIME_KUMA_IMAGE env carries n8nio/n8n:2.9.1 without @sha256:) + restart: unless-stopped → docker pulled newer patch since manifest audit. Exactly the case smoke tests should catch. Will surface as separate small PR (n8n-worker manifest re-audit).

Usage

tests/smoke.sh <module-id>           # human-readable output
tests/smoke.sh --all                 # iterate every module in modules/
tests/smoke.sh --json <module-id>    # machine-readable JSON

Exit codes: 0 = all PASS, 1 = any FAIL, 2 = usage/dependency error.

Dependencies

bash 4+ (macOS: brew install bash for v5+)
python3 with pyyaml + jsonschema (pip3 install jsonschema pyyaml)
ssh with ~/.ssh/config Host entries for module hosts
curl

Intentionally NOT yq — reduced dependency surface by reusing python3 (already required for jsonschema) for manifest field reading via inline scripts. One fewer thing for operator/CI to install.

Validation

Module	PASS	FAIL	SKIP	Notes
n8n-worker	3	1	2	image drift caught (separate fix needed)
uptime-kuma	5	0	1	no smoke-extra defined

NOT in scope (separate PRs)

Per-module smoke-extra.sh implementations (one PR per interesting module)
CI integration (run smoke.sh in Forgejo Actions on PRs touching modules/<id>/)
n8n-worker image drift fix (real finding, separate PR)
README in tests/ (tests/smoke.sh --help is sufficient docs for now)

Cross-module impact

Every wave 2/3/4+ runbook can now mark Smoke test path: tests/smoke.sh declared but file not yet written (cross-module open loop) as FIXED. Future module.yaml writers can rely on this template existing.

Test plan

bash -n tests/smoke.sh — syntax OK
tests/smoke.sh --help — usage prints
tests/smoke.sh n8n-worker — runs end-to-end (drift catch validated)
tests/smoke.sh uptime-kuma — all 5 PASS
tests/smoke.sh --json <id> — valid JSON output (parseable by jq)
tests/smoke.sh nonexistent-module — exits 1 with manifest-exists:FAIL
(Future) Run tests/smoke.sh --all and inspect aggregate pass rate as Phase 02 progress signal

## Summary Cross-module open loop closure: every Phase 02 v2 runbook (PR #29, #30, #32, #33, #34) references `tests/smoke.sh` as declared-but-not-yet-written. This PR provides the template — a generic per-module smoke test that reads each `module.yaml` and runs 5+1 checks against runtime. ## What it checks 1. **`manifest-exists`** — `modules/<id>/module.yaml` file present 2. **`schema-valid`** — `module.yaml` validates against `schema/module.schema.json` 3. **`container-running`** — docker container running on declared `spec.runtime.host` 4. **`image-digest-match`** — live container image digest matches manifest's `spec.runtime.image_observed` (**drift detection**) 5. **`health-http`** — `curl spec.health.http.url` returns `expected_status` 6. **(optional) `smoke-extra`** — `modules/<id>/tests/smoke-extra.sh` hook for module-specific checks (e.g., honcho-postgres can verify pgvector extension installed; karakeep can verify search returns results) ## Real drift caught during validation Running the new script on `n8n-worker` immediately surfaced drift (separate from this PR's scope, just noting): - Manifest `image_observed: sha256:72e2242d5d3b…` (audit 2026-05-02) - Live container: `sha256:74660bbc672e…` Tag-only pinning (`UPTIME_KUMA_IMAGE` env carries `n8nio/n8n:2.9.1` without `@sha256:`) + `restart: unless-stopped` → docker pulled newer patch since manifest audit. Exactly the case smoke tests should catch. Will surface as separate small PR (n8n-worker manifest re-audit). ## Usage ```bash tests/smoke.sh <module-id> # human-readable output tests/smoke.sh --all # iterate every module in modules/ tests/smoke.sh --json <module-id> # machine-readable JSON ``` Exit codes: `0` = all PASS, `1` = any FAIL, `2` = usage/dependency error. ## Dependencies - `bash 4+` (macOS: `brew install bash` for v5+) - `python3` with `pyyaml` + `jsonschema` (`pip3 install jsonschema pyyaml`) - `ssh` with `~/.ssh/config` Host entries for module hosts - `curl` **Intentionally NOT `yq`** — reduced dependency surface by reusing python3 (already required for jsonschema) for manifest field reading via inline scripts. One fewer thing for operator/CI to install. ## Validation | Module | PASS | FAIL | SKIP | Notes | |---|---|---|---|---| | n8n-worker | 3 | 1 | 2 | image drift caught (separate fix needed) | | uptime-kuma | 5 | 0 | 1 | no smoke-extra defined | ## NOT in scope (separate PRs) - Per-module `smoke-extra.sh` implementations (one PR per interesting module) - CI integration (run smoke.sh in Forgejo Actions on PRs touching `modules/<id>/`) - n8n-worker image drift fix (real finding, separate PR) - README in `tests/` (`tests/smoke.sh --help` is sufficient docs for now) ## Cross-module impact Every wave 2/3/4+ runbook can now mark `Smoke test path: tests/smoke.sh declared but file not yet written (cross-module open loop)` as FIXED. Future module.yaml writers can rely on this template existing. ## Test plan - [ ] `bash -n tests/smoke.sh` — syntax OK - [ ] `tests/smoke.sh --help` — usage prints - [ ] `tests/smoke.sh n8n-worker` — runs end-to-end (drift catch validated) - [ ] `tests/smoke.sh uptime-kuma` — all 5 PASS - [ ] `tests/smoke.sh --json <id>` — valid JSON output (parseable by `jq`) - [ ] `tests/smoke.sh nonexistent-module` — exits 1 with `manifest-exists:FAIL` - [ ] (Future) Run `tests/smoke.sh --all` and inspect aggregate pass rate as Phase 02 progress signal

pdurlej added 1 commit

2026-05-03 11:00:28 +02:00

feat(tests): smoke.sh template — generic per-module smoke + drift detection 8ca1259c0e

Cross-module open loop closure: every Phase 02 v2 runbook references
tests/smoke.sh as declared-but-not-yet-written. This PR provides the
template — a generic per-module smoke test that reads each module.yaml
and runs 5 checks against runtime.

What it checks (per module):
1. manifest-exists  — modules/<id>/module.yaml file present
2. schema-valid     — module.yaml validates against schema/module.schema.json
3. container-running — docker container running on declared spec.runtime.host
4. image-digest-match — live container image digest matches manifest's
   spec.runtime.image_observed (DRIFT DETECTION — caught real drift on
   n8n-worker during template validation: live sha256:74660bbc672e vs
   manifest sha256:72e2242d5d3b, audit was 2026-05-02; tag-only pinning +
   restart: unless-stopped pulled newer patch — separate fix, not in scope)
5. health-http      — curl spec.health.http.url returns expected_status
6. (optional) smoke-extra — modules/<id>/tests/smoke-extra.sh hook for
   module-specific checks (e.g., honcho-postgres can verify pgvector
   extension installed, karakeep can verify search returns results)

Usage:
  tests/smoke.sh <module-id>           # human-readable
  tests/smoke.sh --all                 # iterate all modules
  tests/smoke.sh --json <module-id>    # machine-readable

Exit codes: 0 = all PASS, 1 = any FAIL, 2 = usage/dependency error.

Dependencies: bash 4+, python3 + pyyaml + jsonschema, ssh, curl. NOT yq —
intentionally avoided to reduce dependency surface (jsonschema already
requires python3 + pyyaml; reusing same python for manifest reading via
inline scripts).

Validation against current modules:
- n8n-worker: 3 PASS, 1 FAIL (image drift surfaced — real finding, not
  template bug), 2 SKIP (no health.http.url, no smoke-extra)
- uptime-kuma: 5 PASS, 0 FAIL, 1 SKIP (no smoke-extra)

Per-module extra hooks: deferred to module-specific PRs. Template path is
modules/<id>/tests/smoke-extra.sh — when present and executable, called
with module-id as $1; non-zero exit = FAIL with first line of output as
detail. Keeps generic smoke.sh stable while modules can add specific
checks (e.g., postgres extension verification, search index sanity).

Cross-module impact: every wave 2/3/4+ runbook can now mark
"Smoke test path: tests/smoke.sh declared but file not yet written" as
FIXED. Future module.yaml writers can rely on this template existing.

NOT in scope (separate PRs):
- Per-module smoke-extra.sh implementations (one PR per interesting module)
- CI integration (run smoke.sh in Forgejo Actions on relevant PRs)
- n8n-worker image drift fix (real finding from validation, separate PR)
- README in tests/ (script --help is sufficient docs for now)

Length: 292 lines bash. Tested on macOS bash 5+ + linux bash 4+.

pdurlej referenced this pull request from a commit

2026-05-03 11:19:48 +02:00

docs(decisions): ADR 0001 — canary 3+3 mandatory; orchestrator cadence is PM-style

pdurlej referenced this pull request

2026-05-03 11:20:12 +02:00

docs(decisions): ADR 0001 — canary 3+3 mandatory; orchestrator cadence is PM-style #41

pdurlej referenced this pull request from a commit

2026-05-03 19:50:38 +02:00

docs(decisions): ADR 0001 amendment — Rule 1a inline enforcement + cycle-counter

pdurlej merged commit c1c506edf0 into main

2026-05-03 20:56:43 +02:00

pdurlej referenced this pull request from a commit