fix(forgejo): add deploy runner scope doctor #286

Merged

pdurlej merged 1 commit from codex/260/proper-fix into main

2026-05-15 17:02:30 +02:00

codex commented

2026-05-15 16:58:24 +02:00

Collaborator

Canary status: missing — fire canary 3+3 manually before merge

Canary Context Pack

Product story

Future agents must be able to read the deploy lane as green when it is green. Issue #260 made the deploy runner stay active but fail to pick up trusted-main jobs until the watchdog restarted it, which creates a recurring false-fire signal for every later cutover session.

What changed

Added scripts/forgejo/deploy-runner-scope-doctor, a check-first rs2000 tool that diagnoses the deploy runner scope invariant and, only with --apply, backs up Forgejo Actions tables before repairing the known bad runner scope.
Documented the required Forgejo action_runner shape: rs2000-deploy-host must be repository-scoped as owner_id=0, repo_id=35.
Updated the runner runbook to frame the watchdog as belt-and-suspenders diagnostics, not the normal pickup path.
Added regression tests for the script and contract text.

Why it changed

RCA found that the active deploy runner row is in a combined owner+repo scope (owner_id=1, repo_id=35). Forgejo's Actions task-version model tracks global, owner, and repository scopes separately, not combined owner+repo polling scope. The runner can therefore cache a task version that does not change on new repository jobs; a service restart resets the cached version and makes pickup work again.

Files touched

scripts/forgejo/deploy-runner-scope-doctor
docs/ci/runner-contract.md
runbooks/forgejo-actions-runner.md
tests/test_deploy_runner_scope_doctor.py

Relevant context

ADR-0018: agentic-first systems fix root causes, not recurring false-fire workarounds.
Issue #260 runner pickup RCA.
The existing scripts/forgejo/deploy-runner-watchdog remains as diagnostics/belt-and-suspenders only.

Runtime evidence

Read-only live check through the new script confirmed the expected repairable state:

id=4 rs2000-deploy-host owner_id=0 repo_id=35 deleted=1778752131
id=5 rs2000-deploy-host owner_id=1 repo_id=35 deleted=NULL
REPAIRABLE active deploy runner has owner_id=1; expected 0
rerun with --apply after operator approval to repair

Earlier RCA evidence on #260 showed fresh trusted-main runs stuck with task_id=0 until watchdog restart, then assigned immediately after runner restart.

Known constraints

This PR does not run the DB repair. --apply is a sacred-path-adjacent production DB write and remains operator-gated at apply time.
The script refuses unknown runner shapes and only repairs the known owner_id=1, repo_id=35 -> owner_id=0, repo_id=35 state.
Backups are written before mutation and chmodded 0600.

Explicit out-of-scope

No Forgejo service restart.
No watchdog removal.
No production smoke in this PR.
No hard-delete of stale runner id=4.

Requested decision

Review/merge the repo-side safety tool and documentation. After merge, operator can approve the live repair separately with an explicit gate.

Merge blockers

Script can mutate without --apply.
Script can update more than one row or an unexpected row shape.
Backup does not happen before the action_runner update.
Docs describe the watchdog as an accepted primary workaround.

Spec sources read

prompts/codex-260-fix-plus-umami-probe-2026-05-14.md — dispatch and hard gates.
decisions/0018-agentic-first-fix-root-causes.md — no recurring workaround framing.
docs/ci/runner-contract.md — deploy runner contract update target.
runbooks/forgejo-actions-runner.md — operational runbook target.
scripts/forgejo/deploy-runner-watchdog — existing watchdog behavior and diagnostic context.
/tmp/forgejo-src/models/actions/runner.go — runner scope model used in RCA.
/tmp/forgejo-src/models/actions/tasks_version.go — task-version scope model used in RCA.
/tmp/forgejo-src/routers/api/actions/runner/runner.go — runner polling behavior used in RCA.
/tmp/forgejo-runner-src/internal/app/poll/poller.go — runner cached task-version behavior used in RCA.

Test plan

bash -n scripts/forgejo/deploy-runner-scope-doctor
pytest tests/test_deploy_runner_scope_doctor.py tests/test_deploy_runner_watchdog.py -q
# 8 passed

Read-only live diagnostic:

ssh rs2000 'bash -s -- --check' < scripts/forgejo/deploy-runner-scope-doctor
# exits 1 because the known repairable bad scope is still present; no mutation performed

Refs #260

Canary status: missing — fire canary 3+3 manually before merge ## Canary Context Pack ### Product story Future agents must be able to read the deploy lane as green when it is green. Issue #260 made the deploy runner stay active but fail to pick up trusted-main jobs until the watchdog restarted it, which creates a recurring false-fire signal for every later cutover session. ### What changed - Added `scripts/forgejo/deploy-runner-scope-doctor`, a check-first rs2000 tool that diagnoses the deploy runner scope invariant and, only with `--apply`, backs up Forgejo Actions tables before repairing the known bad runner scope. - Documented the required Forgejo `action_runner` shape: `rs2000-deploy-host` must be repository-scoped as `owner_id=0, repo_id=35`. - Updated the runner runbook to frame the watchdog as belt-and-suspenders diagnostics, not the normal pickup path. - Added regression tests for the script and contract text. ### Why it changed RCA found that the active deploy runner row is in a combined owner+repo scope (`owner_id=1, repo_id=35`). Forgejo's Actions task-version model tracks global, owner, and repository scopes separately, not combined owner+repo polling scope. The runner can therefore cache a task version that does not change on new repository jobs; a service restart resets the cached version and makes pickup work again. ### Files touched - `scripts/forgejo/deploy-runner-scope-doctor` - `docs/ci/runner-contract.md` - `runbooks/forgejo-actions-runner.md` - `tests/test_deploy_runner_scope_doctor.py` ### Relevant context - ADR-0018: agentic-first systems fix root causes, not recurring false-fire workarounds. - Issue #260 runner pickup RCA. - The existing `scripts/forgejo/deploy-runner-watchdog` remains as diagnostics/belt-and-suspenders only. ### Runtime evidence Read-only live check through the new script confirmed the expected repairable state: ```text id=4 rs2000-deploy-host owner_id=0 repo_id=35 deleted=1778752131 id=5 rs2000-deploy-host owner_id=1 repo_id=35 deleted=NULL REPAIRABLE active deploy runner has owner_id=1; expected 0 rerun with --apply after operator approval to repair ``` Earlier RCA evidence on #260 showed fresh trusted-main runs stuck with `task_id=0` until watchdog restart, then assigned immediately after runner restart. ### Known constraints - This PR does not run the DB repair. `--apply` is a sacred-path-adjacent production DB write and remains operator-gated at apply time. - The script refuses unknown runner shapes and only repairs the known `owner_id=1, repo_id=35 -> owner_id=0, repo_id=35` state. - Backups are written before mutation and chmodded `0600`. ### Explicit out-of-scope - No Forgejo service restart. - No watchdog removal. - No production smoke in this PR. - No hard-delete of stale runner id=4. ### Requested decision Review/merge the repo-side safety tool and documentation. After merge, operator can approve the live repair separately with an explicit gate. ### Merge blockers - Script can mutate without `--apply`. - Script can update more than one row or an unexpected row shape. - Backup does not happen before the `action_runner` update. - Docs describe the watchdog as an accepted primary workaround. ## Spec sources read - `prompts/codex-260-fix-plus-umami-probe-2026-05-14.md` — dispatch and hard gates. - `decisions/0018-agentic-first-fix-root-causes.md` — no recurring workaround framing. - `docs/ci/runner-contract.md` — deploy runner contract update target. - `runbooks/forgejo-actions-runner.md` — operational runbook target. - `scripts/forgejo/deploy-runner-watchdog` — existing watchdog behavior and diagnostic context. - `/tmp/forgejo-src/models/actions/runner.go` — runner scope model used in RCA. - `/tmp/forgejo-src/models/actions/tasks_version.go` — task-version scope model used in RCA. - `/tmp/forgejo-src/routers/api/actions/runner/runner.go` — runner polling behavior used in RCA. - `/tmp/forgejo-runner-src/internal/app/poll/poller.go` — runner cached task-version behavior used in RCA. ## Test plan ```bash bash -n scripts/forgejo/deploy-runner-scope-doctor pytest tests/test_deploy_runner_scope_doctor.py tests/test_deploy_runner_watchdog.py -q # 8 passed ``` Read-only live diagnostic: ```bash ssh rs2000 'bash -s -- --check' < scripts/forgejo/deploy-runner-scope-doctor # exits 1 because the known repairable bad scope is still present; no mutation performed ``` Refs #260

codex added the

tier/full

risk/runtime

class/security-sensitive

risk/process

labels

2026-05-15 16:58:24 +02:00

codex added 1 commit

2026-05-15 16:58:24 +02:00

fix(forgejo): add deploy runner scope doctor

canary-required / collect-diff (pull_request) Successful in 4s

Details