pdurlej/platform

Fork 0

ops(rs2000): codify disk health follow-ups #802

Merged

pdurlej merged 1 commit from codex/orders/rs2000-runtime-followups into main

2026-06-17 22:07:03 +02:00

codex commented

2026-06-17 22:02:23 +02:00

Collaborator

Canary status: missing - fire canary 3+3 manually before merge

Canary Context Pack

Product story

RS2000 already recovered from the Docker/runtime disk pressure incident in #800. This follow-up makes the remaining policy edges explicit so future agents can tell the difference between safe cache cleanup, staged-but-not-live host policy, and data-bearing runtime state.

What changed

Added root filesystem warning/critical thresholds to docker_disk_policy.py report.
Made missing optional host commands report as explicit returncode=127 command payloads instead of crashing local/non-systemd runs.
Added a journald drop-in to the installer: SystemMaxUse=1G, MaxRetentionSec=14day.
Added journald.policy report metadata: configured, disk_usage_status, live_verified, pending_restart, status, expected/observed config, and warnings.
Documented remaining logopts exception classes and Docker reclaimables classification.
Opened #801 separately for safe TTL cleanup of Forgejo Actions task volumes.

Why it changed

The previous emergency cleanup solved the immediate outage pattern, but the follow-up needed to prevent two agent-visible failure modes: broad unsafe Docker pruning, and false confidence that a staged journald file is already active.

Files touched

scripts/host-ops/docker_disk_policy.py
scripts/host-ops/install_docker_disk_policy.py
tests/test_host_ops_docker_disk_policy.py
runbooks/rs2000-disk-hygiene.md
state/reports/rs2000-docker-reclaimables-classification-2026-06-17.md
state/reports/rs2000-logopts-ownership-plan-2026-06-17.md

Relevant context

#800 closeout runtime evidence: RS2000 root about 58% used, Docker about 135G, smokes green for Matrix/Element/Forgejo, no unhealthy containers.
#801 tracks the separate CI-temp TTL lane.

Runtime evidence

No fresh RS2000 mutation was performed in this PR. Fresh read-only SSH was attempted, but local ssh-agent refused key signing. This PR uses #800 closeout evidence and local deterministic tests only.

Known constraints

The installer writes files only. It does not restart Docker or systemd-journald.
journald.policy.status=active means the configured drop-in is older than the observed journald process entry.
pending_restart, configured_unverified, and unknown are not green states.

Explicit out-of-scope

No live RS2000 deploy.
No journald restart, Docker restart, container recreate, or timer enablement.
No image prune, volume prune, docker system prune --volumes, backup deletion, or docker compose --remove-orphans.
No secret reads or runtime env inspection.

Requested decision

Approve the repo-side policy/logging/docs follow-up. Runtime rollout or any destructive cleanup remains a separate approved maintenance action.

Merge blockers

Any path that can delete Docker volumes, backups, images, or runtime state by default.
Any report output that treats missing journald evidence or staged-only config as green.

Spec sources read

scripts/host-ops/docker_disk_policy.py - implementation target.
scripts/host-ops/install_docker_disk_policy.py - installer target.
tests/test_host_ops_docker_disk_policy.py - focused tests.
runbooks/rs2000-disk-hygiene.md - operator-facing policy.
state/reports/rs2000-runtime-health-closeout-2026-06-17.md - #800 closeout evidence.

Verification

python3 -m pytest tests/test_host_ops_docker_disk_policy.py -> 10 passed.
python3 scripts/host-ops/docker_disk_policy.py report --json >/tmp/rs2000-disk-policy-local-report.json && python3 -m json.tool /tmp/rs2000-disk-policy-local-report.json -> JSON valid.
Local macOS report correctly emitted explicit journald unknown/not_configured state because journalctl is unavailable there.
git diff --check -> clean.
Simple secret-pattern rg over touched files -> no matches.

Model review

Ollama/Kimi first pass requested changes for journald false-confidence risk; addressed with explicit journald.policy status and tests.
Ollama/Kimi second pass: approve, no remaining blocker.
AntiGravity/Gemini Pro: approve, required change null; residual risk is operator ignoring pending_restart, documented in the runbook.

Closes #796
Closes #798
Closes #799
Refs #801

Canary status: missing - fire canary 3+3 manually before merge ## Canary Context Pack ### Product story RS2000 already recovered from the Docker/runtime disk pressure incident in #800. This follow-up makes the remaining policy edges explicit so future agents can tell the difference between safe cache cleanup, staged-but-not-live host policy, and data-bearing runtime state. ### What changed - Added root filesystem warning/critical thresholds to `docker_disk_policy.py report`. - Made missing optional host commands report as explicit `returncode=127` command payloads instead of crashing local/non-systemd runs. - Added a journald drop-in to the installer: `SystemMaxUse=1G`, `MaxRetentionSec=14day`. - Added `journald.policy` report metadata: `configured`, `disk_usage_status`, `live_verified`, `pending_restart`, `status`, expected/observed config, and warnings. - Documented remaining logopts exception classes and Docker reclaimables classification. - Opened #801 separately for safe TTL cleanup of Forgejo Actions task volumes. ### Why it changed The previous emergency cleanup solved the immediate outage pattern, but the follow-up needed to prevent two agent-visible failure modes: broad unsafe Docker pruning, and false confidence that a staged journald file is already active. ### Files touched - `scripts/host-ops/docker_disk_policy.py` - `scripts/host-ops/install_docker_disk_policy.py` - `tests/test_host_ops_docker_disk_policy.py` - `runbooks/rs2000-disk-hygiene.md` - `state/reports/rs2000-docker-reclaimables-classification-2026-06-17.md` - `state/reports/rs2000-logopts-ownership-plan-2026-06-17.md` ### Relevant context - #800 closeout runtime evidence: RS2000 root about 58% used, Docker about 135G, smokes green for Matrix/Element/Forgejo, no unhealthy containers. - #801 tracks the separate CI-temp TTL lane. ### Runtime evidence No fresh RS2000 mutation was performed in this PR. Fresh read-only SSH was attempted, but local `ssh-agent` refused key signing. This PR uses #800 closeout evidence and local deterministic tests only. ### Known constraints - The installer writes files only. It does not restart Docker or `systemd-journald`. - `journald.policy.status=active` means the configured drop-in is older than the observed journald process entry. - `pending_restart`, `configured_unverified`, and `unknown` are not green states. ### Explicit out-of-scope - No live RS2000 deploy. - No journald restart, Docker restart, container recreate, or timer enablement. - No image prune, volume prune, `docker system prune --volumes`, backup deletion, or `docker compose --remove-orphans`. - No secret reads or runtime env inspection. ### Requested decision Approve the repo-side policy/logging/docs follow-up. Runtime rollout or any destructive cleanup remains a separate approved maintenance action. ### Merge blockers - Any path that can delete Docker volumes, backups, images, or runtime state by default. - Any report output that treats missing journald evidence or staged-only config as green. ## Spec sources read - `scripts/host-ops/docker_disk_policy.py` - implementation target. - `scripts/host-ops/install_docker_disk_policy.py` - installer target. - `tests/test_host_ops_docker_disk_policy.py` - focused tests. - `runbooks/rs2000-disk-hygiene.md` - operator-facing policy. - `state/reports/rs2000-runtime-health-closeout-2026-06-17.md` - #800 closeout evidence. ## Verification - `python3 -m pytest tests/test_host_ops_docker_disk_policy.py` -> 10 passed. - `python3 scripts/host-ops/docker_disk_policy.py report --json >/tmp/rs2000-disk-policy-local-report.json && python3 -m json.tool /tmp/rs2000-disk-policy-local-report.json` -> JSON valid. - Local macOS report correctly emitted explicit journald unknown/not_configured state because `journalctl` is unavailable there. - `git diff --check` -> clean. - Simple secret-pattern `rg` over touched files -> no matches. ## Model review - Ollama/Kimi first pass requested changes for journald false-confidence risk; addressed with explicit `journald.policy` status and tests. - Ollama/Kimi second pass: `approve`, no remaining blocker. - AntiGravity/Gemini Pro: `approve`, required change `null`; residual risk is operator ignoring `pending_restart`, documented in the runbook. Closes #796 Closes #798 Closes #799 Refs #801

codex added 1 commit

2026-06-17 22:02:23 +02:00

ops(rs2000): codify disk health follow-ups

python-ci / Python 3.12 (pull_request) Successful in 44s

Details

python-ci / Python 3.13 (pull_request) Successful in 42s

Details

canary-required / collect-diff (pull_request) Successful in 4s

Details

python-ci / Python 3.11 (pull_request) Successful in 42s

Details

base-is-main / guard (pull_request) Successful in 1s

Details

canary-required / canary (pull_request) Has been skipped

Details

patchwarden-client-dry-run / collect-diff (pull_request) Successful in 4s

Details

patchwarden-pr-sanity / collect-diff (pull_request) Successful in 4s

Details

patchwarden-pr-sanity / sanity (pull_request) Successful in 2m4s

Details

patchwarden-client-dry-run / dry-run (pull_request) Successful in 18s

Details

140997a96e

codex added the

labels

2026-06-17 22:02:46 +02:00

codex commented

2026-06-17 22:06:14 +02:00

Author

Collaborator

Role: executor

Terminal action: operator_override.

Reason: Forgejo commit status for 140997a96e8b371705751cc844e90ce5a29129b8 remained pending with all PR jobs at Waiting to run / blocked-by-required-conditions after repeated polling. The PR itself is mergeable and contains no live runtime mutation.

Evidence before override:

python3 -m pytest tests/test_host_ops_docker_disk_policy.py -> 10 passed.
python3 scripts/host-ops/docker_disk_policy.py report --json -> valid JSON via python3 -m json.tool.
git diff --check -> clean.
secret-pattern rg over touched files -> no matches.
Ollama/Kimi second pass -> approve, no remaining blocker.
AntiGravity/Gemini Pro -> approve, required change null.

Active approval: present; scope: live merge-fest for this RS2000 runtime/disk-health follow-up. No exact approval phrase is recorded.

Out of scope remains unchanged: no live RS2000 deploy, no Docker/journald restart, no image/volume/system prune, no backup deletion, and no runtime secret reads.

**Role:** executor Terminal action: `operator_override`. Reason: Forgejo commit status for `140997a96e8b371705751cc844e90ce5a29129b8` remained pending with all PR jobs at `Waiting to run` / blocked-by-required-conditions after repeated polling. The PR itself is mergeable and contains no live runtime mutation. Evidence before override: - `python3 -m pytest tests/test_host_ops_docker_disk_policy.py` -> 10 passed. - `python3 scripts/host-ops/docker_disk_policy.py report --json` -> valid JSON via `python3 -m json.tool`. - `git diff --check` -> clean. - secret-pattern `rg` over touched files -> no matches. - Ollama/Kimi second pass -> approve, no remaining blocker. - AntiGravity/Gemini Pro -> approve, required change null. Active approval: present; scope: live merge-fest for this RS2000 runtime/disk-health follow-up. No exact approval phrase is recorded. Out of scope remains unchanged: no live RS2000 deploy, no Docker/journald restart, no image/volume/system prune, no backup deletion, and no runtime secret reads.

pdurlej merged commit 3f9876794e into main

2026-06-17 22:07:03 +02:00

pdurlej referenced this pull request from a commit

2026-06-17 22:07:05 +02:00

Merge pull request #802 from codex/orders/rs2000-runtime-followups

patchwarden commented

2026-06-17 22:35:25 +02:00

First-time contributor

Patchwarden PR sanity

Status: advisory_findings
PR: 802
Commit: 140997a96e8b371705751cc844e90ce5a29129b8
Security-sensitive label: missing
Authority: advisory model review plus deterministic blockers only
3+3 canary: still alive; this does not replace it

Deterministic findings

No deterministic findings.

Model reviewers

`global-glm` / `glm-5.1:cloud`

Status: ok
Verdict: OK
Findings: none

`global-deepseek` / `deepseek-v4-pro:cloud`

Status: ok
Verdict: ABSTAIN
high Missing diff prevents review
- Evidence: The PR description lists changed files but the diff block is empty. Without the actual code changes, it is impossible to verify the claimed additions (root filesystem thresholds, journald drop-in, returncode=127 handling, etc.) or check for
- Next: Ensure the diff is included in the review request. Re-submit with the full diff of the touched files.

`redteam` / `kimi-k2.6:cloud`

Status: ok
Verdict: NOT_OK
high journald active-status relies on easily backdated mtime
- Evidence: PR description explicitly defines journald.policy.status=activeinscripts/host-ops/docker_disk_policy.py as: 'configured drop-in is older than the observed journald process entry'. This mtime-vs-process-start heuristic is trivially byp
- Next: Do not infer liveness from mtime alone. Add a deterministic verification step: record a SHA-256 of the drop-in into a sentinel file at install time, and only report active when the drop-in checksum matches the sentinel and the journald process start time is newer than the sentinel write time. Alte

Policy notes

GLM 5.1 + DeepSeek V4 Pro are the operator-required model mix for this bot.
Optional red-team model is enabled only when PLATFORMCTL_PR_SANITY_REDTEAM_MODEL is configured.
Auto-merge is not enabled here.

# Patchwarden PR sanity - Status: `advisory_findings` - PR: `802` - Commit: `140997a96e8b371705751cc844e90ce5a29129b8` - Security-sensitive label: `missing` - Authority: advisory model review plus deterministic blockers only - 3+3 canary: still alive; this does not replace it ## Deterministic findings No deterministic findings. ## Model reviewers ### `global-glm` / `glm-5.1:cloud` - Status: `ok` - Verdict: `OK` - Findings: none ### `global-deepseek` / `deepseek-v4-pro:cloud` - Status: `ok` - Verdict: `ABSTAIN` - **`high`** Missing diff prevents review - Evidence: `The PR description lists changed files but the diff block is empty. Without the actual code changes, it is impossible to verify the claimed additions (root filesystem thresholds, journald drop-in, returncode=127 handling, etc.) or check for` - Next: Ensure the diff is included in the review request. Re-submit with the full diff of the touched files. ### `redteam` / `kimi-k2.6:cloud` - Status: `ok` - Verdict: `NOT_OK` - **`high`** journald active-status relies on easily backdated mtime - Evidence: `PR description explicitly defines `journald.policy.status=active` in `scripts/host-ops/docker_disk_policy.py` as: 'configured drop-in is older than the observed journald process entry'. This mtime-vs-process-start heuristic is trivially byp` - Next: Do not infer liveness from mtime alone. Add a deterministic verification step: record a SHA-256 of the drop-in into a sentinel file at install time, and only report `active` when the drop-in checksum matches the sentinel and the journald process start time is newer than the sentinel write time. Alte ## Policy notes - GLM 5.1 + DeepSeek V4 Pro are the operator-required model mix for this bot. - Optional red-team model is enabled only when `PLATFORMCTL_PR_SANITY_REDTEAM_MODEL` is configured. - Auto-merge is not enabled here.

codex referenced this pull request

2026-06-17 23:03:05 +02:00

docs(status): refresh platform situation #791