pdurlej/platform

Fork 0

ops(rs2000): record disk hygiene rollout #793

Merged

pdurlej merged 1 commit from codex/orders/rs2000-disk-hygiene into main

2026-06-17 19:58:57 +02:00

codex commented

2026-06-17 14:45:02 +02:00

Collaborator

Canary status: missing — fire canary 3+3 manually before merge

Canary Context Pack

Product story

RS2000 ran out of disk during the Matrix/Element incident because rebuildable artifacts and backup outputs were not bounded. This PR records the live remediation that keeps Docker cache/log growth controlled without automating deletion of application data.

What changed

Added host-ops backup retention helper and installer used for the keep-2 critical backup policy.
Added Docker disk policy helper and installer for RS2000.
Added RS2000 disk hygiene runbook.
Added rollout evidence report for the 2026-06-17 runtime deployment.
Added focused tests for backup retention and Docker disk policy generation.

Why it changed

The runtime was fixed first under scoped operator approval. This PR makes that deployed state reproducible and reviewable in the canonical platform repo.

Files touched

runbooks/rs2000-disk-hygiene.md
scripts/host-ops/backup_retention.py
scripts/host-ops/install_keep2_backup_policy.py
scripts/host-ops/docker_disk_policy.py
scripts/host-ops/install_docker_disk_policy.py
state/reports/rs2000-disk-hygiene-2026-06-17.md
tests/test_host_ops_backup_retention.py
tests/test_host_ops_docker_disk_policy.py

Relevant context

AGENTS.md hard-stop model for critical infra and destructive cleanup.
docs/forgejo-agent-operations.md for PR identity and Forgejo workflow.
runbooks/rs2000-disk-hygiene.md for the resulting operator runbook.

Runtime evidence

Captured in state/reports/rs2000-disk-hygiene-2026-06-17.md.

Key post-deploy facts:

/ improved from 188G free / 62% used to 206G free / 58% used after BuildKit GC.
Docker Build Cache dropped from 88.04G to 70.98G.
No Docker volumes, backups, or rollback images were pruned.
Element, Matrix, Forgejo, Infisical, n8n, ntfy, Uptime Kuma, and Honcho smokes passed.
Docker had no unhealthy/restarting containers after rollout.

Known constraints

/etc/docker/daemon.json log rotation applies to newly created containers; existing long-lived containers keep their creation-time logging options until recreated.
platform-unique-knowledge-backup.service and hp-backup-critical.service were already failed before this rollout and remain separate follow-ups.
BuildKit GC can make future builds slower because cache may need to rebuild.

Explicit out-of-scope

Docker volume pruning.
Backup archive deletion beyond the already deployed host-ops keep-2 critical policy.
Docker daemon restart.
Application deploy or restart.
Secret reads, credential changes, exposure changes, or network boundary changes.

Requested decision

Review the repo representation of the live rollout and merge if the implementation matches the safety boundary.

Merge blockers

Any path that can delete Docker volumes or application data automatically.
Any secret-bearing artifact in the PR.
Incorrect runtime evidence or missing rollback path.

Spec sources read

AGENTS.md — repo and critical-infra safety contract.
docs/forgejo-agent-operations.md — Forgejo identity and PR contract.
runbooks/unique-knowledge-backup.md — to verify unique-knowledge archives are outside Docker GC scope.
scripts/backup/unique_knowledge_backup.py and tests/test_unique_knowledge_backup.py — to understand existing backup test patterns.

Tests and verification

python3 -m py_compile scripts/host-ops/backup_retention.py scripts/host-ops/install_keep2_backup_policy.py scripts/host-ops/docker_disk_policy.py scripts/host-ops/install_docker_disk_policy.py
python3 -m pytest tests/test_host_ops_backup_retention.py tests/test_host_ops_docker_disk_policy.py — 8 passed
Secret-pattern scan over new scripts/tests/runbook/report returned no matches.
Runtime smokes and disk checks are recorded in state/reports/rs2000-disk-hygiene-2026-06-17.md.

Canary status: missing — fire canary 3+3 manually before merge ## Canary Context Pack ### Product story RS2000 ran out of disk during the Matrix/Element incident because rebuildable artifacts and backup outputs were not bounded. This PR records the live remediation that keeps Docker cache/log growth controlled without automating deletion of application data. ### What changed - Added host-ops backup retention helper and installer used for the keep-2 critical backup policy. - Added Docker disk policy helper and installer for RS2000. - Added RS2000 disk hygiene runbook. - Added rollout evidence report for the 2026-06-17 runtime deployment. - Added focused tests for backup retention and Docker disk policy generation. ### Why it changed The runtime was fixed first under scoped operator approval. This PR makes that deployed state reproducible and reviewable in the canonical platform repo. ### Files touched - runbooks/rs2000-disk-hygiene.md - scripts/host-ops/backup_retention.py - scripts/host-ops/install_keep2_backup_policy.py - scripts/host-ops/docker_disk_policy.py - scripts/host-ops/install_docker_disk_policy.py - state/reports/rs2000-disk-hygiene-2026-06-17.md - tests/test_host_ops_backup_retention.py - tests/test_host_ops_docker_disk_policy.py ### Relevant context - AGENTS.md hard-stop model for critical infra and destructive cleanup. - docs/forgejo-agent-operations.md for PR identity and Forgejo workflow. - runbooks/rs2000-disk-hygiene.md for the resulting operator runbook. ### Runtime evidence Captured in state/reports/rs2000-disk-hygiene-2026-06-17.md. Key post-deploy facts: - `/` improved from 188G free / 62% used to 206G free / 58% used after BuildKit GC. - Docker Build Cache dropped from 88.04G to 70.98G. - No Docker volumes, backups, or rollback images were pruned. - Element, Matrix, Forgejo, Infisical, n8n, ntfy, Uptime Kuma, and Honcho smokes passed. - Docker had no unhealthy/restarting containers after rollout. ### Known constraints - `/etc/docker/daemon.json` log rotation applies to newly created containers; existing long-lived containers keep their creation-time logging options until recreated. - `platform-unique-knowledge-backup.service` and `hp-backup-critical.service` were already failed before this rollout and remain separate follow-ups. - BuildKit GC can make future builds slower because cache may need to rebuild. ### Explicit out-of-scope - Docker volume pruning. - Backup archive deletion beyond the already deployed host-ops keep-2 critical policy. - Docker daemon restart. - Application deploy or restart. - Secret reads, credential changes, exposure changes, or network boundary changes. ### Requested decision Review the repo representation of the live rollout and merge if the implementation matches the safety boundary. ### Merge blockers - Any path that can delete Docker volumes or application data automatically. - Any secret-bearing artifact in the PR. - Incorrect runtime evidence or missing rollback path. ## Spec sources read - AGENTS.md — repo and critical-infra safety contract. - docs/forgejo-agent-operations.md — Forgejo identity and PR contract. - runbooks/unique-knowledge-backup.md — to verify unique-knowledge archives are outside Docker GC scope. - scripts/backup/unique_knowledge_backup.py and tests/test_unique_knowledge_backup.py — to understand existing backup test patterns. ## Tests and verification - `python3 -m py_compile scripts/host-ops/backup_retention.py scripts/host-ops/install_keep2_backup_policy.py scripts/host-ops/docker_disk_policy.py scripts/host-ops/install_docker_disk_policy.py` - `python3 -m pytest tests/test_host_ops_backup_retention.py tests/test_host_ops_docker_disk_policy.py` — 8 passed - Secret-pattern scan over new scripts/tests/runbook/report returned no matches. - Runtime smokes and disk checks are recorded in `state/reports/rs2000-disk-hygiene-2026-06-17.md`.

codex added 1 commit

2026-06-17 14:45:02 +02:00

ops(rs2000): record disk hygiene rollout

canary-required / collect-diff (pull_request) Successful in 5s

Details

python-ci / Python 3.11 (pull_request) Successful in 44s

Details

python-ci / Python 3.12 (pull_request) Successful in 46s

Details

python-ci / Python 3.13 (pull_request) Successful in 42s

Details

canary-required / canary (pull_request) Successful in 16s

Details

base-is-main / guard (pull_request) Successful in 1s

Details

patchwarden-client-dry-run / collect-diff (pull_request) Successful in 4s

Details

patchwarden-client-dry-run / dry-run (pull_request) Successful in 21s

Details

patchwarden-pr-sanity / sanity (pull_request) Successful in 5m15s

Details

patchwarden-pr-sanity / collect-diff (pull_request) Successful in 4s

Details

df98a1e773

codex added the

class/security-sensitive

risk/process

risk/runtime

tier/full

labels

2026-06-17 14:45:35 +02:00

patchwarden commented

2026-06-17 14:52:24 +02:00

First-time contributor

Patchwarden PR sanity

Status: eligible_sanity_clean
PR: 793
Commit: df98a1e7736f0e183f09cd41d2ca0d1266b3df84
Security-sensitive label: present
Authority: advisory model review plus deterministic blockers only
3+3 canary: still alive; this does not replace it

Deterministic findings

No deterministic findings.

Model reviewers

`global-glm` / `glm-5.1:cloud`

Status: ok
Verdict: OK
Findings: none

`global-deepseek` / `deepseek-v4-pro:cloud`

Status: ok
Verdict: OK
medium Missing canary for critical infra change
- Evidence: PR description states 'Canary status: missing — fire canary 3+3 manually before merge'. The change touches host-ops scripts that affect disk hygiene on a production host, which likely falls under AGENTS.md critical-infra canary requirements
- Next: Fire the canary (3+3) before merging, as noted in the PR description, to satisfy the repo's safety contract.

`redteam` / `kimi-k2.6:cloud`

Status: error
Verdict: -
Note: ReadTimeout: The read operation timed out
Findings: none

Policy notes

GLM 5.1 + DeepSeek V4 Pro are the operator-required model mix for this bot.
Optional red-team model is enabled only when PLATFORMCTL_PR_SANITY_REDTEAM_MODEL is configured.
Auto-merge is not enabled here.

# Patchwarden PR sanity - Status: `eligible_sanity_clean` - PR: `793` - Commit: `df98a1e7736f0e183f09cd41d2ca0d1266b3df84` - Security-sensitive label: `present` - Authority: advisory model review plus deterministic blockers only - 3+3 canary: still alive; this does not replace it ## Deterministic findings No deterministic findings. ## Model reviewers ### `global-glm` / `glm-5.1:cloud` - Status: `ok` - Verdict: `OK` - Findings: none ### `global-deepseek` / `deepseek-v4-pro:cloud` - Status: `ok` - Verdict: `OK` - **`medium`** Missing canary for critical infra change - Evidence: `PR description states 'Canary status: missing — fire canary 3+3 manually before merge'. The change touches host-ops scripts that affect disk hygiene on a production host, which likely falls under AGENTS.md critical-infra canary requirements` - Next: Fire the canary (3+3) before merging, as noted in the PR description, to satisfy the repo's safety contract. ### `redteam` / `kimi-k2.6:cloud` - Status: `error` - Verdict: `-` - Note: ReadTimeout: The read operation timed out - Findings: none ## Policy notes - GLM 5.1 + DeepSeek V4 Pro are the operator-required model mix for this bot. - Optional red-team model is enabled only when `PLATFORMCTL_PR_SANITY_REDTEAM_MODEL` is configured. - Auto-merge is not enabled here.