pdurlej/kan-ductor

Fork 0

bug(projection): manual drift detection misclassifies human moves as agent_diverged within audit window #58

New issue

Open

opened 2026-05-10 14:20:10 +02:00 by pdurlej · 2 comments

pdurlej commented

2026-05-10 14:20:10 +02:00

Owner

Source

3+3 review on PR #39 (feat: detect manual projection drift), merged 2026-05-10 via chain drain #52.

"manual drift detection can classify a human move as agent_diverged when a matching agent audit exists within the time window. Use exact audit/activity lineage or audit result activityPublicId; add human-after-agent-within-window regression test."

Problem

Manual drift detection algorithm currently uses a time window to associate card activities with agent audit events. If Piotr (human) moves a card within the same window as a matching agent audit event, the human move is incorrectly classified as agent_diverged (drift).

Result: false drift signals when Piotr legitimately moves cards near agent activity.

Scope

Replace time-window heuristic with exact lineage matching:
- Use audit's result activityPublicId (when present) to bind audit↔activity 1:1
- Fall back to time window only if no exact link exists
Add regression test: human moves card 30s after agent moved different card → must NOT be classified as drift
Add regression test: human moves SAME card 30s after agent → CORRECTLY classified as override

Acceptance criteria

Drift classifier uses exact lineage where available
Time-window fallback marked as confidence:low in result
Regression test for "human-after-agent within window" passes
Existing drift detection still works for genuine cases

Refs

PR #39 (chain drain via #52)
3+3 review comment id 3187

## Source 3+3 review on PR #39 (`feat: detect manual projection drift`), merged 2026-05-10 via chain drain #52. > "manual drift detection can classify a human move as agent_diverged when a matching agent audit exists within the time window. Use exact audit/activity lineage or audit result activityPublicId; add human-after-agent-within-window regression test." ## Problem Manual drift detection algorithm currently uses a **time window** to associate card activities with agent audit events. If Piotr (human) moves a card within the same window as a matching agent audit event, the human move is incorrectly classified as `agent_diverged` (drift). Result: false drift signals when Piotr legitimately moves cards near agent activity. ## Scope - Replace time-window heuristic with exact lineage matching: - Use audit's result `activityPublicId` (when present) to bind audit↔activity 1:1 - Fall back to time window only if no exact link exists - Add regression test: human moves card 30s after agent moved different card → must NOT be classified as drift - Add regression test: human moves SAME card 30s after agent → CORRECTLY classified as override ## Acceptance criteria - Drift classifier uses exact lineage where available - Time-window fallback marked as `confidence:low` in result - Regression test for "human-after-agent within window" passes - Existing drift detection still works for genuine cases ## Refs - PR #39 (chain drain via #52) - 3+3 review comment id 3187

pdurlej added the

priority:p1

3plus3-followup

labels

2026-05-10 14:20:10 +02:00

codex referenced this issue

2026-05-28 01:20:39 +02:00

gemini(w1): scout audit result activityPublicId coverage #102

codex referenced this issue

2026-05-28 01:20:40 +02:00

gemini(w2): manual drift fixture for human move after agent move #113

gemini referenced this issue

2026-05-29 00:18:39 +02:00

fix(api): prefer exact agent audit over time-window fallback for drift detection (#113) #136

codex commented

2026-06-03 10:23:45 +02:00

Collaborator

Codex verification on current origin/main (966de964...): #58 appears resolved.

Evidence:

packages/api/src/routers/agent.ts#detectProjectedCardDrift now only classifies agent_diverged when latestAgentAudit.result.activityPublicId exactly matches latestMoveActivity.publicId.
The old time-window association is still visible as low-confidence metadata, but it does not authorize agent_diverged; if the latest move activity has no exact matching audit, status becomes manual_drift.
packages/api/integration-tests/agent.integration.test.ts includes reports human move after agent move in the same time window as manual drift, covering the original failure mode.

Verification run:

pnpm --filter @kan/api exec vitest integration-tests/agent.integration.test.ts --run -t "human move after agent move"

Passed. Recommendation: close #58 as satisfied by current main.

Codex verification on current `origin/main` (`966de964...`): #58 appears resolved. Evidence: - `packages/api/src/routers/agent.ts#detectProjectedCardDrift` now only classifies `agent_diverged` when `latestAgentAudit.result.activityPublicId` exactly matches `latestMoveActivity.publicId`. - The old time-window association is still visible as low-confidence metadata, but it does not authorize `agent_diverged`; if the latest move activity has no exact matching audit, status becomes `manual_drift`. - `packages/api/integration-tests/agent.integration.test.ts` includes `reports human move after agent move in the same time window as manual drift`, covering the original failure mode. Verification run: ```bash pnpm --filter @kan/api exec vitest integration-tests/agent.integration.test.ts --run -t "human move after agent move" ``` Passed. Recommendation: close #58 as satisfied by current main.

Iskra commented

2026-06-09 03:10:12 +02:00

Collaborator

{
"confidence": 5,
"effort_hint": "medium",
"escalation": {
"kind": "none",
"reason": ""
},
"evidence_refs": [
{
"note": "Issue reports manual projection drift detection misclassifying human moves as agent divergence within an audit window.",
"type": "forgejo",
"value": "issue-title-body-labels-and-target-snapshot"
},
{
"note": "Body states time-window matching can falsely associate Piotr's legitimate card moves with nearby agent audit events.",
"type": "forgejo",
"value": "issue-body-problem"
},
{
"note": "Scope replaces heuristic matching with exact audit-to-activity lineage and adds a human-after-agent regression test.",
"type": "forgejo",
"value": "issue-body-scope"
}
],
"impact": 4,
"judge_actor": {
"name": "iskra",
"runtime": "openclaw"
},
"judged_at": "2026-06-09T01:09:00Z",
"labels_to_apply": [
"judge/p1",
"judge/codex-candidate"
],
"piotr_fit": "high",
"priority": "p1",
"rationale_summary": "This is P1 Codex-ready correctness work because false drift attribution undermines trust in manual operator overrides and projection diagnostics.",
"reach": 4,
"recommended_next_action": "codex_candidate",
"rerun_reason": "no_prior_judgment",
"schema": "openclaw.judge.v0",
"target": {
"kind": "issue",
"number": 58,
"repo": "pdurlej/kan-ductor"
},
"target_snapshot": {
"body_hash": "sha256:24f59ad88697a0248b1b7ea2e6c221a98e6b51ef981948b1129db241fdce82d5",
"commit_count": null,
"evidence_hash": "sha256:241b2c503c9f48cf4953b7efe054ac6df39b28f687fe837224326f6f8ac75e86",
"head_sha": null,
"labels": [
"3plus3-followup",
"priority:p1"
],
"labels_hash": "sha256:eae246ad0747d73dd2fb96aea169d7d74574e5b2de03312edce7ec9f6d87a8f0",
"state": "open",
"title_hash": "sha256:d61ec05924db64a693f1a28e018d8a5a16d5e2d82027dd91bdbeb79bf00dd51d",
"updated_at": "2026-06-03T10:23:45+02:00"
},
"top_caveat": "Use exact lineage when available and keep time-window fallback narrow and visibly lower confidence."
}

{ "confidence": 5, "effort_hint": "medium", "escalation": { "kind": "none", "reason": "" }, "evidence_refs": [ { "note": "Issue reports manual projection drift detection misclassifying human moves as agent divergence within an audit window.", "type": "forgejo", "value": "issue-title-body-labels-and-target-snapshot" }, { "note": "Body states time-window matching can falsely associate Piotr's legitimate card moves with nearby agent audit events.", "type": "forgejo", "value": "issue-body-problem" }, { "note": "Scope replaces heuristic matching with exact audit-to-activity lineage and adds a human-after-agent regression test.", "type": "forgejo", "value": "issue-body-scope" } ], "impact": 4, "judge_actor": { "name": "iskra", "runtime": "openclaw" }, "judged_at": "2026-06-09T01:09:00Z", "labels_to_apply": [ "judge/p1", "judge/codex-candidate" ], "piotr_fit": "high", "priority": "p1", "rationale_summary": "This is P1 Codex-ready correctness work because false drift attribution undermines trust in manual operator overrides and projection diagnostics.", "reach": 4, "recommended_next_action": "codex_candidate", "rerun_reason": "no_prior_judgment", "schema": "openclaw.judge.v0", "target": { "kind": "issue", "number": 58, "repo": "pdurlej/kan-ductor" }, "target_snapshot": { "body_hash": "sha256:24f59ad88697a0248b1b7ea2e6c221a98e6b51ef981948b1129db241fdce82d5", "commit_count": null, "evidence_hash": "sha256:241b2c503c9f48cf4953b7efe054ac6df39b28f687fe837224326f6f8ac75e86", "head_sha": null, "labels": [ "3plus3-followup", "priority:p1" ], "labels_hash": "sha256:eae246ad0747d73dd2fb96aea169d7d74574e5b2de03312edce7ec9f6d87a8f0", "state": "open", "title_hash": "sha256:d61ec05924db64a693f1a28e018d8a5a16d5e2d82027dd91bdbeb79bf00dd51d", "updated_at": "2026-06-03T10:23:45+02:00" }, "top_caveat": "Use exact lineage when available and keep time-window fallback narrow and visibly lower confidence." }

Iskra added the

judge/codex-candidate

judge/p1

labels

2026-06-09 03:10:12 +02:00

Iskra referenced this issue from pdurlej/judging-claw

2026-06-09 03:10:12 +02:00

[Judging Claw] codex_candidate for pdurlej/kan-ductor#58 (p1) #77