pdurlej/skill-surfer
1
0
Fork 0
Code Issues 45 Pull requests Projects Releases Packages Wiki Activity Actions

[Wave 5] Add stratified benchmark fixture set #33

New issue
Open
opened 2026-05-28 01:20:52 +02:00 by codex · 0 comments
codex commented 2026-05-28 01:20:52 +02:00
Collaborator
Copy link

Goal

Evaluate real routing quality by failure category, not only aggregate numbers.

Scope

  • Add 80-120 compact prompts across explicit, implicit, no-skill, near-miss, risky, ambiguous, MCP-only, skill+MCP.
  • Keep cases short and deterministic.
  • Include tags for category-level metrics.

Acceptance criteria

  • Benchmark output includes metrics by tag/category.
  • At least 10 no-skill and 10 near-miss cases exist.
  • Fixtures contain no private secrets or customer data.
  • pnpm check && pnpm build passes.
## Goal Evaluate real routing quality by failure category, not only aggregate numbers. ## Scope - Add 80-120 compact prompts across explicit, implicit, no-skill, near-miss, risky, ambiguous, MCP-only, skill+MCP. - Keep cases short and deterministic. - Include tags for category-level metrics. ## Acceptance criteria - Benchmark output includes metrics by tag/category. - At least 10 no-skill and 10 near-miss cases exist. - Fixtures contain no private secrets or customer data. - `pnpm check && pnpm build` passes.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
wave-6-productization
wave-5-ranking
No results found.
Labels
Clear labels   
agent/claude-code
Vistula actor: Claude Code.

  
agent/codex
Vistula actor: Codex.

  
agent/hermes
Vistula actor: Hermes Upstream.

  
agent/iskra
Vistula actor: Iskra.

  
agent/ollama
Vistula actor: Ollama or compatible model.

  
agent/patchwarden
Vistula actor: Patchwarden.

  
dependency/blocked
Vistula dependency state: blocked.

  
dependency/blocks-others
Vistula dependency state: blocks other work.

  
dependency/cross-repo
Vistula dependency scope: cross-repository.

  
dependency/needs-confirmation
Vistula dependency state: needs confirmation.

  
docs
Documentation and operator guidance.

  
domain:agents
Agent identity, delegation, subagents, routing, or workflows.

  
domain:ci
CI, checks, Actions, runners, or release automation.

  
domain:docs
Documentation, packets, practices, or source-of-truth text.

  
domain:forgejo
Forgejo issues, PRs, labels, repo settings, or identity.

  
domain:infra
Infrastructure, storage, backup, DNS, network, or host substrate.

  
domain:memory
Memory, recall, write gates, salience, or Honcho-adjacent behavior.

  
domain:runtime
Runtime services, deploys, daemons, timers, or live behavior.

  
domain:signal
Signal UX, delivery, context, or outbound behavior.

  
domain:ux
Product UX, operator experience, or conversation ergonomics.

  
flow/architecture
Vistula lifecycle: architecture phase in progress.

  
flow/blocked
Vistula lifecycle: blocked on dependency or decision.

  
flow/deployed
Vistula lifecycle: deployed to runtime where applicable.

  
flow/done
Vistula lifecycle: work merged or otherwise done.

  
flow/implementation
Vistula lifecycle: implementation phase in progress.

  
flow/intake
Vistula lifecycle: incoming signal not refined yet.

  
flow/maintained
Vistula lifecycle: maintained/steady state.

  
flow/observed
Vistula lifecycle: observed after delivery.

  
flow/ready
Vistula lifecycle: ready for architecture or implementation.

  
flow/refining
Vistula lifecycle: Hermes/spec refinement in progress.

  
flow/retired
Vistula lifecycle: retired or intentionally closed.

  
flow/review
Vistula lifecycle: review phase in progress.

  
gemini-ready
Scoped for Gemini 3.5 Flash or another lower-cost coding agent.

  
kind/adr
Vistula work kind: architecture decision record.

  
kind/bug
Vistula work kind: bug fix.

  
kind/chore
Vistula work kind: chore.

  
kind/feature
Vistula work kind: feature.

  
kind/infra
Vistula work kind: infrastructure.

  
kind/ops
Vistula work kind: operations.

  
kind/refactor
Vistula work kind: refactor.

  
kind/research
Vistula work kind: research/discovery.

  
mcp
MCP server or MCP capability inventory work.

  
merge/auto
Vistula merge mode: automated merge.

  
merge/manual
Vistula merge mode: manual merge.

  
merge/manual-dependency-conflict
Manual merge reason: dependency conflict.

  
merge/manual-failing-tests
Manual merge reason: failing tests.

  
merge/manual-merge-conflict
Manual merge reason: merge conflict.

  
merge/manual-missing-review
Manual merge reason: missing review.

  
merge/manual-operator-preference
Manual merge reason: operator preference.

  
merge/manual-red-zone
Manual merge reason: red-zone risk.

  
merge/manual-security-sensitive
Manual merge reason: security-sensitive work.

  
merge/manual-unclear-scope
Manual merge reason: unclear scope.

  
merge/manual-unknown
Manual merge reason: unknown.

  
mode:operator-only
Apply step requires explicit operator approval.

  
mode:patchwarden-iskra-approved
Work eligible for Patchwarden/Iskra approval inside repo policy.

  
mode:safe-auto
Low-risk work eligible for lightweight autonomous handling.

  
observed/erroring
Vistula observation: erroring.

  
observed/needs-followup
Vistula observation: needs follow-up.

  
observed/pending
Vistula observation: pending.

  
observed/retire-candidate
Vistula observation: retire candidate.

  
observed/unused
Vistula observation: unused.

  
observed/used
Vistula observation: used.

  
priority:p0
Immediate safety, uptime, data, or operator-blocking issue.

  
priority:p1
Important active work; should be pulled soon.

  
priority:p2
Useful normal-priority work.

  
priority:p3
Opportunistic cleanup, research, or backlog work.

  
review:claude-reviewed
Reviewed by Claude-family agent.

  
review:codex-reviewed
Reviewed by Codex-family agent.

  
review:dziadek-reviewed
Reviewed by Dziadek pinch-point reviewer.

  
review:needs-human
Needs human review before merge or apply.

  
safety
Security, policy, or no-mutation boundary work.

  
safety:external-write
May write to external systems, users, rooms, repos, or services.

  
safety:no-prod-mutation
Must not mutate production or live external state.

  
safety:prod-impact
May affect production/runtime behavior.

  
safety:secret-touch
Touches secrets, credentials, auth, or secret-adjacent config.

  
size/large
Vistula size: large.

  
size/medium
Vistula size: medium.

  
size/small
Vistula size: small.

  
size/tiny
Vistula size: tiny.

  
size/unknown
Vistula size: not classified yet.

  
source/adr
Vistula source: ADR.

  
source/agent-generated
Vistula source: agent generated.

  
source/manual
Vistula source: manual entry.

  
source/operator-chat
Vistula source: operator chat.

  
source/voice-note
Vistula source: voice note.

  
status:blocked
Blocked by missing dependency, evidence, access, or operator decision.

  
status:codex-ready
Ready for a Codex implementation pass.

  
status:merged:pending-evidence
Merged but waiting for live or artifact evidence before closure.

  
status:needs-evidence
Needs proof, logs, tests, or operator-visible evidence.

  
status:operator-needed
Requires explicit operator input before safe progress.

  
status:parked
Intentionally deferred; not active campaign work.

  
tests
Benchmark, test, and verification work.

  
tier:0-platform-substrate
Data, secrets, backup, storage, or critical platform substrate.

  
tier:1-iskra-value-layer
Iskra/OpenClaw behavior, memory, runtime, or value-layer work.

  
tier:2-tools-products-modules
Tools, products, modules, experiments, and lower-blast-radius work.

  
type:bug
Incorrect behavior or regression.

  
type:chore
Maintenance, cleanup, metadata, or operational task.

  
type:docs
Documentation-only work.

  
type:feat
New capability or product behavior.

  
type:policy
Governance, protocol, boundary, or decision-contract change.

  
type:research
Investigation, spike, or evidence-gathering task.

No labels
agent/claude-code
agent/codex
agent/hermes
agent/iskra
agent/ollama
agent/patchwarden
dependency/blocked
dependency/blocks-others
dependency/cross-repo
dependency/needs-confirmation
docs
domain:agents
domain:ci
domain:docs
domain:forgejo
domain:infra
domain:memory
domain:runtime
domain:signal
domain:ux
flow/architecture
flow/blocked
flow/deployed
flow/done
flow/implementation
flow/intake
flow/maintained
flow/observed
flow/ready
flow/refining
flow/retired
flow/review
gemini-ready
kind/adr
kind/bug
kind/chore
kind/feature
kind/infra
kind/ops
kind/refactor
kind/research
mcp
merge/auto
merge/manual
merge/manual-dependency-conflict
merge/manual-failing-tests
merge/manual-merge-conflict
merge/manual-missing-review
merge/manual-operator-preference
merge/manual-red-zone
merge/manual-security-sensitive
merge/manual-unclear-scope
merge/manual-unknown
mode:operator-only
mode:patchwarden-iskra-approved
mode:safe-auto
observed/erroring
observed/needs-followup
observed/pending
observed/retire-candidate
observed/unused
observed/used
priority:p0
priority:p1
priority:p2
priority:p3
review:claude-reviewed
review:codex-reviewed
review:dziadek-reviewed
review:needs-human
safety
safety:external-write
safety:no-prod-mutation
safety:prod-impact
safety:secret-touch
size/large
size/medium
size/small
size/tiny
size/unknown
source/adr
source/agent-generated
source/manual
source/operator-chat
source/voice-note
status:blocked
status:codex-ready
status:merged:pending-evidence
status:needs-evidence
status:operator-needed
status:parked
tests
tier:0-platform-substrate
tier:1-iskra-value-layer
tier:2-tools-products-modules
type:bug
type:chore
type:docs
type:feat
type:policy
type:research
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
pdurlej/skill-surfer#33
No description provided.