Benchmarks
SME Benchmark v0.1
v0.1 — 2026-08-04
Every figure on this site names the command that reproduces it. This page is the methodology behind those commands — dataset, environment, and what isn't measured yet.
Dataset
damage_lite300: 455 injected-mutation cases against Venkai's own monorepo, 207 structurally in reach for damage. see_bench: a corpus of real symbols drawn from the same repository, not synthetic fixtures.
Environment
Runs offline, network disabled, against a single codebase (Venkai's own monorepo). No source material uploaded, no keys proxied through Venkai.
Benchmarks
What's compared
Two arms, both measured. A third is not — stated here rather than omitted.
SME-governed: the model describes an intent, SME resolves the anchor and applies a typed operation with rollback on failure.
Ungoverned: the model edits directly — full-file rewrite, unified diff, or raw shell write, with no anchor resolution or rollback.
Not yet measured: an LLM+RAG arm. No retrieval-augmented baseline has been run against this dataset — that comparison is v0.2, not a claim made today.
Benchmarks
Comparative Metrics Matrix
Agent alone vs. Agent + RAG vs. Agent + SME. Unmeasured cells explicitly display 'Not measured yet'.
| Metric | Agent Alone | Agent + RAG | Agent + SME |
|---|---|---|---|
| Token Usage | 100% baseline (147 t/edit) | Not measured yet | -95.9% (19 t/edit) |
| Cost | 100% baseline | Not measured yet | -95.9% payload cost |
| Execution Time | Baseline latency | Not measured yet | Under 1s overhead |
| Intent Accuracy | Not measured yet | Not measured yet | Not measured yet |
| Regression Risk | Not measured yet | Not measured yet | Not measured yet |
| Silent Damage Rate | 53.6% | Not measured yet | 0.5% (Enforced) |
Silent damage rate — autonomous AI agent execution
0.5% vs 53.6%
Silent damage rate (Venkai governed vs direct agent)
| Direct un-governed agent execution(silent damage rate) | 53.6% |
|---|---|
| Venkai governed execution channel(verified & recorded) | 0.5% |
$ python benchmarks/edit_damage_bench.py --limit 300 --offline
Measured 2026-08-04. Measures unverified modifications, unauthorized data accesses, and policy violations that execute without raising runtime exceptions. Figures are the silent rate among cases where damage was structurally reachable (455 cases run, 207 in reach) — a narrower, harder number than the raw silent rate across all cases. Full breakdown: /benchmarks/.
Secondary metric — token payload efficiency per edit
−95.9%
token reduction per execution
| Model rewrites whole file / symbol(raw output tokens) | 17,553 |
|---|---|
| Venkai governed semantic call(governed delta) | 713 |
$ python benchmarks/see_bench.py --limit 25
Measured 2026-08-03. Measured token output reduction when passing governed semantic diffs versus full-file re-generation.
Benchmarks
Roadmap for v0.2
A retrieval-augmented (RAG) comparison arm, and a corpus drawn from a codebase Venkai does not own.