What changed, and what it cost to find out.

Everything measured so far compares Venkai against a simulated rewrite. That is a fair baseline but it is our own. So the engine now runs inside mini-SWE-agent — the bash-only harness the SWE-bench leaderboard uses — against the official scorer, which decides by running the repository's real tests.

Two arms, identical in every respect except one: same model, same instances, same step limit, same scorer. One arm edits however it likes. The other has the engine available in its container and a prompt that requires it for existing symbols.

Four infrastructure faults stood between the plan and the first real run, none of them about the engine: a serving flag, a cost-accounting exception on a locally served model, a response format, and a client class that silently overrode that format. A guard refused to score every one of those runs — an agent that never ran produces a 0%, and a 0% from a broken pipe reads exactly like a 0% from a weak engine.

No score yet. It gets published here when it exists, with the command, and whichever way it falls.

25 real symbols, edited two ways. Emitting the whole symbol again costs 17,553 output tokens; emitting a Venkai edit call costs 713. We quote that comparison rather than the whole-file one (112,023) because no serious agent rewrites an entire file, and using that baseline would flatter us.

The same day, the measurement that argues against us: 20 edits spread over 12 files. Regenerating a file costs the file once, however many changes it contains, so there is a point where rewriting wins. We measured it rather than avoided it — roughly 240 edits to a single file. Past that it is not an edit, it is a rewrite.

Six rules in our own analysis layer were edited by the engine and came back without their @decorator. The functions still existed. The signatures matched. The structural check saw nothing wrong. The rules had simply stopped running, and the report announced that false positives had been eliminated.

A decorator is part of the symbol. The engine now refuses an edit that would remove one, instead of applying it and passing the check. This is the failure mode we care most about: not a crash, but a success message over a silent deletion.

The resolver matched definitions anchored at column zero. An indented method matched nothing, fell through to a text fallback, and got replaced along with an arbitrary window of surrounding lines — which could take the end of the file with it. The operation reported success and a 63% saving.

Worse, the replaced region did not parse on its own, so the one check that would have noticed the class had disappeared switched itself off precisely where it was needed. Resolution now goes through the AST, the text path is reserved for text formats, and a test fails if the engine ever truncates a file again.

Every call now records its tokens and its latency where they actually happen, rather than being reported by the component that has an interest in the number. Most of what is on this site comes from that instrumentation.