Journal
What we've learned, and what changed because of it.
Articles on AI memory, RAG and context engineering above; the dated engineering build log — including the edits the engine got wrong — below.
- AI Memory
Why Enterprise AI Still Forgets Your Business
A model's knowledge is fixed at training time and its context is fixed to one conversation. Neither is a record of your company — and retrieval only fixes the part of that problem it can reach.
- RAG
RAG Retrieves. It Does Not Remember.
Retrieval answers "what looks like this question." A large share of real enterprise questions are actually "what depends on what, and is it still true" — a different question that similarity search cannot answer.
- Context Engineering
What Context Actually Means to an AI Model
A model has no concept of 'knowing' something outside its weights. Context is whatever tokens are in the window at inference time, full stop — which makes where those tokens come from the entire engineering problem.
- AI Infrastructure
Bigger Context Windows Are Not Memory
A million-token window and a memory system solve different problems. One increases how much a model can read in a single request. The other decides what a model still knows tomorrow.
Engineering build log
- in progress
SME wired into the official SWE-bench harness
Everything measured so far compares Venkai against a simulated rewrite. That is a fair baseline but it is our own. So the engine now runs inside mini-SWE-agent — the bash-only harness the SWE-bench leaderboard uses — against the official scorer, which decides by running the repository's real tests.
Two arms, identical in every respect except one: same model, same instances, same step limit, same scorer. One arm edits however it likes. The other has the engine available in its container and a prompt that requires it for existing symbols.
Four infrastructure faults stood between the plan and the first real run, none of them about the engine: a serving flag, a cost-accounting exception on a locally served model, a response format, and a client class that silently overrode that format. A guard refused to score every one of those runs — an agent that never ran produces a 0%, and a 0% from a broken pipe reads exactly like a 0% from a weak engine.
No score yet. It gets published here when it exists, with the command, and whichever way it falls.
The number on the front page, and the case against it
25 real symbols, edited two ways. Emitting the whole symbol again costs 17,553 output tokens; emitting a Venkai edit call costs 713. We quote that comparison rather than the whole-file one (112,023) because no serious agent rewrites an entire file, and using that baseline would flatter us.
The same day, the measurement that argues against us: 20 edits spread over 12 files. Regenerating a file costs the file once, however many changes it contains, so there is a point where rewriting wins. We measured it rather than avoided it — roughly 240 edits to a single file. Past that it is not an edit, it is a rewrite.
$ python benchmarks/see_bench.py --limit 25
The engine deleted six decorators and reported success
Six rules in our own analysis layer were edited by the engine and came back without their @decorator. The functions still existed. The signatures matched. The structural check saw nothing wrong. The rules had simply stopped running, and the report announced that false positives had been eliminated.
A decorator is part of the symbol. The engine now refuses an edit that would remove one, instead of applying it and passing the check. This is the failure mode we care most about: not a crash, but a success message over a silent deletion.
Methods go through the AST, not through a text window
The resolver matched definitions anchored at column zero. An indented method matched nothing, fell through to a text fallback, and got replaced along with an arbitrary window of surrounding lines — which could take the end of the file with it. The operation reported success and a 63% saving.
Worse, the replaced region did not parse on its own, so the one check that would have noticed the class had disappeared switched itself off precisely where it was needed. Resolution now goes through the AST, the text path is reserved for text formats, and a test fails if the engine ever truncates a file again.
Cost and latency recorded per call
Every call now records its tokens and its latency where they actually happen, rather than being reported by the component that has an interest in the number. Most of what is on this site comes from that instrumentation.
There is no newsletter and no form. If you want the next entry when it lands, write to contact@venkai.fr and you will get it as a reply, from a person.