Proof over claim

Last time I described the hour a platform shipped the feature I had spent months building. This is the part where I ran the test instead of the emotions, and found two things that were never competing.

One number up front, because it qualifies everything below: this is still one user. Every figure here comes from my corpus, my questions, my four servers. The method transfers. Whether the result does, I cannot tell you yet.

The same test, two different answers

The test from last time is one question: teach it a fact only true in your world, close everything, come back in a fresh session, and count how often the fact comes back.

Both passed. That was the first surprise, and it is worth saying clearly: the built-in memory works. This is not a post about a competitor being bad.

The difference showed up when I changed one thing in the test. I asked the fact from a different editor.

And then again from a different machine. And then I asked a colleague to ask it.

That is where the two answers stopped matching, and it had nothing to do with quality. It had to do with what each thing considers its own boundary.

Session memory and project memory are different products

Here is the distinction I could not put into words on that bad evening, and it took a test to produce it.

A vendor's memory is bound to the vendor's harness. It makes one assistant continuous with itself. That is genuinely valuable and it is what most people mean when they ask for memory.

What I had built is bound to the repository instead. It makes the knowledge about a codebase continuous — across editors, across machines, across people, across model upgrades.

Those are not two implementations of one feature. They are answers to two different questions. "What did I just say?" and "what does this project know?" only look similar until you switch tools.

The clearest way to see it: when someone leaves your team, a session memory leaves with them. A project memory does not, because it was never theirs.

The boundary a similarity search cannot cross

The four boundaries above are about where memory lives. There is a fifth one, and it is about how you find anything in it — I only understood it because someone else wrote it down.

Think about the two entries that matter most together: an error, and the fix that was found three weeks later. Write them out.

deploy hangs at "Build image", worker log says nothing

the runner disk was full; docker prune and restart the service

Those two share almost no vocabulary. One is a symptom, one is a cause; one is about a build step, one is about a disk. In embedding space they sit far apart — not slightly, structurally. And they are the single most valuable pair in the whole store, because together they are the answer and apart they are two anecdotes.

A similarity search cannot connect them. Not because the embeddings are bad. Because semantic closeness and causal connection are different relations, and only one of them is what a vector index measures.

That is why our store keeps a causal path beside the similarity one — which entry led to which, which contradicts which. I had treated that as a secondary feature for a long time. It is the part that similarity cannot do, and I needed an outside article to see it.

This also cuts against my own product, so I will say it plainly: if what you need is "find me the thing that sounds like this", a vector search over a file does that, and the extra machinery earns nothing.

The things my own tool did that I had not noticed

This is the embarrassing part, and it is the reason I am writing it down. I had built features I did not use, because my own use case was one person and four servers.

It can hand knowledge between people. I had built the sharing path months earlier for a technical reason and never once used it, because I work alone on this. For a team it is the entire point: one person debugs a thing at 2 a.m., everyone else inherits the reason.

It can answer why, not just what. Every entry carries what worked and what failed. I had been reading only the first field for months. The second one is where the expensive knowledge lives — the approach that looked right and was not.

It can be asked from things that are not editors. Because it speaks MCP and HTTP, our own operations dashboard queries it: a question box that answers from the event stream plus the stored lessons. I built that as a side project and it turned out to be a second product surface.

And it does not care which model you use. The lessons written by one model are read by the next one. I had treated that as an implementation detail. It is the reason the store survives an upgrade cycle that rewrites everything else.

Where the vendor's version is simply better

A comparison that only finds advantages is an advertisement, so here is the other direction, and I mean it.

Setup: theirs is zero. It exists the moment you install. Mine needed a decision, an account and configuration — and measuring that gap honestly is what pushed us to a trial that needs no sign-up at all.

Depth inside one conversation: theirs sits in the harness and sees everything. Anything from the outside sees what it is told, and that is a real ceiling, not a temporary one.

Trust: the vendor already holds your code. Handing the same knowledge to a second party is an extra decision, and "it is hosted in the EU" is an answer, not a dismissal of the question.

If your work lives in one assistant, on one machine, alone, the built-in memory is the right choice and I would tell you so.

Measure it yourself instead of believing either of us

The test that produced all of this is four variations of one question, and the variations are the whole trick. Same fact, different boundary.

Run it in the tool you taught the fact to. Then in a second editor. Then on a second machine. Then have someone else ask.

Each of those four is a boundary a memory either crosses or does not, and no announcement will tell you which — the crossing is the thing that got designed, and it rarely shows up in release notes.

Write the four numbers down before you form an opinion. The shape of the four tells you which product you are holding.

The harness, extended from last time:

#!/usr/bin/env bash
# Four boundaries, one fact. The SHAPE of the four numbers is the answer.
set -u
FACT="which port the staging database listens on"
EXPECTED="5433"           # what you stored, written down BEFORE you ask

ask() {  # ask <label> <command...>
  local label="$1"; shift
  local hits=0
  for i in 1 2 3 4 5; do
    out=$("$@" 2>/dev/null)
    grep -qF -- "$EXPECTED" <<<"$out" && hits=$((hits + 1))
  done
  printf '%-22s %d/5\n' "$label" "$hits"
}

ask "same tool"      your-assistant --new-session --ask "$FACT"
ask "other editor"   other-editor-cli --ask "$FACT"
ask "other machine"  ssh other-box  your-assistant --ask "$FACT"
# The fourth one is not scriptable, and that is the point:
# ask a colleague to run the same question on their own machine.
# ask "other person"  ...

# 5/5 0/5 0/5 0/5  -> session memory. Continuous with itself.
# 5/5 5/5 5/5 5/5  -> project memory. Continuous with the repository.
# Neither is wrong. They are answers to different questions, and you now
# know which one you have.

One warning from our own numbers: do not measure recall quality by whether the answer sounds right. We once had a stored entry containing the exact address we needed, displayed at session start, and made the mistake anyway — the preview cut off at a hundred characters and the address sat at character three hundred and twenty-three. Delivery is a separate measurement from storage, and it is the one that decides whether any of this pays.

What changes for you

Before: a platform ships something adjacent to your work and you decide, from the announcement, whether you are finished. Half the time you are wrong in the pessimistic direction, which costs you the thing you were building.

After: you run four variations of one test, get four numbers, and find out whether you were building the same product or a neighbouring one. It takes an afternoon and it replaces a week of dread.

The thing I actually learned is not about competition. It is that I had never described my own product, and a platform release forced me to — which turned out to be the most useful thing anyone did for it all year.

I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations; cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks. Every assistant reads the same memory, and every lesson carries the name of whoever learned it.

Try it:

• 30 seconds, no account — npx @cachly-dev/mcp-server@latest demo in any git repo. It reads your log locally and prints what an assistant would already know about the project. After npx fetches the package, the command makes no network calls.
• Claude Code plugin — /plugin marketplace add cachly-dev/cachly-mcp, then /plugin install cachly-brain@cachly.
• 5 minutes, free tier — npx @cachly-dev/mcp-server@latest autopilot writes the MCP configuration for whichever assistant you use.
• Or from the web — cachly.dev · free tier, German servers, no credit card.

This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable.

https://github.com/xbill9/gemma4-dev

The whole exercise cost under three dollars, and that is the part worth keeping. Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts. It also bought five wrong claims, each caught by measuring instead of reasoning. On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.

What is this project trying to Do?

Three rigs in this monorepo serve google/gemma-4-E2B-it on an AWS G5g instance. One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers. The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.

For months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison.

Prerequisites

• An AWS account with G-family quota in us-east-1. Each g5g.2xlarge is 8 vCPU, so 16 vCPU of spot quota runs two at once.
• A subnet, a security group opening TCP 8000, and an instance profile carrying AmazonSSMManagedInstanceCore plus read on the Hugging Face token secret.
• A Hugging Face token in Secrets Manager. It is fetched at boot into a root-only EnvironmentFile and never appears in user data.
• boto3 and the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.

AWS EC2 G5g

G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.

Gemma 4 E2B

google/gemma-4-E2B-it is the reference instruction-tuned release. It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.

The dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.

The Three Runtimes

Turing has no bfloat16 datapath, so all three run float16. It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.

Check the Quotas

check_g5g_quotas

| Quota | vCPUs |
| Running On-Demand G and VT instances (vCPU) | 16 |
| All G and VT Spot Instance Requests (vCPU) | 16 |

`g5g.2xlarge` needs 8 vCPUs.

That is the constraint behind every launch below: two rigs in parallel, and no more.

The Sweep Could Not See vLLM

The sweep script read its throughput figure straight out of the response body:

"decode_tps": usage.get("decode_tokens_per_second", 0.0),

usage.decode_tokens_per_second is a field our own servers invent. vLLM does not emit it, and neither does anything else, so the harness could not be pointed at the vLLM rig at all. The three-way comparison had never actually been run.

Re-running a rig does not fix that. Only a common statistic does.

One Statistic, Three Servers

Every OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.

python3 sweep.py --help | grep -A2 decode-source

  --decode-source {auto,usage,stream,both}
                        where the decode figure comes from; see the module
                        docstring

The stream path uses vllm bench serve's exact TPOT definition, (latency - ttft) / (output_len - 1), so a number from this harness is directly comparable to that tool's published figures. auto probes the endpoint once and picks both where the server emits its own gauge, stream where it does not.

Is the Calibration Transferable?

No, and that is worth a measurement rather than an assumption. Running both measures each rig's offset between the two statistics.

Two percent against 4.6 percent, on the same day and the same instance shape. Borrowing one rig's ratio to convert the other's number would inject a 2.6 percent error into a comparison whose smallest interesting gap is 24 percent. The cross-rig table below is therefore built from stream throughout.

A Gauge Rounded to One Decimal

The JAX server emitted its decode gauge with one decimal place:

f'tpu_jax_decode_tokens_per_second{{model="{MODEL_ID}"}} {METRICS["last_tokens_per_second"]:.1f}',

At about 13 tok/s, one decimal is 0.78 percent resolution. Every sweep that rig had produced showed all three repeats of a cell as byte-identical: 12.8, 12.8, 12.8. That is not reproducibility, it is the measurement floor. The rig had been used to argue about two percent effects it could not resolve.

Two characters fixed it. The first run afterwards reads 12.962, where before it would have said 13.0.

Launch the Instance

Capacity for the whole G5g family was exhausted across all four availability zones several times, so the launcher cycles them with a sixty second backoff.

[12:51:45] round 5 us-east-1c: ❌ AWS InsufficientInstanceCapacity
[12:52:47] us-east-1a: ✅ Launching `i-02e79988a6cbeecbf` (g5g.2xlarge, spot, 1x T4G) in `us-east-1`.

The walkthrough from here follows the PyTorch rig on i-02e79988a6cbeecbf; the other two run the same steps against their own instances. All three landed in us-east-1a within hours of each other. Note that AWS names the other zones as available in every one of those errors — that text describes on-demand capacity and says nothing about spot.

Watch the Install

Cloud-init installs the runtime and then backgrounds itself, so the progress tool reports cloud-init's own state as well as the install log. A dead bootstrap and a slow one must not render identically.

get_install_progress i-02e79988a6cbeecbf

INSTALL COMPLETE
--- cloud-init ---
status: done
errors: []

This is a wheel install, not a build. Across the three timed boots the install finishes a median 113.55 s after launch, against the hours the vLLM rig needs for a from-source build.

Verify the GPU

A config flag being accepted proves nothing, so the probe runs a real matmul on the device.

verify_gpu_arch i-02e79988a6cbeecbf

NVIDIA T4G, 7.5, 15360 MiB
torch: 2.12.0+cu132
arch_list: ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
capability: (7, 5)
compute_dtype: float16
fp16 matmul ok: True

✅ torch reached the GPU and a real fp16 matmul executed.

The DLAMI's torch carries sm_75. Upstream PyPI aarch64 wheels do not, so a pip install torch on this box would serve on CPU without saying so.

Deploy the Server

The payload is the rig's own source, shipped over SSM as a gzipped tarball because user data caps at 16 KiB.

deploy_torch_server i-02e79988a6cbeecbf

✅ Deployed 3 files (16 KiB base64) to `i-02e79988a6cbeecbf`.

Payload root: `/home/xbill/gemma4-dev/gpu-pytorch-g5g-2b`
Build id: `060a572aeb55` — verify_model_health checks the running server reports this.

Verify the Installation

A non-empty reply is not evidence of health. One sibling was once measured answering ': ok: ok: ok…', so the check reads the server's own degenerate-response counter either side of its probe, and compares the served build id against the local payload.

verify_model_health i-02e79988a6cbeecbf

✅ health=200 tokens=5 reply='ok'

- Degenerate (server's own verdict on the full text): **no**
- Build id served: `060a572aeb55`
- Build id matches the local payload (`060a572aeb55`).

Run the Sweep

The same command runs against all three rigs. Only the endpoint changes.

python3 sweep.py --base http://<ip>:8000/v1 --out benchmarks/runs/<run> \
  --contexts 64,512,1024,2048,3072,3800 --outputs 32,128 --repeats 3 \
  --decode-source both

decode-source: both -> both
ctx~512 out=32: in=633 out=32 decode=10.96 tok/s  e2e=9.83 tok/s (warmup 11.08)  stream/usage=0.9616
ctx~2048 out=128: in=2501 out=89 decode=10.66 tok/s  e2e=9.16 tok/s (warmup 10.61)  stream/usage=0.9540
ctx~3800 out=32: FAILED HTTP Error 400 {"detail":"prompt is 4630 tokens and the context
  bound is 4096, leaving no room to decode. Start the server with a larger --seq."}

Cells that cannot exist on the hardware are recorded as infeasible rather than dropped. An absent cell is indistinguishable from an untried one, which is how a sweep overstates its own coverage.

Decode at Concurrency One

How Close Is That to the Hardware?

The ceiling is arithmetic, not a measurement. E2B streams 4.514 GB of weights per decode step against a measured 277 GB/s, giving 16.30 ms per step and 61.4 tok/s.

The PLE table is excluded from that figure because it is a gather and never a matmul. Quantising it from 9.257 GB to 5.752 GB moved decode by 0.00 tok/s across three cells, which is what confirms it never streams.

All three runtimes sit far below the ceiling, so none of them is bandwidth-bound at batch one. The PyTorch profile shows why: about 5,650 kernel launches per step at one to three microseconds each, on a chip whose launch overhead is five to ten.

The Number I Did Not Expect

Time to first token was the result of the run, right up until it was checked.

A 30x advantage, far larger than the 3.2x on decode. It is also impossible. Prefill at 3,746 tokens is roughly 14 TFLOP against a T4G's realistic 20 to 30 TFLOP/s, which is 460 ms at best. vLLM's row says 178 ms.

It did not. vLLM ships enable_prefix_caching=True, and its own metrics say so:

grep -E "^vllm:prefix_cache_(queries|hits)_total" metrics.prom

vllm:prefix_cache_queries_total{engine="0"} 102898.0
vllm:prefix_cache_hits_total{engine="0"}     97440.0

A 94.7 percent hit rate. vLLM genuinely prefilled 5.3 percent of the tokens it was sent, because the harness reused one prompt for a cell's warm-up and all three repeats. Neither sibling has a prefix cache, so both paid full prefill every time.

The fix places a nonce first in the prompt, since a shared prefix is exactly what the cache keys on and a trailing nonce would not have defeated it. That property is now a unit test.

What the Prefill Data Does Support

Strip the contaminated column and a real result remains. Neither of the other two runtimes caches prefixes, and both saw identical prompts.

JAX prefills 2.4x slower than PyTorch, consistently across all five shared context lengths. On an interactive workload with real context that dominates user-visible latency, and it runs in the opposite direction to the 1.24x decode advantage the same rig enjoys.

Boot Time Reverses the Ranking

Nine cold boots, three per runtime, plus nine warm reloads. The start line is the moment run_instances returns an id, because capacity wait measures AWS rather than the rig.

vLLM takes 23m 38s to serve, from a prebuilt AMI that downloads nothing. PyTorch installs its runtime from wheels and pulls the 9.54 GiB checkpoint over the network, and is still 7.3x faster.

Boot variance is 11.5, 11.8 and 12.6 percent — too consistent across three different runtimes to be a property of any of them. This family's decode noise floor is 1.7 percent, measured by running an identical build on two hosts, so boot is about seven times noisier and a single boot measurement is close to worthless.

Is Health 200 the Same as Ready?

Not on every runtime, which is why the harness records two stop lines.

JAX returns health 200 and then compiles XLA per shape bucket on the first real request. Quoting health alone understates its time to serving by 22 seconds, and the cost does not vanish when warm. vLLM is the mirror image: slowest to boot, fastest first token, because graph capture is paid before the port binds.

What Does a Code Change Cost?

vLLM's 264 s warm figure is a systemctl restart, not a code change, so it flatters the comparison. vLLM wins decode 3.2x and loses the iteration loop by 3 to 100x.

Five Theories About 546 Seconds

vLLM's cold boot is dominated by weight loading: 468 to 561 seconds across four measurements. Explaining it took five attempts, four of which were wrong.

1. g5g.2xlarge needs no swapfile and buys that time back. The rig's own documentation. Falsified by the three-boot campaign at 23m 38s.
2. A bigger host will not fix it. Retracted the same day — no large host had been measured.
3. A larger host would very plausibly fix it. Falsified by one g5g.4xlarge boot: available RAM 11.19 to 26.49 GiB, weight loading 546 to 468 s, and total boot 4.7 percent lower, inside the noise band.
4. It is the loader; vLLM's log says auto-prefetch is disabled on EXT4. Falsified by a within-box A/B — 76.13 s as shipped against 75.12 s with --safetensors-load-strategy=prefetch, which is 1.3 percent and therefore nothing.

What that last run did find is the useful part.

Same volume, same filesystem, same engine, differing only in whether the blocks had been read once. 9.54 GiB in 468 s is about 21 MiB/s, which is absurd for gp3 steady state and ordinary for first-touch reads against a snapshot-backed volume.

Theory five is EBS lazily hydrating the volume from the AMI snapshot, and it is written down as untested. Given the strike rate it does not get promoted by reasoning.

Why the First Campaign Was Thrown Away

The first boot campaign was discarded and re-run. Two independent instances had reported 214.4 s and 125.1 s, identical to the tenth. That is not consistency. It is a five second poll quantising two similar boots onto the same tick.

The data was not wrong; the campaign log shows 215 s and 216 s of wall clock. It was unusably coarse. Health polling went to half a second, and the next pair of boots came in at 216.90 s and 193.86 s — an 11.9 percent spread the old harness could not see.

And Price/Performance?

The whole exercise — 19 instances, 4.06 hours of g5g.2xlarge plus 0.38 of g5g.4xlarge, a serving sweep, three cross-rig runs, nine boots and three A/B restarts — came to under $3. That is arithmetic rather than a bill: AWS drops terminated instances after an hour, so the derivation in cost_derivation.md bounds it at $1.84 all-spot and $2.68 all-on-demand.

That 26 to 46 percent premium is the entire spot versus on-demand decision on this hardware, which is to say there is not one. Try spot, fall back, keep working; the automatic fallback cost about $0.24 across a nine-boot campaign.

Cheap hardware is what made the method possible, not merely affordable. Discarding a completed campaign over a poll-interval bug cost twenty minutes and pennies. Where a run is expensive, the same discovery argues for shipping the numbers with a caveat instead.

AWS Services Used

Clean Up

Every instance is terminated as soon as its artifacts are captured. There is no built image to lose, only a pip install and a model cache.

terminate_g5g_instance i-02e79988a6cbeecbf

🗑️ Terminating `i-02e79988a6cbeecbf`. Relaunch costs a pip install, not a build.

aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[?starts_with(InstanceType,`g5g`)].InstanceId' \
  --output text | grep . || echo "🟢 none running"

🟢 none running

What This Does Not Cover

Everything above is concurrency one, which makes it a latency comparison rather than a serving one. Continuous batching is vLLM's whole value proposition and it is untested here.

Only vLLM can serve concurrently at all today. The PyTorch server holds an asyncio.Lock with the comment "one GPU, one process -> serialize requests", and the JAX rig has no batching machinery whatever. The engine-level batch sweep suggests what is on the table — batch eight reaches 84.16 tok/s for an extra 0.258 GB, with per-step time growing two percent across an eight-fold batch — but that number never leaves the engine.

Three further gaps: the JAX leg ran its shipped quantised configuration against two dense runtimes; no output-quality axis was measured at all, on a comparison where one runtime uses a deliberately lossy LM head; and every TTFT figure predates the prompt-uniqueness fix, so only the JAX versus PyTorch half of that table is sound.

Summary

The goal of this article was to compare three inference runtimes on identical silicon without the harness being a variable. The key to the solution was a single client-side statistic that every OpenAI-compatible server can produce. The measured results were:

• Decode at concurrency one: vLLM 32.53, JAX 12.69, PyTorch 10.24 tok/s — 53, 21 and 17 percent of a 61.4 tok/s bandwidth ceiling, so none is bandwidth-bound.
• The ranking reverses on lifecycle. Cold boot 195.2 s for PyTorch against 1417.8 s for vLLM; a code change costs 25 s against a from-source rebuild.
• JAX prefills 2.4x slower than PyTorch, 1.403 against 0.595 ms/token.
• A 94.7 percent prefix-cache hit rate turned a 30x TTFT result into a harness artifact.
• Calibration offsets are per-rig, 0.9799 and 0.9543, and are not transferable.
• Boot variance is about 12 percent on this platform regardless of runtime.
• The whole exercise cost under $3 in total. That is what made discarding a finished campaign over a poll-interval bug a twenty minute decision rather than an argument.

Scope: the decode and boot numbers are one g5g.2xlarge per runtime in us-east-1a on 2026-08-31, three repeats per cell and three repeats per boot, mixed spot and on-demand with the market recorded per run. Three things differed between the legs and are named where they matter: vLLM ran max_model_len 16384 against 4096 for the other two; the JAX leg ran its shipped quantised configuration against two dense runtimes; and vLLM booted from a prebuilt AMI carrying its model cache while the other two installed from wheels and downloaded the checkpoint, which is the point of the boot comparison rather than a flaw in it. Two runs sit outside that envelope and say so in the text: the RAM test was a single g5g.4xlarge, and the prefetch A/B ran 2026-09-01 on on-demand after a spot reclamation killed the first attempt. Decode is unaffected by the prefix-cache issue, which changes prefill only.

The strategy for using MCP for multi-runtime comparison was validated with a incremental step by step approach.

This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable.

https://github.com/xbill9/gemma4-dev

The whole exercise cost under three dollars, and that is the part worth keeping. Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts. It also bought five wrong claims, each caught by measuring instead of reasoning. On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.

What is this project trying to Do?

Three rigs in this monorepo serve google/gemma-4-E2B-it on an AWS G5g instance. One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers. The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.

For months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison.

Prerequisites

• An AWS account with G-family quota in us-east-1. Each g5g.2xlarge is 8 vCPU, so 16 vCPU of spot quota runs two at once.
• A subnet, a security group opening TCP 8000, and an instance profile carrying AmazonSSMManagedInstanceCore plus read on the Hugging Face token secret.
• A Hugging Face token in Secrets Manager. It is fetched at boot into a root-only EnvironmentFile and never appears in user data.
• boto3 and the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow.

AWS EC2 G5g

G5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.

Gemma 4 E2B

google/gemma-4-E2B-it is the reference instruction-tuned release. It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.

The dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.

The Three Runtimes

Turing has no bfloat16 datapath, so all three run float16. It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.

Check the Quotas

check_g5g_quotas

| Quota | vCPUs |
| Running On-Demand G and VT instances (vCPU) | 16 |
| All G and VT Spot Instance Requests (vCPU) | 16 |

`g5g.2xlarge` needs 8 vCPUs.

That is the constraint behind every launch below: two rigs in parallel, and no more.

The Sweep Could Not See vLLM

The sweep script read its throughput figure straight out of the response body:

"decode_tps": usage.get("decode_tokens_per_second", 0.0),

usage.decode_tokens_per_second is a field our own servers invent. vLLM does not emit it, and neither does anything else, so the harness could not be pointed at the vLLM rig at all. The three-way comparison had never actually been run.

Re-running a rig does not fix that. Only a common statistic does.

One Statistic, Three Servers

Every OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.

python3 sweep.py --help | grep -A2 decode-source

  --decode-source {auto,usage,stream,both}
                        where the decode figure comes from; see the module
                        docstring

The stream path uses vllm bench serve's exact TPOT definition, (latency - ttft) / (output_len - 1), so a number from this harness is directly comparable to that tool's published figures. auto probes the endpoint once and picks both where the server emits its own gauge, stream where it does not.

Is the Calibration Transferable?

No, and that is worth a measurement rather than an assumption. Running both measures each rig's offset between the two statistics.

Two percent against 4.6 percent, on the same day and the same instance shape. Borrowing one rig's ratio to convert the other's number would inject a 2.6 percent error into a comparison whose smallest interesting gap is 24 percent. The cross-rig table below is therefore built from stream throughout.

A Gauge Rounded to One Decimal

The JAX server emitted its decode gauge with one decimal place:

f'tpu_jax_decode_tokens_per_second{{model="{MODEL_ID}"}} {METRICS["last_tokens_per_second"]:.1f}',

At about 13 tok/s, one decimal is 0.78 percent resolution. Every sweep that rig had produced showed all three repeats of a cell as byte-identical: 12.8, 12.8, 12.8. That is not reproducibility, it is the measurement floor. The rig had been used to argue about two percent effects it could not resolve.

Two characters fixed it. The first run afterwards reads 12.962, where before it would have said 13.0.

Launch the Instance

Capacity for the whole G5g family was exhausted across all four availability zones several times, so the launcher cycles them with a sixty second backoff.

[12:51:45] round 5 us-east-1c: ❌ AWS InsufficientInstanceCapacity
[12:52:47] us-east-1a: ✅ Launching `i-02e79988a6cbeecbf` (g5g.2xlarge, spot, 1x T4G) in `us-east-1`.

The walkthrough from here follows the PyTorch rig on i-02e79988a6cbeecbf; the other two run the same steps against their own instances. All three landed in us-east-1a within hours of each other. Note that AWS names the other zones as available in every one of those errors — that text describes on-demand capacity and says nothing about spot.

Watch the Install

Cloud-init installs the runtime and then backgrounds itself, so the progress tool reports cloud-init's own state as well as the install log. A dead bootstrap and a slow one must not render identically.

get_install_progress i-02e79988a6cbeecbf

INSTALL COMPLETE
--- cloud-init ---
status: done
errors: []

This is a wheel install, not a build. Across the three timed boots the install finishes a median 113.55 s after launch, against the hours the vLLM rig needs for a from-source build.

Verify the GPU

A config flag being accepted proves nothing, so the probe runs a real matmul on the device.

verify_gpu_arch i-02e79988a6cbeecbf

NVIDIA T4G, 7.5, 15360 MiB
torch: 2.12.0+cu132
arch_list: ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']
capability: (7, 5)
compute_dtype: float16
fp16 matmul ok: True

✅ torch reached the GPU and a real fp16 matmul executed.

The DLAMI's torch carries sm_75. Upstream PyPI aarch64 wheels do not, so a pip install torch on this box would serve on CPU without saying so.

Deploy the Server

The payload is the rig's own source, shipped over SSM as a gzipped tarball because user data caps at 16 KiB.

deploy_torch_server i-02e79988a6cbeecbf

✅ Deployed 3 files (16 KiB base64) to `i-02e79988a6cbeecbf`.

Payload root: `/home/xbill/gemma4-dev/gpu-pytorch-g5g-2b`
Build id: `060a572aeb55` — verify_model_health checks the running server reports this.

Verify the Installation

A non-empty reply is not evidence of health. One sibling was once measured answering ': ok: ok: ok…', so the check reads the server's own degenerate-response counter either side of its probe, and compares the served build id against the local payload.

verify_model_health i-02e79988a6cbeecbf

✅ health=200 tokens=5 reply='ok'

- Degenerate (server's own verdict on the full text): **no**
- Build id served: `060a572aeb55`
- Build id matches the local payload (`060a572aeb55`).

Run the Sweep

The same command runs against all three rigs. Only the endpoint changes.

python3 sweep.py --base http://<ip>:8000/v1 --out benchmarks/runs/<run> \
  --contexts 64,512,1024,2048,3072,3800 --outputs 32,128 --repeats 3 \
  --decode-source both

decode-source: both -> both
ctx~512 out=32: in=633 out=32 decode=10.96 tok/s  e2e=9.83 tok/s (warmup 11.08)  stream/usage=0.9616
ctx~2048 out=128: in=2501 out=89 decode=10.66 tok/s  e2e=9.16 tok/s (warmup 10.61)  stream/usage=0.9540
ctx~3800 out=32: FAILED HTTP Error 400 {"detail":"prompt is 4630 tokens and the context
  bound is 4096, leaving no room to decode. Start the server with a larger --seq."}

Cells that cannot exist on the hardware are recorded as infeasible rather than dropped. An absent cell is indistinguishable from an untried one, which is how a sweep overstates its own coverage.

Decode at Concurrency One

How Close Is That to the Hardware?

The ceiling is arithmetic, not a measurement. E2B streams 4.514 GB of weights per decode step against a measured 277 GB/s, giving 16.30 ms per step and 61.4 tok/s.

The PLE table is excluded from that figure because it is a gather and never a matmul. Quantising it from 9.257 GB to 5.752 GB moved decode by 0.00 tok/s across three cells, which is what confirms it never streams.

All three runtimes sit far below the ceiling, so none of them is bandwidth-bound at batch one. The PyTorch profile shows why: about 5,650 kernel launches per step at one to three microseconds each, on a chip whose launch overhead is five to ten.

The Number I Did Not Expect

Time to first token was the result of the run, right up until it was checked.

A 30x advantage, far larger than the 3.2x on decode. It is also impossible. Prefill at 3,746 tokens is roughly 14 TFLOP against a T4G's realistic 20 to 30 TFLOP/s, which is 460 ms at best. vLLM's row says 178 ms.

It did not. vLLM ships enable_prefix_caching=True, and its own metrics say so:

grep -E "^vllm:prefix_cache_(queries|hits)_total" metrics.prom

vllm:prefix_cache_queries_total{engine="0"} 102898.0
vllm:prefix_cache_hits_total{engine="0"}     97440.0

A 94.7 percent hit rate. vLLM genuinely prefilled 5.3 percent of the tokens it was sent, because the harness reused one prompt for a cell's warm-up and all three repeats. Neither sibling has a prefix cache, so both paid full prefill every time.

The fix places a nonce first in the prompt, since a shared prefix is exactly what the cache keys on and a trailing nonce would not have defeated it. That property is now a unit test.

What the Prefill Data Does Support

Strip the contaminated column and a real result remains. Neither of the other two runtimes caches prefixes, and both saw identical prompts.

JAX prefills 2.4x slower than PyTorch, consistently across all five shared context lengths. On an interactive workload with real context that dominates user-visible latency, and it runs in the opposite direction to the 1.24x decode advantage the same rig enjoys.

Boot Time Reverses the Ranking

Nine cold boots, three per runtime, plus nine warm reloads. The start line is the moment run_instances returns an id, because capacity wait measures AWS rather than the rig.

vLLM takes 23m 38s to serve, from a prebuilt AMI that downloads nothing. PyTorch installs its runtime from wheels and pulls the 9.54 GiB checkpoint over the network, and is still 7.3x faster.

Boot variance is 11.5, 11.8 and 12.6 percent — too consistent across three different runtimes to be a property of any of them. This family's decode noise floor is 1.7 percent, measured by running an identical build on two hosts, so boot is about seven times noisier and a single boot measurement is close to worthless.

Is Health 200 the Same as Ready?

Not on every runtime, which is why the harness records two stop lines.

JAX returns health 200 and then compiles XLA per shape bucket on the first real request. Quoting health alone understates its time to serving by 22 seconds, and the cost does not vanish when warm. vLLM is the mirror image: slowest to boot, fastest first token, because graph capture is paid before the port binds.

What Does a Code Change Cost?

vLLM's 264 s warm figure is a systemctl restart, not a code change, so it flatters the comparison. vLLM wins decode 3.2x and loses the iteration loop by 3 to 100x.

Five Theories About 546 Seconds

vLLM's cold boot is dominated by weight loading: 468 to 561 seconds across four measurements. Explaining it took five attempts, four of which were wrong.

1. g5g.2xlarge needs no swapfile and buys that time back. The rig's own documentation. Falsified by the three-boot campaign at 23m 38s.
2. A bigger host will not fix it. Retracted the same day — no large host had been measured.
3. A larger host would very plausibly fix it. Falsified by one g5g.4xlarge boot: available RAM 11.19 to 26.49 GiB, weight loading 546 to 468 s, and total boot 4.7 percent lower, inside the noise band.
4. It is the loader; vLLM's log says auto-prefetch is disabled on EXT4. Falsified by a within-box A/B — 76.13 s as shipped against 75.12 s with --safetensors-load-strategy=prefetch, which is 1.3 percent and therefore nothing.

What that last run did find is the useful part.

Same volume, same filesystem, same engine, differing only in whether the blocks had been read once. 9.54 GiB in 468 s is about 21 MiB/s, which is absurd for gp3 steady state and ordinary for first-touch reads against a snapshot-backed volume.

Theory five is EBS lazily hydrating the volume from the AMI snapshot, and it is written down as untested. Given the strike rate it does not get promoted by reasoning.

Why the First Campaign Was Thrown Away

The first boot campaign was discarded and re-run. Two independent instances had reported 214.4 s and 125.1 s, identical to the tenth. That is not consistency. It is a five second poll quantising two similar boots onto the same tick.

The data was not wrong; the campaign log shows 215 s and 216 s of wall clock. It was unusably coarse. Health polling went to half a second, and the next pair of boots came in at 216.90 s and 193.86 s — an 11.9 percent spread the old harness could not see.

And Price/Performance?

The whole exercise — 19 instances, 4.06 hours of g5g.2xlarge plus 0.38 of g5g.4xlarge, a serving sweep, three cross-rig runs, nine boots and three A/B restarts — came to under $3. That is arithmetic rather than a bill: AWS drops terminated instances after an hour, so the derivation in cost_derivation.md bounds it at $1.84 all-spot and $2.68 all-on-demand.

That 26 to 46 percent premium is the entire spot versus on-demand decision on this hardware, which is to say there is not one. Try spot, fall back, keep working; the automatic fallback cost about $0.24 across a nine-boot campaign.

Cheap hardware is what made the method possible, not merely affordable. Discarding a completed campaign over a poll-interval bug cost twenty minutes and pennies. Where a run is expensive, the same discovery argues for shipping the numbers with a caveat instead.

AWS Services Used

Clean Up

Every instance is terminated as soon as its artifacts are captured. There is no built image to lose, only a pip install and a model cache.

terminate_g5g_instance i-02e79988a6cbeecbf

🗑️ Terminating `i-02e79988a6cbeecbf`. Relaunch costs a pip install, not a build.

aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[?starts_with(InstanceType,`g5g`)].InstanceId' \
  --output text | grep . || echo "🟢 none running"

🟢 none running

What This Does Not Cover

Everything above is concurrency one, which makes it a latency comparison rather than a serving one. Continuous batching is vLLM's whole value proposition and it is untested here.

Only vLLM can serve concurrently at all today. The PyTorch server holds an asyncio.Lock with the comment "one GPU, one process -> serialize requests", and the JAX rig has no batching machinery whatever. The engine-level batch sweep suggests what is on the table — batch eight reaches 84.16 tok/s for an extra 0.258 GB, with per-step time growing two percent across an eight-fold batch — but that number never leaves the engine.

Three further gaps: the JAX leg ran its shipped quantised configuration against two dense runtimes; no output-quality axis was measured at all, on a comparison where one runtime uses a deliberately lossy LM head; and every TTFT figure predates the prompt-uniqueness fix, so only the JAX versus PyTorch half of that table is sound.

Summary

The goal of this article was to compare three inference runtimes on identical silicon without the harness being a variable. The key to the solution was a single client-side statistic that every OpenAI-compatible server can produce. The measured results were:

• Decode at concurrency one: vLLM 32.53, JAX 12.69, PyTorch 10.24 tok/s — 53, 21 and 17 percent of a 61.4 tok/s bandwidth ceiling, so none is bandwidth-bound.
• The ranking reverses on lifecycle. Cold boot 195.2 s for PyTorch against 1417.8 s for vLLM; a code change costs 25 s against a from-source rebuild.
• JAX prefills 2.4x slower than PyTorch, 1.403 against 0.595 ms/token.
• A 94.7 percent prefix-cache hit rate turned a 30x TTFT result into a harness artifact.
• Calibration offsets are per-rig, 0.9799 and 0.9543, and are not transferable.
• Boot variance is about 12 percent on this platform regardless of runtime.
• The whole exercise cost under $3 in total. That is what made discarding a finished campaign over a poll-interval bug a twenty minute decision rather than an argument.

Scope: the decode and boot numbers are one g5g.2xlarge per runtime in us-east-1a on 2026-08-31, three repeats per cell and three repeats per boot, mixed spot and on-demand with the market recorded per run. Three things differed between the legs and are named where they matter: vLLM ran max_model_len 16384 against 4096 for the other two; the JAX leg ran its shipped quantised configuration against two dense runtimes; and vLLM booted from a prebuilt AMI carrying its model cache while the other two installed from wheels and downloaded the checkpoint, which is the point of the boot comparison rather than a flaw in it. Two runs sit outside that envelope and say so in the text: the RAM test was a single g5g.4xlarge, and the prefetch A/B ran 2026-09-01 on on-demand after a spot reclamation killed the first attempt. Decode is unaffected by the prefix-cache issue, which changes prefill only.

The strategy for using MCP for multi-runtime comparison was validated with a incremental step by step approach.

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Code: github.com/deghosal-2026/agent-self-edit

15 iterations. 4,150 LLM calls. Zero promotions. The gate said no — 15 times in a row.

And it was right every time.

I spent a full session building and testing this system. I thought I'd get a promotion — an edit that the gate approves, the prompt improves, accuracy goes up. Instead, I got the most honest result possible: the gate rejected everything, and the rejections were correct.

Here's what happened, how I found a two-line bug that was letting noise through as "improvement," and why a gate that never promotes is the most valuable thing I built.

The Problem: How Do You Know an Edit Is Actually Better?

Most prompt optimization is vibes. You change a prompt, run it on a few examples, and if it "looks better," you ship it. There's no statistics. No control group. No significance test. Just intuition.

That works fine for a human tweaking a prompt by hand. But when you're building a system that rewrites its own prompts autonomously — an LLM proposing edits, the system applying them, and the loop repeating — vibes aren't enough. You need evidence.

The easy answer is "ask an LLM to judge the edit." But that's a fox guarding the henhouse. An LLM judging its own edits will optimize toward what it likes, not what actually works. The prompt drifts. The system gets worse, not better, and nobody notices because the judge keeps saying "looks good."

The harder answer is: build a deterministic, statistical gate that evaluates edits using evidence, not opinion. Code, not prompts. p-values, not vibes.

That's what I built.

The System: A Closed Loop With a Gate

AgentSelfEdit is a sidecar. It doesn't modify the agent's runtime. It observes execution traces and proposes prompt edits through a closed loop:

1. Analyze — An LLM reviews failed traces and proposes concrete edits, each with a hypothesis
2. Test — Each edit is A/B tested against the current prompt on a held-out task set
3. Gate — A deterministic promotion gate with 6 checks decides: promote, reject, or near-miss
4. Registry — Promoted edits are versioned with full lineage, diff, and rollback

The gate is the most important component. It's the safety mechanism that prevents the system from optimizing itself into a worse state. And it's not an LLM — it's six deterministic checks, running in fail-fast order:

Three outcomes: promote (all 6 pass), near-miss (most pass, logged for human review), reject (a critical check failed). No LLM in the decision. No "looks good to me." Just code.

The Twist: A Bug That Made Everything Look Like Success

I ran the loop. I got a promotion. Accuracy jumped from 20% to 40%. I celebrated.

Then I looked at the code.

The confidence check — the one that's supposed to prevent noise from getting promoted — was checking the wrong threshold:

# What was written:
passed = p < confidence_level  # p < 0.95

# What it should have been:
alpha = 1 - confidence_level    # 0.05
passed = p < alpha              # p < 0.05

The gate was checking p < 0.95 instead of p < 0.05. Think about what that means. A p-value of 0.9 would pass. A p-value of 0.5 would pass. Even a p-value of 0.94 would pass. Almost everything would pass.

My "promotion" at p=0.1 had a 10% chance of being random noise. One in ten. I was celebrating a result that could have been a coin flip.

Standard hypothesis testing works like this: you set a significance level (alpha), typically 0.05. You compute a p-value. If p < alpha, the result is statistically significant. The confidence_level (0.95) is 1 - alpha. So alpha = 1 - 0.95 = 0.05. The gate should have been checking p < 0.05. It was checking p < 0.95.

Two lines of code. That's all it took to turn noise into a "success."

After the fix, the same edit produced p=0.23. The gate rejected it. Correctly. There was a 23% chance the improvement was random — well above the 5% threshold the gate requires.

The Run: 15 Iterations, All Rejected

I ran the corrected loop for 15 iterations against a local Qwen3.5-4B-4bit model on Apple Silicon. Every iteration produced the same result:

• The analyzer reviewed 50 real failure traces and proposed the same edit — adding priority rules to a classification prompt
• The A/B test ran the candidate against the current prompt on 26 hard classification tasks
• The edit fixed 4 tasks, broke 1, for a net +3 improvement (11.5%)
• The gate rejected it (p=0.23, not significant at p<0.05)

The edit was real. The improvement was real. The model genuinely classified 4 tasks better with the priority rules. But it also broke 1 task — a search bug that went from "technical" (correct) to "feature" (wrong) because one of the priority rules was too broad.

Net +3 on 26 tasks. p=0.23. The permutation test asked: "if this edit had no real effect, what's the probability of seeing an improvement this large by chance?" The answer: 23%. The gate requires less than 5%.

The gate was doing its job. It was protecting the system from an edit that, while helpful, hadn't proven itself enough. The evidence bar was set at p<0.05, and the edit couldn't clear it.

What I Caught Along the Way

The confidence check wasn't the only bug. Over the course of this session, I found and fixed 31 issues. Here are the ones that mattered most:

The A/B test was comparing a prompt against itself. The code passed a fragment (the edited section) as prompt_b instead of the full candidate prompt. Both arms used the same prompt. Every A/B test was a tie — not because the edit didn't help, but because there was no edit to test.

The failure traces were fabricated. The script hardcoded final_output: "other" for every trace. But the model actually outputs "billing," "security," "technical." The analyzer was learning from a failure pattern that didn't exist.

The gate received the wrong prompt. check_all got prompt_b (the edited version) instead of prompt_a (the original). The frozen_sections check looked for edit.old_text in the wrong prompt — it was already replaced.

The Docker test skipped the A/B test and gate. It used --dry-run, which skips the two most important stages. "9/9 tests passed" was a smoke test dressed up as an integration test.

Every one of these bugs produced output that looked correct. The summary said "pass." The traffic logs said something different. I caught every one by inspecting raw LLM traffic — 4,150 request/response pairs logged to a JSONL file.

What I Learned

The gate is the product, not the optimizer. The most valuable part of a self-improving system isn't the component that proposes changes — it's the component that decides whether to accept them. The optimizer can be an LLM, a heuristic, or random guessing. The gate must be deterministic, verifiable, and conservative. If the gate is wrong, the system drifts. If the gate is right, the system is safe — even if the optimizer is dumb.

A gate that never promotes is still valuable. If the gate rejects everything for 15 iterations, it's tempting to say "the system doesn't work." But the gate rejecting means the evidence bar is being enforced. The analyzer's proposals aren't strong enough yet — that's a different problem. The gate is working. The optimizer needs to get better.

The goal isn't to make the loop pass. I spent time tweaking thresholds to force a promotion. I raised the drift threshold from 0.3 to 0.5. I expanded the A/B task set from 5 to 26. I got a promotion and celebrated. Then I found the confidence bug and realized the promotion was noise. I had to course-correct: the goal of the field test was to prove the gate behaves honestly, not to get a green outcome. A gate that rejects underpowered improvements is the success condition, not the failure condition.

Statistical significance is not optional. The inverted confidence check (p < 0.95 instead of p < 0.05) let noise through as "improvement." Two lines of code. That's the difference between a system that learns and a system that drifts. If you're building a self-improving system, check your statistics. Read the gate code. Verify the p-value threshold.

Key Takeaways

1. Don't let the LLM judge its own edits. Build a deterministic gate. Code, not prompts. p-values, not vibes.
2. p < 0.05, not p < 0.95. Check your confidence logic. It's the difference between signal and noise.
3. A rejection is a valid outcome. The gate saying "no" 15 times in a row means it's working. The evidence bar is being enforced.
4. Inspect the gate checks, not just the decision. The decision says "reject." The checks tell you why — and which check failed matters.
5. Log raw LLM traffic. Every bug I found was caught by reading request/response pairs, not summary output. One environment variable, one JSONL file.
6. Local 4B is enough. 4,150 calls, 37 minutes, $0.00. No cloud. No API keys. No cost.

Try It

pip install agent-self-edit

The full 15-iteration field test — every A/B result, every gate decision, every per-task output — is open source:

• GitHub: github.com/deghosal-2026/agent-self-edit
• PyPI: pypi.org/project/agent-self-edit
• Field test report: final-field-test-report.md
• Per-iteration A/B artifacts: results/omlx/qwen3.5-4b-4bit/
• All 31 issues fixed: GitHub issues (closed)
• Learnings: learnings.md

What Do You Think?

The gate works. The loop is mechanically sound. The current analyzer doesn't produce edits strong enough to pass — which is a valid finding, not a bug.

But here's the question I keep thinking about: if the gate never promotes, is the system actually "self-improving"?

For v0.2.0, I'm building a rejection-aware analyzer that learns from gate decisions, cumulative evidence across iterations, and larger A/B task sets for more statistical power.

But I'm curious — what would you do differently? Would you lower the significance threshold? Add more tasks? Build a smarter analyzer? Drop your thoughts in the comments.

Your support copilot started as a weekend prototype: one model, one provider, one API key in an env var. Then it became production, and you inherited its weaknesses: the provider's availability is your availability, every retry is your code, spend is a mystery until the invoice, and agents bolt tool-use on however they can. This post migrates that stack onto an enterprise AI gateway — and actually runs the migration, with the raw outputs to show for it.

The gateway here is Bifrost, an open-source (github.com/maximhq/bifrost, Apache-2.0) gateway written in Go, presenting a single OpenAI-compatible API across 23+ providers. I rebuilt the legacy stack locally — mock providers with deterministic latency, a realistic traffic pattern — and moved it behind Bifrost step by step.

The legacy baseline, measured

A support copilot's traffic has a shape: mostly repeated FAQ-style questions, plus one-off queries. My traffic mix: 60 requests — 40 FAQ prompts (8 distinct questions asked 5 times each) plus 20 one-offs. Mock provider latency: 200 ms.

Run 1, direct to the provider:

legacy: 60 ok / 0 fail, 9,335 tokens billed, ~201 ms avg latency

Run 2 — the provider dies mid-sweep, as providers do:

legacy + failure: 34 ok / 26 fail

26 requests — 43% — failed outright. Nothing in the legacy stack retries across providers because nothing can: the app speaks one provider's API. And availability is only the loudest problem. The quieter ones: every team's service embeds the same shared key (one key's quota is everyone's ceiling, and revoking it breaks everyone at once), there is no per-team attribution of spend, and the only way to cut cost on repeated questions is to build caching yourself — request normalization, hash keys, TTLs, invalidation — inside the application. That is the whole argument for a gateway in one row of output.

The migration, step by step

Seven moves, each reversible. Diagrams follow the flow.

1. Deploy the gateway beside the app

docker run -p 8080:8080 maximhq/bifrost

One config file wires your existing provider and key; the app keeps working untouched. Mine, reduced to the bones:

{
  "providers": {
    "openai": {
      "keys": [{ "name": "primary", "value": "mock-key", "weight": 1.0,
                 "models": ["support-chat"] }],
      "network_config": { "base_url": "http://provider:9001",
                          "default_request_timeout_in_seconds": 30 }
    }
  }
}

The gateway setup guide covers the web-UI alternative, and there is a Go SDK if you want the gateway embedded rather than adjacent.

2. Point one low-risk client at the gateway

The OpenAI-compatible API means the client change is a base URL — api.openai.com → localhost:8080 — not a rewrite. Every request now flows through a hop you control. Screenshot of the providers page after this step:

3. Add a fallback provider

A second provider config (anthropic in my bench) plus a request-level fallback chain:

{
  "model": "openai/support-chat",
  "messages": [{"role": "user", "content": "hi"}],
  "fallbacks": ["anthropic/support-chat"]
}

Then the proof. Healthy primary:

"routing_info": {"provider": "openai", "key": "primary", "is_fallback": false}

I killed the primary provider's process and re-sent the identical request:

"routing_info": {
  "provider": "anthropic", "key": "backup",
  "is_fallback": true,
  "primary_provider": "openai", "primary_model": "support-chat"
}

The request succeeded on the backup and the response says exactly what happened — is_fallback: true with the failed primary recorded. That audit trail is what you want at 2 a.m.: not just "it kept working," but "it kept working this way." The retries and fallbacks docs cover chained fallbacks and per-provider retry counts.

One honest caveat from my bench: failover on initial connection-refused (provider already dead before the first connect) was inconsistent in my mock setup — it fired reliably when the upstream errored or the connection dropped mid-pool, but a cold connection-refused sometimes returned a 502 instead of failing through. Validate failover against your providers' real failure modes before you trust it in production.

4. Turn on caching

Semantic caching has two modes: exact-match (direct hash, no embeddings needed) and embedding-based similarity. Config for direct mode with a Redis Stack vector store:

"plugins": [{
  "name": "semantic_cache",
  "config": { "dimension": 1,
              "vector_store_namespace": "BifrostBench",
              "default_cache_key": "support-cache",
              "ttl": "5m" }
}]

Two identical requests, one cache key. The second response:

"cache_debug": {
  "cache_hit": true,
  "cache_id": "1cf8a91b-c115-57bf-97a0-fc821dc4de1e",
  "hit_type": "direct",
  "cache_hit_latency": 0
}

Same created timestamp as the first response — it was replayed, not re-fetched. Zero provider call, zero tokens. (Practical note: this needed Redis Stack with the RediSearch module; plain Redis lacks the FT.* commands the index wants.)

5. Issue virtual keys per team

Virtual keys are the governance primitive: per-team keys carrying model allowlists, budgets, and rate limits. Declared in config for the support team:

"governance": {
  "virtual_keys": [{
    "id": "vk-support-team",
    "value": "sk-bf-support-team",
    "provider_configs": [{
      "provider": "openai",
      "allowed_models": ["support-chat"], "key_ids": ["*"]
    }]
  }]
}

The allowed request routes normally. A request for premium-model with the same key:

"Model 'premium-model' is not allowed for this virtual key"

Denied at the gateway before any provider saw it. The same key machinery carries budgets and rate limits — the mechanism that ends the "who spent $800 on Opus last night" incident review. Screenshot of the key in the governance UI:

6. Wire observability and agent tooling

Bifrost exports Prometheus metrics natively and logs every request with routing context — provider chosen, fallback index, cache behavior, token counts, latency split into gateway vs upstream time. That last distinction matters: when a provider slows down, you see upstream_latency grow while gateway overhead stays flat, so you know whose pager to page. See the observability docs. And for agent traffic, MCP is a first-class surface: the gateway brokers tool calls with explicit execution (no auto-execution unless you opt in).

I hand-rolled a minimal MCP server (one tool, JSON-RPC over HTTP) and registered it as a client. The client list reported state: healthy with get_time discovered. Execution went through the gateway explicitly:

POST /v1/mcp/tool/execute  {"function":{"name":"benchtools-get_time","arguments":"{}"}}
→ {"role":"tool","content":"2026-08-27T07:48:21Z"}

Two security properties surfaced unprompted: tool names are namespaced per client (benchtools-get_time) to prevent collisions between servers, and execution without permission fails closed ("tool is not available or not permitted"). Agent traffic gets the same governance as chat traffic.

7. Cut over with an audit trail

Remaining clients migrate one at a time — each is a base-URL change with the gateway's request log as your audit trail. The Logs view after a few requests:

The measured payoff

Same 60-request traffic, now through Bifrost with the cache on and the fallback wired:

via Bifrost: 60 ok / 0 fail, 32 cache hits, 4,363 tokens billed

At an illustrative $0.0025 per 1K tokens:

The savings came entirely from replayed cache hits — no provider call, no tokens. On a support workload that repeats questions daily, that ratio compounds. The availability delta speaks for itself: 0 failures through a provider kill, against 26 in the legacy run.

The bottom line

Migrating to an enterprise AI gateway is not a rewrite. It is a sequence of small, reversible moves — deploy beside, point one client, add fallback, enable cache, issue keys, wire observability, cut over. Measured on the rebuilt stack: 53% cost reduction on cacheable traffic, zero failed requests through a provider kill, and governance the legacy stack never had. The migration risk is low; the legacy risk is already on your pager.

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Code: github.com/deghosal-2026/agent-self-edit

I built an AI that rewrites its own prompts. It looked like it worked. It didn't.

Over a single session, I found and fixed 31 issues. Nine of them were fundamental — each one made the system look like it was working when it wasn't. The most dangerous was this: the promotion gate was letting noise through as "improvement." It checked p < 0.95 instead of p < 0.05. Almost everything passed. A "promotion" at p=0.1 had a 10% chance of being random noise. Two lines of code separated a system that learns from a system that drifts.

But that wasn't the only one. The A/B test "passed" because it compared a prompt against itself. The scoring "passed" because it accepted any non-empty response. The Docker test "passed" because it skipped the hard parts. The failure traces were fabricated. The gate received the wrong prompt. The CLI talked to a mock instead of a real LLM. The config silently ignored the endpoint. And the field test runner was measuring the wrong thing entirely.

The most dangerous bugs aren't the ones that crash. They're the ones that produce output that looks correct. The system always produces something — the question is whether that something is real.

Here's every bug, how it hid, how I caught it, and what it taught me about building systems on top of LLMs.

Bug 1: The Gate Was Letting Noise Through as "Improvement"

This was the most insidious bug. The gate promoted an edit. Accuracy jumped from 20% to 40%. I celebrated.

Then I looked at the code.

# What was written:
passed = p < confidence_level  # p < 0.95

# What it should have been:
alpha = 1 - confidence_level    # 0.05
passed = p < alpha              # p < 0.05

The gate was checking p < 0.95 instead of p < 0.05. Almost everything passed. A p-value of 0.9 would pass. Even 0.5 would pass. The "promotion" at p=0.1 had a 10% chance of being random noise.

Standard hypothesis testing requires p < alpha, where alpha = 1 - confidence_level. With confidence_level = 0.95, alpha = 0.05. The gate should have been checking p < 0.05. It was checking p < 0.95.

After the fix, the same edit produced p=0.23. The gate rejected it. Correctly. There was a 23% chance the improvement was noise — more than the 5% threshold the gate requires.

Two lines of code. That's the difference between a system that learns and a system that drifts.

The lesson: Check your statistics. p < 0.95 is not p < 0.05. The confidence_level is not the p-value threshold — alpha is. Read the code.

Bug 2: The A/B Test Compared a Prompt Against Itself

This was the first one I found, and it set the tone for everything that followed.

The system has an A/B test engine. It runs a candidate prompt against the current prompt on a held-out task set, scores both, and computes whether the candidate is statistically better. The output looked like this:

A/B test: tie (p=1.0000, n=5)
Gate: reject

A tie. p=1.0. That means both prompts produced identical results on all 5 tasks. The gate correctly rejected — a tie means no improvement.

But a tie with p=1.0 is suspicious. It means zero variance. Not one task changed. In practice, even a bad prompt produces some difference. A perfect tie is a red flag.

I dug into the traffic logs. Every single A/B test call — all 10 of them (5 tasks × 2 prompts) — used the exact same prompt text. The system was comparing a prompt against itself.

The root cause was in run.py. The code passed proposal.new_text as prompt_b:

# BUG: passes the edited fragment, not the full prompt
ab_result = run_ab_test(
    registry.current_prompt, proposal.new_text, task_set, llm, scorer, config
)

proposal.new_text is a fragment — like "You are a technical support ticket classifier." It's the edited section, not the full prompt. The A/B engine expected a complete system prompt. The fragment wasn't valid, so the engine fell back to the current prompt for both arms.

The fix was one line:

candidate_prompt = registry.current_prompt.replace(proposal.old_text, proposal.new_text)
ab_result = run_ab_test(
    registry.current_prompt, candidate_prompt, task_set, llm, scorer, config
)

Construct the full candidate prompt by applying the edit to the current prompt. Then test that.

The lesson: When an A/B test produces a perfect tie, check the traffic. A tie means either (a) the edit doesn't change behavior, or (b) you're not actually testing two different prompts. Inspect before you trust the result.

Bug 3: Scoring Marked Everything as "Passed"

The system had a scoring mode called label. It was designed for real traces where the "expected output" is a success label like "no hallucination, no loop, no degradation" — not an actual answer. In label mode, the scorer checked one thing: bool(llm_output.strip()). If the LLM produced any non-empty response, the trace was marked "passed."

This meant every trace — including failure traces — scored 100%. The LLM always writes something. A trace with success: false was marked "passed" because the LLM wrote a paragraph.

100% pass rate is impossible unless the scoring is broken. I saw it and thought: "That can't be right." It wasn't.

I deleted the scoring script entirely. The production scoring system (scorers.py) was correct — it uses ExactMatchScorer, ContainsScorer, and LLMJudgeScorer. The label mode only existed in a standalone eval script that shouldn't have been part of the self-edit loop at all.

The lesson: A 100% pass rate is a red flag, not a success. If your scoring system never fails, it's not testing anything.

Bug 4: Docker Tests Skipped the A/B Test and Gate

"9/9 Docker tests passed." The WBS row was marked done. Everything looked fine.

The Docker integration test ran agent-self-edit run --once --dry-run. The --dry-run flag causes run.py to skip the A/B test and the promotion gate entirely. The test only verified that the system could ingest traces and run the analyzer. It never tested the A/B test or the gate — the two most important components.

This was a smoke test dressed up as an integration test. The WBS acceptance criteria said "A/B test and promotion gate" — but the test skipped both.

I caught it by looking at the test output. There was no "A/B test" line. No "Gate:" line. Just "Analysis complete" and "Loop stopped." The most important stages never ran.

The fix was to remove --dry-run, add a task_set_path to the config so the A/B test could execute, and run the full loop: ingest → analyze → A/B test → gate → reject. After the fix, the Docker test took 62 seconds instead of 5 — because it was actually doing real LLM calls for the A/B test.

The lesson: --dry-run is not an integration test. If your test skips the hardest part, it's a smoke test. Label it accordingly.

Bug 5: Failure Traces Were Fabricated

This was the bug that explained why the A/B test always tied. The failure traces — the data fed to the analyzer — were fabricated.

The _seed_trace_store() function created traces like this:

store.ingest({
    "task_input": task["input"],
    "final_output": "other",  # HARDCODED
    "expected_output": task["expected_output"],
    "success": False,
})

Every trace said the model output "other" when it should have output "technical" or "urgent" or "billing." But the model doesn't output "other" — it outputs "billing," "security," "technical." The analyzer was learning from a failure pattern that didn't exist.

Imagine a doctor trying to diagnose patients, but every patient's chart says "symptom: headache" regardless of what they actually have. The doctor would propose treatments for headaches. None of them would work, because the patients don't have headaches.

That's what was happening. The analyzer saw 10 traces all saying the model output "other." It proposed edits aimed at fixing "other" outputs. But the model never outputs "other" — it outputs "billing" when it should output "technical," or "security" when it should output "urgent." The edit was aimed at the wrong problem.

The fix: run the current prompt against the task set, capture the model's actual outputs, and seed only the real failures. After this fix, the A/B test immediately showed non-zero deltas for the first time. The analyzer started proposing relevant edits.

The lesson: Your feedback loop is only as good as the data you feed it. If the failure traces don't match reality, the system optimizes against fiction. Always seed real data.

Bug 6: The Gate Received the Wrong Prompt

After fixing the confidence check, the gate was still failing — but on a different check: frozen_sections. The error message said "edit.old_text not found in current_prompt."

I assumed the analyzer was modifying frozen content. It wasn't.

The check_all function takes current_prompt as its third argument. The code was passing prompt_b (the edited version) instead of prompt_a (the original):

# BUG: passes the edited prompt
gate_result = check_all(proposal, ab_result, prompt_b, prompt_a, config)

# FIX: pass the original prompt
gate_result = check_all(proposal, ab_result, prompt_a, prompt_a, config)

The frozen_sections check looks for edit.old_text in current_prompt. If current_prompt is prompt_b (the edited version), the old text has already been replaced. It's not there. The check fails — not because the edit modified frozen content, but because the check was looking at the wrong prompt.

This bug was hiding behind the confidence bug. While the confidence check was inverted (p < 0.95), it was always the first check to pass, and the frozen_sections failure never mattered. Once I fixed the confidence check, the frozen_sections bug surfaced.

The lesson: Fixing one bug can reveal another. When you fix the top of the fail-fast stack, the next failure surfaces. Keep going.

Bug 7: run.py Talked to a Mock Instead of a Real LLM

The loop ran and completed. The system was "making LLM calls." The output showed "Analysis complete: 1 proposals."

But run.py:37 had this:

llm = MockProvider(responses="[]")

A debugging leftover. Even with provider: openai in the config, the code hardcoded a MockProvider that returned empty strings. The analyzer was receiving [] as its input — no traces, no failures, nothing to analyze. It still "produced a proposal" — but the proposal was based on nothing.

The loop completed in under a second. Real LLM calls take minutes. That was the red flag.

The lesson: Debugging leftovers are dangerous. If you hardcode a mock during development, replace it before shipping. And if your LLM loop completes instantly, you're not calling an LLM.

Bug 8: Config Silently Ignored the LLM Endpoint

The config file had base_url: http://localhost:8000/v1. The system was "configured" to use the local OMLX server.

But LLMConfig — the dataclass that reads the config — didn't have a base_url field. The YAML's base_url was silently dropped. The OpenAI client used its default endpoint (api.openai.com) instead of the local server.

Every call went to the cloud — or failed silently. The OMLX server never logged any requests because it never received any.

This was a silent config failure. No error, no warning. The field just didn't exist, so the value was ignored. The system ran, made calls, and produced output — just not to the endpoint the user configured.

The lesson: Silent config failures are the worst kind. If a config field doesn't map to a dataclass field, either validate it or log a warning. Don't silently drop it.

Bug 9: The Field Test Runner Was Measuring the Wrong Thing

The field test produced results — accuracy, latency, token counts. The "field test" was "running." The numbers looked reasonable.

But run_traces.py was a generic LLM eval runner. It sent each trace's task_input to the LLM as a standalone chat completion. It didn't call any agent_self_edit modules. It wasn't running the self-edit loop at all — it was measuring the model's raw output on individual tasks.

The script didn't import anything from the package it was supposed to test. It was a standalone OpenAI client — not the self-edit loop. The "field test results" were measuring the model's baseline behavior, not the loop's ability to improve.

I deleted it and built run_improvement_loop.py — a script that calls the internal API directly, runs the full loop (analyze → A/B test → gate → promote/reject), and writes per-iteration artifacts (prompt-a/b, results-a/b, ab-comparison) for every iteration.

The lesson: Make sure your test runner is actually testing the thing you think it's testing. If it doesn't import the package, it's not testing the package.

The Pattern: Read the Traffic, Not the Summary

Every single bug was caught the same way: read the raw LLM traffic, not the summary output.

The summary said "pass." The traffic said "you're comparing a prompt against yourself."

I used AGENT_SELF_EDIT_LLM_LOG — one environment variable that causes every LLM request/response pair to be written to a JSONL file. 4,150 entries across 15 iterations. Every bug was found by reading this file.

The first red flag was always the same: suspicious speed + perfect result.

• A 54-second A/B test with a perfect tie
• A 100% pass rate on real traces
• A loop completing in under a second
• A promotion at p=0.1

When the result looks too clean, it usually is. Real LLM calls have latency. Real A/B tests have variance. Real scoring produces failures. If everything passes, check what "passing" actually means.

What I Learned

1. Log raw LLM traffic. Always. Summary output lies. Request/response pairs don't. One environment variable, one JSONL file, and every bug becomes findable.

2. "Too fast + too clean" is a red flag. Real LLM calls take time and have variance. If your A/B test completes in 54 seconds with a perfect tie, something is wrong. If your scoring never fails, something is wrong. If your loop completes instantly, something is wrong.

3. Every bug looked like success. That's the danger of building on top of LLMs — the system always produces something. The question is whether that something is meaningful. The A/B test produced a "result." The scoring produced a "pass." The gate produced a "promotion." None of them were real.

4. 31 issues in one session. The system went from "looks like it works" to "actually works." The difference was inspecting the data underneath the summary. Two hours of reading traffic logs. No magic, just verification.

Try It

pip install agent-self-edit

• GitHub: github.com/deghosal-2026/agent-self-edit
• All 31 issues fixed: GitHub issues (closed)
• Field test report: final-field-test-report.md
• Learnings: learnings.md
• Per-iteration A/B artifacts: results/omlx/qwen3.5-4b-4bit/

What's the worst "it looked like it was working" bug you've found in an AI system? I'd love to hear about it — drop it in the comment

I published a LoRA fine-tuning run two weeks ago. The headline was that my first test set had lied to me: on data I generated myself the fine-tune scored 100% and prompting scored 94%, so fine-tuning looked pointless. On a test set rebuilt from real public corpora the fine-tune scored 95% and prompting collapsed to 66%.

A reader called Max Quimby left a comment that reframed the whole thing:

Five points and twenty-eight points. Your fine-tune lost the first moving to the rebuilt set, prompting lost the second, and the distance between those two drops tells you more than either one does. It says the original set was differentially easy for the prompted model rather than uniformly easy for both.

He then named a mechanism. The few-shot examples are part of the prompted method's input, so any kinship between the example pool and the test set inflates one arm and not the other. And he closed with a rule: when two methods move by very different amounts after a test-set swap, suspect the set before you suspect the methods.

I spent an afternoon checking it. He was right that my table was misleading, wrong about which arm was inflated, and the direct test of his mechanism produced a third answer that neither of us predicted.

The error in my table

Here is the comparison as I published it.

Read that as three methods measured on two sets and his conclusion follows immediately. But one row is not what it appears to be.

The few-shot and zero-shot rows are the same method twice: the same base model, the same prompt, the same six hand-written examples, evaluated on two different sets. Clean.

The fine-tune row is two different adapters. The 100% is an adapter trained on v1 data scoring on the v1 set. The 95% is a different adapter, trained on v2 data, scoring on the v2 set. I changed the training data and the test data at the same time, put both numbers in one row, and labelled the difference a drop.

That five was never a measure of set difficulty. Retraining on harder data was quietly absorbing the loss, and my table gave a reader no way to see it.

Holding the artifact fixed

The fix is to evaluate one artifact against both sets. The v1 adapter still exists in the repo, so this is one command:

.venv/bin/python evaluate.py --mode lora \
  --adapter-path ./adapters_v1_synthetic \
  --limit 400 --tag _v1adapter_realset

Thirty-three points. The same weights, the same code, the same prompt shape. Only the test set changed.

So the corrected picture, every row now a single method or artifact measured twice:

His instinct was right and his conclusion was backwards. The differential is real, but the fine-tune is the most inflated arm, not the least. And the ordering is the part worth keeping:

The more a method had been fitted to the v1 distribution, the more it lost when that distribution went away. Trained on v1 data: 33 points. Merely prompted with v1-shaped examples: 28. Never shown v1 at all: 22.

That monotonicity is the cleanest statement of the whole affair. It also rescues the 95%: retraining the same recipe on representative data took it from 67 back to 95, which makes that number a statement about the data, not about the method.

Testing his mechanism directly

The 22-point floor is what the set costs any model. The 6 points above it that few-shot paid are the candidate for his example-pool kinship, and my examples make the case look strong. Here are three of the six, next to the v1 generator templates they instantiate:

Same templates, different instances. Git history confirms I never touched them between the two evaluations, so the prompted arm carried v1-shaped hints into a v1-shaped test set, and then carried the same hints into a set they no longer matched.

If that kinship is what bought the 6 points, swapping the pool for one drawn from the v2 training data should recover them. So I built a second set of six: same count, same 3 positive / 3 negative balance, same three PII types, but every example lifted from data/train.jsonl and verified absent from the test set by exact string match.

One and a half points. Both arms ran on the identical test set, so this is a paired comparison and deserves a paired test rather than a glance at the margin of error. McNemar's exact test on the 400 prediction pairs: 21 items only the v1 pool got right, 27 only the v2 pool got right, p = 0.47.

Nothing. Matching the example pool's provenance to the test set recovers no measurable accuracy.

Then where did the 6 points go?

Back to the v1 set, where the examples were worth something, and compare them on the v2 set with the same paired test:

On the real set, adding six examples to the prompt does exactly nothing: 53 items only zero-shot got right, 54 only few-shot got right. A coin flip.

So the six examples were worth 6 points on the set that shared their provenance and 0 points on the set that did not. That is his differential inflation, confirmed. But his implied remedy, using examples of matching provenance, does not recover the loss, because the deficit was never really about provenance. Six examples of any origin cannot express this task's rules. On the easy v1 set they were enough. On the real one, no pool of six helps, so there is nothing for a better-chosen six to win back.

The per-type breakdown shows what in-domain examples actually do:

They redistribute rather than add. The types the examples demonstrate get better, the types they omit get worse, and the total barely moves. Few-shot prompting teaches what it shows and distracts from everything else.

What survives all of this

The fine-tune's advantage on the real set is not in doubt. Same paired test, fine-tune against the better of the two prompted arms: 4 items only few-shot got right, 113 only the fine-tune got right, p < 0.0001. A 113-to-4 split is not a benchmark artifact.

The rules I would keep

Report same-artifact numbers, or say plainly that you are not. My five-point drop was two different adapters wearing one row. Nobody could have caught that from the article, which is my fault and not the reader's.

Keep an arm with no exposure to the training distribution. Zero-shot was the only reason I could decompose 28 into 22 plus 6. Without a control that has never seen your data, a drop is just a drop.

A differential drop points at the set, but does not tell you which arm to blame. Rank your methods by how much they were fitted to the old distribution and check whether the drops follow that order. Here they did, exactly.

Use paired tests when arms share a test set. Two of the three differences that looked meaningful at a glance (1.5 points, 0.2 points) are indistinguishable from noise under McNemar, while the one that mattered came back at p < 0.0001. Margins of error on independent proportions would have let me argue for all three.

And the one that stings: a perfect score gets audited, a merely good one gets a slide. Max's point, and he is right. My 100% is what made me rebuild the set. If it had been 0.94 I would have shipped it, and if it had been 0.94 nobody would ever have found the two-adapters error in my table either.

Reproduce it

Everything is in the repository, including both example pools, the new results JSONs and the paired predictions the McNemar tests read:

git clone https://github.com/jguillaumesio/lora-pii-detection-mlx
cd lora-pii-detection-mlx
python3 -m venv .venv && .venv/bin/pip install mlx-lm datasets
.venv/bin/python build_dataset.py
.venv/bin/python evaluate.py --mode few-shot --few-shot-set v2 --limit 400 --tag _v2examples

Thanks to Max Quimby for the comment. It cost me an afternoon and a correction, which is the best possible outcome for a comment.

Originally published on jguillaumesio.com. Follow-up to My fine-tuned model scored 100%. The benchmark was lying.

This is a companion to the PlannerCritic series. Article 5 was about what happened when I tried to break my own engine. This one is about the three seams I know it can't close — and why I wrote them down before anyone else had to.

An open-source safety tool that claims to be bulletproof is less trustworthy than one that publishes its own holes. Here are mine.

v0.2.3 Update (Aug 29): The failure-mode register grew to 15+ rows with the addition of F-20 (#296) — documenting the deterministic-corruption blind spot where redaction/transit layers silently corrupt data. The transit-integrity check (verify_transit_integrity) now validates that numeric JSON fields survive redaction. See v0.2.3 release notes.

The Temptation I Resisted

I built an agent planning engine that blocked 11 of 11 adversarial goals, 35 of 35 SWE-bench-derived flawed variants, and 21 generated injection traps. The Article 5 headline writes itself: "It Didn't Work. Here's Why."

The temptation is to stop there. Eleven-for-eleven. Architecture, not the prompt, is what makes it safe. Done.

But three of the smartest comments on that article weren't congratulating me. They were naming the seams I'd hand-waved past. And the most useful thing I can do with this piece is publish those seams clearly, link the issues I opened for each, and explain why publishing them is the credibility move — not the weakness.

The Three Open Seams

Seam 1: Indirect injection through tool outputs

Every injection test I ran put the payload in the initial goal text. That's the easy case. The deterministic gates don't read goal text, so the payload never reaches gate logic. That's why 11/11 held.

The hard case is a payload that arrives after the goal was audited — a fetched webpage, a compromised database record, an untrusted API response. The critic evaluates the plan the planner produces; it does not re-audit every tool result the planner consumed. If a tool output contains a well-crafted payload, the planner may incorporate it into a sub-plan that the critic's initial check never sees.

A commenter (@seasonkoh) put the fix better than I had:

"Re-auditing every tool result with another LLM still leaves data and instructions entangled. A stronger contract is for tools to return typed data plus provenance, while deterministic policy decides whether that source may influence a particular state transition. Text fetched from a product page may inform discovery, but it should never be able to alter the payee, amount, approval requirements, or destination authority."

That's the v0.3.0 work. It's tracked as #249: typed tool-result provenance plus capability-scoped state transitions, so untrusted sources can inform discovery but can never acquire the ability to alter high-stakes fields. The critical path stays deterministic — the gate inspects the provenance tag, not the content, so it can't be prompt-injected.

I can't claim this works yet. I can claim it's the right shape of fix, and I can point at the issue.

Seam 2: Well-formed malicious plans defeat structural checks

The deterministic gates check structural completeness, not semantic intent. An attacker who crafts a plan that includes a dummy rollback and a dummy verification step can satisfy the linter while carrying malicious actions. The gates pass what looks structurally sound.

This is the floor of deterministic authority I wrote about in the companion piece: code is authoritative everywhere it can be, but "can be" stops at structure. A plan that is structurally perfect and semantically wrong is invisible to every gate I've shipped.

The honest answer is that this seam is only partially closeable. The critic can catch semantic malice — sometimes — but the critic is an LLM, which brings us to seam 3. The partial mitigation is the requirement-traceability gate (#255): every plan step must trace back to a bound acceptance criterion, so a plan that satisfies every gate but delivers the wrong user story gets caught. That closes drift; it does not close malice. I'm being specific about which is which.

Seam 3: The critic is itself an LLM

Relying on an LLM to catch adversarial intent relies entirely on the critic model's semantic comprehension. Sophisticated jailbreaks — multi-step logical traps, encoded payloads, social-engineering phrasing framed as legitimate edge-case testing — can blind-spot even an adversarial system prompt.

This is exactly why the critical path is deterministic and the critic is downgraded to warning outside eligible families. The architecture works because it does not bet the security contract on the LLM being clever. But the semantic layer alone is not sufficient, and a well-formed malicious plan is the case where both the structural gates and the semantic critic can fail together.

The uncomfortable part is that I can measure one direction of this and not the other. The v0.2.1 boundary evaluator showed label_flip_rate = 1.0 and underclaim_approvals = 0 — the critic is maximally non-deterministic yet never lets a seeded defect through. I wrote a whole piece about why that paradox holds (My LLM Critic Flip-Flops on Every Run); the short version is that the deterministic gates own the under-claim direction. But "never under-claims a seeded defect" is not "never under-claims a novel jailbreak." I can measure the first. I can't measure the second — and a jailbreak is exactly the case where the critic being an LLM is the seam.

Why Publishing This Is the Credibility Move

There's a pattern I noticed in how people read Article 5. The readers who took the safety story seriously were the ones who read the "Honest Limitation" section first. The 11/11 number alone read like marketing. The 11/11 number next to three named seams read like engineering.

I think there's a reason. A safety tool that claims to be complete is making a claim its author cannot fully verify — and a careful reader knows that. A safety tool that publishes its own holes is making a smaller, checkable claim: "here's what I catch, here's what I don't, here's what I'm doing about the gap." The second is auditable. The first is an assertion.

The failure-mode register has 15 rows (plus F-20 added in v0.2.3). Each row is a known way the engine can fail or be wrong, with the mitigation named. v0.3.0 is scoped to add F-15 (indirect injection via tool output) and F-16 (critic satisfaction over/under-endorsement) — proposed in #249 and #254, not yet shipped. The register is the artifact version of this article — a maintained list of the engine's known limits, in the repo, versioned with the code.

I'd rather a reader open that file and find 14 honest rows than find a README that says "safe by design" and nothing else.

What the Comments Taught Me About Writing Limitations

Three things, from the Article 5 thread:

1. Name the seam precisely, or a reader will name it for you. I wrote "indirect injection is a different threat surface." @seasonkoh came back with the exact capability-scoped state-transition contract that closes it. If I'd written the seam precisely the first time, the fix would already be half-specified. Vague limitations invite vague comments.
2. A limitation with an issue link is more credible than a limitation with a promise. "Deferred to v0.3.0" is a promise. #249 is a tracked, scoped, acceptance-criteria'd commitment. Readers can watch the issue. Promises disappear; issues either close or don't.
3. The limitation section is where the expert readers show up. The congratulatory comments were on the 11/11 number. The substantive comments — the ones that changed the roadmap — were all on the limitations. If you want to find the people who actually understand your domain, write the limitations carefully and watch who responds.

What I'm Not Claiming

I'm not claiming that publishing limitations makes a tool safe. A tool with a great limitations section and no working safety layer is just well-documented danger. The limitations section is credible only because the 11/11, the 35/35, and the deterministic gates are real and in the repo. The limitations are the honesty layer on top of a real floor.

I'm also not claiming the three seams above are the complete list. They're the three I know about. The fourth seam — the one I haven't noticed yet — is the one a future commenter will name, and I'd rather have a register that's ready to accept it than a README that implies the list is closed.

Questions for Anyone Shipping a Safety-Critical Tool

• Which limitation of your tool are you most afraid to write down? Mine was "the critic is an LLM and can be jailbroken." It felt like it undermined the whole pitch. Writing it down didn't reduce adoption; it increased trust. I'm curious whether that generalizes.
• Do you maintain a failure-mode register, or do your limitations live in a README paragraph? The register changed how I think about the engine — it forced me to name each failure with a mitigation, which exposed the ones with no mitigation. A paragraph hides those.
• When a commenter names a seam you missed, do you argue or do you file an issue? I almost argued one of these. Filing the issue was faster and produced a better artifact. I'm trying to make that the default.

I don't have a clean ending for this one. The seams are open. The issues are filed. The register is maintained. That's the state of the engine, and I'd rather say that plainly than wrap it in a bow.

Series: Article 1 · Article 2 · Article 3 · Article 4 · Article 5

Links:

• Repo: github.com/deghosal-2026/planner-critic-engine
• The failure-mode register (15+ rows, maintained): docs/reference/failure-modes.md
• The open-seam issues: #249 (indirect injection) · #255 (requirement traceability) · #254 (critic satisfaction)
• v0.2.3 addition: #296 — F-20 deterministic-corruption blind spot (new failure-mode row)
• The comment threads that shaped this piece: Article 5 comments

A practical way to evaluate computer vision models before you commit

If you work on a vision project, model choice is rarely just about the biggest name on the leaderboard. You usually need to answer a more specific question:

• Which model handles my prompt or image style well?
• Which one is better for the task I actually need?
• Which option should I benchmark more deeply before I build around it?

Roboflow Playground is useful because it turns those questions into a workflow. You can start trying, comparing, and evaluating supported vision models for free, without having to build the whole evaluation stack yourself first.

What Playground gives you

At a high level, Playground is a place to experiment with 134 models from providers like Google, OpenAI, Anthropic, Meta, and Qwen.

That matters because model selection often starts broad and gets narrow quickly. A directory with this many options makes it easier to move from “What should I use?” to “What performs best for my case?”

The basic entry point is simple:

• Open a prompt
• Run it across supported models
• Inspect the results

That may sound lightweight, but for builders it is often the fastest way to surface differences in behavior before you invest time in deeper testing.

A quick comparison workflow

A useful way to think about Playground is as a first-pass comparison layer.

Instead of guessing which model is strongest for a vision use case, you can put a prompt into the system and review how different models respond. For object detection, that can help you see where results differ in interpretation or coverage.

The source example points to a comparison flow for object detection models. The important part is not a specific prompt recipe, but the process:

1. Submit a prompt
2. Review model outputs side by side
3. Decide which candidates deserve more evaluation

That workflow is especially helpful when you are still narrowing down a model shortlist. It reduces the risk of starting with a favorite model and only later discovering that another option is a better fit.

When you need ground truth, use Vision Evals

Playground is good for experimentation, but experimentation is not the same thing as evaluation against a standard.

For that, Roboflow Vision Evals evaluates 34 frontier vision-language models across six standardized ground-truth tasks. The source specifically calls out object detection and counting among those tasks.

This distinction is important for developers:

• Playground helps you explore and compare
• Vision Evals helps you measure against ground truth

That separation gives you a more disciplined workflow. You can use Playground to narrow the field, then use Vision Evals when you need a standardized assessment of model behavior on known tasks.

In practice, that means you are not relying only on intuition or ad hoc spot checks. You can move from qualitative exploration into a more structured evaluation path.

Side-by-side technical comparison with Compare

There are cases where you already know the models you want to test head-to-head.

That is where the Compare tool comes in. When you need to evaluate specific model matchups directly, Compare generates a technical side-by-side breakdown.

For builders, that is a different kind of decision support than a broad model directory. Compare is more focused:

• You pick the matchup
• You inspect the technical breakdown
• You use that to make a sharper decision

This is useful when the question is no longer “Which model should I start with?” and has become “Which of these two or three candidates is better for this implementation?”

That distinction matters because different evaluation stages call for different tools. A broad playground is for discovery. A comparison tool is for targeted decisions.

Why the model directory matters

The directory is not just a list for browsing. It also helps explain the shape of the model ecosystem inside Playground.

Among the 130+ models, there are 49 specialized single-task models. The source names YOLO26 and RF-DETR as examples of models built specifically for high frame rates and production accuracy.

That tells you something useful about how to navigate the directory:

• Some models are general-purpose
• Some are specialized for a single task
• Some are designed with production constraints in mind

For developers, that means the right choice depends on the deployment target as much as the benchmark. A model that looks attractive in a general demo may not be the best fit if your priority is high frame rate or production accuracy.

So the directory becomes a practical filter, not just a catalog.

A builder-friendly way to use all three layers

If you want a clean process, the three pieces fit together well:

1. Use Playground for fast exploration

Start by trying supported models for free. This is the quickest way to get a feel for how different systems respond to the same prompt.

2. Use Compare for direct matchups

When you already have a shortlist, compare models side by side and focus on the technical differences that matter for your implementation.

3. Use Vision Evals for standardized benchmarking

When you need a ground-truth view, use Vision Evals and its six standardized tasks to evaluate frontier vision-language models more rigorously.

That sequence keeps the evaluation process organized. You do not jump straight into a full benchmarking effort before you know which models are worth that time.

Tradeoffs to keep in mind

This kind of workflow is useful, but it helps to be clear about what each tool is for.

Playground is not the same as a benchmark suite. It is excellent for trying models and comparing outputs, but it is not a replacement for ground-truth evaluation.

Compare is not meant to solve every possible selection question. It is best when you already have a specific matchup in mind.

Vision Evals gives you standardized tasks, but that does not eliminate the need to choose the right model class for your use case. A specialized single-task model may still be more appropriate than a general model, depending on your goals.

So the practical takeaway is not “pick the highest-performing model everywhere.” It is “match the tool to the stage of evaluation.”

Bottom line

If you are selecting vision models, Roboflow Playground gives you a simple entry point: try models for free, compare responses, and move into deeper evaluation when needed.

The useful part for builders is the structure around it:

• Playground for discovery
• Compare for head-to-head technical review
• Vision Evals for standardized ground-truth benchmarking
• The directory for finding both general and specialized models

That makes the platform less like a demo page and more like a model selection workflow you can actually use while building.

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

There is a simple question at the heart of modern LLM training:

Where did the data come from?

If you train a model on answers generated by itself, you are doing something fundamentally different from training it on answers generated by an older model, a human, or a static dataset.

That difference is called on-policy vs. off-policy learning.

It sounds like reinforcement-learning terminology from the 1990s. It is. But it has become one of the most important ideas for understanding what is happening in modern LLM post-training: RLHF, PPO, rejection sampling, preference optimization, self-play, synthetic data, verifiable rewards, and increasingly, models that generate their own training trajectories.

The interesting part is that the distinction is not really about "online" versus "offline" data.

It is about a much more precise question:

Is the model learning from behavior produced by the policy it is currently trying to improve?

Once you see that distinction, a surprising number of LLM training techniques fall into place.

1. Imagine teaching a programmer

Suppose you are training an LLM to write Python.

You give it:

Write a function that returns the longest increasing subsequence.

The model produces:

def lis(a):
    ...

An evaluator gives it a score.

Now imagine two training systems.

System A: on-policy

The current model generates the solution.

You evaluate that solution.

You update the model based on the result.

Then the new model generates another solution.

The loop is:

current model
     |
     v
generate solution
     |
     v
evaluate solution
     |
     v
update model
     |
     v
new model
     |
     +----> generate again

The data continuously moves with the model.

System B: off-policy

Instead, you have 10 million solutions sitting in a dataset.

Some were written by humans.

Some came from GPT-4.

Some came from an older checkpoint.

Some came from a specialized coding model.

You train your current model on those examples.

The behavior that produced the data is not necessarily the behavior of the model you are currently training.

That is off-policy learning.

The distinction matters because the model's mistakes determine what it gets to learn from.

If the current model is terrible at recursion, an on-policy system will naturally generate lots of terrible recursive solutions. An off-policy dataset might contain excellent recursive solutions that the current model would never have generated.

That sounds like an obvious advantage for off-policy learning.

And sometimes it is.

But there is a catch.

2. The RL concept is older than LLMs

The terminology comes from reinforcement learning.

Chris Watkins' Q-learning work in the late 1980s and early 1990s gave one of the classic examples of off-policy learning. Q-learning can learn about an optimal policy while the agent is actually behaving according to another policy.

The conceptual trick is powerful:

The policy generating the experience does not have to be the policy being learned.

Contrast that with policy-gradient methods, where you typically generate trajectories using the current policy and then use those trajectories to estimate how changing that policy would affect expected reward.

This distinction became especially important as reinforcement learning moved from toy environments to expensive neural-network systems.

By the time John Schulman and colleagues introduced PPO in 2017, the engineering problem was familiar:

Generate experience with a policy, then update the policy without moving it so far that the experience becomes useless.

PPO explicitly alternates between collecting samples from the current policy and optimizing on those samples. It permits multiple optimization epochs over the collected data while constraining how far the updated policy moves from the policy that generated the samples.

That constraint is not cosmetic.

It is the central operational problem.

Suppose your model generated:

"The answer is 42."

You update the model ten times using that batch.

After those ten updates, your model may have changed substantially.

The sample was generated by policy P_old.

You are now optimizing policy P_new.

The more different P_new becomes from P_old, the less directly the old sample tells you about P_new.

That is the basic tension behind on-policy RL.

3. Why LLMs make this unusually expensive

In a game like Atari, generating another million actions may be relatively cheap.

For an LLM, generating another million trajectories can mean running a giant transformer for billions of tokens.

And the reward might require an expensive evaluator.

Consider a deliberately simple calculation.

Suppose:

• model inference costs $2 per million generated tokens
• you generate 100 million tokens per training iteration
• you perform 100 iterations

Generation alone costs roughly:

100M tokens x 100
= 10B generated tokens

10B / 1M x $2
= $20,000

That is a toy number, but the scaling relationship is real.

At frontier-model scale, the expensive resource is often not the gradient update.

It is producing useful experience.

This is why off-policy learning is so attractive.

If you have already paid to generate 10 billion tokens, you would very much like to reuse them.

And you would like to reuse them more than once.

That is the economic argument for off-policy training:

Experience is an asset. Don't throw it away after one gradient update.

This is also why replay buffers became such an important idea in classical RL.

But LLMs introduce an even more interesting problem: the "environment" is often other models, humans, tools, or verifiers rather than a game simulator.

4. InstructGPT shows the transition

One of the clearest historical examples is OpenAI's 2022 InstructGPT work.

The basic pipeline was:

GPT-3
  |
  v
human demonstrations
  |
  v
SFT model
  |
  v
generate multiple answers
  |
  v
human preference rankings
  |
  v
reward model
  |
  v
PPO
  |
  v
InstructGPT

The important part for our discussion is the final stage.

The reward model evaluates outputs generated by the policy, and PPO uses those interactions to improve the policy.

That is much closer to the classic on-policy RL loop than ordinary supervised fine-tuning.

And it produced a remarkable result.

OpenAI reported that evaluators preferred outputs from the 1.3B-parameter InstructGPT model over the 175B-parameter GPT-3 model on their instruction-following evaluation.

In other words, changing how the model learned from experience could matter more than making the model roughly 100x larger.

This was an important moment in LLM history because it demonstrated that post-training was not merely polishing a pretrained model.

It could substantially change what the model optimized for.

And PPO's on-policy nature was part of that story.

5. So why not always use on-policy learning?

Because on-policy learning has a brutal property:

The data expires quickly.

Imagine training version 1 of your model.

It generates:

Prompt: Prove that sqrt(2) is irrational.

Answer: ...
Reward: 0.8

You update the model.

Now you have version 2.

Why should version 2 be restricted to learning from version 1's trajectories?

It might be able to solve the problem much better.

Conversely, version 1 may have generated a brilliant proof that version 2 will almost never discover again.

This creates a strange asymmetry.

On-policy learning gives you highly relevant data:

data ~= current behavior

but potentially wastes enormous amounts of useful historical data.

Off-policy learning gives you reusable historical data:

data != necessarily current behavior

but now you have to deal with the distribution mismatch.

At a high level:

On-policy:
    fresh + relevant
    expensive + disposable

Off-policy:
    reusable + diverse
    potentially stale + mismatched

This is one reason modern LLM training increasingly looks like a hybrid system rather than a pure on-policy or pure off-policy system.

You want the freshness of on-policy data and the economics of off-policy data.

6. The math: what exactly goes wrong?

Let the model be a policy:

pi_theta(y | x)

This means:

Given prompt x, what probability does the model with parameters theta assign to answer y?

Suppose the model gets reward R(x, y).

The objective is conceptually:

J(theta) = E[R(x, y)]

where y is sampled from the model itself.

For a policy-gradient method, a basic gradient estimator looks like:

grad J(theta)
    ~= E[ grad log pi_theta(y | x) * R ]

The important thing is that y was sampled from:

pi_theta

Now suppose we have old data generated by another policy:

pi_old

but we want to optimize:

pi_theta

The expectation is now being taken under the wrong distribution.

One classical solution is importance sampling.

Conceptually:

E_pi_theta[f(y)]

    =
E_pi_old[
    pi_theta(y|x) / pi_old(y|x) * f(y)
]

The ratio

pi_theta(y|x) / pi_old(y|x)

corrects for the fact that the data came from the old policy.

This looks elegant.

It can also become horrible.

Suppose a sequence has probability:

pi_old(y|x) = 1e-6

under the old model but

pi_theta(y|x) = 1e-3

under the new model.

The importance weight is:

1e-3 / 1e-6 = 1000

One example can suddenly have 1,000 times the influence of another.

With long autoregressive sequences, probability ratios can become extremely volatile because token-level ratios multiply across the sequence.

That creates a classic RL engineering problem:

How much old data can we safely reuse before the correction becomes statistically ugly?

PPO takes a pragmatic route.

Instead of allowing arbitrary policy movement, it clips the probability ratio:

r(theta) =
    pi_theta(y|x) / pi_old(y|x)

and uses an objective that effectively says:

Improve the policy, but don't benefit too much from moving far away from the policy that generated this experience.

This is one reason PPO became so popular: it turns a theoretically nasty distribution-shift problem into something engineers can actually operate.

7. The LLM world is now blurring the boundary

Here is where things get interesting.

People often casually say:

"DPO is off-policy RL."

That is useful shorthand, but technically it is better to be precise.

DPO, introduced by Rafael Rafailov and colleagues in 2023, takes preference pairs such as:

prompt
chosen answer
rejected answer

and directly optimizes the language model using those preferences.

There is no PPO rollout loop.

No reward-model inference is required during optimization.

No requirement that the current model generate every training example.

So operationally it looks much more like learning from a fixed preference dataset.

This is one reason DPO was attractive: it removed much of the machinery associated with RLHF while retaining a principled connection to the underlying reward-maximization problem.

But now consider what happens if we repeatedly generate preference data using the current model:

Model v1
   |
   v
generate candidates
   |
   v
judge / verifier
   |
   v
preference dataset
   |
   v
train Model v2
   |
   v
generate candidates
   |
   v
judge / verifier
   |
   v
train Model v3

You have built something that is neither simply "offline training" nor simply classical on-policy PPO.

You have a data-generation loop whose policy evolves over time.

That distinction is becoming increasingly important for reasoning models.

A model might generate thousands of candidate proofs, programs, mathematical solutions, tool-use trajectories, or chains of actions. A verifier selects successful ones. Those successful trajectories become training data.

The resulting system has an economic structure very different from ordinary SFT:

compute
  -> generate attempts
  -> evaluate attempts
  -> retain valuable experience
  -> train
  -> generate better attempts
  -> repeat

The bottleneck can move from gradient computation to experience generation and evaluation.

That is the deeper reason on-policy versus off-policy matters for LLM developers.

It is not merely a taxonomy of RL algorithms.

It is a question about how efficiently you turn inference compute into learning signal.

8. The practical rule for LLM engineers

A useful mental model is:

The most important engineering questions therefore become:

1. Who generated this data?

Not just "is it synthetic?"

A dataset generated by your current checkpoint is very different from one generated by a model six generations ago.

2. How far away is the behavior policy?

If your current model assigns very different probabilities to the training trajectories, you have distribution shift.

3. How expensive is experience?

If generation is cheap, throwing data away may be fine.

If generation requires a giant reasoning model plus a verifier, replay becomes much more attractive.

4. How reliable is the reward?

On-policy learning can repeatedly exploit a flawed reward function.

The model gets better at finding whatever the evaluator rewards, rather than what you actually wanted.

5. Is diversity valuable?

Off-policy datasets can contain behaviors that the current model would never discover.

This can be extraordinarily valuable in reasoning and coding.

Imagine your model has a 0.01% probability of discovering a particular algorithm.

An on-policy system might need an enormous number of rollouts to find it.

An external expert, stronger model, or historical checkpoint might already have produced it.

In that situation, insisting on on-policy data is throwing away information.

The interesting design space is therefore not:

"Should I use on-policy or off-policy training?"

It is:

Which experiences should be generated by the current policy, which should be harvested from other policies, and how aggressively should each kind be reused?

That is a much more useful question.

And it points toward a future where the training system itself looks increasingly like an experience-management system:

                  +------------------+
                  |  Current model   |
                  +--------+---------+
                           |
                     generate
                           |
                           v
                    +-------------+
                    |  Evaluator  |
                    +------+------+ 
                           |
                 +---------+---------+
                 |                   |
                 v                   v
          fresh experience      replay buffer
                 |                   |
                 +---------+---------+
                           |
                           v
                    policy update
                           |
                           v
                     new model

The winning systems may not be the ones that are purely on-policy or purely off-policy.

They may be the ones that are best at deciding when fresh experience is worth paying for and when old experience is still valuable.

That is ultimately an economics problem disguised as a reinforcement-learning problem.

And for LLMs, the economics are unusually stark: every trajectory is potentially an expensive experiment, and every successful trajectory is potentially reusable training capital.

9. The takeaway

The simplest way to remember the distinction is:

On-policy learning learns from what the model currently does. Off-policy learning learns from what some behavior did.

On-policy training gives you data that is tightly matched to the policy being optimized, but generating that data can be enormously expensive.

Off-policy training lets you reuse experience, mix data from different models, and learn from behavior the current model might never discover. But now you inherit the statistical problem of distribution mismatch.

Classical RL encountered this decades ago with Q-learning, actor-critic methods, and policy-gradient algorithms. LLMs have simply made the underlying tradeoff vastly more expensive and more consequential.

InstructGPT demonstrated the power of policy optimization for language models. PPO provided a practical mechanism for repeatedly improving a policy from its own rollouts. DPO subsequently showed how much of the preference-learning problem could be reformulated as direct optimization on a fixed dataset.

The next step is arguably more interesting.

Once models can generate enormous amounts of candidate reasoning, code, proofs, tool trajectories, and other experiences—and automated evaluators can decide which experiences are valuable—the central training question becomes:

How should an LLM decide which of its experiences to learn from, how many times to reuse them, and when to spend compute generating new ones?

That sounds less like traditional fine-tuning and more like building a learning system with a memory.

Do you think future frontier-model training will converge toward predominantly on-policy learning, or will replaying and intelligently curating experience become the bigger competitive advantage?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.

For a long time I had a model that trained cleanly and produced nothing usable. The loss fell. Gradients were finite. Nothing crashed. It simply never learned to stop — every generation ran to the length cap and got truncated.

The cause was two lines of my own source that were individually correct.

The collision

In a neural codec language model the audio vocabulary has a fixed size, and the stop token is one extra class on top. So the output layer is one wider than the codebook:

nn.Linear(d_model, NUM_AUDIO_TOKENS + 1)   # 1025 classes: 0..1023 audio, 1024 = EOS
eos_id = NUM_AUDIO_TOKENS                  # 1024

Correct. EOS is the last class and the layer has room for it.

Then, elsewhere, the loss:

F.cross_entropy(logits, targets, ignore_index=NUM_AUDIO_TOKENS)

Also reasonable on its own. ignore_index is how you skip padding.

But the sentinel and the stop token are the same integer. Every position whose target was "stop" was discarded before the loss was computed. Not down-weighted — removed. The model was never once shown an example of stopping, across every epoch it ever ran.

PyTorch defaults ignore_index to -100 precisely because it must be a value that can never be a real class. Replace it with a real vocabulary constant and that guarantee is gone, silently: shapes valid, loss finite, run healthy.

The curves are identical

Minimal reproduction, two arms differing only in the sentinel value.

The broken arm finished at 0.0035. The fixed arm finished at 0.0034.

Same curve to any human, any dashboard, any threshold you would write. One has a working objective and one has an objective with a hole in it, and the loss cannot distinguish them — because the loss is computed over what survived the mask. A metric cannot report on examples it never received.

The second half: lowest loss, worse model

200 epochs at lr = 2e-5, evaluated every 50 on a fixed split held out by utterance (n = 32). Rank is the stop token's position among 1025 classes at the true terminal frame.

Between epochs 100 and 150 the training loss improved by 20% while mean P(stop) fell 54%, top-1 stop accuracy went 18/32 to 8/32, and autonomous termination halved.

Selecting by lowest training loss returns epoch 150. The model that terminates reliably is epoch 100. The reversal was observed independently in a shorter run, which is why I am willing to state it.

"Save the checkpoint with the lowest validation loss" is the default in more or less every training script in existence, including mine. On this run it was actively the wrong rule, and the number it optimised looked better the whole way down.

For completeness, what fixing the collision bought on the real model, same held-out split:

End-to-end synthesis then terminated on its own at frame 203 against a 350-frame ceiling. Before the fix that was impossible by two independent mechanisms: the training-time collision above, and an inference-time mask that set the logit of every index at or beyond the codebook size — including end-of-sequence — to negative infinity before sampling.

One honest loose end: P(stop) plateaus in the range 0.35–0.47 across two learning rates and a 5.9x increase in training data (224 to 1313 utterances, with speaker, language, emotion and the held-out split all held constant). So the plateau is not a data-quantity limit. I have no confirmed explanation for it. Terminal timing in speech is genuinely ambiguous, and hedging with the stop token ranked second of 1025 may simply be correct behaviour.

The same integer, safe one stage over

The detail I find most instructive: the identical line is harmless in the next stage of the same model.

The autoregressive stage predicts the first codebook plus EOS — 1025 classes, so 1024 is a real class and using it as a sentinel is fatal. The non-autoregressive stage predicts audio codes only — 1024 classes, so 1024 is out of range and the exact same ignore_index is correct.

One + 1 in a different file decides whether that line destroys your objective. Both stages read identically at the call site. That is not a mistake you catch by reading carefully; it is a mistake you catch by checking a relationship between two numbers that never appear together.

Which is what a linter is for

This is now two rules in trainproof, my linter for training runs:

• Sentinel collision. Compare the output layer's class count against ignore_index. If the sentinel is a valid class, fail. If it sits exactly one past the end, say so explicitly — because the same integer is fatal one class earlier, and that distinction deserves to be visible rather than silently passed.
• Dead class. Accumulate which classes ever reach the loss as a positive target during the first epoch, then flag any class the output layer can emit but that never once appears as an answer.

My own two stages are the regression fixture — the fatal case and its safe twin, one integer apart. Not a synthetic example.

The design decision worth stating: the dead-class rule only fires when coverage is already broad and few classes are missing. One unseen class out of 1025 is a structural exclusion. Nine hundred unseen is a small sample. Without that guard the check screams on every short run and gets switched off — which is how good checks die.

If you take one thing from this

I do not think this is rare. Any codebase where a padding sentinel, an end-of-sequence id and a vocabulary size are all defined as named constants in different files can produce it, and none of your instrumentation will complain.

If you fine-tune anything with a custom ignore_index, go and check it against your output layer's width right now. It takes thirty seconds and the failure mode is completely silent.

Paper: The Loss Curve Is Not a Sufficient Statistic — Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

DOI: 10.5281/zenodo.21864658

pip install trainproof — GitHub, MIT

The best enterprise MCP gateway is not the product with the longest feature list. It is the one whose identity, policy, deployment, and failure model match your agents. After reviewing the current MCP specification and the leading gateway options, Bifrost is one of my strongest shortlist choices for application teams that want model routing and MCP tool access in the same self-hostable gateway, an embeddable Go SDK, and an explicit application-controlled tool-execution step. Its open-source codebase also gives platform teams an inspectable starting point.

That recommendation has boundaries. Bifrost Enterprise, not the open-source edition alone, is the relevant tier if you require high-availability clustering, enterprise identity federation, admin RBAC, and audit-grade logs. And if your main problem is container isolation, Kubernetes lifecycle management, or extending an existing API gateway, another product may fit better.

The short answer

Bifrost is a strong enterprise MCP gateway for teams that want one Go-based, self-hosted data plane for LLM provider traffic and MCP tools. It connects to multiple MCP servers, exposes their tools through one endpoint, supports shared and per-user upstream authentication, applies layered tool allow-lists, and keeps tool execution explicit by default.

It is not universally “the best.” Docker MCP Gateway is compelling for isolated local server runtimes; Kong is a natural extension of an existing Kong estate; Microsoft MCP Gateway targets Kubernetes-managed server lifecycle; and Lunar MCPX focuses on dedicated MCP aggregation and tool controls.

What is an enterprise MCP gateway?

An enterprise MCP gateway is an infrastructure layer between AI agents and Model Context Protocol servers. It gives agents one governed entry point for discovering and calling tools while centralizing identity, credential handling, authorization, routing, logging, and policy enforcement.

Without a gateway, every agent must connect to every server independently:

Direct connections

Agent A ─┬─ GitHub MCP server
         ├─ database MCP server
         └─ internal API MCP server
Agent B ─┬─ GitHub MCP server
         └─ database MCP server

With a gateway

Agents ── MCP gateway ─┬─ GitHub MCP server
                       ├─ database MCP server
                       └─ internal API MCP server
             │
             └─ identity, tool policy, credentials,
                approvals, traces, limits, audit

The direct model is fine for one developer and a few trusted tools. At enterprise scale, it duplicates configuration and secrets, scatters logs, and makes it difficult to answer a basic incident question: which user, through which agent, called which tool with what authority?

“MCP gateway” describes several different products

Flat comparison tables are misleading because the category contains at least four architectures:

1. Combined LLM and MCP gateways, such as Bifrost, govern model requests and MCP tool access in one gateway.
2. Dedicated MCP aggregation and control layers, such as Lunar MCPX, emphasize MCP federation, tool policy, and observability.
3. API gateways with MCP support, such as Kong, apply an established gateway and plugin ecosystem to MCP traffic and API-to-tool conversion.
4. Runtime and lifecycle gateways, such as Docker and Microsoft MCP Gateway, focus on where MCP servers run and how they are isolated or managed.

Before picking a vendor, decide which problem you actually have. A platform team replacing separate model and tool proxies has a different requirement from a security team placing policy in front of existing remote MCP servers.

Six tests for an enterprise MCP gateway

1. Whose identity reaches the tool?

Ask whether every caller collapses into one shared service account or whether the gateway preserves end-user identity. Shared credentials are simpler, but per-user OAuth lets the downstream system retain its own permission model and audit trail. Bifrost documents the available patterns in its MCP connection and authentication guide and lets operators inspect and revoke per-user MCP sessions.

The current MCP authorization specification requires each access token to be used only for its intended service. An MCP gateway must not reuse a client’s token to call another API; it should use a separate token with only the permissions that API requires.

2. Where is tool policy enforced?

Filtering tools/list reduces what the model sees, but discovery-time filtering alone is not authorization. Re-check access when tools/call executes. For destructive operations, the gateway or application should also support approval, argument validation, or a hardened read-only variant. Bifrost’s virtual-key MCP controls enforce an allow-list at inference and again at tool execution.

3. How are credentials stored and refreshed?

Verify shared OAuth, per-user OAuth, workload identities, static-key storage, token refresh, revocation, and secret redaction. Also check whether request headers are forwarded automatically. Bifrost’s connection documentation says incoming headers are not forwarded by default and describes per-client allow-lists. Safe defaults matter because a convenience feature can become a credential-exfiltration path.

4. Are logs actually audit evidence?

Operational logs, OpenTelemetry traces, and immutable administrative audit trails solve different problems. You normally need all three: traces for latency and failures, request logs for debugging, and retained audit events for security investigations and change accountability. Bifrost documents built-in observability and request logging, OpenTelemetry export, and separate Enterprise audit logs.

5. What fails, and how?

Test a dead upstream server, expired OAuth token, changed tool schema, slow tool, gateway-node loss, and duplicate call. “Retries supported” is not enough; retrying a read is different from retrying create_invoice after a timeout. Bifrost documents its MCP connection states, health checks, and retry behavior, but your proof of concept should still validate the failure semantics of each tool.

6. Which MCP specification does it implement?

The finalized MCP 2026-07-28 release made the HTTP protocol core stateless and added Mcp-Method and Mcp-Name routing headers. Many product pages still describe legacy SSE or session-affinity behavior. Require a version-compatibility matrix for your actual clients and servers rather than accepting “MCP compatible” as a complete answer.

How Bifrost’s MCP gateway works

Bifrost occupies a useful position because it is both an AI model gateway and an MCP gateway. According to its MCP architecture documentation, it acts as an MCP client to external tool servers and can act as an MCP server to clients such as Claude Desktop.

The verified request path looks like this:

1. Bifrost connects to upstream MCP servers and discovers their tools. Its connection guide documents STDIO, HTTP, and SSE connections.
2. Remote connections can use static headers, shared OAuth, or per-user OAuth. The connection and authentication documentation explains when each option applies.
3. Bifrost applies stacked tool filtering at client configuration, request, and virtual-key levels. Empty client tool lists deny access by default.
4. The model receives only the allowed tool definitions. Tool names are prefixed by client, avoiding collisions between servers.
5. By default, the model only proposes a tool call. Your application reviews it and invokes the separate tool-execution endpoint. Agent Mode can opt selected tools into automatic execution.
6. Requests and model operations can be exported through Bifrost’s OpenTelemetry integration.

That separation between proposal and execution is valuable. It creates a clean approval and validation point without claiming that a human is automatically in the loop. Your application still has to implement the approval policy.

Open source versus Enterprise

Bifrost’s open-source gateway is Apache 2.0 and includes the MCP connection/aggregation path, virtual-key governance, tool filtering, rate and budget controls, and observability plugins. The Enterprise overview describes a strict superset that adds high-availability clustering, identity-provider integrations, RBAC, audit-grade logging, and private deployment options. The clustering documentation covers peer discovery, state synchronization, and node failover.

Keep that boundary in your evaluation sheet. “Open source” does not mean every enterprise control is in the community edition. For deployment planning, Bifrost also provides an official Helm guide for OSS and Enterprise installations on Kubernetes.

Bifrost publishes impressive gateway-overhead figures, but I would treat them as vendor benchmarks. They measure Bifrost’s gateway path, not your end-to-end MCP tool latency, and there is no common independent benchmark here for ranking all six products.

Bifrost compared with enterprise MCP gateway alternatives

Why Bifrost belongs on the shortlist

Bifrost's case is strongest when the same application needs governed model routing and MCP tool access. Three documented design choices support that fit:

• Bifrost returns proposed tool calls for review before the application invokes a separate execution API, creating an explicit point for approvals and argument validation.
• Teams can deploy it as a gateway or integrate it directly through the Go SDK.
• Its MCP Sessions UI supports inspection, re-authentication, editing, and revocation of per-user OAuth and header credentials.

Those capabilities do not make Bifrost the automatic winner for every deployment. They explain why it is a credible proof-of-concept candidate for the buyer profile in this article.

Kong’s configuration surface is an advantage if you already run Kong and overhead if you do not. Docker’s container isolation is excellent for local STDIO servers but does not automatically answer enterprise identity governance. Microsoft’s lifecycle manager is useful when the gateway should deploy servers, which Bifrost does not position as its primary job.

When I would choose Bifrost

I would put Bifrost on the proof-of-concept shortlist when:

• the platform already needs multi-provider model routing as well as MCP;
• the team prefers an inspectable, self-hosted Go service or wants to embed the gateway through its Go SDK;
• explicit tool execution fits the approval design;
• virtual-key and per-request tool allow-lists are sufficient for application-level access;
• OpenTelemetry and private deployment are operational requirements; and
• there is a clear path to Enterprise if clustering, SSO/SCIM, RBAC, and audit logs become mandatory.

I would look elsewhere first when the primary requirement is a curated connector catalog, container-per-server isolation, Kubernetes-managed MCP server deployment, or deep integration with an existing API gateway.

A proof-of-concept checklist

Do not evaluate an MCP security gateway from a demo dashboard alone. Run these tests with one read tool and one destructive tool:

• two users with different upstream permissions;
• discovery and execution denial for the restricted user;
• OAuth expiry, refresh, and revocation;
• attempted forwarding of an unapproved authorization header;
• a tool description/schema change after approval;
• gateway restart and upstream-server failure during a call;
• trace correlation from agent through gateway to server;
• audit export and actor attribution;
• concurrent clients using the exact MCP protocol versions you deploy; and
• edition/license mapping for every control in the acceptance criteria.

Final recommendation

Bifrost is one of the strongest enterprise MCP gateway options in 2026 for application teams that want unified LLM and MCP infrastructure, self-hosting or Go embedding, per-user MCP credential operations, layered tool governance, and an explicit execution boundary. That is a specific architectural fit, not a universal victory.

Choose the shape before the brand. Then require primary-source evidence and a failure-oriented proof of concept. For a Bifrost evaluation, compare the OSS and Enterprise editions explicitly and make 2026-07-28 protocol conformance part of acceptance testing.

Frequently asked questions

Do AI agents always need an MCP gateway?

No. A single trusted agent connected to a few local servers may be simpler without one. A gateway becomes valuable when multiple agents, users, credentials, policies, or audit requirements need one enforcement point.

What is the difference between an MCP gateway and an API gateway?

An API gateway primarily understands routes, services, and HTTP consumers. An MCP gateway understands MCP operations and entities such as tool discovery and execution. Products like Kong add MCP-aware capabilities to an API gateway; Bifrost combines MCP with model-provider routing.

Can Bifrost manage multiple MCP servers?

Yes. Bifrost connects to multiple upstream MCP servers, discovers their tools, filters them, and can expose the allowed aggregate through its /mcp gateway endpoint. It calls each upstream connection an MCP client.

Is Bifrost an open-source MCP gateway?

Yes. The Bifrost repository uses the Apache 2.0 license. Clustering, enterprise identity/RBAC, audit-grade logs, and some private deployment capabilities belong to Bifrost Enterprise, so evaluate the edition as well as the project.

How do you secure MCP servers behind a gateway?

Authenticate users and workloads, use audience-bound tokens, obtain separate upstream credentials, allow-list tools, enforce policy again at execution, require approval for risky calls, restrict header forwarding, trace every call, and retain security-relevant audit events. Also test protocol versions and tool-schema changes rather than treating the gateway as a complete security boundary by itself.