Coding agents like Codex, Cursor, and OpenCode change the economics of model selection. When a person chats with a model directly, one prompt usually produces one response. When an agent works a task, a single request turns into a ten-step loop: listing directories, reading files, applying a small patch, running a test, and formatting the result.
Most of those turns are mechanical. Only a small fraction require a frontier reasoning model to design an architecture or untangle a race condition. Yet asking developers to switch models by hand rarely works. People either leave the frontier model selected all day and pay frontier rates to read files, or stay on a fast model and get frustrated when a harder change goes sideways.
Some commercial harnesses already solve this for general interactive coding. Cursor’s Auto mode does a good job of picking between fast and frontier models behind the scenes, and other frontier harnesses ship their own internal routing. The catch is that vendor routers are opaque and untunable. You cannot see the decision criteria, adjust the escalation threshold, or reuse that router across other tools. More importantly, a vendor’s router does not know your company, your domain, or your recurring workflows. It cannot tell whether a prompt is part of a scheduled operational check, an internal bookkeeping job, or a high-risk production migration.
Putting model routing in our own LiteLLM gateway lets us apply one explicit, auditable policy across every agent harness and background workflow we run. Over the past week I replaced our gateway’s local embedding classifier with Cloudflare’s clef-flash decision model, fixed the protocol bridge so Codex sessions route cleanly to DeepSeek V4.1 Flash, and cut classification latency to 184ms on new prompts and 0ms inside tool loops.
Why Session Heuristics Fail in Agent Workflows
Our first gateway router combined local semantic embeddings with session metadata. On every request to the coding model group, the gateway generated a 768-dimensional embedding on the local machine hosting Ollama (nomic-embed-text:v1.5), reduced it to 32 dimensions via PCA, and added five structural signals: prompt length, conversation turn count, tool count, and whether the session referenced source files. A logistic regression model then routed the turn to either our fast coding waterfall (DeepSeek V4.1 Flash 1017) or our frontier thinking tier (Gemini 4 Argon).
That design looked reasonable on paper, but it optimized for the wrong signals in production.
Session length is not task difficulty. In an interactive agent workflow, a session accumulates dozens of tool definitions, file references, and prior turns within a few minutes. Because our regression weights treated turn count, tool count, and file extensions (.md, .py, .json) as indicators of complexity, routine work inside long sessions steadily drifted into the frontier tier. In our shadow logs, 28 out of 66 scheduled 30-minute heartbeat checks (42%) were routed to our frontier model at 92.2% confidence simply because the session carried tool definitions and referenced HEARTBEAT.md. Conversely, a single-turn prompt asking to “Design a caching layer for the LiteLLM proxy to handle 10k concurrent requests” went to the fast tier because it arrived with zero prior turns and no attached files.
Cold-start tail latency broke the budget. When the embedding model stayed warm in memory on the local Ollama host, classification took 27.6ms at the median. Whenever the model was evicted or the host was busy, latency spiked to 3,622ms, tripping our 1.5-second router timeout.
Maintaining a custom classifier is the wrong job. Every time prompt templates, domain workflows, or agent harnesses changed, fixing the drift meant pulling logs, labeling examples, and re-fitting PCA and regression weights. We did not need a custom machine learning pipeline. We needed a tunable policy check where we could express our own workflow rules in plain English.
Moving From Trained Weights to Tunable Domain Policy
To replace the regression model, we shadow-tested Cloudflare’s @cf/cloudflare/clef-flash on Workers AI. Clef is a 9B decision model that takes a state string (we pass up to 2,000 characters of the user’s latest request) and scores it from 0 to 1 against natural-language questions.
That shifts routing from model training to product and operations policy. Unlike an opaque vendor router, we can define exactly what warrants a frontier model in our environment and update those rules in seconds as our workflows evolve:
- Hard criteria: Is this a complex or multi-step task requiring deep reasoning, architecture design, or debugging complex systems? Simple code generation, file edits, and routine tasks are not hard.
- Fast criteria: Is this a simple, routine, or mechanical task such as a short code edit, simple question, summarization, or file operation?
Running both classifiers side by side across 114 production calls surfaced one important calibration lesson. Because Clef is conservative when scoring open-ended complexity, raw hard scores tend to cluster in the middle of the scale even on genuinely difficult architecture tasks. We added a multiplier to boost sensitivity on complex prompts, which initially caused a few routine requests to clear the escalation bar even when Clef was much more confident the task was fast.
Requiring two conditions before escalating, that the boosted hard score clears our threshold and that hard outscores fast head-to-head, eliminated all 19 false escalations immediately:
| Traffic Segment | Calls | Legacy vs. Uncalibrated Clef | Legacy vs. Calibrated Clef | Better Decision on Disagreements |
|---|---|---|---|---|
| Interactive Coding & User Turns | 48 | 28 / 48 (58.3%) | 47 / 48 (97.9%) | Clef (1 / 1) |
| Scheduled Heartbeat Checks | 66 | 38 / 66 (57.6%) | 38 / 66 (57.6%) | Clef (28 / 28) |
| Trivial Sanity Checks | 3 | 0 / 3 (0.0%) | 0 / 3 (0.0%) | Clef (3 / 3) |
| Total | 117 | 66 / 117 (56.4%) | 85 / 117 (72.6%) | Clef (32 / 32) |
Across all 32 disagreements between the legacy classifier and calibrated Clef, Clef made the right call every time. It kept 100% of scheduled heartbeats and sanity checks on the fast tier while escalating the single-turn 10k-concurrent-request architecture prompt to Argon.
Reducing Classifier Latency: First-Pass Results and What Is Next
Calling an external API before routing an LLM request adds latency to the critical path. In our initial shadow test using standard Python HTTP calls (Connection: close), median classification latency was 312.5ms, with a 95th percentile of 729.6ms.
An extra 300ms on every new prompt is noticeable in interactive work. We did a first optimization pass to bring that overhead down, with measurable gains from two changes:
- Persistent HTTP/2 connection pooling. Keeping a warm TLS connection pool to Cloudflare eliminated the per-request handshake, cutting median network latency from 312.5ms to 184.4ms and tightening p95 latency from 729.6ms to 286.5ms.
- Caching across multi-step tool loops. Inspecting the 48 interactive coding calls in our shadow log showed that only 15 were new user prompts. The other 33 (nearly 70%) were tool-result continuations where the agent executed up to 10 consecutive tool calls under the same user instruction. Because the user’s objective does not change while the agent reads files and runs tests, caching the routing decision by the SHA-256 hash of the user query resolves steps 2 through N in 0.0ms.
| Classifier / Transport | Min | p25 | p50 (Median) | Mean | p90 | p95 / Max |
|---|---|---|---|---|---|---|
Initial Shadow (Connection: close, n=114) | 211.8ms | 277.2ms | 312.5ms | 345.9ms | 434.2ms | 729.6ms / 1,278.0ms |
| Benchmark Baseline (no keep-alive, n=20) | 186.3ms | 211.6ms | 230.6ms | 262.1ms | 534.2ms | 615.9ms |
| Persistent HTTP/2 Pool (keep-alive, n=20) | 149.9ms | 170.8ms | 184.4ms | 186.0ms | 219.0ms | 286.5ms |
| Tool-Loop Cache Hit (~70% of agent turns) | 0.0ms | 0.0ms | 0.0ms | 0.0ms | 0.0ms | 0.0ms |
| Local Ollama + PCA/LogReg (n=10) | 20.4ms | 21.3ms | 27.6ms (hot) | 385.5ms | 3,622.6ms (cold) | 3,622.6ms (timeout) |
If Cloudflare ever times out or returns an error, the router automatically falls back to the local classifier so gateway availability never depends on a single external endpoint.
At 184ms median on uncached prompts, there is still room to improve:
- Running the router at the edge: Right now our self-hosted gateway makes a WAN round trip to Workers AI on every uncached turn. Moving the classification step into a Cloudflare Worker or AI Gateway directly adjacent to
clef-flashwould cut the network hop between the proxy and the model. - Session-intent pinning: Our current SHA-256 cache only matches identical user prompts across tool-loop continuations. Pinning a session’s tier across short follow-up turns (such as “now run the tests” or “fix that lint error”) unless the prompt shape changes would avoid re-classifying every conversational nudge.
- Speculative fast-tier dispatch: Because roughly 75% to 85% of coding turns stay on the fast tier, the gateway could start the fast-model request in parallel with the Clef check and only cancel and escalate on the minority of turns that score
hard.
The Unit Economics of Routing With Clef-Flash
Adding an external model call in front of every prompt raises an obvious question: how much does the classifier itself cost compared with the model spend it saves?
Cloudflare prices @cf/cloudflare/clef-flash at $0.038 per million input tokens (compared with $0.15 per million for clef-omni and $0.24 per million for the 27B clef model). Because Clef returns structured option probabilities rather than generated text, there is no output token charge.
In our gateway, the router only sends the user’s latest prompt (capped at 2,000 characters) plus our two short policy questions, not the entire 100,000-token repository context. A typical classification call uses roughly 300 input tokens (and at most ~600 tokens at the 2,000-character cap):
| Metric | Value | Notes |
|---|---|---|
clef-flash input price | $0.038 / 1M tokens | Cloudflare Workers AI catalog rate ($0 output cost) |
| Average classifier payload | ~300 input tokens | Latest user prompt (<= 2,000 chars) + 2 routing criteria |
| Cost per uncached classification | $0.0000114 | $0.0114 (1.14 cents) per 1,000 uncached prompts |
| Effective cost per 1,000 agent turns (70% cached) | $0.0034 | 300 uncached calls + 700 tool-loop cache hits |
| Estimated monthly classifier spend (1,000 turns/day) | ~$0.10 / month | 30,000 total turns (~9,000 uncached Clef evaluations) |
Average frontier turn (Gemini 4 Argon) | $0.01155 / call | Measured across 14,948 calls (100k avg prompt tokens) |
Put simply, preventing a single false escalation to our frontier model ($0.01155) pays for roughly 1,000 uncached Clef classifications, or 3,300 cached agent turns. When our legacy classifier misrouted 28 heartbeat checks in a single sample window, that drift alone cost roughly 30x more than running clef-flash across a full month of agent traffic.
Protecting the Agent Loop: Protocol Bridges and Self-Healing Retries
A cheaper fast tier only delivers value if agents behave identically when routed there. When we moved Codex traffic onto our routed coding group, we had to close three operational gaps at the gateway boundary so sessions would not silently fail back to the frontier model:
Bridging /v1/responses tool definitions. Codex communicates using OpenAI’s /v1/responses endpoint and passes deferred tools inside additional_tools items. When LiteLLM forwarded /v1/responses payloads natively to OpenAI-compatible upstreams, those deferred tool definitions were either rejected with HTTP 400 or silently dropped, forcing Codex onto our frontier fallback on every tool call. Configuring our fast DeepSeek V4.1 Flash deployments to bridge /v1/responses through standard chat-completions mode (mode: chat) preserved every tool definition across all three upstream providers.
Making interrupted streams self-healing. When an agent session disconnects mid-stream or hits a content filter after emitting a reasoning block, Codex automatically retries with that partial reasoning item at the end of the conversation history. Google’s Gemini endpoint rejects any request that ends on a model turn with an HTTP 400 error ("Requests ending with a model turn are not supported"), turning a brief network hiccup into five failed retries and an aborted task. Sanitizing the conversation boundary at the gateway, stripping orphaned trailing reasoning blocks before forwarding the retry, eliminated those 400s completely.
End-to-end auditability in traces. Unlike a closed vendor router, a gateway router lets you inspect every decision. By propagating the router’s tier choice, raw hard and fast probabilities, cache-hit status, and latency into our Opik trace store and Postgres spend logs, we can audit model distribution and tune our routing rules against real workload data in a single query.