The Bill That Made Me Question Everything

Last month, I analyzed the API bill for a project with LangGraph-orchestrated agents. The cost was 40x higher than estimated. Why? Because at every turn, the orchestrator injected new instructions, routed calls, managed history — and each of these operations consumed tokens from a frontier model.

I knew external orchestration was expensive. I didn’t know it was that expensive. And when I found the paper “Compiling Agentic Workflows into LLM Weights” (arXiv:2605.22502, published May 21, 2026), I discovered I wasn’t the only one thinking about this — and that the solution is more radical than I imagined.

The proposal: instead of maintaining an external orchestrator running above the LLM, compile the entire workflow directly into a smaller model’s weights. The result? A 3B-parameter model delivering 87-98% of quality from a frontier model 70x larger, at 128-462x lower cost, with 2.8x more speed.

And privacy? Data never leaves your infrastructure.

The Dirty Secret of External Orchestration

Agent orchestration frameworks have accumulated over 290,000 GitHub stars — LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, Semantic Kernel, Strands, LlamaIndex. All follow the same pattern: an external orchestrator above the LLM, injecting instructions and routing decisions every turn.

And all carry the same problems:

Absurd complexity. Dozens of external components talking to each other. Each integration is a failure point. In complex tasks like travel booking (14 decision nodes), the external orchestrator routing failure rate can reach 24%.

Token bloat. Every turn, the orchestrator re-injects instructions, history, and context. Token consumption spikes because instructions change every step — but the frontier model charges for every token.

IP exposure. Your operational procedures — your company’s business logic — are sent in plain text to third-party APIs on every call. Each system prompt contains your workflow. Each call exposes your competitive advantage.

As the paper’s authors identified: orchestration “fragments reasoning by generating only from local node context, introduces routing failure modes absent in non-orchestrated architectures, and constrains the model’s capacity.”

Subterranean Agents

The proposed solution is elegant: Subterranean Agents. The process works like traditional software compilation:

Step 1: Define the workflow as a directed graph. Nodes represent procedure steps (with prompt templates, agent/user roles), edges represent transitions. A travel booking workflow has 14 nodes. Zoom tech support also has 14. Insurance claims processing can have more.

Step 2: Generate thousands of synthetic conversations. The system traverses the graph and generates complete dialogues covering all possible paths — including edge cases, user errors, and exceptions.

Step 3: Fine-tune on a smaller model. These conversations are used to tune a compact model (3B to 8B parameters). The model internalizes the entire workflow — not as prompt instructions, but as neural weights.

Step 4: Deploy without an orchestrator. In production, the smaller model runs alone. No LangGraph. No CrewAI. No intermediaries. The workflow is “baked” into the weights.

The Numbers That Justify the Switch

Tests covered three procedural domains: travel booking (14 nodes), Zoom technical support (14 nodes with product-specific knowledge), and insurance claims processing.

Quality. A compiled 3B model beat the same 3B model with explicit orchestration on 4 of 5 quality metrics (p < 0.001). Against frontier models 70x larger, it achieved 87-98% quality.

Cost. 128 to 462x cheaper per request. If you spend $10,000/month on frontier API for orchestrated agents, the same compiled workflow costs $21-78/month.

Speed. 2.8x faster in latency. Without the orchestrator’s multiple round-trips, the response reaches the user faster.

Privacy. The model runs locally. No data leaves your infrastructure. No operational procedures exposed in plain text to third-party APIs.

The Obvious Objection (And the Answer)

The immediate question every architect asks: “If the workflow changes, do I have to retrain?”

Yes. But the cycle is fast. The authors integrated recompilation into traditional CI/CD pipelines. If the workflow changes, the system generates new synthetic data from the updated graph and recompiles the smaller model in 30 to 50 minutes using common production hardware.

Not months. Not weeks. Minutes. The smaller model treats the workflow as a compilable artifact — like compiled code. Source changed? Recompile.

Where This Works (And Doesn’t)

Works brilliantly for procedural, structured tasks: customer service with defined scripts, claims processing, client onboarding, tech support with decision trees, approval workflows, bookings and scheduling.

Doesn’t work for tasks requiring genuinely unpredictable decisions — open web research, ambiguous problem-solving, creative exploration. For those, agents with orchestration (or sophisticated harnesses like SkillOpt) remain necessary.

The rule I proposed in the Agent vs Workflow post applies perfectly: if the flow is predictable, compile. If unpredictable, orchestrate. And for most enterprise workflows, the flow is predictable.

The Connection to the Series

This paper is the convergence of at least four themes:

SkillOpt optimizes a .md file without touching weights. Subterranean Agents compile the workflow into weights. Complementary approaches on the same spectrum: the more procedural the task, the more deeply you can “bake” the logic into the model.

Agent vs Workflow: the paper confirms that for procedural tasks, orchestrated agents are dominated — outperformed in cost, speed, and often quality by simpler alternatives.

Fine-Tuning vs RAG: this is a perfect justified fine-tuning case — when behavior is fixed and procedural, internalizing into weights drastically outperforms prompt or RAG delivery.

95% of projects fail: many enterprise projects use expensive frontier models for tasks a compiled 3B model would do better. The waste is structural.

Conclusion: AI Is Getting Leaner

AI software architecture is maturing. The era of using massive, expensive models to carry simple workflows through external prompt loops is ending.

By compiling business logic directly into smaller models’ neural structure, companies achieve the development Holy Grail: frontier AI performance at local model cost and speed.

462x cheaper. 2.8x faster. 87-98% quality. Data never leaves your infra. Recompilation in 30 minutes.

If you still use LangGraph or CrewAI for procedural workflows, this paper is the signal to reconsider.

Share if this changed your architecture:

290,000 GitHub stars for orchestration frameworks. And a paper just proved that, for procedural tasks, they’re a 462x waste.

Read Also