The Economics of Long-Context Windows

The Seductive Million-Token Window

Gemini 1.5 Pro ships with a 1M token context. Claude 3 Opus handles 200K. GPT-4o supports 128K. The immediate reaction from most engineers: "Great — I'll just dump everything in and skip the RAG complexity."

Sometimes that instinct is correct. Often it is not. The difference comes down to arithmetic that most teams skip because they are in a hurry to ship. This post does the math.

What You Are Actually Paying For

LLM pricing has two levers: input tokens and output tokens. Output is usually 3–5× more expensive per token than input, but output volume is bounded by your max_tokens setting. The variable cost in production is almost always input tokens.

For a 1M-token context window, the economics depend entirely on:

1. How often the context changes — if it is static, prompt caching changes everything.
2. How many questions get answered per context load — amortization matters.
3. What retrieval infrastructure costs — RAG is not free.

Let us build a concrete model.

The Cost Model

Assumptions (June 2025 pricing, approximate)

RAG infrastructure costs (self-hosted):

Total RAG infra: ~$2,200/month minimum for a serious deployment.

Long-Context Break-Even

If you have a 400K-token corpus and use Claude 3.5 Sonnet:

• Without caching: 400K × $3.00/M = $1.20 per request
• With prompt caching (90% hit rate): 40K × $3.00/M + 360K × $0.30/M = $0.23 per request

At 10,000 requests/month: $2,300/month uncached, $2,300 cached. RAG breaks even around the same figure — but RAG scales sub-linearly while long-context scales linearly.

At 100,000 requests/month: Long-context (cached) = $23,000. RAG infra stays at $2,200 + retrieval API costs ($100). RAG wins decisively.

The crossover is approximately 10,000 requests/month for a 400K-token corpus. Below that, long-context with caching is cheaper and simpler. Above it, RAG wins.

Decision Flowchart

Latency: The Hidden Cost

Cost is only half the picture. Latency matters for interactive applications.

Time to first token (TTFT) comparison:

• RAG retrieval: 50–200ms (vector search) + 200–500ms (LLM with 2K context) = 250–700ms total
• Long-context (50K tokens): 800ms–2s TTFT typical
• Long-context (400K tokens): 3–8s TTFT typical

For a chat interface, 8 seconds to first token is unusable even with streaming. RAG wins on latency at large context sizes.

For batch processing (document analysis pipelines, nightly jobs), latency is irrelevant and long-context simplicity has real value.

When Long-Context Wins Unconditionally

1. Corpus Smaller Than ~100K Tokens

If your entire knowledge base fits in context, use it. RAG adds complexity, retrieval errors, chunking decisions, and embedding drift — all of which degrade answer quality. A perfect recall rate of 100% beats a 95% retrieval hit rate at essentially any cost level when requests are low volume.

2. Reasoning Over the Full Document Is Required

RAG retrieves relevant chunks. It cannot reason about relationships between distant sections of a document. "What are all the contradictions in this 80-page contract?" requires seeing the full document. Legal, financial, and technical review tasks often fall here.

3. You Need to Ship in a Week

RAG is at minimum a three-week project to build correctly. Chunking strategy, embedding model selection, retrieval evaluation, re-ranking, hybrid search — each is a decision with production consequences. Long-context with a good system prompt ships in a day.

When RAG Wins Unconditionally

1. Corpus Grows Unboundedly

If your knowledge base is a live database, Confluence wiki, or codebase that grows weekly, you cannot stuff it in context. RAG scales; context windows do not.

2. You Need Source Attribution

RAG retrieval gives you the exact chunks that generated the answer. Long-context does not tell you which sentences the model drew from. For compliance-sensitive applications (legal, medical, financial), source attribution is often mandatory.

3. Multi-Tenant Isolation

With RAG you can scope retrieval to a specific user or tenant. With long-context you would have to construct a separate context per tenant — which at scale is prohibitively expensive.

The Hybrid Pattern

The most robust production architecture is not a binary choice:

Retrieve chunks for the 90% of queries that are straightforward. Route "hard" queries — where retrieval confidence is low, or where the question explicitly asks about the full document — to a long-context pass over the full source. This keeps median cost and latency low while handling the tail correctly.

Practical Cost Tracking

Track token spend per request from day one. Most teams discover they are spending 10× more than expected because they forgot to count system prompts, conversation history, and tool definitions.

# Simple token cost tracker
PRICES = {
    "claude-3-5-sonnet": {"input": 3.0, "output": 15.0, "cache_read": 0.30},
}
 
def compute_cost(model: str, usage: dict) -> float:
    p = PRICES[model]
    input_cost = usage.get("input_tokens", 0) * p["input"] / 1_000_000
    output_cost = usage.get("output_tokens", 0) * p["output"] / 1_000_000
    cache_cost = usage.get("cache_read_input_tokens", 0) * p["cache_read"] / 1_000_000
    return input_cost + output_cost + cache_cost
 
# Log this per-request to your observability stack
response = client.messages.create(...)
cost = compute_cost("claude-3-5-sonnet", response.usage.model_dump())
logger.info("request_cost_usd", extra={"cost": cost, "request_id": req_id})

Without per-request tracking you are flying blind. Cost surprises in production are almost always caused by unexpectedly long contexts in edge cases — conversation threads that grow to 100+ turns, system prompts that crept up to 20K tokens, tool schemas nobody audited.

Key Takeaways

• The long-context vs RAG decision is primarily a cost and latency calculation, not a capability question — do the math for your specific corpus size and request volume.
• The break-even point is roughly 10,000 requests/month for a ~400K-token corpus with prompt caching enabled.
• Long-context wins decisively for small corpora, full-document reasoning tasks, and projects that need to ship fast.
• RAG wins decisively at high request volumes, growing corpora, multi-tenant isolation, and when source attribution is required.
• Time to first token at 400K+ context is 3–8 seconds — unacceptable for interactive UX without streaming and careful latency budgeting.
• Track per-request token cost from day one; cost surprises are almost always long tails from edge-case context sizes.