Is RAG quietly dying? What we're actually shipping instead in 2026
Two years ago RAG was the default answer to "how do I make an LLM know my data." Chunk it, embed it, retrieve top-k, stuff it in context. Every tutorial, every framework, every "AI wrapper" startup pitch deck had the same diagram.
I don't think that default holds anymore, and I've been rebuilding around it for the past few months.
Three things broke the old assumption for me:
Context windows got cheap. When you can dump 200k+ tokens for pennies, "retrieve only the relevant chunk" stops being the obvious optimization - it becomes a lossy compression step you're paying latency and accuracy for, not saving cost on.
Chunking is still the weakest link nobody wants to admit. Semantic chunking, sentence-window, parent-child - every trick papers over the same problem: you're guessing where meaning boundaries are before you know the query. Agentic/iterative retrieval (let the model decide what to fetch next, over multiple turns) beats static top-k retrieval on every eval I've run, just at higher latency cost.
Fine-tuning got more accessible again. For narrow, stable domains, a lightly fine-tuned small model + light retrieval now often beats a heavy RAG pipeline on a frontier model - cheaper per query and more consistent.
So what I'm actually shipping now looks more like: long-context-first for anything under ~100k tokens of source material, agentic multi-hop retrieval for anything bigger or dynamic, and RAG-as-embeddings-search only as a last resort for pure "needle in haystack" lookup.
What's actually in your retrieval stack right now - and did you rip out RAG, or double down on it?
Replies