There is a story that repeats itself in enterprise technology teams with quiet, expensive regularity. A senior leader sees a compelling demo. A proof-of-concept gets green-lit. Engineers are excited. A small team spends weeks wiring together a retrieval pipeline, a vector store, an embedding model, and a prompt layer.
In the test environment, things seem great. But when the project hits production, it falls apart due to messy real-world data.
This is the tale of DIY Retrieval-Augmented Generation (RAG). By 2025, it caused a huge reversal in how enterprise technology was viewed historically.
One finding stood out across the research. Organizations often attributed AI failures to model limitations when the root causes were far more mundane: poor metadata, fragmented repositories, and retrieval architectures that could not scale beyond pilot environments.
The Numbers Boards Are Not Talking About Loudly Enough
According to S&P Global Market Intelligence's 2025 Voice of the Enterprise survey, which polled over 1,000 IT and business leaders across North America and Europe, 42% of companies abandoned most of their AI initiatives in 2025, up from just 17% the year prior. 1
That is not a gradual trend. It is a near-tripling of the failure rate in a single year, at a moment when AI investment was accelerating, not contracting.
The same study found that the average organization scrapped 46% of AI proofs-of-concept before they ever reached the people they were built for. Nearly half of everything built is discarded before deployment. 1
According to Gartner’s April 2026 survey of 782 infrastructure and operations leaders, just 28% of AI uses met their ROI goals, and a full 20% didn't work at all. So, only about a quarter really succeeded. 2
38% of leaders who encountered setbacks cited poor data quality or limited data availability as a direct cause of failure. 2
According to McKinsey's November 2025 State of AI report, while 88% of organizations surveyed reported using AI in at least one function, only 39% said AI had any meaningful enterprise-wide impact. 3
More than 80% reported no significant EBIT contribution despite adoption. And the differentiator between the companies winning with AI and those watching pilots stall?
McKinsey found that high performers were nearly three times as likely to have fundamentally redesigned workflows and invested in data infrastructure before scaling AI, not after.
Progress’ ebook, The RAG Cookbook: Turning Enterprise Knowledge Into Reliable AI Answers, offers a practical guide to designing RAG systems that make enterprise AI responses more accurate, grounded, and useful.
Download the ebook
When DIY RAG Appears Inexpensive but Breaks the Budget
Retrieval-Augmented Generation is the right architectural answer to a real problem. LLMs hallucinate when they operate on memory alone. RAG grounds them in your actual data, live, governed, and sourced. The logic is sound. The execution, when done in-house from scratch, is where the economics quietly collapse.
The visible costs of building RAG internally are straightforward: engineering time, cloud infrastructure, and model API spend. These are the costs that get put into a business case, and they are almost always underestimated. But the hidden costs are where projects die.
Data readiness debt
Enterprise data is rarely in a state suitable for retrieval. Documents live in silos. Metadata is inconsistent. Older files use formats that require custom parsers. Before a single query can be reliably answered, weeks of data engineering work are required, and this is work that needs to be redone every time the data changes.
Ongoing maintenance load
A RAG system is not a project with an end date. It is an operational service. Chunking strategies degrade as document types evolve. Embedding models have been updated, requiring reindexing.
Retrieval relevance drifts as the knowledge base grows. Each of these requires dedicated engineering attention. Teams that planned for a six-week build discover they have taken on a permanent infrastructure commitment, one with no product manager, no roadmap, and no SLA.
Hallucination tuning that never quite finishes
Enterprises use RAG to cut down on hallucinations – when models spout plausible but wrong information. To hit those stats on your own, you need ongoing evaluations, true data sets, and good retrieval metrics. Most in-house crews can't keep up with maintaining all that. Luckily, using RAG slashes those demo glitches.
Escalating total cost of ownership
Teams frequently underestimate how quickly retrieval quality deteriorates as content volumes expand. What works across a few thousand documents often struggles when exposed to years of accumulated enterprise knowledge.
The Organizational Toll Is the Harder Number to Quantify
Beyond the financial costs, there is an organizational cost to repeated failure that does not show up in any budget line.
As organizations explore AI-powered search, content intelligence, website copilots, and answer-driven digital experiences, engineering and architecture teams need to understand how existing content storage can support more intelligent, reliable, and scalable answer systems.
When a proof of concept gets scrapped, the engineers who worked on it know something's off, but the real hit comes for the business people who pushed for it; they lose some serious credibility. As a result, the next AI project starts with weaker executive backing, way less cash, and tighter deadlines.
The report attributed this not to the quality of the models, but to integration gaps, data readiness failures, and governance deficits, exactly the territory that DIY RAG projects consistently struggle to navigate.
Download Now
What Separates the Ones That Succeed
Across the research, the organizations that shipped AI that actually worked shared a consistent profile. They did not treat infrastructure as an afterthought. They audited data readiness before project kickoff.
They built governance into the system from day one, not as a remediation layer added after audit. And critically, they recognized the difference between what is worth building internally and what is worth deploying from a purpose-built platform.
The economics here are no longer speculative. Managed RAG platforms, solutions that handle indexing, retrieval quality, evaluation, and governance as a service, have matured to the point where the build-vs-buy decision has a clear answer for most organizations.
McKinsey makes the bigger picture simple: the firms nailing AI's value in 2025 didn't build the fanciest models. Instead, they crafted robust frameworks combining data, governance, and process improvements. This lets AI actually work dependably for them, not just look good on paper.
Managed platforms reduce much of the operational burden associated with RAG, but they do not eliminate the need for governance. Successful organizations still invest heavily in data stewardship, accountability, and ongoing performance measurement.
The Real Cost of Waiting
Every month, a DIY RAG project runs over the timeline, and a competitor's AI capability pulls ahead. Every abandoned proof of concept represents not just a sunk budget but also lost organizational conviction, and conviction is harder to rebuild than a technical pipeline.
The World Economic Forum's 2026 assessment of enterprise AI readiness put it plainly: "Without clean, secure, and well-governed enterprise data, models falter, trust erodes, and transformation stalls." The teams that internalized this earliest, that recognized the foundation as the work, not the model, are the ones delivering in production today.
References
- S&P Global Market Intelligence (2025) Generative AI shows rapid growth but yields mixed results. S&P Global Market Intelligence, October. Available at: S&P Global Market Intelligence. Accessed: 4 June 2026.
- Gartner (2026) Gartner says artificial intelligence projects in infrastructure and operations stall ahead of meaningful ROI returns. Gartner Newsroom, 7 April. Available at: Gartner. Accessed: 4 June 2026.
- Singla, A., Sukharevsky, A., Yee, L. and Chui, M. (2025) The state of AI in 2025: Agents, innovation, and transformation. McKinsey & Company, November. Available at: McKinsey & Company. Accessed: 4 June 2026.
- Gartner (2026) The Hidden Workforce Costs of AI. Available at: https://www.gartner.com/en/articles/ai-workforce-costs (Accessed: 3 June 2026).
- MIT Project NANDA (2025) The GenAI Divide: State of AI in Business 2025. Massachusetts Institute of Technology. Available at: MIT Project NANDA. Accessed: 4 June 2026.
