Why 87% of AI Projects
Never Reach Production —
And What the 13% Do Differently

Somewhere in your organisation, there is a data science team that has built an impressive model. It runs in a Jupyter notebook, produces predictions that business stakeholders found exciting during the demo, and has been labelled a "proof of concept" for long enough that the label has become permanent. It will never reach production.

This is not unusual. It is, in fact, the default outcome for enterprise AI in 2025. The majority of AI investments are absorbed by initiatives that produce genuine technical work, real business interest, and zero deployed value. The organisations that are pulling ahead are not those spending the most on AI — they are those that have understood and solved the structural problem that kills most AI projects before they leave the lab.

This article examines what that structural problem actually is, what a genuine data and AI foundation looks like, and what separates the organisations producing compounding AI value from those still presenting pilot results at quarterly reviews.

The AI investment paradox

The scale of enterprise AI investment in 2024 and 2025 is genuinely extraordinary. Global corporate AI investment exceeded $200 billion in 2024, with hyperscalers alone committing more than $300 billion in AI infrastructure capex in 2025. Every major consultancy, every technology vendor, and every board-level strategy document positions AI as the defining competitive variable of the decade. IDC projects that global spending on AI solutions will reach $632 billion by 2028.

Against this backdrop, the production failure rate is startling. Gartner has repeatedly found that the majority of AI models built by enterprises never make it into production — the widely cited figure of 87% comes from VentureBeat's analysis of enterprise ML deployments, and it has held remarkably stable across multiple years and multiple sources. McKinsey's 2024 State of AI survey found that fewer than one in four organisations report that more than 5% of their AI use cases have been scaled across the business. Forrester found that 74% of firms struggle to achieve or scale value from AI.

The paradox is not that organisations are failing to invest — they are investing at unprecedented scale. The paradox is that investment levels and realised value have almost no relationship. Organisations spending ten times more on AI than their peers are not generating ten times more business value. The constraint is not capital. It is structure.

Part of what sustains this investment cycle despite poor returns is the genuine success of the 13%. The companies that have cracked production AI — the Amazons, Netflixes, and JPMorgans, but increasingly mid-market firms with strong data leadership — are generating compounding competitive advantages. AI-powered personalisation at Netflix is estimated to save $1 billion annually in customer retention. Amazon's ML-driven supply chain optimisation reduces inventory costs by an estimated 20–30%. These outcomes are visible and real. They are also the product of specific structural capabilities that most organisations have not built.

Why AI projects fail — the structural causes

The surface-level explanation for AI project failure is usually "data quality issues" — and that is true, but it is not deep enough to be useful. The real explanation is a cascade of structural gaps that individually would be manageable, but together make production deployment effectively impossible.

The data quality trap

Garbage in, garbage out is a cliché because it is accurate. The most sophisticated model architecture in the world cannot compensate for training data that is incomplete, inconsistently labelled, temporally skewed, or contaminated by operational anomalies. IBM's Global AI Adoption Index found that 39% of organisations cite data complexity as the primary barrier to AI adoption — more than cost, skills gaps, or technology immaturity. But data quality problems are usually symptoms of a deeper issue: the absence of data governance.

Organisations without formal data governance have no single source of truth for key business entities. Customer records exist in CRM, billing, support ticketing, and e-commerce systems with different identifiers, different update cycles, and different definitions of what "active" means. A model trained on one version of customer data will not generalise to the operational definition used in the production system it is supposed to serve. This is not a model problem. It is a data architecture problem.

The "model in a notebook" problem

A data scientist builds a model in Python. It achieves 94% accuracy on the holdout set. Business stakeholders are impressed. And then nothing happens for six months. This is the most common failure mode in enterprise AI, and its cause is structural: there is no pathway from a trained model to a deployed model that can serve predictions at scale, be monitored for performance degradation, be retrained when data drift occurs, and be integrated into the business systems that actually need it.

Moving a model from a notebook to production requires ML engineering capabilities that most data science teams do not have. It requires containerisation, API development, CI/CD pipelines for model deployment, infrastructure for serving predictions at low latency, logging and monitoring systems, and — critically — the organisational process to tie model outputs to business decisions. Most organisations that hire data scientists have not hired the ML engineers, data engineers, and platform engineers that turn data science work into deployed product.

Organisational misalignment

Even when data quality and MLOps infrastructure are addressed, AI projects frequently stall at the boundary between data science and business ownership. A model that predicts customer churn is useful only if there is a business owner who owns the churn reduction process, has defined the intervention logic, has agreed on the decision threshold, and has the authority to act on model outputs at scale. Without a business owner who is genuinely invested in the model's deployment and usage, data science teams produce predictions that no one uses.

This alignment problem is exacerbated by the typical project structure: data scientists report into a centralised analytics team, business stakeholders engage as sponsors for the initial business case but disengage during the long technical build phase, and by the time the model is ready, the business problem has evolved or the stakeholder who championed the project has moved on. The organisations that consistently deploy AI have fundamentally different governance structures — typically embedding data science capability within business units rather than centralising it in a function that serves everyone and is owned by no one.

The data architecture that makes AI possible

The prerequisite for any serious AI programme is not a model — it is a data foundation. This sounds obvious and is routinely ignored. Organisations that skip to model development without first establishing clean, governed, accessible data infrastructure will repeat the 87% failure pattern regardless of how much they spend on tools, talent, or compute.

The data lakehouse

The architecture that has emerged as the practical standard for enterprise AI infrastructure is the data lakehouse — a hybrid of the scalability and flexibility of a data lake with the structure, governance, and query performance of a data warehouse. Platforms like Databricks, Snowflake, and Apache Iceberg-based architectures allow organisations to store raw data at scale while providing the structured access patterns that both BI tools and ML training pipelines need. The key capability is a unified metadata layer that makes data discoverable, governable, and auditable without sacrificing flexibility.

Data mesh vs. centralised warehouse

The data mesh paradigm — advocated by Zhamak Dehghani and widely adopted in the post-2022 period — argues that centralised data teams cannot scale to meet the data needs of large, complex organisations. The alternative is treating data as a product, with domain teams owning and publishing their own data products to a federated platform with shared standards for interoperability and governance. Data mesh does not eliminate the need for central infrastructure and governance — it distributes the ownership of data production while maintaining central standards.

For AI specifically, the data mesh debate matters because model training requires clean, well-documented, consistently updated datasets. Whether that is achieved through a centralised data engineering team or a federated model with domain ownership is a question of organisational scale and maturity — but the outcome requirement is the same: data that is trustworthy, documented, and accessible to the teams building models.

Feature stores and real-time pipelines

A feature store is a centralised repository for the engineered features used in ML models — the computed, transformed inputs that models actually receive rather than the raw data from source systems. Without a feature store, every data scientist re-engineers the same features from scratch, features computed in training cannot be reproduced at inference time (causing training-serving skew), and there is no reuse across models or teams. Feature stores — whether managed (Feast, Hopsworks, Tecton) or built in-house — are a prerequisite for MLOps at any meaningful scale.

Real-time vs. batch is a critical architectural choice for AI applications. Batch pipelines — running on a schedule, transforming large volumes of historical data — are appropriate for model training and for use cases where prediction latency of hours or days is acceptable. Real-time streaming pipelines using Kafka, Flink, or Spark Streaming are required for use cases where model outputs need to affect decisions within seconds: fraud detection, dynamic pricing, personalisation. Most organisations need both, which means investing in data engineering capability that spans both paradigms.

Data catalogue and lineage

Data catalogue tools — Alation, Collibra, DataHub, Apache Atlas — provide the metadata layer that makes data discoverable across an organisation. For AI specifically, lineage is critical: the ability to trace exactly which source data, which transformations, and which feature engineering steps produced the training dataset for a given model version. Without lineage, model debugging is guesswork, regulatory compliance (particularly under GDPR and the EU AI Act) is impossible to demonstrate, and model retraining is fragile. Data lineage is not a luxury for regulated industries — it is a foundation requirement for any AI programme that needs to be trusted.

What production AI actually looks like

The gap between a model that works in a notebook and a model that delivers business value in production is larger than most organisations appreciate before they attempt it. Production AI is not data science plus deployment — it is a fundamentally different engineering discipline with its own toolchain, processes, and failure modes.

Automated retraining pipelines

A model trained once and deployed indefinitely will degrade. The question is not whether to retrain — it is how frequently, triggered by what signals, and with what level of human oversight. Production ML systems require automated retraining pipelines that detect drift signals, pull fresh training data from the feature store, re-train and evaluate the new model against the incumbent, and deploy the updated model if it meets performance thresholds. Building this pipeline — end to end, reliably, with appropriate guardrails — typically takes experienced ML engineers three to six months. Most organisations have not built it. Most organisations therefore deploy models that gradually stop working, often without realising it.

Before and after: the maturity gap

The difference between a data-immature and a data-mature organisation is not primarily about technology investment — it is about structural capability, process discipline, and the alignment of data and AI work with business ownership. The table below captures what this maturity gap looks like across six critical dimensions.

Capability	Data-Immature Organisation	Data-Mature Organisation
Decision-making speed	Strategic decisions wait days to weeks for analysts to pull and reconcile data from disparate systems; reports are often outdated by the time they are read	Self-service dashboards and automated reporting pipelines deliver current data to decision-makers in real time; analyst capacity redirected to interpretation rather than extraction
AI project success rate	Most AI projects stall at POC stage; deployment pathway undefined; MLOps infrastructure absent; business ownership weak; majority of model work discarded	Defined model deployment pathway with MLOps toolchain; clear business ownership requirements before project initiation; 60–70%+ of initiated projects reach production
Data quality	No single source of truth; duplicate records, conflicting definitions, and unstandardised fields across systems; data quality issues discovered at point of use	Master data management and data governance framework; data quality metrics monitored continuously; issues identified and remediated upstream, not discovered by downstream consumers
Model deployment time	Months to deploy a model from trained state to production, requiring manual handoffs between data science and engineering; deployment is a one-time event, not a repeatable process	Automated CI/CD pipeline deploys validated models in hours to days; retraining and deployment are automated processes triggered by monitoring signals, not manual interventions
Analytical capability	Descriptive analytics dominant; historical reporting the norm; predictive and prescriptive analytics limited to isolated experiments with no production deployment	Predictive models embedded in operational systems; prescriptive analytics driving automated decisions; analytical capability compounding as more models deployed and monitored
Cost of data operations	High manual overhead in data extraction, reconciliation, and report production; significant duplicate effort across teams; data infrastructure costs poorly governed	Automated pipelines eliminate manual extraction; feature stores and data products enable reuse; infrastructure costs governed through platform FinOps practices; total cost declining as scale increases

The maturity gap is not primarily a technology gap. Immature data organisations typically have modern tools — cloud data warehouses, ML platforms, BI software — that they are underutilising because the process, governance, and organisational structure required to use them effectively has not been built. Buying better tools is rarely the answer. Building the capability to use tools systematically is.

The LLM moment: what large language models actually change for enterprise

The emergence of large language models — GPT-4, Claude 3, Gemini 1.5, and their successors — has changed the enterprise AI landscape in specific and important ways. It has also generated a substantial amount of hype that obscures what the genuine opportunity is and what responsible deployment requires.

Where LLMs genuinely change the game

The pre-LLM era of enterprise AI was dominated by narrow models: a fraud detection model, a churn prediction model, a demand forecasting model. Each required substantial labelled training data, domain-specific feature engineering, and significant ML engineering work to deploy. LLMs change the equation for a specific class of problem — those involving unstructured text — by providing a general-purpose capability that can be adapted to enterprise use cases with dramatically less labelled data and faster deployment timelines.

The use cases where LLMs are delivering measurable enterprise value in 2025–2026 are specific and well-evidenced:

Document intelligence. Organisations with large volumes of contracts, policies, reports, or correspondence can use LLMs to extract structured information, classify documents, summarise content, and flag deviations from standard terms. Law firms and financial institutions processing thousands of contracts monthly report 60–80% reductions in review time for routine document types. This is not a pilot outcome — it is being deployed at scale by firms including Allen & Overy, Goldman Sachs, and KPMG.

Internal knowledge retrieval (RAG). Retrieval-Augmented Generation — connecting an LLM to a curated internal knowledge base and having it answer employee questions with citations — is the single most widely deployed enterprise LLM use case in 2025. McKinsey's own implementation of its internal knowledge assistant is reported to have increased consultant productivity by 30–40% for research-intensive tasks. The key to a functioning RAG system is not the LLM — it is the quality, structure, and currency of the underlying knowledge base.

Code generation and engineering productivity. GitHub Copilot's 2023 productivity study found that developers using AI coding assistance completed tasks 55% faster, with higher self-reported satisfaction. By 2025, AI coding assistants are used by an estimated 65% of professional developers, and organisations with mature adoption are reporting 20–30% reductions in time-to-feature for software development work. The productivity gains are real, though they require thoughtful workflow integration and code review discipline to avoid accumulating AI-generated technical debt.

Customer service automation. LLM-powered customer service — handling tier-1 enquiries, drafting responses for agent review, and providing agents with real-time knowledge retrieval — has become a mainstream deployment. Klarna's AI assistant handled the equivalent of 700 full-time agents' volume in its first year of deployment. Organisations that have integrated LLMs into customer service workflows report 25–40% reductions in average handling time and measurable improvements in first-contact resolution rates.

Report and content generation. Automated generation of first-draft reports, market summaries, performance analyses, and executive briefings from structured data inputs is delivering measurable time savings in financial services, consulting, and corporate strategy functions. Bloomberg's internal LLM deployment for earnings analysis is reported to reduce analyst time on routine report generation by more than 50%.

What responsible enterprise LLM deployment requires

The failure modes for enterprise LLM deployment are different from those for traditional ML, but they are equally consequential. Hallucination risk — models generating plausible but factually incorrect outputs — is the primary concern for any use case where the model's output will be acted on without expert review. The mitigation is architectural: RAG systems that ground model outputs in verified documents, confidence scoring, mandatory human review for high-stakes outputs, and monitoring of factual accuracy rates in production.

Data security and privacy are non-negotiable concerns. Sending proprietary business data to a third-party LLM API exposes that data to the model provider and, depending on the provider's terms, potentially to model training. Organisations processing sensitive customer data, trade secrets, or regulated personal information must evaluate on-premises or private cloud deployment options. Microsoft Azure OpenAI Service and AWS Bedrock provide enterprise-grade API access with data isolation commitments that most compliance teams can accept — public API access often cannot.

Fine-tuning vs. RAG is a decision that most organisations over-complicate. For the majority of enterprise use cases, a well-implemented RAG architecture — with high-quality retrieval, strong prompting, and appropriate output validation — will outperform a fine-tuned model at a fraction of the cost and complexity. Fine-tuning is appropriate when the organisation needs to instil specific style, format, or domain vocabulary that cannot be achieved through prompting and retrieval. It is rarely the right first choice.

Our view

The 13% of organisations that successfully deploy AI at scale have three things in common. They built the data foundation before they built the models. They invested in production engineering as seriously as they invested in data science. And they made business ownership of AI outcomes a hard requirement, not an optional stakeholder engagement activity.

None of these things are technology decisions. They are structural decisions — about how a data and AI programme is organised, governed, and resourced. The organisations still presenting proof-of-concept results at quarterly reviews are not there because they have the wrong tools or the wrong algorithms. They are there because they have not built the structural capability that turns good models into deployed business value.

The data foundation — clean pipelines, governed data assets, a single source of truth for key business entities — is not a prerequisite for starting AI experiments. It is a prerequisite for deploying AI at scale. Organisations that skip this step will keep producing impressive notebooks and stalled pilots. Organisations that build it gain a compounding advantage: every model trained on clean, well-governed data performs better; every model deployed on solid MLOps infrastructure is easier to monitor, retrain, and improve; every business team with real data ownership is more capable of extracting value from AI outputs.

The LLM moment is real. The productivity gains from document intelligence, internal knowledge retrieval, and code generation are measurable and achievable in months, not years. But LLMs built on a weak data foundation — unstructured internal knowledge, inconsistent data quality, no lineage — will produce the same failure pattern as every AI initiative before them.

The organisations that will extract genuine competitive advantage from AI in the next three years are not those that invest the most. They are those that invest most carefully — in sequence, with discipline, starting with the foundation that makes everything else work.