One logistics company spent the better part of a year interviewing vendors for a demand-forecasting model. Engineers argued over transformer architectures. Executives asked about GPU costs and rollout timelines. Nobody, for the first several months, asked to see the shipping records the model would actually learn from.
When someone finally did, those records turned up in 9 formats spread across 6 systems, some of them untouched since 2014. Ask an engineer who has actually shipped one of these systems where the hours go, and the honest answer is almost never the model. That kind of quiet discovery sits at the start of nearly every serious engagement built around artificial intelligence and machine learning development, and it rarely matches what the pitch deck promised. Stripped of the marketing language, the real work of building AI and machine learning systems for a paying client turns out to be mostly about old account records, sensor logs, support tickets, and spreadsheets a company has been quietly accumulating for a decade. Cleaning that material and labelling it, then testing whether it can be trusted enough for a model to learn from, can eat up to 80% of a project's hours. The network itself gets an afternoon's attention by comparison.
Sales decks favour a certain kind of image: a glowing diagram of interconnected nodes, a chatbot answering with unsettling fluency, a dashboard predicting next quarter's churn before anyone asks it to. None of it shows the part that actually decides whether the project works, a data engineer somewhere reconciling 3 different definitions of "customer" across departments that have never spoken to each other.
That mismatch between what gets pitched and what gets built explains a strange pattern showing up across the industry right now. Adoption keeps climbing, and budgets keep growing, yet the money keeps disappearing without much to show for it. Ambition, in most of these projects, has been running well ahead of the plumbing underneath it, and few vendors lead with that fact, because a filing system doesn't sell the way a live demo does.
The gap shows up plainly in the numbers, and the numbers are not flattering. Companies giving up on most of their AI work more than doubled in a single year, moving from 17% in 2024 to 42% in 2025, a shift traced by Fortune to research from S&P Global Market Intelligence. The average firm scrapped roughly half its proofs of concept before any of them reached production. None of this reads like an outlier year anymore. It reads like the new baseline against which every AI budget now gets measured.
Generative AI tells a blunter version of the same story. MIT's Project NANDA spent the first half of 2025 studying more than 300 enterprise deployments and landed on an uncomfortable number: with tens of billions of dollars already spent, only a thin slice of those projects produced any return a business could actually measure, as one industry outlet summarised the findings. The models behind that failure rate were rarely worse than the models behind the rare successes. What differed, almost every time, was the ground they were standing on.
Firms that specialise in artificial intelligence and ML development have started treating the audit itself as the product, not a preamble to it. N-iX, among others working in this space, now runs a formal data-readiness review before any model gets discussed, on the theory that a forecasting tool trained on inconsistent records will simply forecast the inconsistency with more confidence.
Ask what data engineering services actually cover in a serious AI engagement, and the list runs longer than most people expect. It tends to move through a company's data in roughly this order, though the sequence bends to fit whatever mess is already sitting there:
The obstacles this work runs into are rarely about tools. Data leaders surveyed for Informatica's CDO Insights study pointed to a cluster of concerns holding back measurable AI value: security and compliance worries, uncertainty over what counts as responsible AI use, doubts about how reliable the results really are, and a plain lack of trust in the data underneath it all. Trust, in other words, gets earned at the data layer well before it reaches the interface.
Some prospective clients hear about a 6-week audit and lose interest immediately. They came in the door already convinced that AI and machine learning development means bolting a language model onto whatever database they already have, and they wanted a demo by Friday. That reaction tends to predict how the relationship goes if it continues past the first meeting.
The clients who stay are usually the ones who already sat through a failed pilot, or who work with regulated data and know exactly what a governance gap costs later. They tend to be further along than they look at first glance, already burned once by a vendor who promised speed and delivered a prototype nobody could put into production. They come to machine learning consulting conversations asking about lineage and access before they ask about model accuracy. N-iX, like a handful of firms taking this approach, tends to describe the audit stage not as overhead but as the only honest starting point on offer.
Neural networks make for better keynote slides than data pipelines ever will. But the pipeline is where nearly all the real work happens, and where nearly all the real failures start. A data audit will never trend online. It will, more often than the alternative, be the reason a model still works a year after it ships, quietly, long after the keynote is forgotten.
Posted on Friday 16 October 2026 at 11:46
Whilst we have checked the links in this article before publishing we cannot guarantee the content on these sites, please use your own caution and report any issues. All images used in this article remain copyrighted to the original author who's link is placed in the article.
Some links in this article are paid placements.
Click to help with your selection
CONTINUE >>