How Data Hydration Enables Scalable and Trusted AI

A large, realistic water droplet suspended in the center, filled with scattered binary code (zeros and ones). Binary code also surrounds the droplet, suggesting raw data. The text 'What is Data Hydration?' is at the top left. The image visually connects the concept of water/hydration with digital data, symbolizing the cleansing and enrichment of data for AI.

It’s accepted wisdom that AI models require access to extensive quantities of data for model training so that they can identify patterns, understand context and reduce bias, so helping the models to perform inference and produce accurate predictions.

For sure, there's an awful lot of data in this world - one prediction suggests there will be 181 zettabytes of it by the end of 2025. But to be useful, AI models need to be trained on data that is appropriately high quality, and that is in much more limited supply. Enriching data to a quality level that’s suitable for AI requires a set of processes that have become collectively referred to as Data Hydration.

So what’s the quality problem with data? For starters, datasets are often not sufficiently accurate. They can contain erroneous data elements (sometimes created on purpose) that have not been validated and cleansed (by removing data or correcting it) - activities that are time consuming and costly to perform. 

Furthermore, datasets are often not complete, either because specific data is hard to come by, or because the owners of the data want to keep it private for personal or commercial reasons. AI model builders are already finding it increasingly difficult to harvest high quality (free) data from the public internet, forcing them to enter into commercial deals with content owners, such as media companies, book publishers, healthcare providers or retailers.

Commercial arrangements often require model builders to comply with governance controls and implement provenance functions to ensure that data sourcing and enrichment is tracked, and that data itself is leveraged ethically, according to contractual terms.

Understandably, training AI models on erroneous or incomplete data can cause them to produce outputs that are below par. Outputs can be biased or include hallucinations. In such cases, the models can generate outputs that are misleading, or even nonsensical, but which are presented as facts. All too often, the impression is that AI models guess what a response should be, though they don’t always state that it’s not certain when presenting results.

Sometimes the best route to compiling a complete dataset is to feed it with synthesized data. But this process often requires the use of  cutting-edge but power-hungry cryptography to ensure that the synthesized data resembles actual data but does not inadvertently reveal it, or the owners of it.

Even when datasets are accurate and complete, they may lack organization or structure, making it difficult to extract data elements to use for AI processing. This includes determining the relationship between data elements and the context of what the data might relate to. 

Data organization approaches - including contextual labelling, construction of ontologies and creation of knowledge graphs and time series - can be adopted to organize data in a way that is optimized for efficient AI processing.

And efficiency is important given the high cost of setting up and running AI infrastructure - including costly GPU chips, datacenter facilities, power costs and water costs (for cooling). Scaling AI to meet the demands of business and consumers is a constant challenge, especially as interest in models to power AI agents or to generate video is rapidly increasing. 

Meanwhile, AI model builders are still attempting to determine what business and consumer customers will actually pay for AI services. There is no doubt that many AI services provide value and make humans more efficient, but quantifying that value to set price points is tricky, especially in a competitive market.

Enter Data Hydration, an emerging term that was coined by venture capitalist Gerry Buggy. His firm, Iona Star, is focused on investing in technologies that operate at the convergence of AI and access to data. A key focus of the firm’s investment thesis is minimizing the cost base of delivering AI to customers that want to use it.

So where does the hydration word come from? In nature, hydration is the process of absorbing essential water, or the state of an organism being supplied with adequate water to thrive. Relatedly, Data Hydration covers all of the activities that support AI models being supplied with an adequate supply of essential data to thrive. Such activities include:

• Validation and Cleansing: Ensuring data is accurate by eliminating (and correcting where possible) errors and bias.

• Enrichment and Organization: Combining data inputs and leveraging synthetic data, to fill critical data gaps, and leveraging ontologies, knowledge graphs, and time-series to organize and add context to datasets.

• Governance and Provenance: Establishing transparent data sourcing and audit trails for data ownership and usage compliance and to ensure ethical AI.

The technology landscape of companies engaged in developing Data Hydration tools and services is expanding as demand from AI model builders and service providers come to understand the data challenges associated with delivering AI efficiently at scale, and which is trusted by customers.

While Data Hydration is a new term to cover a set of disciplines that have often been addressed separately in the past, practitioners are increasingly considering holistic approaches to the data quality needs of AI. So stay tuned and watch this space.

In the meantime, you are encouraged to join our new LinkedIn Group for Data Hydration to learn from peers and contribute your own experiences and viewpoints on delivering scalable and trusted AI.