Why (and how) we need to measure data diversity in language models
Recent advances in language models are typically credited to improvements of architecture and ever-larger datasets. But a parallel conversation has been building – in research labs and across industry – around a more fundamental question: not just how much data these models are trained on, but what that data actually contains.
Within this context, Dong Nguyen – Associate Professor of Computer Science at Utrecht University – points out that data diversity is frequently cited as an important factor, and as one of the main reasons why models may show bias or fail on unusual cases during interaction.
“But when you take a step back,’ Nguyen says, “it’s very unclear what we mean with data diversity.”
The term, she notes, can refer to many different things, and without the necessary theoretical and practical insights, it’s difficult to make substantial claims about data diversity’s role. “How much does diversity matter? Which types of diversity are relevant? At what stages of training does diversity matter the most, and does its importance vary across tasks?”
These open questions are at the core of DataDivers, the research project Nguyen is currently leading. Funding by the EU and part of the NLP and Society Lab of Utrecht University, the project aims to examine how data diversity can be defined and measured in datasets, how it relates to the behaviour of models, and how it might be taken into account to develop fairer and more robust models.
“We’re focusing on LLMs at the moment, but a lot of the things that we do are quite general,” Nguyen notes, adding that the methods are intended to apply more broadly across Natural Language Processing (NLP) beyond large language models alone.
Measuring data diversity
According to Nguyen, the team’s main goal is to develop different methods that make it possible to measure data diversity more systematically. At the moment, their work focuses on two broad axes. The first is what she describes as stylistic diversity.
“Within any language,” she explains, “people don’t always write or speak in the same way. Expression varies depending on background, but also on the situation: writing a job application, for instance, is very different from speaking with a friend. The question then becomes how this kind of variation is represented in datasets, and whether it can be measured in a meaningful way.”
The second axis is semantic diversity, which relates more directly to the meaning of text. This includes variation in topics and subject matter, and how such differences are distributed across a dataset. Although the project will later look at other types of diversity, these two were chosen first because they’re expected to be relevant across many different applications.
From a technical perspective, Nguyen explains that the approaches to measuring stylistic and semantic diversity are quite similar. The core idea in both cases is to map texts into a vector space. For semantic diversity, texts that are similar in meaning end up close to each other in that space. For stylistic diversity, the goal is comparable: texts written in similar styles are placed near each other, while texts written in very different styles are positioned further apart.
“In this way, a dataset can be represented as a large cloud of points in a space,” Nguyen says. From there, it’s possible to apply different metrics to quantify diversity.
“We’re currently trying to categorize the different ways of measuring diversity and look at the pros and cons of each metric. At the same time, we’re setting up more systematic experiments to investigate which metrics work better than others and to formalize them further.”
Potential impact on models
Once clearer metrics for data diversity are in place, they can be used to investigate more systematically the impact of data composition on model behavior, Nguyen explains.
“If we vary the diversity in the dataset along different axes,” she says, “it becomes possible to study whether this influences how well the models perform on certain tasks or the kind of linguistic variation present in their outputs.”
Building on this, Nguyen ties data diversity more broadly to the learning process of language models. She compares it to human learning: when acquiring a new language or skill, it would be limiting to rely on a narrow set of examples, instead of exposure to a broad and varied range of material.
Similarly, she expects diversity in training data to improve the learning process and points to early – though still preliminary – evidence that better and more diverse datasets may allow models to learn more effectively with less data.
From this perspective, a related goal of the project is to support the creation of better datasets from the outset. “What often happens nowadays is that, once a model's released, we discover all kinds of problems.”
“But,” she argues, “it all starts with the data. If we can understand the data and its influence on model behavior better, we could intervene earlier, before the training and deployment phases.”
At present, however, many decisions about how datasets are constructed and adjusted are still guided by heuristics and remain an “ad hoc” practice. This is partly due to constraints in data availability, but also because, as Nguyen notes, it is difficult to investigate these questions systematically without concrete frameworks or tools to measure diversity in the first place.
Challenges
Nguyen points to several challenges when attempting to measure data diversity in practice. A first difficulty is conceptual: what actually constitutes a good diversity metric is not always straightforward. “There isn’t just one definition, is there? It really depends on the specific application.”
A second challenge concerns scalability. “Many metrics that are theoretically appealing don’t scale well to the large datasets used for LLMs,” she adds, which might call for certain trade-offs between theoretical elegance and practical usability.
Finally, there is the issue of missing information. Many important axes of diversity, such as dialect or demographic representation, are simply not labeled in most large datasets, which are often collected from the web without rich metadata. “While it may be possible to infer some of this information automatically, the absence of reliable annotations makes it harder to develop fine-grained measures of diversity,” Nguyen says. “This is why our project focuses primarily on metrics that don’t rely on metadata."
But these challenges, she argues, aren’t insurmountable. The team is currently at the first stage of the project, which will result in the release of a software library, containing tools and methods that can be used to measure data diversity. The focus will then shift to investigating the impact of data diversity on model behaviour.
“If we can understand that link better, we can create fairer and more robust models that can handle different types of output more effectively,” Nguyen says.
