The drawbacks of benchmark-driven AI research
By Ioanna Lykiardopoulou
When the ImageNet paper was published in 2009, it set a defining ML benchmark, shaping the course of modern AI research and development – from introducing a data-centric approach to paving the way for deep learning’s dominance.
“At the time, this benchmark played a great role,” says Maarten Stol, Chief Scientific Officer at Braincreators. “Having ImageNet’s dataset available triggered a surge in new research for novel methods.”
“But the first drawback with benchmarks,” he adds, “is that eventually we all score very high. As a result, they lose their differentiating power.”
Maarten also points to a related concern: the familiar graphs often shown in AI talks, where model accuracy rises steadily and gradually approaches the horizontal “human performance” line.
Consider, for instance, an image-classification task such as detecting cancer in X-ray scans. “Let's say human performance is at 90%, while a machine learning model performs higher, at 92%,” he says.
Looking at the numbers alone, the conclusion is that the model is better at detecting cancer than a human doctor.
But this, according to Maarten, is misleading. He argues that the most revealing evaluation of performance isn’t what a model or a human did right. It’s what they did wrong.
If a human doctor, for example, misclassifies 10% of the cases, the most likely reason behind this misclassification would be that the examples are too ambiguous or extremely difficult. When a model, however, fails in 8% of the cases, some of them would also be challenging – but others would be obvious mistakes that a trained professional would rarely, if ever, make.
“The fact that the model scores better on the frequency of a benchmark doesn’t mean that it has visual understanding, or that it understands what it’s doing,” Maarten notes. “It only means that it scores well on that particular benchmark.”
“If we combine this with the problem of differentiation, we’re eventually running out of interesting error cases. These are the cases that show us how a model makes mistakes and what we should improve.”
“I think that the sometimes unhealthy benchmarking culture, the hunting for incremental results, only focuses on how much you're doing right. But it's not looking at the quality and the nature of the mistakes that you're making.”
The quest for meaning
But then the question is: if we move from quantity into quality, how will we be able to evaluate it? After all, quality is about interpretation, which is inherently linked to subjectivity – a non-fitting element for science.
“We’d need another leap like ImageNet,” Stol says. “One that would come not only with millions of images, but also with hundreds of thousands of interpretations and a specific, statistical distribution of these text descriptions in the latent space, in their vector database, and so on.”
This direction could be plausible, he adds, but it would require significant research focus. From his side, Stol is researching what he calls “notions of hardness:” what makes a problem or instance hard.
Stol comes back to the X-ray example. What he’s looking for is an explanation of what makes it hard for a machine learning model to accurately identify the scans.
“I think we need to push machine learning research into this sort of area where we start to understand these error cases we’re left with, that 8% that we never solve,” he says.
“What do these error cases have in common? Can we start to group them? Can we start to generate them? Can we start to control them? Because if we can have controlled notions of hardness, we can actually 'benchmark' on meaning instead of on frequency, accuracy, and scores.”
Stol links this to the idea of “model provenance” as proposed in “A Hierarchy of Limitations in Machine Learning.” In that work, modeling is understood as a series of choices that narrow what a system can capture, locking in certain meanings while setting others aside. Once you commit to a quantitative framing, Stol explains, forms of qualitative human interpretation and context fall outside of the scope unless you deliberately bring them back in.
Although the paper is a preprint, it does provide an interesting thought experiment.
“What if, at some point, we could manage this provenance? What if we could track this hierarchy of limitations and say: we have a quantitative model. It’s a black box neural network. It performs well at a predefined task, but around it we’ve also collected a number of videos, audio, and any other qualitative input that has meaning.”
“And maybe there’s a way to reinterpret the quantitative model on the fly, – or in a more science-fiction scenario – even integrate that meaning directly.”
“This is a future dream,” Stol says, adding that he doesn't have the answers. But he thinks it’s important to have a clear vision towards this type of future.
“If we care about ethics and safety, our AI models must be about meaning. They must be about people.”
👥 Every month, we spotlight one member for our community. At KickstartAI, we believe that meaningful progress starts with individuals – builders, researchers, founders, and practitioners – who are applying AI in bold, practical ways. Our Community Spotlight series celebrates their work, their challenges, and the lessons they’re learning along the way.
💡 Want to learn more about the KickstartAI community and our mission to accelerate AI adoption in the Netherlands? Check out our website, listen to our podcast, or follow us on LinkedIn!
