Exploring multimodal AI models for biodiversity
By Ioanna Lykiardopoulou
Biodiversity is under mounting pressure as a result of climate change and human activity, with species and ecosystems shifting faster than our ability to monitor and understand them. Conservation efforts increasingly depend on new tools to track these changes, identify emerging risks, and inform where interventions are most needed
Against this backdrop, AI is becoming an important part of biodiversity research. Generative AI in particular, together with a new generation of multimodal models, can combine different types of data into unified systems for biodiversity analysis, monitoring, and prediction. Some aim to forecast biodiversity change across entire regions; others focus on extracting patterns from visual data to track individual animals.
Two examples of how this is taking shape in practice can be seen in the work of Thanasis Trantas, AI Research Scientist at TNO, and Rita Pucci, Assistant Professor at the Leiden Institute for Advanced Computer Science (LIACS) at Leiden University.
A foundation model for biodiversity
According to its makers, BioAnalyst is the world’s first large-scale, multimodal foundation model specifically designed for biodiversity. “Most existing foundation models for ecology are focusing on very concrete use cases, usually at the classification layer, but we found none that works as a spatio-temporal forecaster or predictor,” says Thanasis Trantas, AI Research Scientist at TNO.
This was one of the main reasons the team decided to build its own, which they released in June 2025. Their main source of inspiration was Microsoft’s Aurora, a large-scale foundation model for weather and climate forecasting.
BioAnalyst is tailored to biodiversity analysis and conservation planning. It combines 10 different modalities, such as climate, vegetation, and species distribution. It can be fine-tuned for a series of downstream tasks, including habitat suitability assessments and population trend forecasting.
“A key differentiation of our architecture is that we don’t rely on frozen backbones or foundation models to generate representations for each modality,” Trantas explains. “By using Perceiver for both encoding and decoding, we instead take all modalities in their raw form and fuse them directly in the encoder. The idea is to do as little pre-processing as possible, just normalization, and let the model make sense of the raw information itself.”
With this unified setup, the model can capture interactions among modalities at different data scales and distinguish features across domains such as ground conditions, atmospheric processes, and species distributions.
Training challenges
The lack of training data was the first challenge the team had to address. “When we started, we came into a wall,” Trantas says. “There was no data that could facilitate the pre-training or fine-tuning of the model. We then decided to create our own dataset.”
The team built BioCube, which compiles multiple datasets (from atmospheric and climate to edaphic and species distribution variables), all grounded to the same spatiotemporal system. It spans 20 years, with data from 2000 to 2020.
Another major challenge was access to computing power for training and development. The team managed to secure access for one year and a half to Snellius, the supercomputer located at SURF, via an NWO grant.
Trantas explains that the compute resources available to them – in terms of hours of use, consultancy, and storage capacity – shaped significantly the model’s development.
One example is BioAnalyst’s resolution. “Imagine that for 20 years of data, looking at a temporal dimension of 6-hours and 0.25 degrees (~27km) of spatial grid sampling for multiple data modalities, the storage requirements can reach up to more than 1,000 terabytes,” he says. “We chose a different approach and upsampled our model to support a monthly resolution, as in any case there aren’t too many biodiversity observations that change day to day.”
Trantas adds that they needed to build two different architectures: a big one to actually pre-train the model, and a smaller one to use for debugging and to run experiments faster. “We also had to have a distributed training framework, so that the model could be distributed in multiple GPUs and compute modes, to be trained faster.”
Creating a baseline for future models
With no existing benchmarks, the team evaluated BioAnalyst using standard metrics, ecological measures, and comparisons with baseline machine learning models and Aurora, showing the model generalizes well across diverse biodiversity prediction tasks.
Current limitations include BioAnalyst’s specific focus on European land-based biodiversity as well as the model’s deterministic nature and the challenge of uncertainty quantification of its predictions – all of which can be part of future work.
“The innovation here isn't the model itself. It's the engineering of the whole system that can produce an adequate prediction,” Trantas says. “We don’t have the perfect prediction results yet – and there aren’t any to begin with.”
“What we want to be is the baseline and we invite people to start building on top of what we created.” This is why the team has open-sourced all parts of its work, including their preprint, the model codebase, the fine-tuning techniques, the model weights, and the dataset creation codebase.
Bio-inspired computer vision for population monitoring
Identifying individual animals in the wild is central to biodiversity monitoring. For species like tigers, zebras, leopards, and zebrafish, unique skin patterns, such as stripes and spots, offer a natural biometric signature. Yet teaching AI systems to reliably recognize individuals from these patterns is no easy task.
“Individual identification through skin pattern formation is a key method for population monitoring,” says Rita Pucci, Assistant Professor at LIACS. “But automating it is challenging. In images of animals in nature, you have to tell the model that the background doesn’t matter, the species doesn’t matter – only the skin pattern matters.”
“During my post-doc, I tried traditional methods, like convolutional neural networks, but they didn’t work. The model was focusing on the background. Cutting out the animal didn’t help either.”
Pucci is now following a different approach in a PhD project she’s supervising, titled “TATTOOs: Pattern formation through bio-inspired computer vision,” with PhD student Jens van Bijsterveld.
“We need to make the model focus on patterns alone to identify individuals, which means that it needs the knowledge necessary to understand these patterns,” she says.
To do that, the project focuses on building a machine learning model, based on computer vision, that integrates the mechanisms of pattern formation directly into its learning process. In practice, this means incorporating a mechanistic model – specifically the Turing reaction–diffusion model – to explain to the ML system how patterns form.
Architecture and training
Pucci describes the model as an autoencoder architecture consisting of two modules: an encoder which focuses on extracting features from the pattern in the input image (close to representational learning); and a decoder which uses these features to learn how to reconstruct the visual representation of the pattern as in the input image. These processes happen simultaneously during training, allowing the model to develop knowledge about the different skin patterns.
After training, the decoder is removed. “That’s because we’re not interested in generating images for individuals,” Pucci says. The encoding part that remains is the most important one for the project: it retains all the necessary pixel information to represent and regenerate the pattern.
These encoded features are then used as classifiers. The model learns to cluster the slightly different representations for the same individual animal under a single label.
The team is currently applying this method to zebrafish, a species for which extensive data is available. Large video datasets allow them to track individuals and assemble thousands of labeled images per fish, a scale necessary for developing and validating a pattern-focused representation.
Key challenges
When asked about the technical challenges involved in developing this model, Pucci begins with the training images. “We’re studying fish and underwater images aren’t ideal, even in the best possible setup of a fish tank,” she says. Meanwhile, all data comes from video frames, which introduces its own complications. A significant issue is the movement of the fish, changing the angle and shape of their body. As a result, the visible pattern also shifts slightly in every frame.
Still, the team chose zebrafish for a practical reason: compared to other maculate animals, they make it possible to build much larger datasets. “It’s much easier to collect enough images of 200 zebrafish individuals than, say, 200 tigers,” Pucci notes.
The integration of a mechanistic model inside or outside of the main ML model is another big challenge, as the two perform "completely different” functions. On the one hand, the mechanistic model is rule-based and time-dependent; on the other hand, the ML model is an adaptive system that learns through optimization. “The combination of different types of data – in this case, images and information from the mechanistic model – is also an open challenge in multimodal models like ours,” Pucci adds.
“These are big, but not impossible challenges,” she says. The team is currently at the early stages of research, running ablation studies to evaluate the contribution of the mechanistic model. If the project proves successful, the model can be used for population monitoring combined with the detection of related issues, such as inbreeding.
Pucci herself is particularly interested in various types of multimodal models, such as those that can fuse heterogeneous types of data like images, audio, and DNA sequences. She adds that interest in multimodal models for biodiversity is gradually increasing. “We now have enormous amounts of data we can use to train these models and we also have state-of-the-art technology already tested in medicine.”
