Navigating the future: how AI is transforming autonomous systems

By Paulette van Bossum

Autonomous systems have evolved from futuristic concept to increasingly common reality, thanks largely to advances in AI.

From self-driving cars to agricultural drones and industrial robots, they are set to reshape how we live and work. They, of course, use advanced hardware, but the real breakthrough is the AI that drives their decision-making.

But autonomy is not binary — it's a spectrum. And as our experts emphasize, the most successful autonomous systems combine AI with human supervision, particularly when safety is a critical concern.

To explore this fast-changing field, we talked to five experts from across the autonomous systems landscape in the Netherlands:

Albert Maas, founder of Avular, which develops autonomous robots and drones for specialized industry applications.

Andras Palffy, co-founder of Perciv AI, which develops radar perception technology for autonomous vehicles.

Justin Spratt, Chief Business Officer at Vay, pioneering remote driving technology for driverless car sharing.

Mehdi Dastani, Professor chairing the Intelligence Systems research group at Utrecht University, specializing in multi-agent systems.

Martijn Wisse, Professor and Chair of the Department of Cognitive Robotics at TU Delft, focusing on bio-inspired AI and robotics.

What You'll Learn:

• How AI is transforming autonomy across industries — from transport to agriculture

• Why the future of autonomous systems involves AI and humans working together

• Why bio-inspired AI could lead to more efficient autonomous systems

• The challenges of implementing AI in real-world environments

• The regulatory and ethical considerations shaping the field

The evolution of AI in autonomous systems

"Initially, AI was about modelling — trying to model individual faculties of humans, like how we see things, how we reason, how we learn, how we walk, how we talk," explains Mehdi Dastani, professor at Utrecht University. "But in the 90s, the focus turned to combining these techniques to build autonomous systems — systems that are embedded in an environment and make decisions autonomously."

This shift to more integrated systems marked a turning point, but progress was constrained by computing limitations, sensor capabilities, and algorithmic approaches. Early systems required carefully structured environments and extensive human programming.

But the last decade, and especially the last five years, has brought dramatic acceleration in autonomous systems. Why? AI. 

As Albert Maas of Avular notes: "You see such acceleration in jobs that can be done by AI now. It's mind-boggling how quickly it's going. Every quarter there is something groundbreaking and new."

This has allowed autonomous systems to emerge from research labs and controlled industrial settings into more complex, real-world environments.

It has also led to a spectrum of autonomous approaches depending on the environment in which the systems are deployed. At one end are systems that rely totally on AI, with minimal human intervention. At the other are hybrid systems that blend human control with AI enhancements.

"We use a lot of AI, but we're not big advocates of full neural networks," Maas explains. "Some companies, like Tesla or Figure AI, go from what they call 'photon to action' — the photon into a camera that goes through a complete neural network and creates an action. It works, but it has limitations because it takes a lot of computing power and it is a complete black box."

So Avular takes a more hybrid approach: "We think the physical world needs safety layers. So we take another approach where you have a lot of prior knowledge, a database and AI enhances that. And people are still important too."

Human supervision remains important for many autonomous systems right now. Vay's approach exemplifies this human-AI collaboration model with their remote driving technology. "For pickup and drop-off, we use a human Remote Driver. We've built technology that abstracts the human from needing to be in the vehicle. Then AI software supports the human driving the car," Spratt explains. 

Andras Palffy, co-founder of Perciv AI, observed how perception technologies for autonomous systems have evolved: "For AI, you need a lot of data, and you need good-quality data." He explains that recent advances in AI tools and techniques have made data collection and processing more efficient. His team is now pushing to develop foundational AI models for radars, similar to the models that have transformed other domains.

A bio-inspired approach offers another promising recent direction. As Wisse explains: "My group is researching a novel cognitive theory about how the brain works, and trying to copy that into algorithms for robots.”

According to all our experts, the current state of AI in autonomous systems reflects a balance between technical realities and societal concerns about safety and reliability. The approaches tend to combine systems that leverage the strengths of both AI and humans.

Innovations and applications

AI-powered autonomous systems have amazing potential. But we wanted to know about real-world applications that are making an impact right now. Here's what our experts shared:

1. Perception: the eyes of autonomous systems

For autonomous systems to function effectively, they first need to understand their environment. This requires sophisticated perception technologies that are powered by advanced AI. As Albert Maas notes, "The interpretation of the world — knowing that this is a door, that this is a human, all the visual kind of processing commands — are very important and interesting."

Perciv AI specializes in radar perception technology. "Radars are significantly cheaper than cameras and LiDAR. We're talking orders of magnitudes cheaper," Palffy explains. "And they work in bad weather — snow, rain, smoke, fog. Radar doesn't care — but cameras and LiDAR really suffer there."

"It's also easier to integrate. You can hide radar in the bumper, for example, and you never see it. You can’t hide a camera or a LiDAR scanner. It’s not just about aesthetics, you cannot scratch it, or if there's mud on it, radar still works. Imagine a dirty construction environment — you can’t expect workers to clean the LiDAR every hour."

While the radar hardware itself offers advantages in adverse conditions, it's Perciv’s AI that transforms raw sensor data into meaningful information. Their AI system translates sensor information to an understanding of objects and their relationships:

"We use AI to interpret the radar data: that's a pedestrian, that's a car, that's a cyclist," Palffy explains. Their approach combines various AI technologies, from convolutional networks to transformer-based models and graph neural networks, depending on the specific use case and computational constraints.

Palffy notes that working with radar data presents unique challenges: "Radar data is simply very weird. Sometimes it's a point cloud, but with different channels. Sometimes it's an image, but with a different representation than a camera." This is why they're developing their own foundational models specifically for radar perception, to make radar data more usable for autonomous systems.

Palffy's team also uses AI throughout their development process: "We use AI to make AI. We record data and use AI to annotate it, to label it, and to mine for good examples and difficult scenes."

Palffy shared how Perciv AI's technology is being applied in the real world:

"We have a drone project that is in production now — an agricultural drone with a collision avoidance system that uses our software," he explains. "You literally cannot crash it — you try to push and crash into the tree, but our AI won't let you."

Another application involves airport shuttles: "We work with airport shuttles companies who try to take either luggage or people from one location to another at an airport," Palffy notes. "When it's foggy or raining, LiDARs and cameras struggle. What we provide is a better perception where false positives significantly drop, while the shuttle still stops when it has to."

Developing these capabilities presents unique challenges: "The electronics and processor architecture are important," Maas notes. "You now see more and more dedicated chips for certain problems. There are chips to estimate the depth in the world using low computing power. That's very handy for a robot."

However, Wisse points out that current AI perception systems still struggle with unexpected scenarios: "Current robots don't recognize that this is a novel situation, and they just apply whatever rules they have. They are not always aware that it doesn't make sense and sometimes they make big, big errors." 

2. Remotely driven cars: the human-AI hybrid approach

Vay's AI-supported Remote Driving technology enables a new kind of car-sharing service that removes a major point of friction: the need to travel to pick up the car and park it when you’ve finished.

"Our initial go-to-market is driverless car share - we believe that the car share market could be a multiple of itself, as we take the friction out of going to pick up and park the car," Justin Spratt explains. Vay’s service is already operational in Las Vegas with nearly 50 cars.

Unlike fully autonomous vehicles that rely on complex AI systems to make all driving decisions, Vay operates a Remote Driving car-sharing service where professionally-trained Remote Drivers control the vehicles from a distance. They sit at a Remote Driving Station with a steering wheel, pedals, and other vehicle controls that have been developed to meet the highest automotive industry standards. The car surroundings are reproduced via camera sensors and transmitted to the screens of the Remote Driving Station. Road traffic sounds, such as emergency vehicles and other warning signals, are transmitted via microphones to the Remote Driver’s headphones.

How it works: Users can find and request an electric car through the Vay app. A Remote Driver then delivers the vehicle to the customer's location within minutes. Once the car arrives, the customer takes over and drives it like a regular vehicle. 

The technology faces technical challenges that AI helps address. And their AI performs multiple supporting functions: "We can pick up emergency services way before it's audible or picked up by the driver," Spratt explains. "We know where these vehicles are and can make sure we drive around them, as well as avoid getting into traffic jams or more difficult situations."

Unlike fully autonomous vehicles, which remain expensive to operate, Vay's approach offers a practical middle ground — a practical compromise that's economically viable now, unlike fully autonomous vehicles: "Fully autonomous will compete with ride-hail as it is today," says Spratt. "McKinsey reports that full autonomy (self-driving cars) costs $8 per mile. And Uber comes in at around $1.50 to $2.50. We're price sensitive - and our promise is 50% below ride-hail because we don't have a driver seated in the car."

3. Specialized robots and drones: finding success in specific tasks

As Martijn Wisse observes, "The field of robotics is not ready for general purpose robots, so current success stories are always coupled to application-specific robots." In other words: companies are creating specialized autonomous robots to tackle well-defined challenges.

Avular is a great example of this approach. Albert Maas shared how they are working with industry partners to develop robots for very specific tasks.

One example is Deep Forestry: "The drone flies autonomously through the forest. They have AI algorithms to detect whether the bark is okay, what kind of tree it is, to get an inventory of the forest and whether to cut them or wait a year, to make that much more insightful."

Another is in agriculture: “We’ve created a robot with smart technology that naturally protects crops from diseases, reducing the need for pesticides.”

Martijn Wisse points to several successful Dutch companies that have found success by focusing on specific niche applications:

"We have quite some successful examples like Lely, the cow-milking robot company. They are the world market leader. They did the same thing. They just had one product - the cow milking robot. But now they're spreading out their robots everywhere on farms," Wisse explains.

He also highlights another success story: "Some students here at TU Delft started a decade ago with a robot that cleans the outside of a ship. It really has just that one task. But it saves the shipping industry a lot of money, because they can do the cleaning while at harbor. They are a super successful company now as well."

4. Multi-agent coordination: AI systems working together

While individual autonomous systems are impressive, many real-world applications require multiple systems working together. This is where multi-agent systems become crucial, as Professor Mehdi Dastani explains.

"The autonomous car will be on highways eventually, and there are also other autonomous cars. Then you start to think about how these autonomous cars can help each other, or can coordinate their actions," Dastani says.

He offers some concrete examples: "Platooning is one kind of collaboration, where they can connect to each other and form a kind of platoon. If they come from a side road and they have to merge, they can communicate to see who gives rights to whom and which speeds they need, to make the general traffic more fluent."

The challenge of developing these coordination capabilities is addressed through multi-agent reinforcement learning (MARL), where multiple systems learn simultaneously how to act in a shared environment.

"Reinforcement learning is a machine learning technique that tries to learn which action to take in which situation," Dastani explains. "In multi-agent reinforcement learning, the question is how different agents can collaborate so that they learn their policies in a situation where other agents are also learning what to do. If you learn to go left and the other one learns just to go right, it might lead to an accident, so they have to learn in a coordinated way."

This approach has applications far beyond just traffic management. As autonomous systems become more common across different domains, their ability to coordinate effectively becomes increasingly important — whether they're drones monitoring crops, robots in a warehouse, or vehicles navigating city streets.

5. Bio-inspired AI: more efficient autonomous systems

Martijn Wisse, whose research at TU Delft focuses on biology-inspired robotics, suggests that the human brain offers lessons for developing more efficient AI for autonomous systems.

"The biological brain is interesting because it's super efficient. A lot more efficient than current AI algorithms," Wisse explains. "The way it does that is by working a lot with prejudice — or prior knowledge."

"The brain is very good at taking a lot of prior knowledge in the form of an internal mental model of how the whole world works, and then requiring only a little bit of sensor data. You need to see something only very briefly, and you already know what's there, because all of the rest is painted by the internal model in your brain. This is super effective. Sometimes, of course, it leads to small problems. It’s why a magician can fool us, for example, because they count on our brain doing this. It's why we don't see the hidden movements that the magician makes."

This approach could help overcome one of the biggest challenges in the field — handling the complexity of unstructured, real-world environments. This is particularly valuable for autonomous systems because, as Wisse points out, "In the factory, robots work well because everything is structured, everything is predictable. But in the real-world, out there, the amount of variety is enormous. So it's impossible to make a training set of data that has seen every different option already."

His team is working on translating these biological principles into practical algorithms: "We are following Karl Friston, a leading neuroscientist who has translated ideas about biological brains having an internal model into mathematics, and it connects very well to Bayesian inference. Bayesian probabilistic theory says how you can calculate with uncertainties and beliefs, and how you can combine prior beliefs with new information."

These examples demonstrate that AI-powered autonomous systems are already providing value in specific niches. However, more general applications, like fully self-driving cars and humanoid robots, remain challenging.

Technical challenges 

Implementing AI in autonomous systems involves overcoming numerous technical challenges. Our experts highlighted several key areas where innovation is still needed.

1. Data

Data quality and availability is fundamental. As Andras Palffy explains, "For AI, you need a lot of data, and you need good quality data. One of our secret sauces is that we are really good at recording our own data."

2. Compute

Computing requirements present another significant challenge. Albert Maas highlights how his team addresses this:

"It has a lot to do with optimization, but also prior knowledge. Only using these AI algorithms when and where needed. Then you can also make them smaller, because you also have neural networks that are very small but do a dedicated thing, so they can go through a loop very quickly."

3. Latency

Latency is a particular concern for Remote Driving. As Spratt explains, "The real challenge is the latency between the car and the Remote Driver." To address this, they've developed sophisticated networking solutions: "We have four modems in our car, so we can theoretically run across four different mobile networks."

4. Rule-based approaches

Dastani points out the limitations of purely rule-based approaches for autonomous systems: "Autonomous systems can be designed as rule-based systems that encapsulate explicit domain knowledge, but that can't really be scaled-up, because most of the time we cannot capture the dynamic or changing environment in terms of explicit rules."

That's where machine learning comes in, but it brings its own challenges, particularly around safety and verification: "With the machine learning approaches, usually it would be very difficult to make guarantees that this car will never cause this type of accident," Dastani notes.

His team is developing approaches to address this: "What we try to do is build a shelter on these learning systems to prevent them from doing bad things. It's like teaching a child to ride a bike — as the parent of a child, you run beside them to see when you should intervene to prevent a bad event."

Safety, ethical and regulatory considerations

As AI-powered autonomous systems grow more common, ethical and regulatory questions are becoming more important.

Part of Mehdi Dastani's research focuses on accountability and responsibility:

"If we want to find out what has caused what, or who is responsible for a bad outcome — or in a multi-agent setting, which subset of agents or systems have been responsible — you should have a very concrete notion of responsibility."

This requires collecting specific information: "What kind of information do we need to identify the responsible systems?" Dastani emphasises that correlation isn't sufficient: "We cannot determine who is responsible for a bad outcome by just relying on correlation between events. We should really find out which event caused what, or which action caused which outcome."

Andras Palffy emphasises the importance of developing systems that balance functionality with privacy, highlighting a unique advantage of radar-based AI perception: "Radar doesn't record your face. It's really hard to recognize you in radar data, but it's still enough to not crash into you. So maybe that's a direction where robots still see us but don't record us in a way that's intruding."

On the regulatory front, the European Union is developing frameworks to govern AI and autonomous systems. The EU AI Act categorizes AI systems based on risk levels and establishes different requirements for each category.

"They basically characterize Intelligent Systems, or AI systems, at different risk levels," Dastani explains. "They also start to regulate by prohibiting certain types of AI systems and marking the others as high risk or low risk."

For companies developing autonomous and remote driving systems, navigating regulations presents both challenges and opportunities. Justin Spratt describes Vay's experience with the regulatory process in Germany:

"Our founder, Thomas, was told he was mad to try to build a global driverless business out of Germany, " Spratt recounts. "But he was embraced by the German regulatory authorities in Hamburg. And now Hamburg has positioned itself as the urban mobility center of Europe."

Despite initial support, the process hasn't been straightforward: "We have a driverless car running in Hamburg in Germany, which is unheard of, but we cannot collect money on that yet. So it's the commercial piece to this that's still being worked out."

Albert Maas contrasts the cautious European approach with China's: "China said in 2015 already that they wanted to be the biggest in robotics. And they succeeded. They have fewer rules for testing and deploying AI in robots. You can go to Beijing and order Starbucks wherever you are, and a drone will come and drop it off."

This creates a complex dynamic: "I find it difficult, because I think how the European Union does it according to the right norms and values, but there's so much commercial and economical gain in removing some of the regulation and friction. But then it might be a reason for us to lose grip on society."

Martijn Wisse sees the cautious European regulatory approach as generally positive: "It's really good to be careful with this technology, because the more powerful the technology, the more quickly a small one-person's mistake can affect many people negatively."

He expresses concern about less regulated approaches: "I'm a bit afraid that somebody in the US might make a mistake that affects the whole field. It's not like robots are going to take over the world, but robots and AI, have a lot of reach, a lot of effect. An error somewhere in one of the optimization criteria might have a big effect worldwide."

The Dutch ecosystem

The Dutch ecosystem, with its combination of academic excellence, innovative startups, and collaborative approach, is well-positioned to play a significant role in autonomous systems development.

As Martijn Wisse's examples of successful Dutch robotics companies demonstrate, sometimes the key to success lies not in building the most advanced or general-purpose system, but in focusing on practical applications that solve real problems and create tangible value.

Martijn Wisse highlights the unique strengths of the Dutch environment: "I would say we compete when it comes to the insights and the ideas, and also when it comes to our network infrastructure, education, the quality of the students that we have access to. I think all of that is highly competitive with any region in the world."

However, he also notes challenges: "At this moment, it is a little bit difficult to really know where the main developments are going to be made. Is that university or are they big tech companies? And I have the feeling that we don't have enough big tech companies in the Netherlands at this moment."

Albert Maas echoes this sentiment about capital: "I think the talent and the technical universities [are the biggest advantages]. Talent is here. We can use it." However, he admits, "When it comes to fundraising, that's difficult if I can be honest. Looking at China, they just invest from the government side. And in the US, there's just much more risk capital."

Despite these challenges, Wisse remains optimistic about the Dutch approach: "In the US, everybody just starts companies, and then most of them fail, and then they go on. And here you really think hard, and you start a company, you're very careful, and it goes slower, but not so many fail. I started two and neither of them failed. That’s just a snapshot, but still, it's a different approach, and who knows — maybe in the long term, it's even a better approach for autonomous systems."

When will we see autonomous vehicles in NL?

When it comes to autonomous vehicles in the Netherlands, our experts noted some uncertainty and challenges. None could provide a specific timeline for when fully autonomous vehicles might become widespread on Dutch roads. 

Andras Palffy has a positive view: "I don't think anything is holding back the Netherlands [with autonomous vehicles] compared to the EU in general. We have the talent, and actually the government agencies (RDW, Rijkswaterstaat) are very proactive on the topic."

Justin Spratt from Vay, which operates in Hamburg and Las Vegas but not yet in the Netherlands, mentions specific hurdles they've faced: "We are engaged with the Netherlands, but it's tricky. We'd love to be more heavily engaged and operating here. We've made some good progress recently, but there is still work to do to achieve the right regulatory framework.” 

This suggests that while the Netherlands has the technological capability and talent to develop self-driving cars, there may be regulatory or market hurdles specifically for autonomous and remotely driven vehicles that companies need to navigate before we see widespread adoption here.

The future of AI in autonomous systems

Looking ahead, our experts shared their visions for how AI-powered autonomous systems will evolve.

Justin Spratt suggests that while fully autonomous vehicles may eventually arrive, the more immediate future belongs to systems with human oversight:

"We see a world where there's pretty much always a human in the loop. Robots will take the burden off humans, but we think people still have a big role to play, probably at least for another decade, either through remotely driven operations or remote guidance."

Palffy agrees: "I'm a huge fan of AI assistance, driver assistance. I did multiple academic experiments where I measured that it's simply better than a human driver," he explains. "For that, you don't need a fully self-driving car. You need a system that reacts when trouble happens."

This approach addresses both technical limitations and societal readiness: "Even if the technology became available, and we flew down the cost curve, would we want robots doing this without any human oversight? I don't know if we're there yet as a society."

Mehdi Dastani envisions a gradual adoption of self-driving, starting in more controlled environments:

"I hope to see them earlier on the highways rather than on the streets, because highways are more controlled environments," he notes. "But the problem is the transition period. Human drivers create a lot of unpredictability on the roads that autonomous cars might not be able to deal with."

Albert Maas sees autonomous systems transforming almost every industry:

"With the labor shortage around the whole globe, and a lot of work that people don't want to do anymore, I foresee a crazy boost in these kinds of devices," he predicts. "It won't be winner-takes-all. There will be specialized robots in agriculture, healthcare, restaurants, waste departments."

Martijn Wisse provides a contrarian view on the current hype surrounding humanoid robots:

"I started my career by building humanoid robots. But I lost my faith in it," he admits. "There's a strange hype around humanoid robots at the moment, but I think it's going to die out again. All of the investors are gambling that their investment might break through and become the general purpose robot. But I don't believe it is time yet."

Instead, Wisse foresees a future dominated by more specialized robots: "I think that we're going to see lots of specific purpose robots now, one for your task, one for that task. And most of these tasks are in environments where wheels are enough."

Like Maas, he predicts a trajectory where specialized robots achieve commercial success first. But that might then evolve into more general-purpose systems: "Robotics requires a long-term commitment to focus on one product and use-case to make it succeed. Then after, say 10 years, the underlying technology will have evolved so far that suddenly, from a strong position in one market, they branch out and become the next big thing."

He provides a specific example of how this might evolve into a winner-take-all scenario: "It might be John Deere that does autonomous tractors, and at some point takes the whole market of autonomous outdoor machines. So there will be a winner-take-all moment, but not yet. That's what humanoid robot investors are hoping for now — that a clear winner will emerge. But I think they're too early."

Advice for AI professionals

For those interested in working on AI for autonomous systems, our experts offered advice based on their experiences.

Albert Maas emphasizes the importance of focusing on real needs:

"Use case, use case, use case. Get somebody that knows what he wants, then you can deploy all your technical knowledge to that use case and create business and grow," he advises. "I think that's the biggest pitfall –—building something for the technology, or just for the sake of it."

Andras Palffy stresses the importance of adaptability and understanding the bigger picture:

"Our field is changing so rapidly. And coding just became so much easier. It's crucial to stay agile and to learn every day," he notes. "The skills you need these days is the approach, or how you think, not 'how do I import a library in C++?' because in a day or two, it won't be relevant, or an agent can do it instead of you."

What will remain valuable, he suggests, is higher-level thinking: "What we will still need is smart people who can delegate tasks, can make sure that it works, have the overall picture, the thinking, the logic behind it."

Mehdi Dastani sees the integration of advanced language models as an exciting area for new developers:

"These language models, like ChatGPT, are used massively by companies. But I also see that these will be used as models for autonomous systems," he explains. "Some companies are building multi-agent systems based on different pre-trained models. Every one of these agents are large language models that can answer one question, and together they can also answer very complicated and combined questions."

Martijn Wisse has already implemented this approach in a practical way and suggests people just experiment with things like this: "I wrote a small connection to ChatGPT, so whenever I type something, it's as if I'm typing to the robot," Wisse explains. "I say, 'Hi, robot, how are you doing?' Your robot says, 'Oh, I'm fine. What can I do for you?' And I say, 'Well, can you drive there? Okay, now deliver a package.' And what ChatGPT will do is it will check internally in the robot what kind of commands are available for execution."

The system effectively bridges natural language and robot actions: "It notices that there's a command for opening the gripper and there's a command for driving to a location, there's a list of locations, then it combines those things and it actually drives and delivers. ChatGPT gives the natural language interface, and basically it translates my natural language into a set of available executable commands inside the robot."

Wisse believes this approach has significant untapped potential: "I wouldn't even say that's research, but I have the feeling that people underestimate the power of this."

Conclusion

AI is set to transform autonomous systems across industries. While the hardware — sensors, motors, and mechanical components — provides the physical capabilities, it's the AI that gives these systems their intelligence and adaptability.

What emerges from our experts' insights is that the future of autonomy isn't about removing humans from the loop, but rather finding a balance between AI capabilities and human oversight. The most successful approaches blend sophisticated AI with human judgment, creating systems that leverage the strengths of both.

The technical challenges remain significant — from developing more efficient AI perception systems to ensuring different autonomous agents can coordinate effectively. Ethical and regulatory considerations add another layer of complexity. But across academia and industry, innovators are making impressive progress here in the Netherlands.

Safety emerges as a fundamental consideration throughout this evolution. Whether through regulation, technical safeguards, or maintaining human oversight, ensuring these increasingly capable systems operate safely remains paramount.

The question isn't whether AI-powered autonomous systems will transform our world, but how quickly and in what ways — and perhaps most importantly, how we can ensure they do so safely, ethically, and in service of humans.