Detecting Unattended Objects: The NS Challenge
by Evertjan Peer
At KickstartAI, we're always looking for opportunities to apply AI to solve complex real-world problems. Our recent collaboration with NS (Nederlandse Spoorwegen), the Dutch railway company, presented a unique challenge: detecting unattended objects in busy train stations to enhance social security.
In this post, I'll walk you through our journey of our most recent KickstartAI challenge: developing an AI-powered solution to identify unattended objects, a crucial aspect of maintaining safety and security in high-traffic public spaces. We’ll explore how a three-week KickstartAI challenge can lead to concrete results and be a valuable starting point for the implementation phase!
The problem: ensuring station safety
NS places a high priority on social safety within its stations. A significant part of this effort involves identifying unattended goods, of which there are over 2,000 cases per year. Timely identification is crucial for determining the appropriate response, whether it's locating the owner or taking security measures.
Currently, the control room has access to approximately 15,000 cameras. However, the workload is high on the camera team, leaving little time to proactively search for unattended objects in video streams. This project researches whether we can utilize these existing cameras to proactively search for unattended objects and alert the camera team directly for further inspection.
The challenge: automating unattended object detection
The KickstartAI way of working involves aligning on a use case with a partner, organizing a "challenge" (a structured hackathon), and then moving the solution into production. During a challenge, the KickstartAI team joins forces with engineers from our partner organizations to develop a technical proof of concept. The KickstartAI challenges are a mix of hard work, interesting company tours, and social events!
The NS challenge was clear: develop a first baseline end-to-end solution for detecting unattended objects. We assembled a team of 15 participants from NS, KickstartAI, KLM, bol. (Ahold Delhaize), and Ampère (Ahold Delhaize) for a three-week development sprint. The challenge was divided into five work packages to ensure a comprehensive solution and timely results:
The work packages:
Anonymization & data pre-processing: Exploring anonymization techniques to protect privacy while preparing video data for labeling, and converting videos to frames.
Annotating data: Creating "ground truth" data by labeling objects of interest (people, bags, luggage etc.) in video frames to train machine learning models.
Detecting objects: Developing a model to detect people and relevant objects on video frames.
Tracking objects through time: Tracking potential unattended objects over time and determining when to inform the control room.
Evaluation: Evaluating the system from both a technical and a user point of view.
Our approach: building an end-to-end system
Data pre-processing and anonymization:
We started by pre-processing the video data, creating datasets for training, validation, and testing. Given the sensitive nature of video data with people, we explored various anonymization techniques (blurring, mosaic, solid) to protect privacy during the annotation phase. We learned that training on the same data type as the final model (original or anonymized) is crucial and that ongoing discussions with internal legal experts are essential to understand requirements and limitations.
Building a data labeling platform:
To train our object detection model, we needed labeled data. We investigated several labeling tools and ultimately chose Azure ML Studio's data labeling tool due to data privacy requirements. To speed up the labeling process, we explored machine-assisted labeling using pre-trained models like DETIC and Faster-RCNN. While DETIC detected more objects, it also produced many false positives, highlighting the need for human review and a semi-automatic labeling pipeline.
Object detection with YOLO:
For object detection, we chose YOLO (You Only Look Once) for its speed and reliability. However, YOLO's pre-trained model on the COCO dataset didn't have all the categories we needed (bags, luggage, boxes, backpacks etc.). We addressed this by training a custom YOLO model on a combination of publicly sourced data (LVIS, Open-Images) and NS station data, significantly improving performance on our specific use case.
Tracking objects through time:
Tracking objects involved determining when an object becomes "unattended." We started with a simple rule: an object is unattended if it remains static for 10 minutes. We aim to refine this definition only if evaluation shows a high false positive rate. Adding complexity — such as modeling ownership— comes with higher computational costs and privacy implications, so any improvements must be carefully balanced against their impact on the final system. We used Intersection over Union (IoU) to track objects across frames and incorporated confidence scores from the YOLO model to filter out false detections. We developed a tracker and updater algorithm to manage object memory and apply business rules.
Evaluation and trade-offs:
We evaluated our system based on metrics like precision, recall, and F1 score. We achieved a precision of 91% and a recall of 63% on our test set. We also identified trade-offs in parameters like patience (how long before deleting an object from memory), IoU threshold, and model confidence. We noted limitations, such as static objects with owners nearby and objects behind glass, and explored extensions like object re-identification using color histograms and image embeddings.
Results and looking ahead:
Within three weeks, we developed a working baseline for unattended object detection. The tracking and alerting system proved effective, and we developed an implementation plan for pilot testing and scaling. Future work includes optimizing model prediction speed, conducting more extensive training experiments, and further refining object re-identification and ownership detection. Currently we are working on defining how to bring the solution to production (organizational and technical).
This KickstartAI challenge showed the power of collaborative problem-solving and the potential of AI to enhance security and safety in public spaces. It was great to see engineers from different companies work together to a common goal in a fast-paced, startup-like atmosphere. If you work at one of our partner companies, stay tuned for our upcoming challenges and make sure to join. For everyone else, stay tuned for updates on this interesting application of AI!
