Balancing privacy and performance for anonymization pipelines

by Lesley-Ann de Wit

As many technologies lean ever more on data-heavy developments, handling personal data the right way is an important priority for many projects. Regulations like the GDPR have set a high bar for working with individuals' privacy. When handling personal data, one must be very careful in how, when, and until that data is processed. This, however, can form a challenge for developers handling sensitive data; how can we anonymize data effectively without compromising the performance of our AI models?

Whilst some use cases might require access to unprocessed video feeds for real-time monitoring, engineers developing AI models likely do not need raw data for projects. So, anonymization is the logical solution to working with sensitive data. However, the big question is: does anonymization of data hurt model performance? This is a question the engineers at KickstartAI have experimented with. Our experiments involved processing raw video streams filled with identifiable information in the form of faces, entire bodies, and more recognizable features. So, this is how we set out with our goal to find the sweet spot where ethical engineering and high-performing models give optimal results. In this article, we will share the trade-offs we made, what it took to build a robust anonymisation pipeline, and the lessons we learned getting there.

Objectives

We set out on this project with three clear objectives, with our final aim to create a scalable pipeline that supports ethical data practices and high performance:

• Privacy compliance: Adhering to privacy regulations when processing data that contained any personal information.

• Preserve performance: Making sure the models we trained on anonymized data remained accurate and efficient.

• Cost efficiency: Minimizing computational overhead without cutting corners when it comes to quality.

Anonymization levels and techniques

We evaluated three levels for anonymization. These different levels were identified as they require unique approaches with different demands on computation time.

• Full-body anonymization: Masking bodies in their entirety using segmentation.

• Face anonymization: Focusing on faces using object detection.

• Head anonymization: Detecting and anonymizing heads with pose estimation.

We also investigated three different techniques regarding anonymization of the face and body:

• Blur: Using Gaussian blur to obscure features.

• Mosaic: Pixelating areas of interest.

• Solid masks: Placing solid rectangles over detected regions.

Our hypothesis was that more computation time is needed for segmentation than for pose estimation and object detection.

The first step in data anonymization is to select which parts of our data need to be anonymized. Using pre-trained computer vision algorithms, we detected people's faces, heads, or bodies, depending on the anonymization level. We used yolo11s-seg, a YOLO (You Only Look Once) model trained on faces and heads in addition to the COCO (Common Objects in Context) dataset and CenterFace model for our first experiments with this task. We will touch on the results from this method later in the blog.

Another initial experiment for object detection was run with Detic from Meta Research. One advantage of Detic is that it allows the specification of any type of object for detection, without the need for additional training on a specific dataset. If you’re interested in knowing more about Detic, read our blog post on image annotation for object detection using Detic

Our hope was that Detic could improve on the initial results generated with YOLO without us having to train a model. However, we found that Detic struggled to consistently detect faces and heads even in very straightforward examples where heads are not partially obscured. In LVIS, a training set used for Detic, the categories "head" and "face" are not present, resulting in the aforementioned model's inability to consistently detect faces and heads. A large part of our dataset contains such instances of partially obscured faces and heads, making Detic insufficient for our purposes.

We found YOLO to be more consistent and reliable for facial, head, and full body recognition. So, using YOLO, we conducted experiments to test each anonymization method’s computational efficiency and its impact on the model performance. Our experiments relied on a little over 100 frames from diverse validation datasets. These datasets cover different locations at different times of day but also cover a variety of crowd densities. The variety of the dataset helped us evaluate the pipeline’s performance across the test scenarios.

In the end, we created a pipeline that takes videos, splits them into frames and, as the last step, anonymizes those frames. Now let’s dive into the results of this pipeline.

Computation efficiency

We measured the time that each anonymiszation method took per frame. For Body and Head anonymisation, we used a YOLO model that specializes in larger resolution images. Whereas we used the CenterFace model for Face, this model specializes in smaller resolution images. These are the results:

What stood out to us first is that full-body anonymization was the most computationally demanding, likely due to the segmentation requirements. We also found that the choice between blur, mosaic, or solid mask had little effect on computation time. When it comes to face anonymization, increasing the resolution drastically increased compute time for the CenterFace model.

Note that the timing is influenced by the number of detections made by the anonymizer. However, we don’t account for that in this table. So, there is a possibility that a method is faster simply because it detected fewer people.

Anonymization quality

Besides computation time, we also assessed detection models' precision and recall after anonymization. For this, we manually labelled data and determined the intersection of union between the labeled bounding boxes and the boxes identified by the anonymization algorithm.

Results came out as follows:

The results indicate that the face and head detection models struggle to identify smaller or more distant faces and heads. We were strict during labeling, even tagging faces and heads far away in the image that were often so pixelated that identifying a person would be nearly impossible. To focus on performance for identifiable faces/heads, we excluded labels smaller than 30 pixels in width or height.

As shown in the tables above, ignoring smaller faces and heads significantly improves recall. While precision drops slightly, this is likely because the model occasionally predicts smaller faces and heads, which would have been correct but are excluded from evaluation. Removing these predictions would provide a fairer precision measurement.

Comparison model trained on original or anonymized data

We evaluated how anonymization affects both the object detection model and the overall outcome. For the object detection model, we trained a YOLO model to identify people and objects in a little over 100 validation images. The table below compares its performance to a baseline model trained and tested on original, non-anonymized data:

The results indicate minor performance differences, which are not significant enough to suggest that anonymization affects the model's accuracy. However, when evaluating the outcome, a clear difference appears when original data is used with a model trained on anonymized data, as shown in the table below.

This result indicates that combining different data types for training and inference negatively impacts performance (e.g., using original data with a model trained on anonymiszed data). Additionally, the performance of the original model on original data is comparable to that of the anonymised model on anonymised data. However, it is important to note that these results are derived from a limited dataset of only 15 videos, meaning that even a single error can significantly affect the recall and precision metrics.

Combining anonymization and practicality, final thoughts

Clearly, anonymizing data involves trade-offs. During testing, we observed that models trained on anonymized data exhibited minimal performance loss compared to those trained on original data. However, the process of fully anonymizing the data pipeline increased computational costs and introduced latency, making it less practical for high-efficiency use cases.

The analysis also revealed limitations in the anonymization process. For example, the model occasionally failed to detect certain faces, leaving parts of the data un-anonymized. This points to a clear need for better detection algorithms that can handle tricky scenarios, especially in challenging scenarios like low-resolution or obscured images.

Our findings suggest a hybrid approach as the most efficient solution. Allowing models to be trained on original data while using anonymized data only when data scientists need to review the data strikes a good balance between performance and privacy concerns. This method avoids the added costs and delays of fully anonymizing an entire pipeline while still delivering performance that is near identical.