Contrastive Privacy
A Semantic Approach to Measuring
Privacy of AI-based Sanitization

George Bissias · Eugene Bagdasarian · Brian Neil Levine
University of Massachusetts Amherst
Paper (arXiv) PDF Code Cite
photo A contains secret photo B contains secret sanitized A sanitized B should be the shorter distance should be longer distance sanitize sanitize
The contrastive privacy test in one picture. If sanitization actually erased what makes both photos identifiable, the green distance is shorter than the red one. Any pair where it isn't is a privacy leak you can inspect.

Abstract

To sanitize specific concepts from imagery and text, privacy mechanisms with formal guarantees are often eschewed in practice in favor of more intuitive techniques. AI-based sanitization is poised to grow in popularity because it can work with the semantics of natural language concepts; e.g., a prompt to “remove faces, clothing, and body shape.” Many approaches exist commercially and as prior work, but their evaluation has been bespoke and without formal guarantees.

We propose contrastive privacy, a formal definition of privacy that provides a systematic and quantitative test of sanitized media that has a semantic interpretation. It is independent of the model and mechanism used and operates across multiple media modalities. We show how to operationalize the definition with imperfect measures of semantics, provided by models like CLIP, that can connect concepts latently. The algorithm contrasts sanitized media with other items from the same corpus to arrive at a determination — no manual labeling is involved.

We apply our test to images and text using frontier models, quantifying sanitization success across 34 image pipelines and 15 text pipelines, identifying specific failures from a sanitized corpus.

The intuition

You ask a model to “remove anything that identifies Leonardo DiCaprio” from a folder of photos. Did it work? Hand-checking is tedious. Asking another LLM “is this private now?” gives different answers depending on how you phrase the question.

The trick is to compare items against each other. Take two photos that both show DiCaprio. Sanitize each. Now ask:

Is the sanitized version of photo A more similar to the original photo B, or to the sanitized photo B?

If the AI really erased what makes either photo a DiCaprio photo, the two sanitized images should look more like each other than either does to the originals. If instead the sanitized A still resembles the original B, something residual is binding them together — and that residual is your privacy leak. Better still, you can open the failing pair and see what the model missed: a forgotten logo, a side-character's face, text behind a black box.

Contributions

  1. A formal privacy definition for AI sanitization. Contrastive privacy reduces the question “did the AI hide it?” to a single inequality involving semantic distances, with provable guarantees under stated conditions.
  2. An automated, label-free test. The test contrasts each sanitized item against others in the same corpus — no hand-labeled ground truth, no per-item human inspection.
  3. Mechanism- and modality-agnostic. Works for blackouts, blurring, inpainting, LLM rewrites; for images, text, and in principle audio and video.
  4. An empirical comparison of frontier sanitizers. We benchmark 34 image and 15 text pipelines built from GPT-Image, Gemini-3, FLUX2, SAM3, GPT-5, Opus, and others. None achieve ideal privacy, and bigger or newer models don't reliably win.
  5. A diagnostic tool, not just a score. Failures point to specific pairs of items, exposing concrete leaks that engineers can fix.

Use it

Plug in your own AI

Pass any sanitizer (item → sanitized item) and any embedder (item → vector). The framework does the rest.

your_eval.py 
from contrastive_privacy import contrastive_privacy_test

def my_sanitizer(x):
    ...   # your AI: blackout, blur, inpaint, LLM rewrite, anything

def my_embedder(x):
    ...   # any model that returns a vector (CLIP, SBERT, OpenAI, ...)

result = contrastive_privacy_test(
    items=load_my_proxy_set(),     # ~30-50 items that all contain the secret
    sanitize=my_sanitizer,
    embed=my_embedder,
    delta=0.0,                     # 0.0 = ideal privacy
)

print(result.summary())
for f in result.top_failures(5):
    print(f"  pair ({f.i}, {f.j})  leak={f.leak:.3f}")

Returns a ContrastivePrivacyResult with passed, worst_leak, and a list of failing FailurePairs you can open and inspect to see what your AI missed.

Reproduce the paper

The repo also ships console scripts that wrap the test with concept generation and frontier sanitizers (GPT-Image, Gemini, FLUX, SAM3, …):

terminal 
git clone https://github.com/umass-forensics/contrastive-privacy
cd contrastive-privacy && pip install -e .

resolution-analysis data/dicaprio \
    --segmenter ai-gen \
    --fal-image-model fal-ai/gpt-image-1.5 \
    --privacy-concept "the identity of the celebrities" \
    --output runs/dicaprio_gpt-image-1.5

Headline findings

Citation

If you use contrastive privacy in your work, please cite:

@article{bissias2026contrastive,
  title   = {Contrastive Privacy: A Semantic Approach to
             Measuring Privacy of AI-based Sanitization},
  author  = {Bissias, George and Bagdasarian, Eugene
             and Levine, Brian Neil},
  journal = {arXiv preprint arXiv:2605.02977},
  year    = {2026}
}