Skip to content
MDM
2025

Automatic concept labels with CLIP

Labelling the concepts that explainable AI models need, without any manual labelling.

Kind
Research
Context
Project AI course, KU Leuven, team of four
Result
Beat the Stochastic-CBM baseline on all three datasets, from 73.1% to 84.1% balanced accuracy on Pascal VOC.
Built with
Python, PyTorch, CLIP, SciPy, NumPy

The problem

Concept bottleneck models explain their predictions through human-readable concepts, such as “the bird has blue wings”. The catch is labelling: 10,000 images with 100 concepts each already means a million yes-or-no labels by hand. That makes these models hard to scale.

What we built

An unsupervised pipeline that produces those labels with the vision-language model CLIP, in four steps:

  1. Similarity. Embed every image and every concept description with CLIP, and compute their cosine similarity.
  2. Double Gaussian fit. For each concept, fit two Gaussians to the histogram of similarities: one for “absent”, one for “present”. Custom penalty heuristics and a refinement step handle the tails.
  3. Concept ranking. Score how cleanly CLIP separates each concept, so unreliable concepts are dropped automatically.
  4. Thresholding. Derive a separate yes-or-no threshold per concept from its fitted Gaussians.

The codebase was covered by continuous-integration tests.

  1. Step 1

    Similarity

    CLIP embeds every image x and concept description t. Their cosine similarities fill the matrix: sij = EI(xi)⊤ ET(tj).

  2. Step 2

    Two Gaussians

    For each concept (one column), two Gaussians are fitted to the histogram of similarities.

  3. Step 3

    Ranking

    A custom score measures how well the curves separate. Poorly separated concepts are removed.

  4. Step 4

    Thresholds

    Each kept concept gets its own threshold, computed from its fitted Gaussians.

The four steps. Only step 1 uses the images; the labels are used to evaluate, never by the method.

Result

We evaluated on three datasets: CUB-200 (11,788 bird images, 312 concepts), CelebA (50,000 faces, 39 concepts) and Pascal VOC (11,540 images, 20 concepts). The method beat the Stochastic-CBM baseline on every one.

DatasetBaselineOur method
CUB-20075.8%81.4%
CelebA61.3%68.9%
Pascal VOC73.1%84.1%

Balanced accuracy on the 20 concepts the method selected.

The main finding: using all concepts hurts. On CUB, accuracy drops sharply beyond about 40 concepts, so selecting the right ones automatically is essential.

Line chart of balanced accuracy against the number of chosen attributes on CUB, for our method and the baseline. Our method starts near 0.84 and stays above the baseline until about 230 attributes; both fall steeply between 40 and 70 attributes.

Balanced accuracy on CUB as more concepts are added in ranking order, against the baseline. Figure from the team’s report.

What else we tried

  • Supervised generalisation. Networks trained on CLIP embeddings to label unseen concepts. Only CUB looked promising, and even there it did not generalise beyond the concepts it was trained on.
  • LLM concept selection. Can GPT-4o tell from the descriptions alone which concepts CLIP will recognise? It worked for CelebA's simple facial attributes (ROC AUC 0.73 to 0.79) but not for CUB's fine-grained bird features (0.36 to 0.41), and repeating a prompt gave different selections. Fitting the data, as the method does, is more reliable.