Automatic concept labels with CLIP
Labelling the concepts that explainable AI models need, without any manual labelling.
- Kind
- Research
- Context
- Project AI course, KU Leuven, team of four
- Result
- Beat the Stochastic-CBM baseline on all three datasets, from 73.1% to 84.1% balanced accuracy on Pascal VOC.
- Built with
- Python, PyTorch, CLIP, SciPy, NumPy
- Status
The problem
Concept bottleneck models explain their predictions through human-readable concepts, such as “the bird has blue wings”. The catch is labelling: 10,000 images with 100 concepts each already means a million yes-or-no labels by hand. That makes these models hard to scale.
What we built
An unsupervised pipeline that produces those labels with the vision-language model CLIP, in four steps:
- Similarity. Embed every image and every concept description with CLIP, and compute their cosine similarity.
- Double Gaussian fit. For each concept, fit two Gaussians to the histogram of similarities: one for “absent”, one for “present”. Custom penalty heuristics and a refinement step handle the tails.
- Concept ranking. Score how cleanly CLIP separates each concept, so unreliable concepts are dropped automatically.
- Thresholding. Derive a separate yes-or-no threshold per concept from its fitted Gaussians.
The codebase was covered by continuous-integration tests.
Step 1
Similarity
CLIP embeds every image x and concept description t. Their cosine similarities fill the matrix: sij = EI(xi)⊤ ET(tj).
Step 2
Two Gaussians
For each concept (one column), two Gaussians are fitted to the histogram of similarities.
Step 3
Ranking
A custom score measures how well the curves separate. Poorly separated concepts are removed.
Step 4
Thresholds
Each kept concept gets its own threshold, computed from its fitted Gaussians.
Result
We evaluated on three datasets: CUB-200 (11,788 bird images, 312 concepts), CelebA (50,000 faces, 39 concepts) and Pascal VOC (11,540 images, 20 concepts). The method beat the Stochastic-CBM baseline on every one.
| Dataset | Baseline | Our method |
|---|---|---|
| CUB-200 | 75.8% | 81.4% |
| CelebA | 61.3% | 68.9% |
| Pascal VOC | 73.1% | 84.1% |
Balanced accuracy on the 20 concepts the method selected.
The main finding: using all concepts hurts. On CUB, accuracy drops sharply beyond about 40 concepts, so selecting the right ones automatically is essential.

Balanced accuracy on CUB as more concepts are added in ranking order, against the baseline. Figure from the team’s report.
What else we tried
- Supervised generalisation. Networks trained on CLIP embeddings to label unseen concepts. Only CUB looked promising, and even there it did not generalise beyond the concepts it was trained on.
- LLM concept selection. Can GPT-4o tell from the descriptions alone which concepts CLIP will recognise? It worked for CelebA's simple facial attributes (ROC AUC 0.73 to 0.79) but not for CUB's fine-grained bird features (0.36 to 0.41), and repeating a prompt gave different selections. Fitting the data, as the method does, is more reliable.