BirdCLAP

## Overview

BirdCLAP is a machine learning project I led during a mentorship at RBC Borealis, building a system that lets you search bird sounds with plain language. It fine-tunes LAION CLAP, an audio-language model, so that a text query and the matching bird call land near each other in a shared embedding space, which is the same embedding-and-similarity-search idea behind a vector database.

## Problem

Bird audio is messy, heavily imbalanced across species, and hard to search without expert labels. A general-purpose audio-language model understands almost none of it out of the box. The goal was to teach the model real acoustic structure so it could retrieve the right call from a natural-language description, even for species it had never seen during training.

## Solution

I owned the full pipeline end to end. I built the data pipeline combining GBIF taxonomy, GPT-4 retrieval-augmented audio descriptions, and xeno-canto contrastive pairs, cleaning and structuring 33,813 audio-text pairs across 502 species. I then fine-tuned CLAP with a multi-positive InfoNCE objective using genus-level hard negatives, SpecAugment, mixup, and inverse-frequency sampling to fight class imbalance, and built the evaluation that measured whether it actually worked.

## Results

- **Hit@10 = 77%** and **Hit@1 = 30%** on a 2,436-clip gallery, meaning the correct clip landed in the top 10 for three out of four queries
- A **21x improvement** over base CLAP, with a median correct rank of 3 out of 2,436
- **Zero-shot generalization to held-out unseen species at 10x base-model performance**, confirming the model learned real acoustic structure rather than memorizing species

One of the most important things I did was catch a subtle flaw in my own evaluation that was quietly inflating results, then fix it and rerun everything, because a metric you cannot trust is worse than no metric at all.

## Technologies Used

- **Language**: Python
- **ML**: PyTorch, LAION CLAP, Hugging Face, contrastive learning (multi-positive InfoNCE)
- **Data**: GBIF taxonomy, xeno-canto, GPT-4 retrieval-augmented descriptions

## What I Learned

How to take a foundation model and adapt it to a hard, imbalanced real-world domain, and how much of real machine learning is the data pipeline and honest evaluation rather than the model itself.