The situation
The Publisher's product is a curated view of what is genuinely discounted across a large set of fashion retailers. The value is entirely in coverage and freshness — a deals site that is a day late, or that misses half the retailers, has no reason to exist.
Doing that by hand does not scale past a handful of brands, and the Publisher wanted more than thirty.
The problem
Two problems, and they are different in kind.
The first is collection. Every retailer's site is structured differently, and they change without telling anyone. A scraper that works today can quietly return nothing next week, and a pipeline that fails silently is worse than no pipeline — you keep publishing, just with a hole in your coverage.
The second is understanding. Once you have pulled in thousands of products a day, you still have to know what they are in order to make them findable. Product categorisation on retail sites is inconsistent, frequently wrong, and sometimes simply absent. Categorising that volume by hand every day is not an option.
What we built
The collection side runs as per-brand Python scrapers on a scheduled GitHub Actions pipeline, so the whole set runs daily on infrastructure the Publisher was already paying for, with per-brand failures visible rather than silent.
The understanding side is where it gets more interesting. Products are classified using an AI classifier working from the product imagery, which handles the cases where the retailer's own categorisation is missing or wrong.
- Per-brand scrapers covering 30+ retailers, run daily on scheduled GitHub Actions
- An AI product classifier using Claude vision, categorising from product imagery rather than trusting the retailer's own labels
- An admin enrichment panel where the team reviews and corrects what the classifier was not confident about
- A consumer-facing discovery interface built in Next.js on top of the cleaned, classified data
The part we would emphasise
The classifier is not trusted blindly, and that was a deliberate design decision rather than a limitation we ran into.
An automated classifier at this volume will be confidently wrong some of the time, and on a consumer product, visibly wrong categorisation is the thing that costs you trust. So the pipeline routes low-confidence classifications to the admin panel for a human to resolve, instead of publishing everything the model produces and hoping.
That keeps the economics of automation — nobody is hand-categorising thousands of products — while keeping a person on the cases where judgement actually earns its cost.
Where it landed
The Publisher gets daily coverage across the full set of brands without a proportional amount of manual work, and the team's time goes to the edge cases rather than to the bulk.
The architecture is also cheap to extend: adding a retailer means adding a scraper, not redesigning the pipeline.
Client identity withheld by agreement. This account describes our own engineering work and deliberately omits any detail that could identify the client, including their name, brands, location and commercial figures.