Data annotation case studies: hard data, clean labels, models that ship
Blurry field photos, call-centre audio that switches languages mid-sentence, safety evaluations in languages most vendors don’t cover. Here is how we turned each one into training data a model could trust, and what changed.
Polygon segmentation from our agritech engagement: five disease classes.
42,000crop-leaf images segmented in nine weeks
3,800 hof code-switched Swahili–English audio
180,000LLM preference comparisons, two weeks early
−34%word error rate on a client’s speech model
Four projects, four very different problems
Every engagement started with a dataset that had already frustrated someone. Client names are kept confidential, as they are for every team we work with.
The client’s app lets farmers photograph a sick maize plant and get an instant diagnosis. The photos come from budget phones, in harsh sun or deep shade, often slightly out of focus. Their previous vendor kept confusing maize lethal necrosis, a devastating viral disease, with ordinary nutrient deficiency.
What we did
Built a dedicated team and brought in agronomy expertise so annotators understood the biology, not just the colours.
Created a reference library of 600 difficult examples showing where one disease ends and another begins.
Wrote tight definitions for all five disease classes and trained every annotator against them.
Routed every low-confidence label to a specialist reviewer.
Helping a voice assistant understand how Kenyans actually talk
Client: Regional telecom and mobile money provider
The challenge
Real customers switch between Swahili and English mid-sentence, drop in Sheng and bring accents from across the region. The client’s voice assistant had been trained on clean, single-language speech, and callers were misunderstood and transferred. The recordings also contained national ID numbers and M-Pesa transaction codes.
What we did
Staffed the project with native speakers fluent in Swahili, English and Sheng.
Wrote conventions for language switches, speaker turns, fillers and overlapping speech.
Ran a dedicated redaction pass for ID numbers, phone numbers and M-Pesa codes.
Re-checked the hardest utterances and audited consistency across batches.
Making a frontier model safer in languages most vendors skip
Client: AI research lab
The challenge
Large language models are increasingly used across Africa, yet mostly evaluated in English. This lab needed preference rankings and red-team safety testing in Swahili and Amharic. Preference data is subjective, so the real challenge was getting reviewers to judge consistently.
What we did
Assembled language-qualified reviewers who understand cultural context, idiom and tone.
Documented ranking criteria for helpfulness, accuracy and safety, with borderline examples.
Held regular calibration sessions with the client’s research team.
Sent ambiguous safety cases to senior reviewers instead of deciding them in isolation.
A labeling workflow built for the messiest road scenes
Client: Capability workflow for mobility and robotics datasets
The challenge
Perception teams struggle with the pedestrian half-hidden behind a matatu, the sparse point cloud at 80 metres and the boda boda weaving between lanes. These edge cases are where annotators disagree, and inconsistent labels make a model hesitate.
How we approach it
Agree class definitions and cuboid conventions up front, covering heading, truncation and occlusion.
Qualify annotators on gold-standard scenes before production.
Route uncertain scenes to a lead annotator for a final decision.
Track objects across frames so identities stay consistent.
Measure agreement and defect rates class by class.
About these examples: client names are anonymised. The agriculture, speech and LLM results are drawn from project summaries. The 3D and LiDAR example describes our standard workflow for this kind of data rather than a single named engagement. References can be discussed during scoping, subject to client permission.
The same disciplined process behind every result
Scope and calibrate
We review a sample, define classes and edge cases, agree acceptance criteria and write guidelines full of real examples.
Train and pilot
Annotators qualify on a representative sample, and we study every disagreement before production begins.
Produce and review
Production runs with layered QA, clear escalation and documented feedback between annotators and reviewers.
Report and improve
You get the agreed quality and throughput measures, and guideline updates go through change control.
Why are the clients in these case studies anonymous?
Most AI teams treat their training data and vendors as confidential, so we never publish a client’s name, logo or identifiable detail without written permission. Where clients agree, we can arrange reference conversations during scoping.
Can we start with a pilot before committing to full production?
Yes, and we recommend it. A free pilot of up to 500 units tests guideline clarity, quality, review effort and throughput on your own data before you scale.
How do you report quality?
We agree the metrics with you before production starts. Reports can include gold-set accuracy, reviewer agreement, defect and rework rates, throughput and any model-facing metrics you choose to share.
How quickly can a project get started?
Once we’ve reviewed a data sample and agreed scope, we give you a clear start date and delivery plan in your quote. Small projects often start within days.
Your dataset could be our next case study.
Tell us what you’re labeling, how much of it you have, the quality bar and the deadline. We’ll come back with a scoped pilot plan and the metrics that will prove it’s working.
