Building Human-Labeled Data for AI Fact-Checking
How a distributed annotation team helped create structured training and evaluation data for information verification systems
MyDeepTech Team · Case Study · 5 min read
- Four-stage workflow — onboarding, production, quality review and performance management.
- Consistency over volume — the annotation framework had to be applied the same way, every time.
- Fully managed operations — recruitment through reporting, handled end-to-end by MyDeepTech.
- Pattern-based quality review — looking for recurring issues, not just counting finished tasks.
The Challenge
An AI-focused technology company was developing systems designed to assess and verify information presented in online content.
To improve these systems, the company required large volumes of human-reviewed text data. Each piece of content needed to be assessed against defined criteria and assigned the appropriate annotation based on the available evidence.
The challenge was both scale and consistency.
Fact-checking and information verification tasks require annotators to understand instructions carefully, distinguish between different types of claims, and apply the same decision framework repeatedly across thousands of examples.
Our Approach
MyDeepTech built and managed a distributed team of text annotators to support the project.
The workflow was structured around four core stages:
- Onboarding
Annotators were introduced to the project objectives, annotation guidelines and examples. - Production
Annotators reviewed assigned text and applied the required labels according to the project framework. - Quality Review
Completed work was reviewed to identify inconsistencies, incorrect labels and areas where additional guidance was required. - Performance Management
Contributor productivity and quality were monitored throughout the project, allowing the team to address issues and maintain delivery standards.
The project relied on human judgment to create structured data that could subsequently be used in AI development and evaluation.
Human Review at Scale
The project relied on human judgment to create structured data that could subsequently be used in AI development and evaluation.
Annotators worked through large volumes of text while following a common set of rules. Where cases were unclear or errors appeared repeatedly, the project team used feedback and additional clarification to improve consistency.
This approach allowed the workforce to operate at scale without treating every contributor as an isolated worker.
Instead, contributors were managed as part of a coordinated production and quality-control system.
Managing Distributed Contributors
The project required contributors to work remotely across different locations.
MyDeepTech managed the operational layer, including:
- Contributor recruitment
- Screening and onboarding
- Training and guideline distribution
- Task allocation
- Productivity monitoring
- Quality monitoring
- Review and feedback
- Project reporting
- Communication around deadlines and deliverables
This allowed the client to focus on the underlying AI-data requirements while our team managed the workforce required to execute them.
Quality and Consistency
For fact-checking annotation, consistency is particularly important because inconsistent human labels can reduce the usefulness of the resulting dataset.
Our quality process therefore focused on identifying patterns rather than simply counting completed tasks.
When issues were identified, contributors could receive additional guidance or have their work reviewed more closely.
The goal was to maintain alignment with the project's annotation guidelines throughout production.
The Result
The project provided the client with a scalable human-annotation operation capable of processing substantial volumes of text while maintaining a structured quality-control process.
It also demonstrated the value of combining local workforce access with centralised project management for AI data programmes.
Why It Matters
AI systems that evaluate claims and information depend on high-quality human-labeled data.
The effectiveness of that data depends not only on the number of annotations produced, but also on how consistently the annotation framework is applied.
Our model combines:
Human judgment + structured guidelines + quality review + distributed workforce management.
to support organisations building AI systems that need reliable, human-generated training and evaluation data.
AI Data. Human Intelligence.
1 Comment
[…] Building Human-Labeled Data for AI Fact-Checking → […]