Senior Data Labeling Specialist (Stockholm, Remote) — United States

About The Role

Rex.zone is hiring a Senior Data Labeling Specialist to support AI/ML training workflows across NLP and computer vision projects. In this remote, full-time role you will deliver high-accuracy RLHF and evaluation datasets, drive annotation guidelines compliance, and improve training data quality that contributes to model performance improvement.

What You Will Do

• Perform complex data labeling and data annotation for LLM training pipelines, including RLHF preference ranking and QA evaluation

• Run prompt evaluation and large language model evaluation using rubrics, scoring scales, and pairwise comparisons

• Create, refine, and enforce annotation guidelines; adjudicate edge cases and document decisions for auditability

• Execute NLP tasks such as named entity recognition, text classification, toxicity, factuality checks, and summarization evaluation

• Execute computer vision annotation tasks (bounding boxes, polygons, keypoints, segmentation masks, attribute labeling)

• Perform content safety labeling (violence, self-harm, hate, sexual content) with consistent taxonomy and escalation pathways

• Conduct QA: inter-annotator agreement checks, sampling plans, error analysis, audits, and corrective feedback

• Partner with engineering and operations to improve tooling, reduce label noise, and support model performance improvement

Required Qualifications

• Mid-Senior experience in data labeling, data annotation, or QA evaluation for AI/ML datasets

• Familiarity with RLHF workflows, prompt evaluation, and LLM evaluation rubrics

• Strong understanding of annotation guidelines compliance and handling ambiguous edge cases

• Experience with training data quality methods: sampling, audits, adjudication, and error taxonomies

• Comfort working in a remote, metric-driven environment (accuracy, throughput, consistency)

How Success Is Measured

• Gold-set accuracy, audit pass rates, and reduced label noise

• Inter-annotator agreement and consistent guideline compliance

• On-time delivery, predictable throughput, and high audit readiness

Notes

This is a remote role. “Stockholm” is used for candidate search intent while work is performed remotely with distributed teams on Rex.zone.