Senior AI Data Annotation Specialist (Portland) — Portland, OR

About The Role

You will lead end-to-end data annotation workflows for LLM and multimodal training pipelines, ensuring training data quality and consistent labeling across projects. You will drive annotation guidelines compliance, perform QA evaluation and audit sampling, and collaborate with engineers and researchers to resolve edge cases that impact model behavior.

Responsibilities

• Own dataset-level quality strategy, including gold sets, inter-annotator agreement, and error taxonomies

• Execute RLHF and large language model evaluation tasks such as pairwise ranking and rubric-based scoring

• Perform QA evaluation on labeled data, identify systematic issues, and drive corrective actions

• Improve annotation guidelines, ambiguity handling, and escalation workflows for complex edge cases

• Coordinate with AI/ML engineering stakeholders on data schemas, prompts, and evaluation metrics

• Support content safety labeling and policy-aligned decisions for safety-critical datasets

Qualifications

• Professional experience in AI data annotation, data labeling operations, or model evaluation programs

• Strong command of sampling methods, audits, and training data quality measurement

• Familiarity with RLHF, prompt evaluation, and LLM evaluation workflows

• Exposure to NLP tasks such as named entity recognition and classification taxonomies

• Ability to document decisions clearly and maintain traceability for dataset changes

Workplace

Remote, aligned to Portland-area candidates.

Company: Rex.zone