Senior AI Data Annotation Jobs in Austin — Austin, TX

Overview

Senior AI data annotation jobs in Austin focused on producing high-quality training data for AI/ML systems. You will support LLM training pipelines through RLHF, prompt evaluation, QA evaluation, named entity recognition, computer vision annotation, and content safety labeling. This is a remote, full-time role open to candidates in the US with Austin-aligned hiring intent.

What You’ll Do

• Perform expert AI data annotation and data labeling for NLP, LLM training, and computer vision datasets

• Run RLHF workflows: preference ranking, pairwise comparisons, critique writing, and rubric-based scoring

• Execute prompt/response evaluation to identify failure modes, hallucinations, and unsafe outputs

• Apply content safety labeling (toxicity, self-harm, hate, harassment, sexual content, policy violations)

• Conduct QA evaluation and audits to ensure annotation guideline compliance and consistent labeling

• Create decision logs for ambiguous cases and propose guideline refinements

• Monitor training data quality metrics and sampling strategies to improve model performance

• Collaborate with engineering and AI/ML stakeholders to integrate labeled data into LLM training pipelines

Requirements

• Mid-senior experience with annotation, labeling, or evaluation workflows

• Strong judgment and consistency on edge cases and nuanced content

• Experience with QA evaluation, error analysis, and training data quality improvement

• Comfort with LLM outputs, prompt evaluation, and RLHF ranking/scoring frameworks

• Clear written communication and documentation for auditability

Nice to Have

• Named entity recognition and taxonomy/policy-driven labeling experience

• Computer vision annotation tooling (polygons/segmentation) experience

• Familiarity with calibration exercises and inter-annotator agreement concepts