Chatbot Quality Evaluation Specialist — Dearborn, MI

Location:, Dearborn, MI

Duration: 12 Months+ Contract

Payrate: $38+ PTO + Benefits

Job Summary

We are seeking a highly detail-oriented Data Annotator / Validation Engineer 2 to support the evaluation and improvement of AI-powered chatbot systems. This role will focus on reviewing conversational AI interactions, assessing response quality, and applying structured labeling guidelines to improve model performance. The ideal candidate will have experience working with conversational AI data, strong analytical skills, and the ability to consistently apply detailed annotation standards.

Key Responsibilities

• Review and evaluate chatbot conversations, model responses, retrieved content, citations, and metadata.

• Apply approved annotation guidelines, taxonomies, and decision frameworks to label AI-generated content.

• Assess chatbot performance for:

• Accuracy and correctness

• Task completion

• Relevance and completeness

• Groundedness and source attribution

• Safety and policy compliance

• Escalation requirements and failure categorization

• Document annotation decisions with clear rationales and confidence assessments.

• Identify and escalate ambiguous, sensitive, or policy-related content requiring additional review.

• Perform quality checks and correct annotation errors identified during audits and reviews.

• Maintain accurate records of completed annotations, quality results, escalations, and open issues.

• Participate in onboarding, calibration sessions, training programs, and quality review meetings.

• Prepare weekly reports on productivity, quality metrics, backlog status, risks, and process improvements.

• Recommend enhancements to annotation guidelines and taxonomy based on recurring patterns and findings.

Required Qualifications

• Bachelor's Degree required.

• Experience labeling conversational AI data, including text and audio content.

• Strong English communication skills with excellent reading comprehension and writing abilities.

• Ability to interpret and consistently apply detailed guidelines and decision rules.

• Strong analytical and critical-thinking skills.

• Exceptional attention to detail and accuracy.

• Experience documenting decisions and providing concise justifications.

• Ability to manage repetitive tasks while maintaining high-quality standards.

• Ability to adapt to evolving guidelines and incorporate feedback.

Preferred Qualifications

• Experience with companies such as Amazon, Google, Apple, Scale AI, or similar AI/ML organizations.

• Automotive industry experience preferred.

• Master's Degree preferred.