AI Evaluation Analyst (Remote) — Crewlia, United States

• Role: AI Evaluation Analyst (Remote)

• Location: Remote (Australia, Canada, Ireland, New Zealand, United Kingdom, United States)

Applicants must currently reside in Australia, Canada, Ireland, New Zealand, the United Kingdom or the United States.

Overview

We are recruiting analytical contributors to rate and compare AI responses across defined performance dimensions. You will apply scoring rules, analyze why outputs differ, and produce structured evidence for each evaluation decision.

Key Responsibilities

- Rate responses separately for correctness, relevance, completeness, reasoning quality, and instruction adherence.

- Compare paired outputs and select the stronger response using the stated tie-breaking rules.

- Identify the failure mode that most influenced a low score or preference decision.

- Check rating consistency against calibration items and revise decisions when the rubric requires it.

- Summarize evaluation evidence without speculating about how the underlying system was built.

Requirements

- Must be at least 18 years old and able to communicate in English.

- Ability to keep multiple scoring dimensions separate while forming an overall evaluation.

- Ability to compare responses and explain a preference with text-based evidence.

- Reliable high-speed internet and a suitable home-working setup.

- Previous experience with AI evaluation, analytical review, or rubric-based scoring is helpful but not required.

Equal Opportunity

We welcome applications from qualified individuals of all backgrounds. Applicants are considered based on relevant skills, qualifications and project eligibility requirements, without discrimination on grounds protected by applicable law.

Apply Now!