AI Evaluation Specialist — United States

This is a Fully Remote Job

About Our Client:

This organization operates within the technology services industry, focusing on advanced AI evaluation and integration. It addresses the challenge of assessing the effectiveness, safety, usability, and readiness of enterprise AI tools for business adoption. The company partners with various technical and business stakeholders to develop structured evaluation frameworks for generative AI products, including well-known models and specialized life sciences AI solutions.

About the Opportunity:

The AI EVALUATION SPECIALIST role is designed to assess and benchmark enterprise AI tools to ensure their suitability for business use. This position involves developing evaluation criteria, conducting hands-on assessments, and providing clear recommendations regarding AI product quality, safety, and performance. The specialist directly influences the organization's ability to adopt reliable AI technologies by delivering detailed comparative analyses and maintaining evaluation standards.

Responsibilities:

• Plan and execute evaluations of AI tools using representative business scenarios and datasets.

• Assess AI solutions for quality, accuracy, relevance, reliability, usability, and task effectiveness.

• Design and maintain standardized evaluation frameworks, scorecards, and acceptance criteria.

• Collaborate with security, privacy, legal, and governance teams to evaluate safety and compliance aspects.

• Conduct comparative assessments of multiple AI products, analyzing functionality, integration, performance, and cost.

• Evaluate AI agents and tool-enabled workflows, including advanced integration capabilities.

• Stay updated on generative AI advancements and incorporate new evaluation methods.

• Participate in knowledge sharing and contribute to a collaborative team environment.

Requirements:

• Minimum of 5 years experience in model evaluation or AI-enabled product assessment.

• Proficiency in developing structured evaluation plans and measurable acceptance criteria.

• Hands-on experience with generative AI tools, prompt design, and failure mode identification.

• Ability to assess AI systems across multiple dimensions, including safety, bias, privacy, and cost.

• Familiarity with generative AI architectures, including large language models and retrieval-augmented generation.

• Bachelor's degree required.

Preferred Qualifications:

• Experience with enterprise AI platforms such as OpenAI and life sciences models.

• Skills in automated evaluation frameworks and adversarial testing.

• Knowledge of RAG systems, AI agents, and multi-step AI workflows.

• Ability to develop dashboards and analyze evaluation data using Python, SQL, or analytics tools.

• Relevant certifications or a Master’s degree.

Pay Range and Compensation Package:

• Final salary determination based on skill-set, qualifications, and approved funding.

Benefits & Perks:

• Paid Time Off including vacation accrual and paid holidays.

• Medical, dental, vision, and prescription coverage.

• Employee assistance and disability insurance programs.

• Flexible spending and health savings accounts.

• Commuter benefits.

• Retirement plans including 401(k) and 401a.

• Voluntary benefits such as hospital indemnity and legal insurance.

Equal Opportunity Statement: Our client is an equal opportunity employer. They celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, or national origin.

Note:

4Kw8nrFev is a recruitment partner of this role. Please note that all employment decisions, including candidate assessment, interviews, hiring, compensation, and employment terms, are made exclusively by the hiring employer.