AI Evaluation Specialist — KR3 Information Systems, Inc., United States

Senior AI Evaluation Specialist

Location: 100% Remote – Atlanta, GA (Continental U.S. candidates)

Employment Type: Full-Time / Long-Term

Company: KR3 Information Systems

Work Authorization: U.S. Citizens and Green Card Holders only

CDC experience or Badge Preferred but not mandatory

About the Opportunity

KR3 Information Systems is seeking an experienced Senior AI Evaluation Specialist to support an AI/ML initiative.

This is an exciting opportunity for an experienced AI professional to evaluate enterprise AI products, generative AI platforms, and emerging AI capabilities to determine their effectiveness, accuracy, reliability, safety, and readiness for enterprise adoption.

You will work closely with product managers, AI/ML engineers, software engineers, data scientists, security and privacy teams, and business stakeholders to assess AI solutions, develop structured evaluation frameworks, test real-world business scenarios, and translate technical findings into actionable recommendations for leadership.

The role will involve evaluating leading AI platforms and products, including OpenAI solutions and life sciences models such as Rosalind, along with AI agents, RAG-based applications, enterprise search, and tool-enabled workflows.

We are looking for someone who can independently lead hands-on evaluations, think critically, identify AI limitations and risks, and communicate findings effectively to both technical and non-technical stakeholders.

Key Responsibilities

• Plan and execute hands-on evaluations of enterprise AI products, generative AI platforms, LLMs, and AI-enabled applications.

• Develop structured evaluation frameworks, test cases, scorecards, scoring rubrics, benchmarks, and measurable acceptance criteria.

• Assess AI solutions for factual accuracy, relevance, consistency, reasoning, instruction following, hallucinations, reliability, latency, usability, and overall effectiveness.

• Evaluate AI safety, bias, privacy, security, data handling, governance, and responsible AI considerations.

• Conduct comparative evaluations and proof-of-concept assessments across different AI products, models, configurations, and vendors.

• Evaluate RAG-based applications, AI agents, enterprise search, APIs, connectors, and multi-step AI workflows.

• Assess retrieval quality, response grounding, citation accuracy, tool execution, and task completion.

• Establish repeatable testing methodologies and regression evaluation processes to monitor AI performance as models and products evolve.

• Collaborate with engineering, product, security, privacy, and business teams to identify risks and recommend improvements.

• Prepare evaluation reports, dashboards, scorecards, and executive summaries outlining findings, limitations, risks, and adoption recommendations.

• Stay current with emerging AI technologies, evaluation methodologies, responsible AI practices, and enterprise AI products.

Required Qualifications

• Bachelor's degree in Computer Science, Data Science, Information Technology, Engineering, or a related discipline.

• 5+ years of experience in AI product evaluation, model evaluation, model quality assurance, or evaluating AI-enabled products and services.

• Hands-on experience with Generative AI tools and Large Language Models (LLMs).

• Experience designing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.

• Strong understanding of AI evaluation dimensions, including accuracy, relevance, consistency, hallucinations, safety, bias, privacy, security, usability, performance, and cost.

• Practical experience with prompt engineering, systematic testing, output analysis, and identifying AI failure modes.

• Familiarity with LLM architectures, RAG, AI agents, APIs, enterprise search, and tool/connector integrations.

• Ability to independently conduct technical evaluations, analyze results, and communicate findings and recommendations clearly.

Preferred Qualifications

• Experience evaluating enterprise AI platforms and assistants, including OpenAI products, life sciences models such as Rosalind, or comparable solutions.

• Experience with automated LLM evaluation frameworks such as Ragas, DeepEval, promptfoo, LangSmith, OpenAI Evals, or similar tools.

• Familiarity with LLM-as-a-judge, human-in-the-loop evaluation, red teaming, and adversarial testing.

• Hands-on experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, and multi-step AI workflows.

• Experience setting up AI evaluation environments using Azure AI Foundry, Azure OpenAI, AWS Bedrock, SageMaker, or open-source platforms.

• Familiarity with Python, SQL, Jupyter notebooks, spreadsheets, or analytics platforms for evaluating datasets and calculating performance metrics.

• Experience developing dashboards and reporting mechanisms to communicate AI evaluation results, quality trends, and risk findings.

• Industry-recognized certifications in AI, Machine Learning, Data Science, Cybersecurity, Cloud Technologies, or Responsible AI.

• Previous CDC or federal government project experience and an active CDC badge are preferred.

What We're Looking For

We are particularly interested in professionals who:

• Have evaluated AI products for actual business adoption decisions rather than simply using AI tools.

• Can independently design and execute structured AI evaluation programs.

• Understand how to test AI systems beyond basic prompts and chatbot interactions.

• Can identify hallucinations, inaccurate outputs, security vulnerabilities, and other AI failure modes.

• Are comfortable working across multiple AI platforms, cloud environments, and emerging technologies.

• Can communicate complex technical findings clearly and confidently to leadership.

• Thrive in a remote, collaborative environment and require minimal handholding.