AI Quality Assurance & Evaluation Specialist — Onsite IT Advisors, Atlanta, GA

AI Evaluation SpecialistLocation: Continental United States

Position OverviewWe are looking for an experienced AI Evaluation Specialist to test and evaluate AI tools and help determine whether they are accurate, reliable, safe, secure, and ready for business use.

In this role, you will work with different AI platforms and Generative AI solutions, including OpenAI products, life sciences AI models such as Rosalind, and similar enterprise AI tools.

You will work closely with product managers, data scientists, AI engineers, software developers, security and privacy teams, and business users. Your primary responsibility will be to test AI tools using real-world business scenarios, check the quality of their responses, identify issues, and recommend improvements.

You will compare different AI products based on accuracy, response quality, reliability, performance, security, privacy, ease of use, and cost. You will also develop standard testing methods and evaluation reports to help organizations select and adopt the right AI solutions.

Key Responsibilities• Test different AI platforms, including OpenAI and life sciences AI models such as Rosalind, using real-world business scenarios, prompts, datasets, and workflows.

• Check AI-generated responses for accuracy, relevance, completeness, consistency, reasoning, and ability to follow instructions.

• Evaluate how well AI tools perform, including response speed, reliability, ease of use, accessibility, and their ability to complete assigned tasks.

• Create and maintain AI testing plans, test cases, evaluation scorecards, rating guidelines, and success criteria.

• Identify AI-related issues such as incorrect answers, misleading information, hallucinations, bias, and inconsistent results.

• Review AI tools for security, privacy, data protection, governance, and responsible use in coordination with the appropriate teams.

• Compare multiple AI products and vendors through hands-on testing and proof-of-concept exercises. Evaluate their features, performance, integration needs, and overall cost.

• Test advanced AI capabilities, including AI agents, Retrieval-Augmented Generation (RAG), enterprise search, document and data analysis, APIs, connectors, and Model Context Protocol (MCP)-based tools.

• Work with technical and business teams to share test results, explain identified problems, and recommend the best AI solutions for business needs.

• Keep up with new developments in Generative AI, AI testing methods, model evaluation, AI safety, and responsible AI practices.

• Share knowledge with team members and contribute to a positive and collaborative work environment.

Required Qualifications• 5+ years of experience in software quality assurance, AI testing, model evaluation, or a related quality assurance field, including hands-on experience testing AI-based products or services.

• Experience creating test plans, test scenarios, test cases, evaluation scorecards, benchmarks, and measurable success criteria.

• Hands-on experience using Generative AI tools and Large Language Models (LLMs), including writing prompts, testing different scenarios, reviewing responses, and identifying incorrect answers or other AI issues.

• Ability to evaluate AI tools based on accuracy, response quality, consistency, reliability, safety, bias, privacy, security, usability, performance, and cost.

• Good understanding of Generative AI technologies, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), AI agents, APIs, enterprise search, and integrations with external tools and systems.

• Strong analytical, problem-solving, communication, and documentation skills.

• Bachelor's degree in any relevant discipline.

• Must be a U.S. Citizen or Lawful Permanent Resident (Green Card Holder).

Preferred Qualifications – A Plus• Experience testing enterprise AI platforms and AI assistants, including OpenAI products, Rosalind, or similar Generative AI solutions.

• Experience with automated AI testing and evaluation tools, AI-as-a-judge approaches, human review of AI responses, red teaming, or security and adversarial testing.

• Experience testing RAG applications, AI agents, MCP-enabled tools, enterprise connectors, and multi-step AI workflows.

• Experience creating dashboards and reports to track AI quality, testing results, performance trends, risks, and regression issues.

• Ability to work with test data and evaluation results using Python, SQL, Excel, spreadsheets, Jupyter notebooks, or other analytics tools.

• Industry-recognized certifications in Artificial Intelligence, Machine Learning, Data Science, Cybersecurity, Cloud Computing, or Responsible AI.

• Master's degree preferred.

•