AI SDET / AI Evaluation Engineer — New York, NY

About The Opportunity

A dynamic startup in the AI validation and quality assurance space, we specialize in building rigorous, scalable evaluation frameworks for generative AI systems. Our team partners with enterprise clients to ensure LLMs, RAG pipelines, and agentic architectures deliver accurate, safe, and reliable outputs in production environments. We operate at the intersection of software testing, AI safety, and human-in-the-loop validation.

Role & Responsibilities

• Design and execute AI evaluation test plans for LLMs, including correctness, hallucination, bias, and safety benchmarks.

• Build automated evaluation pipelines using Python, LangChain, and open-source frameworks like DeepEval or TruLens.

• Curate and label ground-truth datasets to validate model outputs across functional, ethical, and domain-specific dimensions.

• Collaborate with ML engineers to instrument models with observability hooks and log evaluation metrics in real time.

• Define and track AI-specific QA KPIs (e.g., precision@k, safety violation rate, context adherence score).

• Lead human-annotator sessions to calibrate evaluation rubrics and resolve edge-case discrepancies.

Skills & Qualifications

• Must-Have

• Python

• LangChain

• LLM evaluation frameworks (DeepEval, TruLens, RAGAS)

• Generative AI testing

• Test automation

• AI safety protocols

• Structured dataset annotation

• ML observability

• Preferred

• Experience with HF Evaluate or LlamaIndex evaluation modules

• Familiarity with LLM fine-tuning pipelines

• Knowledge of red-teaming or adversarial testing for AI

Benefits & Culture Highlights

• On-site innovation lab with cutting-edge AI tooling and sandbox environments.

• Direct impact on AI safety standards for enterprise adopters.

• Opportunity to co-author evaluation benchmarks and industry whitepapers.

Skills: api,python,ai coding agents,test automation,llms, ai applications