Machine Learning Engineer, AI Agent Evaluation — United States
Зарплата не указана
ОткликнутьсяКомпания: ••••••••
Локация: United States
Формат: не указан
Занятость: Полная занятость
Грейд: не определён
US Remote · Periodic Bay Area travel required
$200,000–$225,000 base salary + equity
About the company
ye2qBqq9 is helping a seed-stage portfolio company hire a machine learning engineer to develop evaluations for AI agents working on complex technical tasks. The team combines software engineering, machine learning, and specialized domain expertise. More about the company and its work will be shared during the interview process.
About the role
You’ll help determine whether AI agents can reliably complete meaningful work in interactive environments. Agents use tools, receive new information, change the environment, and decide what to do next. Your job is to design evaluations that distinguish genuine success from convincing outputs, incomplete work, or behavior that exploits the grading.
This is an ML engineering role with substantial software ownership. Working with domain experts, you’ll turn specialized workflows into tasks, success criteria, rubrics, and automated graders. You’ll investigate misleading results, improve the evaluations, and build and operate the systems that make runs repeatable and failures understandable.
An initial simulation and evaluation stack already exists. The largest opportunity is to strengthen evaluation design and judgment while taking ownership of reliable execution, monitoring, and reporting. You’ll contribute to the environment as needed, with support from the existing software and domain teams.
As an early individual contributor, you’ll work across these areas in a small team where priorities evolve and ownership matters.
What you’ll do
• Design meaningful tasks and grading criteria with domain experts, including cases where multiple approaches can produce a valid result.
• Develop and validate rubrics and automated graders for complete, multi-turn agent sessions.
• Identify misleading scores, grading loopholes, and reward-hacking behavior; investigate the cause and improve the evaluation.
• Analyze trajectories to distinguish agent failures from problems in tools, environments, or graders.
• Build and operate secure evaluation systems with repeatable runs, parallel execution, recovery, monitoring, and alerting.
• Scope technical projects, deliver evaluations, and communicate findings to colleagues and customers.
What you’ll bring
• Hands-on experience evaluating AI agents across complete, multi-step sessions in interactive environments.
• Substantive experience designing or improving tasks, rubrics, success criteria, or graders, with the judgment to recognize when an evaluation rewards the wrong behavior.
• The ability to explain a concrete evaluation problem, how you investigated it, and what changed as a result. Team projects and internal work count; you don’t need to have independently owned every component.
• Strong Python and software engineering skills, including experience deploying, monitoring, debugging, and maintaining production systems.
• Practical experience using coding agents while retaining responsibility for design, review, testing, and delivery.
• Clear communication and comfort collaborating with domain experts, engineers, and customers.
• Willingness to own both ML evaluation problems and the engineering needed to make the work dependable.
Relevant backgrounds
Relevant experience may come from applied ML, agent development, agent benchmarks, evaluation platforms, or RL environments. The important connection is experience designing and interpreting evaluations of agents acting over multiple steps.
Experience with experimental design, human-calibrated grading, reproducibility, sandboxing, distributed execution, and simulation is useful.
The current focus is agent evaluation, not training foundation models. Prior model-training experience can be valuable, and knowledge of a particular scientific domain is not required.
Location
The software team is distributed, with a Bay Area center of gravity. This US-remote role includes periodic in-person offsites and Bay Area visits.
Please note
There are no fees associated with any of the support we provide our investments. 7azaHNU5 m2uytk provides free candidate referrals and introductions to all of our active investments, one of the many services we provide.
Due to the volume of applicants we typically receive, a follow-up email will not be sent unless a match is identified.