Evaluation Researcher — San Francisco, CA

I'm partnering with a well-funded frontier AI research startup developing low-latency speech and multimodal AI systems capable of interacting with humans in real time.

They're looking for an experienced Evaluation Researcher to design and build evaluation systems that directly improve the quality of production speech and language models.

The role:

• Design and develop evaluation frameworks for speech and text models

• Build automated evaluation pipelines, benchmarks and quality metrics

• Evaluate conversational AI performance, including response quality, latency and turn-taking

• Identify model weaknesses and translate evaluation findings into measurable product improvements

• Work closely with research and engineering teams to improve production AI systems

What they're looking for:

• Strong experience building evaluations for production speech or text models

• Proven impact on model quality and end-user experience

• Experience with LLM evaluation, speech evaluation or conversational AI

• Strong Python and ML research engineering skills

• Knowledge of modern evaluation methods, benchmarking and experimental design

Why consider it?

• Join a well-backed frontier AI research lab developing next-generation conversational and multimodal AI systems.

• You'll have direct ownership of model evaluation and play a key role in improving how intelligent agents interact with people in real time.

Location: Downtown San Francisco | Fully On-site

Interested? Apply directly or message me for a confidential discussion.