Research Scientist, Open Benchmarks Red Team — United States
Зарплата не указана
ОткликнутьсяFor our client, we are seeking a Research Scientist, Open Benchmarks Red Team to join the team of a leader in the Information Technology space.
This role focuses on strengthening the reliability and rigor of AI evaluation by finding weaknesses in frontier benchmarks and helping improve how they are assessed. You will work at the intersection of research and engineering, developing practical methods to identify issues, validate findings, and support better benchmark releases. The position calls for close collaboration with researchers and engineers while contributing to technical reports, tools, and publications that influence the broader AI evaluation ecosystem.
The work is highly technical and research-driven, with a strong emphasis on LLMs, AI agents, and experimental analysis of model behavior.
Location: Remote
Compensation: $198,000 – $340,000 annually
Responsibilities
• Identify and reproduce issues in frontier benchmarks, such as reward hacking and task ambiguity
• Propose and implement fixes for validated benchmark issues in collaboration with authors
• Develop reusable methods to detect recurring benchmark weaknesses
• Collaborate with benchmark authors to review findings and support improved releases
• Contribute to technical reports, open-source tools, and publications on benchmark evaluation
Qualifications
• Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation
• Proficient in Python programming with experience in AI agents or evaluation systems
• Ability to identify weaknesses in evaluation methods and develop new approaches
• Experience in designing experiments and analyzing model behavior
• Familiarity with LLM/agent benchmarks and evaluation frameworks
• Proven track record of independently driving research and engineering projects
• Strong communication and collaboration skills
Benefits
• Opportunity to work on cutting-edge AI evaluation projects
• Collaborative environment with leading researchers and engineers
• Access to significant funding support for innovative AI research
• Support for professional development through publications and open-source contributions
• Engagement in impactful research that shapes the future of AI benchmarks
Our client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.