Research Engineer — San Francisco Bay Area

Research Engineer, Coding Agent Optimization Engine

San Francisco, in person · Full-time · Entelligence

Coding agents ship more code every week, and they fail in repeatable ways: the same wrong assumption, the same missed test, the same review comment on PR after PR. Our Coding Agent Optimization Engine closes that loop. It mines agent sessions and PR review comments for failures, traces them to where the harness went wrong, and feeds the lessons back so the next session does not repeat them. It also routes every turn to the most efficient model that can solve it. It runs on live traffic from Claude Code, Codex, Cursor and API clients. We want a research engineer to make it measurably better at both.

Key Focus Areas• Failure mining. Captures coding-agent sessions and links them to the PRs they produced, then mines transcripts and review comments for errors, struggles and recurring root causes.

• Learning loop. Turns review findings into root-cause patterns and tested rules, and delivers them to the next agent session as guidance cards through CLI hooks.

• Pre-PR judge. A reviewer model checks an agent's change before the PR is opened, and the harness reports back what happened in each session.

• Cost-aware routing. Picks a difficulty tier per instruction turn and weights models by expected cost per solved task, with prompt-cache affinity, failover and per-org constraints.

What you will work on• Failure attribution. Given a session transcript and the review comments on its PR, find where the harness went wrong: missing context, a bad tool loop, a skipped check, the wrong model.

• Prevention that measurably works. Decide which lessons become rules, cards or judge checks, and prove with held-out data that they cut repeat findings without flooding agents with noise.

• Evaluation. Own the benchmark harness of record: replay real agentic tasks headlessly against competing harness configs and routing policies, and measure quality vs. cost honestly.

• Engineering world model. Build the comprehensive data layer connecting sessions, PRs, projects, errors and incidents. Leverage the world model to catch issues from happening in future sessions.

• Cost and outcome modeling. Turn measured usage, cache shares, latency and review outcomes into continuously refreshed model and harness parameters.

Your Background• 3+ years building ML or LLM systems that run in production, or equivalent research experience with shipped code.

• Strong Python and comfort reading a large production codebase.

• Experiment design you can defend: held-out sets, variance, confidence intervals, leakage checks.

• Hands-on experience with LLM APIs, agents or tool-use loops, and their failure modes.

• Work on agent evaluation, failure analysis, LLM-as-judge or automated code review.

• Work on routing, cascades, bandits or learning-to-defer.

• Experience with LLM inference economics: prompt caching, batching, token pricing.

• Built or maintained public evals (SWE-bench, Terminal-Bench style harnesses).

• Published research or well-known open-source work in ML systems or developer tools.

Compensation and benefits

Base salary: $170,000 - $210,000 plus meaningful equity. Final offer depends on level and experience. Healthcare and a generous compute budget for experiments.