Senior jQ SVWR4Pph Engineer — United States

Senior 2r zMmkvsjn Engineer

Location: Remote (US) | Type: Full Time | Experience: 5+ years | Pay: $165K-$205K + bonus

About the Role

Before an Mz agent goes in front of customers, someone has to prove it's ready. In this role, that's you. You'll design the test sets, scoring methods, and release checks that show whether an agent read the request correctly, pulled the right data, called its tools properly, and stayed inside policy. You'll also own the evidence behind every model or prompt change that ships.

The team builds the shared Mz platform at a publicly listed fintech whose payment products are used by millions of shoppers and merchants. That platform runs a customer-facing assistant plus a growing set of internal agents, so the quality systems you build will cover both. You'll partner with product and subject-matter owners who define what "correct" means, and with reliability and security engineers who keep production healthy.

This is a senior individual contributor seat, fully remote for US-based engineers. It suits someone who distrusts a single impressive demo and would rather see a regression report.

What You'll Do

• Build and version reference datasets from approved customer interactions and internal workflows, with clear labeling, source tracking, and strict separation between test and development data

• Score full agent runs: whether intent was understood, answers were grounded, tool calls had the right arguments, policy was followed, and hand-offs to a person happened when they should

• Write deterministic checks for anything that can be verified directly, and save model-based grading for what can't

• Stand up model-graded evaluations, then measure how often they agree with human reviewers and where they skew

• Create synthetic and adversarial cases (vague requests, prompt injection, failed tool calls, actions the agent shouldn't take) and confirm with domain experts that they reflect real risk

• Wire regression suites and release gates into CI/CD so every model, prompt, dataset, and config version is logged and results can be reproduced

• Run validation when the team moves to a new model, and help plan shadow traffic, canary rollouts, and rollback

• Feed production traces back into reviewed test cases and quality dashboards, and partner with the reliability team on alerting and post-incident fixes

Qualifications

• 5+ years of engineering experience across software, data, or machine learning, including building systems that test, compare, or monitor Mz quality in production

• Strong Python and SQL, with experience building testing harnesses, data pipelines, and CI/CD hooks that hold up over time

• Hands-on familiarity with how LLM agents use retrieval and tool calls, and how a mistake in one step carries through to the final answer

• Experience building labeled evaluation sets, guarding against data leakage, and slicing failures by user intent and risk level

• Comfort with statistical comparisons (sample size, variance, significance) and with explaining tradeoffs among quality, latency, and cost

• Experience pairing model-graded scoring with deterministic checks and human review, plus a realistic view of where automated judges fall short

• Background in model and data versioning, production telemetry, and staged releases

• You present evidence clearly and work well without close supervision

Nice to Have

• Google Cloud evaluation tooling or a similar experiment-tracking platform

• Synthetic data generation, red-teaming, judge calibration, or evaluating multi-step tool use

• A model migration or two under your belt

• Fintech or another regulated industry, where datasets must respect privacy and release decisions need an audit trail

Why Join Us

The first milestone is concrete: a representative test set and an automated release suite for one customer-facing use case and one internal workflow, able to catch failures you plant on purpose and produce comparisons anyone can audit. You'll own evaluation engineering, while product owners sign off on acceptance criteria and reliability engineers keep ownership of uptime.

If you've watched an Mz launch go sideways because nobody could say whether the new model was actually better, this is the role where you fix that.

To apply, click Apply on LinkedIn and visit ••••••••