ML Platform Engineer — United States

This is a Fully Remote Job

About Our Client:

The organization operates in the technology consulting and software development industry, specializing in delivering cloud, artificial intelligence, data, and enterprise solutions across the United States. It addresses the challenge of integrating advanced AI and software technologies into business environments, supporting clients with scalable and reliable solutions.

About the Opportunity:

The ML PLATFORM ENGINEER is responsible for designing, building, and operating high-performance inference platforms for large machine learning models in production. This role focuses on optimizing AI deployment systems to improve efficiency, reliability, and scalability. The position contributes significantly by ensuring the organization can deliver robust AI services that balance latency, throughput, cost, and quality, thereby supporting diverse model workloads and advancing AI infrastructure capabilities.

Responsibilities:

• Design and operate model serving platforms for various workloads including language models, vision models, and recommendation systems.

• Optimize inference performance through techniques such as batching, speculative decoding, and request multiplexing.

• Implement multi-tenant routing, rate limiting, and quality-of-service policies.

• Develop autoscaling and capacity management systems balancing latency, throughput, and cost.

• Tune GPU utilization, memory management, and caching strategies for large model serving.

• Integrate model serving with API gateways, identity systems, and observability tools.

• Implement caching, prompt deduplication, and response reuse strategies.

• Manage end-to-end observability covering latency, queue dynamics, GPU utilization, and error tracking.

• Develop deployment workflows including canary releases and automated rollback.

• Lead incident response and reliability improvements for AI services.

• Collaborate with ML and product teams on new model releases and features.

• Implement security controls like request signing, content filtering, and abuse detection.

• Document operational procedures, performance metrics, and tuning guidance.

• Stay current with AI serving research and apply advancements to production.

Requirements:

• Bachelor’s or Master’s degree in Computer Science or related field.

• Over 10 years of experience in distributed systems, infrastructure, or ML platform engineering.

• Proficiency in Python and a systems programming language such as Go, Rust, or C++.

• Experience operating high-throughput, low-latency production services.

• Hands-on experience with large language model inference frameworks.

• Strong understanding of GPU architecture and accelerator utilization.

• Familiarity with Kubernetes, autoscaling, and cloud platforms.

• Experience with observability tools including metrics, tracing, and logging.

• Expertise in performance engineering and capacity planning.

• Strong communication and incident response abilities.

Pay Range and Compensation Package:

• Salary range: $100,000 to $160,000 annually.

Equal Opportunity Statement: Our client is an equal opportunity employer. They celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, or national origin.

Note:

hcEzJ64Sq is a recruitment partner of this role. Please note that all employment decisions, including candidate assessment, interviews, hiring, compensation, and employment terms, are made exclusively by the hiring employer.