MLOps Engineer — Reston, VA
Зарплата не указана
ОткликнутьсяDescription
Role summary:
We’re looking for a MLOps Engineer to own the infrastructure that powers our AI/ML platform. Our products rely on generative AI models — image generation, video synthesis, LoRA fine-tuning, and LLM orchestration — running across GPU clusters, distributed training pipelines, and multi-environment deployments. We need someone who can make all of that faster, more reliable, and easier to scale.
You’ll be responsible for the full ML infrastructure lifecycle: building and maintaining training pipelines, managing GPU compute clusters, automating model deployment, optimizing inference performance, and ensuring our systems run reliably across cloud, and air-gapped environments. You’ll work closely with our ML engineers and researchers to translate their work into production-grade systems.
This is a high-autonomy role on a small team where infrastructure decisions have immediate, visible impact on the product. If you want to build the backbone of an AI company — not just maintain someone else’s — this is the opportunity.
What You Will Do
• Design, build, and maintain GPU inference infrastructure on AWS and Azure - including EC2 accelerated computing instances, and SageMaker inference endpoints
• Build and optimize CI/CD pipelines for ML models, from training through evaluation, packaging, and deployment across multiple environments (cloud, air-gapped)
• Automate model deployment workflows — packaging trained models, building optimized inference containers, managing model versioning, and rolling deployments with zero downtime
• Build and maintain custom AMIs and Docker images for GPU workloads using Packer, ensuring consistent environments across development, staging, and production
• Optimize inference performance — GPU memory management, batching strategies and latency reduction for real-time generation endpoints
• Implement monitoring, alerting, and observability for ML systems — GPU utilization, queue depths, model latency, training job health, and cost tracking
• Manage infrastructure as code using Terraform/OpenTofu
• Use AI coding harnesses such as Codex, Claude Code, Pi or Opencode to augment your strong judgement
• Collaborate with ML engineers to debug training failures, optimize data pipelines, and reduce the time from research prototype to production deployment
Work Environment
This role is hybrid but requires occasional in-person work in Reston, VA.
Schedule: Monday – Friday. Occasional off-hours availability for infrastructure incidents, deployment windows, or training job monitoring.
Requirements
Required qualifications:
• U.S. citizenship (required for work with federal customers)
• Willingness to obtain a TS/SCI clearance
• 5+ years of experience in infrastructure engineering, DevOps, or MLOps, with at least 2 years specifically supporting ML/AI workloads in production
• Deep hands-on experience with AWS
• Strong proficiency with infrastructure as code (Terraform required; CloudFormation, or CDK also valued)
• Experience building and managing Docker containers and container orchestration for ML workloads
• Production experience with GPU compute management — CUDA environments, NVIDIA driver management, multi-GPU configuration, and GPU memory optimization
• Strong scripting and automation skills in Python and Bash
• Experience with CI/CD systems (GitHub Actions, GitLab CI, or equivalent) including automated testing, build, and deployment pipelines
• Solid understanding of ML model lifecycle: training, evaluation, packaging, versioning, deployment, and monitoring
• Bachelor’s degree in Computer Science, Systems Engineering, or a related technical field (or equivalent professional experience)
Preferred Qualifications
• Familiarity with generative AI model serving – specifically diffusion or GAN-based models
• Background in air-gapped or disconnected environment deployments (NIST 800-171, CUI compliance, classified networks)
• Familiarity with ML experiment tracking tools (Weights & Biases, MLflow, or similar)
• Experience with cost optimization for GPU compute — spot instances, reserved capacity, right-sizing, and usage monitoring
• Prior work supporting federal law enforcement, intelligence community, or defense infrastructure
• Familiarity with Electron apps, or other desktop apps