ML Ops Engineer — United States

About The Company

ICF is a global advisory and technology services provider specializing in information technology and management consulting. Our Digital Modernization Division offers integrated, strategic solutions to federal clients, leveraging expertise in cloud computing, cybersecurity, enterprise architecture, data modernization, and digital transformation. We are committed to helping government agencies modernize mission-critical programs through innovative technology solutions that drive efficiency and impact. Our teams collaborate across disciplines to deliver measurable results that support public sector missions.

About The Role

The ML Ops Engineer will be a key contributor to the modernization of a large federal agency’s enterprise data and analytics ecosystem. This role involves developing and maintaining machine learning pipelines, model registries, deployment frameworks, monitoring solutions, and automated workflows that support enterprise-wide AI initiatives. The position requires close collaboration with data scientists, data engineers, cloud architects, platform engineers, and Trustworthy AI stakeholders to ensure successful operationalization of machine learning capabilities in cloud environments. This is a remote position with a strong preference for candidates residing in the Washington DC Metro Area, due to occasional onsite meetings. Note that this role is contingent upon contract award and all work must be performed within the U.S. under federal contract requirements.

Qualifications

• U.S. Citizenship is required due to federal contract requirements.

• Candidate must reside and be authorized to work in the U.S.; all work must be performed in the U.S.

• Candidate must have lived in the U.S. for at least three (3) full years out of the last five (5) years.

• Bachelor's degree in Computer Science, Software Engineering, Machine Learning, Data Science, Engineering, or related field, or a high school diploma with four (4) additional years of relevant experience in lieu of a degree.

• 6+ years of experience designing and supporting production machine learning solutions in cloud environments.

• Master's degree may substitute for two (2) years of relevant experience.

• Experience with MLOps, automated deployment pipelines, and model lifecycle management.

• Proficiency in Python and modern software development practices.

• Experience with source control systems and CI/CD frameworks.

• Hands-on experience with Azure Machine Learning, Databricks, MLflow, or comparable platforms.

• Knowledge of cloud-native architecture, containerization, and scalable model serving patterns.

• Experience supporting Responsible AI, AI governance, or Trustworthy AI initiatives is preferred.

• Familiarity with infrastructure-as-code and automated environment provisioning.

• Strong understanding of cloud cost optimization for AI/ML workloads.

Responsibilities

• Design, build, and maintain production-grade MLOps pipelines for model training, evaluation, validation, registration, deployment, monitoring, retraining, rollback, and retirement.

• Develop reusable MLOps and LLMOps patterns using Azure Databricks, Databricks ML, MLflow, Unity Catalog, Azure Machine Learning, Python, and approved Azure AI services.

• Implement experiment tracking, model versioning, registries, lineage, quality metrics, model cards, and approval workflows to ensure discoverability, reusability, and auditability of models.

• Build automated CI/CD pipelines using GitHub Actions, Azure DevOps, Databricks Asset Bundles, and Terraform to promote models, code, configurations, and infrastructure across environments.

• Deploy and maintain scalable model-serving infrastructure supporting batch, streaming, real-time, and API-based inference workloads.

• Establish automated testing and evaluation gates for model performance, data quality, fairness, safety, explainability, robustness, privacy, and release readiness.

• Implement model and feature monitoring, drift and bias detection, logging, tracing, alerting, and operational dashboards.

• Develop automated retraining, champion/challenger promotion, rollback, and model-retirement workflows to prevent performance degradation.

• Integrate ML workflows with source control, infrastructure provisioning, secrets management, identity and access control, environment management, and enterprise release processes.

• Implement Trustworthy AI controls aligned with Government-approved standards and the NIST AI Risk Management Framework.

• Partner with multidisciplinary teams including data scientists, AI engineers, DevSecOps, security, and governance personnel to ensure models are repeatable, testable, deployable, and operationally supportable.

• Monitor compute and serving utilization, tune configurations, apply tagging conventions, and implement cost-optimization strategies.

• Troubleshoot pipeline failures, deployment issues, model-serving incidents, and performance degradations, supporting root-cause analysis and corrective actions.

• Create and maintain deployment templates, reusable repositories, operational runbooks, model documentation, release evidence, and technical standards.

Benefits

• Competitive salary range from $98,614 to $167,644 based on experience and qualifications.

• Remote work opportunity with occasional onsite meetings in Washington, DC.

• Comprehensive benefits package including health, dental, and vision insurance.

• 401(k) retirement savings plan with company match.

• Paid time off and holiday leave.

• Professional development and training opportunities.

• Supportive and inclusive workplace culture.

• Reasonable accommodations available for disabilities and religious beliefs.

Equal Opportunity

ICF is an equal opportunity employer. We are committed to building a diverse and inclusive workplace where all employees feel valued and empowered. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other legally protected status. We provide reasonable accommodations to qualified individuals with disabilities and for sincerely held religious beliefs in all phases of the employment process. Our commitment to equal opportunity extends to all aspects of employment, including hiring, promotion, compensation, benefits, and training. For assistance or accommodation requests, please contact ••••••••.