ML Ops Engineer — United States

About The Company

ICF is a global advisory and technology services provider specializing in delivering integrated, strategic solutions that address complex challenges faced by federal clients. Our Digital Modernization Division focuses on information technology and management consulting, bringing deep expertise in cloud computing, cybersecurity, enterprise architecture, data modernization, and digital transformation. We are committed to helping government programs achieve mission-critical objectives through innovative technology adoption and modernization strategies. At ICF, we foster a collaborative environment that empowers employees to grow professionally while contributing to impactful projects that shape the future.

About The Role

The ML Ops Engineer will play a key role in accelerating the modernization of a large federal agency's enterprise data and analytics ecosystem. This role involves developing and maintaining machine learning pipelines, deployment frameworks, model registries, monitoring solutions, and automated workflows that support enterprise-wide AI initiatives. You will collaborate closely with data scientists, data engineers, cloud architects, platform engineers, and Trustworthy AI stakeholders to operationalize machine learning capabilities across cloud environments. This position is remote with a strong preference for candidates located in the Washington DC Metro Area due to occasional onsite meetings at the client site. Candidates must be U.S. citizens and meet federal contract requirements. The role requires expertise in Azure Databricks, MLflow, Azure Machine Learning, Python, and other modern MLOps tools and practices.

Qualifications

• U.S. Citizenship is required due to federal contract requirements.

• Must reside and perform all work within the U.S.

• Must have lived in the U.S. for three full years out of the last five years.

• Bachelor’s degree in Computer Science, Software Engineering, Machine Learning, Data Science, Engineering, or related field, or a high school diploma with four additional years of relevant experience.

• 6+ years of experience designing and supporting production machine learning solutions in cloud environments.

• Master’s degree may substitute for two years of relevant experience.

• Proficiency in Python and modern software development practices.

• Experience with machine learning operations (MLOps), automated deployment pipelines, and model lifecycle management.

• Hands-on experience with Azure Machine Learning, Databricks, MLflow, or comparable ML platforms.

• Experience with source control systems and CI/CD frameworks such as GitHub Actions and Azure DevOps.

• Knowledge of cloud-native architecture, containerization, and scalable model serving patterns.

• Familiarity with infrastructure-as-code tools like Terraform and automated environment provisioning.

• Experience supporting Responsible AI, AI governance, or Trustworthy AI initiatives.

• Strong understanding of cloud cost optimization practices for AI/ML workloads.

Responsibilities

• Design, build, and maintain production-grade MLOps pipelines for model training, evaluation, validation, registration, deployment, monitoring, retraining, rollback, and retirement.

• Develop reusable MLOps and LLMOps patterns using Azure Databricks, MLflow, Unity Catalog, Azure Machine Learning, Python, and approved Azure AI services.

• Implement experiment tracking, model versioning, registries, lineage, quality metrics, model cards, and approval workflows to ensure models are discoverable, reusable, and auditable.

• Build automated CI/CD pipelines using GitHub Actions, Azure DevOps, Databricks Asset Bundles, and Terraform to promote models and infrastructure consistently across environments.

• Deploy and maintain scalable model-serving infrastructure for batch, streaming, real-time, and API-based inference workloads.

• Establish automated testing and evaluation gates for model performance, data quality, fairness, safety, explainability, robustness, privacy, and release readiness.

• Implement model and feature monitoring, drift and bias detection, logging, tracing, alerting, and operational dashboards using MLflow Model Monitoring and Lakehouse Monitoring.

• Develop automated retraining, champion/challenger promotion, rollback, and model-retirement workflows to maintain reliable production operations.

• Integrate ML workflows with source control, infrastructure provisioning, secrets management, identity and access controls, and enterprise release processes.

• Implement Trustworthy AI controls aligned with Government-approved standards and the NIST AI Risk Management Framework.

• Collaborate with cross-functional teams including data scientists, AI engineers, platform engineers, DevSecOps, Security, and Governance personnel.

• Monitor compute and serving utilization, tune configurations, apply tagging conventions, and implement cost-optimization strategies.

• Troubleshoot pipeline failures, deployment issues, model-serving incidents, and monitoring alerts; support root-cause analysis and corrective actions.

• Create and maintain deployment templates, reusable repositories, operational runbooks, model documentation, release evidence, and technical standards.

Benefits

• Competitive salary range from $98,614 to $167,644 based on experience and qualifications.

• Remote work flexibility with occasional onsite meetings in Washington, DC.

• Comprehensive benefits package including health, dental, and vision insurance.

• Retirement savings plans and employer contributions.

• Professional development opportunities and support for continuing education.

• Inclusive and collaborative work environment fostering diversity and innovation.

• Reasonable accommodations for disabilities and religious beliefs throughout the hiring process.

Equal Opportunity

ICF is an equal opportunity employer committed to building a diverse and inclusive workforce. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other protected characteristic. We provide reasonable accommodations for applicants with disabilities and those with sincerely held religious beliefs. Our hiring practices comply with all applicable federal, state, and local laws. We strive to create a workplace where everyone can thrive and contribute to our mission of delivering impactful solutions to our clients.