AI/ML Engineer — Chapel Hill, NC

About the Role

Our client, a growing technology consultancy, is seeking a Lead AI/ML Engineer to help build the intelligence layer behind an AI platform designed specifically for the banking industry. The consultancy focuses on helping banks build and deploy secure, production-grade AI solutions.

This is a hands-on engineering role focused on production LLM applications, Retrieval-Augmented Generation (RAG), retrieval, model serving, tool orchestration, and evaluation. The goal is to develop AI systems that deliver accurate, verifiable, and auditable answers in highly regulated environments, rather than simply building AI demonstrations.

You will work closely with data engineering, platform engineering, and product teams to build secure AI systems for banking environments, including self-hosted and on-premises deployments.

Key Responsibilities

• Design and build production-grade RAG and retrieval systems that provide verifiable answers with supporting queries, sources, and citations.

• Develop retrieval and verification workflows across structured and unstructured banking data.

• Deploy and operate self-hosted open-weight LLMs and embedding models, including within individual bank environments.

• Build and evolve standardized RAG architectures and implementation patterns.

• Design and implement an MCP and tooling layer that enables AI systems to securely interact with governed data, metrics, documents, and customer information.

• Develop tool and function-calling workflows with strong auditing, access controls, and clearly defined boundaries around AI system access.

• Build and own an AI evaluation framework covering golden-question regression testing, retrieval quality, groundedness, answer accuracy, and appropriate "I don't know" behavior.

• Establish evaluation as a formal release gate for production AI systems.

• Implement LLM guardrails, including PII redaction, prompt-injection defenses, context controls, and query and row limits.

• Partner with data engineering teams to ensure AI systems use governed semantic-layer metrics rather than generating unreliable ad hoc SQL.

• Collaborate with platform and security engineering teams to deploy reliable AI infrastructure in controlled environments.

• Monitor and optimize latency, inference costs, retrieval quality, and model performance across different models and deployment configurations.

• Document model architecture, evaluation methodology, security controls, and guardrails for customer security reviews and audits.

• Provide technical leadership while remaining hands-on with implementation and production troubleshooting.

Required Skills & Qualifications

• 6+ years of software or machine learning engineering experience, including at least 2–3 years building and delivering production LLM, RAG, NLP, or generative AI systems.

• Proven experience taking AI systems from prototype to production and delivering them to real users.

• Strong Python skills and solid software engineering fundamentals.

• Hands-on experience building production RAG and retrieval pipelines.

• Strong understanding of LLM evaluation, including groundedness, retrieval quality, regression testing, and hallucination reduction.

• Experience implementing AI guardrails and security controls.

• Experience deploying or operating self-hosted and open-weight models.

• Ability to evaluate trade-offs between model quality, latency, infrastructure, and cost.

• Strong understanding of prompt engineering, tool calling, structured outputs, and LLM application architecture.

• An engineering mindset focused on accuracy and verifiability rather than simply building compelling AI demonstrations.

• Ability to collaborate across data, platform, security, and product teams.

Technology Environment

The team works with a modern AI/ML technology stack that includes:

• Languages: Python

• Model Serving: vLLM, Ollama

• Infrastructure: NVIDIA GPUs, CUDA, NVIDIA AI Enterprise

• RAG & Retrieval: LlamaIndex, Haystack, Qdrant, pgvector

• Embeddings: Modern embedding models and vector search

• AI Orchestration: MCP, tool/function calling

• Structured AI: Text-to-SQL, semantic layers, Cube, dbt MetricFlow

• Evaluation: LLM evaluation frameworks, groundedness testing, retrieval evaluation

• AI Security: PII redaction, prompt-injection defense, access controls

Experience with every technology listed is not required. The team is particularly interested in engineers who understand the architecture and engineering principles behind production AI systems and can quickly adapt to new tools.

Preferred Qualifications

• Experience building AI systems in banking, financial services, healthcare, cybersecurity, or another high-stakes or regulated environment.

• Experience deploying AI systems in on-premises or highly controlled environments.

• Experience with fine-tuning, adapters, or retrieval-quality optimization.

• Familiarity with banking, financial services, or financial data.

• Experience with AI security, governance, or compliance frameworks.

• Experience working with GPUs, CUDA, and production inference infrastructure.

About the Ideal Candidate

This role is suited to an engineer who has moved beyond LLM experimentation and has experience operating AI systems in production. Relevant backgrounds may include Lead AI Engineer, Senior ML Engineer, LLM Engineer, Applied AI Engineer, or Staff AI Engineer.

The strongest candidates will have built RAG systems, deployed models, implemented evaluation frameworks, and addressed the practical challenges of production AI, with a strong focus on ensuring that AI-generated answers are accurate and reliable.

Why Join This Opportunity?

• Build AI systems that move beyond prototypes and are deployed for use by banking professionals.

• Take ownership of AI architecture across RAG, model serving, evaluation, tooling, and guardrails.

• Work with self-hosted AI infrastructure where security, privacy, and deployment constraints require alternatives to cloud-only approaches.

• Solve complex reliability challenges by making AI systems accurate, grounded, observable, and resistant to failure.

• Work at the intersection of LLM engineering, structured data, semantic layers, and intelligent retrieval.

• Help address high-impact challenges in banking, where inaccurate answers can have significant business and regulatory consequences.

• Establish engineering standards and best practices for production AI systems while remaining actively involved in development.