AI Quality Analyst, AI & Innovation — Kansas City, MO

Your Responsibilities

The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team’s AI data extraction processes, which convert commercial risk insurance documents — such as policies, endorsements, submissions, schedules of values, and loss runs — into structured data. Rather than building the extraction systems, this role exists to support the organization’s Enterprise AI Evaluation Strategies — the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use. The role spans both the foundation and the practice of evaluation: building and curating the ground-truth (“golden”) datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process.

A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what “correct” looks like and to safeguard extraction quality end to end. The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement — without requiring hands-on AI/ML development experience.

Key Responsibilities

• Own, create, and maintain the curated ground-truth (“golden”) datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality — the foundation of the team’s evaluation work.

• Contribute to the design, execution, and continuous improvement of the organization’s Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored.

• Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity.

• Participate in evaluation cycles — baseline, regression, and pre-release testing — whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production.

• Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human-reviewed ground truth to confirm those methods can be trusted at scale.

• Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction.

• Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured consistently and reproducibly.

• Apply commercial risk insurance knowledge to define what a correct extraction looks like for key policy and risk data — such as named insureds, lines of coverage, limits, deductibles and retentions, premiums, exclusions, and loss history — and to resolve ambiguous or complex cases.

• Engage with and train business users on the ground-truth application, guide them through creating and reviewing ground-truth data, and collect the associated source files and validated outputs needed to build and expand the ground-truth library.

• Review AI extraction outputs against ground truth, identify and categorize discrepancies and error types, and deliver actionable diagnostics and feedback to the team advancing the extraction process.

• Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement, to keep ground-truth data accurate, consistent, and trustworthy.

• Manage the versioning and lifecycle of ground-truth datasets so quality results remain reproducible and comparable as models, prompts, and processes change.

• Partner with developers to define and interpret quality metrics and acceptance criteria — accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance — and help translate results into clear findings.

• Monitor quality trends across extraction runs to surface drift, degradation, and anomalies before they reach downstream products, escalating issues to the team.

• Maintain a comprehensive, working understanding of the existing AI data extraction processes the team leads, partnering closely with the Manager and team in support of their continuous improvement.

• Support responsible-AI review of AI outputs, including robustness, bias, safety, and compliance considerations at the output level.

• Define and maintain quality metrics, dashboards, and reporting that give the team and stakeholders clear visibility into extraction performance and ground-truth coverage.

• Document ground-truth methodology, metric definitions, annotation standards, and quality practices to establish and mature the team’s AI quality practice.

• Collaborate with the Analytics Engineering function, which consumes validated AI-extracted data into Gold-layer assets, and the Applied AI Analyst, which evaluates AI from a business and product perspective, to align quality standards with downstream needs.

Qualifications

• 2–5+ years of experience in data quality, quality assurance, data analysis, annotation/labeling, or a related analytical or business-facing role, ideally with exposure to AI/ML, LLM, or data-extraction systems. Hands-on AI/ML or software development experience is not required.

• Experience contributing to structured testing, validation, or evaluation processes, such as QA test plans, user acceptance testing, or data or model validation.

• Working knowledge of commercial risk insurance — including policy structure, coverage terms, limits, deductibles, exclusions, endorsements, submissions, and loss runs — gained through underwriting, brokerage, claims, risk management, or insurance operations, sufficient to judge whether extracted data is accurate and complete.

• General familiarity with large language models and generative AI — what they do well, where they fall short, and why they can produce inaccurate or fabricated outputs — sufficient to understand, review, and reason about AI outputs.

• Strong analytical rigor and exceptional attention to detail, with the ability to assess extraction accuracy and translate findings into clear, actionable recommendations.

• Excellent communication and stakeholder-engagement skills, with the ability to train and support business users and to build effective ground-truth workflows with non-technical partners.

• Comfort working with data and data tools.

Preferred Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field.

• Familiarity with technical generative-AI concepts such as prompt engineering, retrieval-augmented generation (RAG), and LLM-as-judge evaluation.

• Understanding of evaluation methodology and metrics for AI/ML and information extraction, such as precision, recall, F1 score, accuracy, and hallucination detection.

• Experience building, curating, or maintaining labeled or ground-truth datasets and annotation workflows, including inter-annotator agreement.

• Insurance industry designations such as CPCU, AINS, or ARM.

• Basic proficiency in SQL for querying evaluation and production data.

• Basic scripting or Python familiarity; the role is centered on quality review and data stewardship rather than software development.

• Familiarity with AI/LLM observability and evaluation tooling such as LangSmith.

• Familiarity with the Databricks platform and tools such as MLflow and Unity Catalog.

Success Measures

• Coverage, accuracy, and freshness of the ground-truth datasets, and how well they support the evaluations they are built for.

• Meaningful contribution to Enterprise AI Evaluation Strategies, including reliable evaluation cycles, clear acceptance criteria, and adoption of evaluation standards across AI initiatives.

• The degree to which developers can quickly and confidently measure and validate extraction quality using the ground-truth files provided.

• Adoption and proficiency of business users on the ground-truth application, and the volume and quality of source files and validated outputs collected.

• Early detection of quality drift, degradation, and anomalies before they impact downstream products.

• Clarity and usefulness of the quality metrics, dashboards, diagnostics, and feedback provided to the team and stakeholders.