U6tjqsK28V7wmg bkGwXXJJ — United States

apugsTdZWcwB qDDNxE6D – AI Chatbot Evaluations

Location: 100% Remote (US) – must work Pacific Time business hours

Role Type: Contract, 12 months (W-2)

About Our Client

Our client is a global financial technology company, headquartered in the San Francisco Bay Area. Its tax, accounting, personal finance and marketing products serve roughly 100 million consumers and small businesses worldwide, and its small-business accounting platform is one of the most widely used in the US. Over the past few years, the company has been transforming into an AI-driven platform, building generative-AI assistants and agents into its products. Its mission is to power prosperity around the world, and it has a reputation for customer obsession, innovation and a strong engineering and design culture.

Job Description

Our client's small-business platform includes an AI chatbot, built on a leading large language model and extended with company-specific skills and agents, that helps business owners get answers and complete tasks in the product. As more customers rely on it, making sure every answer is accurate, relevant and on-brand is critical. That's why the team is adding three bsUQ447RCPSF xHxvu6gCc focused on AI chatbot evaluation and quality.

In this role, you'll build the question-and-answer datasets used to test the chatbot, review and score its responses against approved source content and quality rubrics, and track accuracy over time. When quality drops in an area, you'll dig in, find the pattern behind the errors and give Engineering clear, actionable feedback. You'll work closely with conversation designers, product managers, engineers and data scientists on the conversational AI team.

This is not a visual UX/UI design role and not a software engineering role, and no Python or coding is required. It's ideal for an analytical, detail-oriented person who loves AI, enjoys figuring out why a chatbot got something wrong, and uses AI tools (such as Claude Code or ChatGPT) to work faster and smarter.

Duties and Responsibilities

• Create large datasets of realistic customer questions with verified correct answers.

• Develop question-and-answer pairs used to evaluate chatbot performance.

• Review AI-generated responses for accuracy, relevance and overall quality.

• Compare chatbot answers against approved source content and documentation.

• Score responses using established evaluation rubrics and conversation design principles.

• Identify incorrect answers, inconsistencies and recurring error patterns.

• Calculate accuracy percentages and monitor performance trends over time.

• Investigate areas where chatbot quality is declining and pinpoint likely causes.

• Provide clear, actionable feedback to Engineering and product partners.

• Use AI tools to streamline evaluation and quality-assurance workflows.

• Build simple productivity tools or automations with AI-assisted tools (e.g., Claude Code) when helpful.

Required Experience/Skills

• 3–5 years of experience in conversation design, chatbot design, conversational AI, AI/LLM evaluation or closely related work (strong candidates with less experience will be considered).

• Hands-on experience evaluating chatbot or AI responses for accuracy and relevance.

• Experience working with structured content, evaluation criteria or quality rubrics.

• Comfortable reviewing large datasets and identifying patterns and trends.

• Ability to calculate basic metrics (percentages, accuracy rates) and analyze trends.

• Strong analytical and logical reasoning skills.

• Ability to understand and consistently apply existing design principles and guidelines.

• Exceptional attention to detail.

• Comfortable learning and using AI-powered tools in daily work.

• Clear written communication for documenting findings and giving feedback.

• Able to work Pacific Time business hours.

Nice-to-Haves

• Experience with Claude Code or similar AI-assisted tools.

• Experience with LLM evaluations, AI data labeling/annotation or response quality assessment.

• Experience designing chatbot or voice bot conversations (e.g., Dialogflow, Amazon Lex, ••••••••, Voiceflow).

• Familiarity with generative AI workflows and prompt writing.

• Experience at a SaaS or technology company.

• Knowledge of small-business accounting software or bookkeeping concepts.

• Experience building simple tools or automations using AI.

• Background as a technical writer or knowledge base specialist who has worked on chatbot content and AI evaluation.

Education

Bachelor's degree preferred (e.g., HCI, Interaction Design, Linguistics, Psychology, Computer Science or a related field), or equivalent practical experience.

Pay & Benefits Summary

• Pay rate: $50.35/hr (W-2 contract).

• 12-month contract.

• 100% remote – work from anywhere in the US on Pacific Time hours.

• Hands-on work with cutting-edge generative AI on a product used by millions of small businesses.

Love AI and have a sharp eye for when a chatbot gets it wrong? Work from the comfort of your home and help shape the quality of an AI assistant used by millions.

apugsTdZWcwB qDDNxE6D | uJgrBxySp6QWXE Xq | Chatbot | LLM Evaluation | AI Evaluation | AI Quality Analyst | Prompt Engineering | Generative AI | Remote