AI Quality Architect — Казахстан, Алматы

We are looking for a senior hands-on architect who designs how software quality is assured when AI agents generate, execute, and maintain most of the testing. They own the architecture of the quality system: agent pipelines, context and oracle strategy, trust and risk boundaries, and cost and reliability of AI-driven testing. They are not a test manager and not a script writer. They decide what agents may do autonomously, how we know their results are correct, and where humans must stay in the loop.

Seniority: Staff / Principal level (typically 10+ years in software quality or engineering, at least 1–2 years of real production work with LLMs or agents).

1. Core responsibilities

• Design the end-to-end AI-driven test architecture (generation, execution, self-healing, triage, reporting).

• Define the oracle strategy: how correctness is specified so that agents can verify it.

• Build the evaluation framework for the agents themselves (evals, regression sets, drift detection, precision/recall of defects found).

• Define autonomy levels, guardrails, and human-in-the-loop checkpoints; own AI-in-QA governance and risk.

• Manage context architecture: specs, requirements, code, and test knowledge exposed to agents (RAG, MCP tools, knowledge graphs).

• Control cost, latency, and flakiness of AI-based pipelines; integrate into CI/CD.

• Set standards and mentor QA engineers, analysts, and developers across teams.

2. Must-have (hard requirements)

• Background in test architecture or quality engineering at scale: SDET, test automation architecture, or developer productivity.

• Production experience building or operating LLM/agent-based workflows (not only using Copilot or ChatGPT).

• Solid engineering skills: Python and/or TypeScript/Java, API and service architecture, CI/CD, containers.

• Understanding of LLM limitations: hallucination, non-determinism, prompt and context design, evaluation methods.

• Experience designing test strategy for distributed systems (microservices, event-driven, API-first).

• Ability to show measurable outcomes (escaped defects, cycle time, cost per test, automation reliability).

3. Nice-to-have

• Experience testing AI/ML products (LLM evals, red-teaming, safety and bias testing).

• MCP (Model Context Protocol) server or tool development; multi-agent orchestration.

• Observability and test analytics (OpenTelemetry, test impact analysis, flaky-test analytics).

• Domain with complex business logic (telecom, billing, fintech, enterprise B2B).

• Public footprint: talks, articles, open-source, patents.

4. We offer:

• Competitive salary;

• Medical insurance, covering dental services;

• More than 300 hard and soft-skills programs by the corporate career development center;

• Open environment and encouraging knowledge sharing culture;

• Free English courses and opportunity to practice foreign languages daily;

• Flexible working hours.