AI Quality Engineer

1.Quality Engineering Leadership Across the Platform 

- Represent and elevate the Central QE community, partnering closely with product and engineering teams to embed quality at every stage of development. 

- Contribute to a modern Quality strategy for a fast-scaling AI First SaaS architecture. 

- Improve CI/CD workflows with better test isolation, data setup, and execution reliability. 

2. AI-Focused Test Strategy & Execution 

- Develop primarily deterministic and contribute into non-deterministic testing strategies for LLM-based agentic systems. 

- Build automation to validate the offline and online evaluation pipelines for correctness, safety, drift, and reliability. 

- Design and implement test harnesses for agent endpoints, tool-calling flows, and planning sequences. 

- Build automation to validate AI artifacts including prompt libraries, golden datasets, rubric scoring, eval outputs and model versions. 

- Validate RAG pipelines end-to-end and integrate AI-specific metrics into CI/CD. 

- Build safety, compliance, and guardrail validation mechanisms. 

3. Functional & Cross-Team Testing 

- Partner across teams to ensure timely, high-quality delivery of roadmap commitments. 

4. Quality Metrics, Telemetry & Observability 

- Reporting discipline for DORA metrics, test coverage, test effectiveness, and health execution. 

- Extend metrics with AI-specific quality indicators. 

- Expand observability for AI systems including degradation patterns and anomalous outputs. 

5. Culture, Leadership & Collaboration 

- Promote a quality-first mindset across Engineering. 

- Lead cross-functional initiatives to ensure quality influences design, development, deployment, and incident reviews. 

- 5+ years experience in Quality Engineering within cloud-native SaaS organizations. 

- Expertise in automated testing infrastructure and CI/CD pipelines. 

- Strong understanding of testing strategies across the pyramid. 

- Solid frontend and full-stack testing collaboration skills. 

- Familiarity with observability tooling. 

- Experience testing LLMs, AI agents, RAG pipelines. 

- Knowledge of AI safety risks including hallucinations and prompt injections.