AI Engineer – AI Observability and Quality

• Design and build the evaluation infrastructure that ensures the platform's AI systems produce accurate, well-sourced, high-quality responses
• Build automated test suites that validate answer quality across agent pipeline changes
• Develop regression detection systems that catch quality degradation before it reaches users
• Create evaluation frameworks that measure response accuracy, citation correctness, and source quality
• Work closely with the product team to translate quality criteria into measurable, automatable test scenarios
• Build cost and latency monitoring that tracks the operational efficiency of AI pipeline execution
• Define evaluation standards and practices that scale as the platform and team grow

• 6+ years of software engineering experience, with significant work in testing infrastructure, ML evaluation, or quality systems
• Experience building evaluation or testing frameworks for LLM-based or ML-based systems
• Understanding of how to measure response quality for generative AI: accuracy, groundedness, citation correctness, relevance
• Proficiency in Python
• Ability to operate independently and define your own roadmap. You will be the first hire in this function
• Experience working at the intersection of engineering and product, translating qualitative quality criteria into quantitative measurements
• Experience with LLM evaluation frameworks (e.g., RAGAS, DeepEval, or custom)
• Familiarity with LLM observability tools (e.g., Langfuse, LangSmith, Weights & Biases)
• Background in statistical methods for quality measurement (significance testing, distribution analysis)
• Experience building A/B testing or experimentation infrastructure
• Background in search relevance evaluation or information retrieval metrics

IDC is committed to providing equal employment opportunities for all qualified persons. Employment eligibility verification required. We participate in E-Verify.