Senior Data Scientist AI Evaluation

We are seeking a senior-level data scientist to establish and advance the evaluation practices that measure the quality, reliability, and correctness of AI models and intelligent agents. In this role, you will transform complex quality questions into measurable standards, trusted ground truth, and rigorous evaluation frameworks. You will design repeatable testing processes that identify performance regressions and support safer, more confident AI releases. Working closely with Product, Engineering, Analytics Engineering, and business stakeholders, you will turn evaluation results into practical improvements. This is a high-impact individual contributor position offering ownership of a developing AI evaluation practice within a fast-growing financial technology environment. You will help shape how AI quality is measured, validated, documented, and continuously improved across the organization.

• Approximately 6–10 years of experience in quantitative data science, machine learning, or a related technical discipline, with focused experience in measurement, evaluation, experimentation, or model validation.

• Strong quantitative and statistical foundations, including experience designing rigorous experiments and interpreting results with appropriate uncertainty measures.

• Demonstrated experience defining meaningful evaluation metrics, establishing ground truth, and assessing the quality of complex or ambiguous model outputs.

• Proficiency in Python and SQL, with experience applying these skills to data analysis, model assessment, and evaluation workflows.

• Experience evaluating machine learning models in production environments and translating findings into practical improvements.

• Ability to validate automated graders against human judgments and account for variability, non-determinism, and potential measurement bias.

• Strong problem-solving and analytical judgment, particularly when developing evaluation approaches for new or evolving AI systems.

• Excellent communication skills, with the ability to explain evaluation results, trade-offs, and recommendations to technical teams and business stakeholders.

• Proven ability to collaborate across functions while independently owning complex analytical projects in a fast-paced environment.

• A quantitative degree in data science, statistics, mathematics, computer science, or a related field is an advantage; equivalent practical experience is also welcome.

• Hands-on experience evaluating large language models (LLMs) or AI agents in production, including evaluation harnesses, LLM-as-judge calibration, and continuous integration regression gates, is a plus.

• Experience evaluating text-to-SQL systems, analytics agents, or other AI applications where outputs can be verified against underlying data is an advantage.

• Background in fintech, brokerage, financial services, or other domains where incorrect outputs can create significant business or risk consequences is desirable.

• Familiarity with AI tools used in research, analysis, and engineering workflows is beneficial.