AI Evaluation Lead

Responsibilities: Own the measurement and quality assurance of AI-generated financial advice by defining evaluation sets and analyzing scoring results. Translate findings into actionable recommendations for the AI/ML team to improve system performance and scale.

Requirements: Requires experience in AI/ML system quality within high-stakes contexts and a baseline literacy in personal finance. Candidates must be comfortable with ambiguity and possess fluency in LLM failure modes and automated scoring limits.

Key skills: AI Evaluation, ML Quality Assurance, Personal Finance Literacy, LLM Production Behavior, Data Analysis, Model Risk Validation, Observability Tooling, Analytical Thinking, Judgment Call Making, Evaluation Framework Design, Case Study Analysis, Cross-functional Collaboration

Keywords: AI Evaluation, Machine Learning, LLM, Generative AI, Fintech, Model Risk, QA, Data Science, Observability, Financial Advice, Automated Scoring, Production AI, Evaluation Framework, Model Validation, Personal Finance