Senior Analyst Model Risk & AI, Gen AI Model Validation

What you'll do

• Perform model validations on GenAI and agentic AI models to ensure they meet internal standards and regulatory expectations.
• Design and build challenger solutions and testing methods for tasks like summarization, question answering, and data synthesis.
• Assess key data inputs, assumptions, prompt engineering, and context engineering for accuracy and appropriateness.
• Design and execute testing approaches for non-deterministic AI systems, including variability analysis and statistical evaluation.
• Identify model risk findings and provide recommendations to mitigate risks through clear and concise validation reports.
• Manage governance accountabilities including findings tracking, remediation testing, and validation.
• Enhance the GenAI model validation framework by developing standardized evaluation metrics and deploying validation tools.
• Evaluate and test cutting-edge AI tools such as VertexAI, LangChain, RAG frameworks, and OpenAI APIs.

What we're looking for

• Advanced degree (M.S. or Ph.D.) in a relevant field such as AI, Machine Learning, Statistics, or Computer Science.
• 3+ years of industry experience in machine learning or data science.
• 1+ years of experience focused on GenAI and/or agentic AI.
• Strong programming and technology experience across platforms including Python, SQL, and Git.
• Solid understanding of GenAI concepts including prompt engineering, RAG, LLM internals, and multi-agent systems.
• Experience in modern cloud and AI tools such as GCP/Vertex AI, AWS/SageMaker, LangChain, and OpenAI APIs.
• Excellent communication skills to explain complex technical concepts to non-technical stakeholders.
• P&C, Group, Life or related insurance product experience (preferred).