Manager, AI Engineering

What you'll do

• Design and own end-to-end LLM evaluation frameworks including automated prompt regression, scoring, and hallucination detection.
• Build comprehensive test suites for agentic AI systems to validate tool selection, coordination, and multi-step reasoning workflows.
• Develop RAG pipeline evaluation frameworks to assess retrieval precision, context faithfulness, and answer grounding.
• Lead structured red-teaming and adversarial testing to defend against prompt injection, jailbreaks, and data leakage.
• Execute fairness, bias, and Responsible AI audits to ensure demographic equity and explainability.
• Perform inference performance benchmarking and enforce LLM quality gates within CI/CD pipelines on Databricks.
• Build production monitoring and drift detection pipelines using observability tools like Grafana and Datadog.
• Define the AI testing roadmap, quality standards, and evaluation metrics for all Generative AI workstreams.

What we're looking for

• Bachelor's or Master's degree in Computer Science, AI/ML, or Software Engineering.
• Experience leading AI/ML quality engineering or LLM testing programs in production environments.
• Expertise in testing LLM and Gen AI systems including prompt testing, hallucination detection, and RAG pipeline assessment.
• Proficiency with evaluation tools such as RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, or Weights & Biases Evals.
• Strong Python programming skills for building test automation scripts and SQL proficiency.
• Knowledge of LLM ecosystems including OpenAI, Anthropic, Hugging Face, and LangChain/LangGraph.
• Experience with MLOps/LLMOps pipelines (MLflow, Databricks, SageMaker) and integrating quality gates into CI/CD workflows.
• Familiarity with cloud AI infrastructure (AWS, Azure, or GCP) and observability tooling like Grafana, Datadog, or CloudWatch.