Staff Software Engineer, Applied AI

What you'll do

• Build agent harnesses for planning, tool use, memory, context compaction, and recovery during long-running tasks.
• Integrate OpenAI and Anthropic model APIs while selecting models based on quality, latency, and cost.
• Evaluate existing harnesses and build missing components connected to AWS workflow execution and approval states.
• Enforce application policy checks for authorization and sensitive writes, requiring evidence of state before completion.
• Build evaluation datasets and replay tests for valid findings, access decisions, and AI agent onboarding.
• Measure false positives, missed risks, and unsafe actions to turn production failures into regression tests.
• Test for prompt injection, stale evidence, conflicting instructions, and unsafe tool use to protect tenant boundaries.
• Improve prompts, retrieval, tools, and model routing, including evaluating fine-tuning or post-training methods.

What we're looking for

• Strong Python engineering skills and experience shipping an LLM application that uses tools.
• Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.
• Experience integrating model APIs, structured outputs, and asynchronous services.
• Experience with OpenAI or Anthropic is especially relevant.
• Depth in agent execution or evaluation systems with working code and failure analysis.
• Ability to distinguish a plausible model response from a supported finding or verified action.
• Open-source contributions, agent observability, security evaluations, or synthetic evaluation data (preferred).
• Fine-tuning, preference data, post-training evaluation, or model distillation using approved datasets (preferred).