AI Observability Engineering Technical Leader

What you'll do

• Define and execute the end-to-end observability platform vision for AI software services.
• Partner with software teams to instrument services consistently and achieve 100% distributed tracing coverage.
• Own and evolve OpenTelemetry enablement, including libraries, design patterns, and process documents for Go and Python.
• Drive best practices for logging standards and cardinality management across the organization.
• Operate and maintain platform components including Splunk Cloud, OTEL collectors, and telemetry pipelines.
• Develop agentic observability outcomes to enable AI agents to analyze health signals and deliver actionable information.
• Build automation and operational tooling in Python or Go to support the observability platform.

What we're looking for

• Bachelor’s degree in Computer Science, Engineering, or related field plus 12 years of experience in platform services or DevOps.
• Master’s degree plus 8 years of related experience.
• PhD plus 5 years of related experience.
• At least 2 years of experience in Splunk Cloud and Splunk Observability (or SignalFx).
• Experience with distributed tracing concepts, implementation, and instrumentation strategies.
• Experience coding in Python and/or Go to build libraries, automation, or operational tooling.
• Experience with Kubernetes, OpenTelemetry collectors, infrastructure-as-code (Terraform), and Helm charts.
• Experience driving large-scale distributed tracing initiatives or using AI agents to refine observability outcomes (preferred).