About the Role
Join an AI team building systems that help teams identify and address failures in production AI agents. You will own the intelligence behind failure detection, evaluation, and proposed fixes, helping make agents more reliable and trustworthy.
What You'll Do
Detect subtle agent failures, including omissions, compliant but incorrect answers, and patterns across large volumes of traces.
Use user behavior signals such as rephrasing, abandonment, and retries to identify issues that users do not explicitly report.
Develop evaluation methods to measure detection quality when labeled ground truth is unavailable.
Build trustworthy patch generation workflows that reproduce failures, verify fixes, and avoid proposing unreliable changes.
Improve the cost-effectiveness of detection by choosing appropriate models and methods for each task.
Review production customer traces regularly to find opportunities for improvement.
What We're Looking For
Applicants with 0 to 10 years of experience are welcome, including new graduates and candidates without a degree.
Experience shipping products or systems that people use.
Hands-on experience with production LLMs, evaluations, LLM-as-judge approaches, and embeddings.
Good judgment in assessing research results and distinguishing meaningful improvements from noise, including when ground truth is unavailable.
Experience operating AI agents in production is a plus.
Must be authorized to work in the United States without visa sponsorship.
Compensation & Benefits
Salary range: $120,000 to $200,000 USD annually. Visa sponsorship is not available.
Location
On-site in San Francisco, United States.