About the Role
Lead the data quality team at an early-stage AI evaluation infrastructure startup. You will shape the systems and research practices used to measure and improve training data for AI agents, helping ensure it is useful, reliable, and representative of real-world tasks.
What You'll Do
Lead development of systems to evaluate tasks across reinforcement learning environments, synthetic data, benchmarks, and domain-specific workflows.
Set data quality strategy by defining standards, building quality control systems, and designing experiments to assess agent outputs.
Develop scalable synthetic data validation methods, including failure analysis, task mutation checks, and trajectory audits.
Work with research engineers, subject-matter experts, and data providers to investigate quality issues and improve data generation workflows.
Turn research insights into production tools, dashboards, validation pipelines, and feedback loops.
Mentor research engineers and promote technical rigor, clear communication, and effective execution.
What We're Looking For
At least 5 years of research or engineering experience, including building AI or machine learning data evaluation systems.
Experience leading technical projects or teams through ambiguous problems, from definition to implementation and iteration.
Advanced proficiency in Python, Docker, and Linux.
Background in AI evaluations, post-training, synthetic data validation, benchmarks, or machine learning data quality.
Strong judgment about what makes agent training data realistic, learnable, diverse, reliable, and useful.
Experience designing metrics and quality processes, translating research into production systems, and working with domain experts.
Clear written communication and experience in an early-stage startup environment.
Compensation & Benefits
Annual salary range: $130,000 to $225,000 USD. Visa sponsorship is available.
Location
On-site in the United States.