About the Role
Own complex data and evaluation programs for AI research teams and internal partners. This role brings structure to evolving technical requirements and helps ensure that data and evaluation workflows produce reliable, useful results.
What You'll Do
Lead data and evaluation programs from scoping and requirements gathering through production, quality assurance, delivery, and retrospectives.
Turn ambiguous requests into clear specifications with milestones, owners, dependencies, and acceptance criteria.
Improve data quality processes through quantitative measures and qualitative review, including assessment of task coverage, difficulty, and reward distributions.
Manage external vendors across the delivery lifecycle and improve delivery pipelines at scale.
Improve workflows, documentation, automation, and cross-functional handoffs for data production and evaluation.
Work with research, platform, and marketplace teams to turn feedback and learnings into product improvements.
What We're Looking For
At least 3 years of experience owning complex technical programs in AI/ML, data operations, support engineering, applied research, forward-deployed engineering, or a related field.
Experience as a Technical Program Manager or strategic program lead at a startup, with a focus on project delivery rather than product management alone.
Experience translating ambiguous technical requirements into executable plans and improving complex data quality processes.
Ability to assess dataset quality using both aggregate metrics and individual examples.
Experience managing external vendors, scaling delivery pipelines, and supporting AI research teams or technical enterprise customers with evolving needs.
Familiarity with RLHF, reinforcement-learning environments, agent evaluations, human-in-the-loop data pipelines, annotation, or expert data collection.
Strong written and verbal communication skills, sound judgment, and comfort working independently in a fast-paced environment.
Compensation & Benefits
Annual salary range: $130,000 to $225,000. Visa sponsorship and relocation support are available for qualified full-time candidates. Benefits include medical, dental, and vision coverage, a retirement plan, paid time off, and commuter benefits.
Location
On-site in San Francisco, California, United States.