Job Summary
This role focuses on the design and upkeep of reusable evaluation infrastructure, with an emphasis on streaming evaluations, autorater pipelines, and metric aggregation. The successful candidate will be tasked with enhancing the scalability, dependability, and effectiveness of extensive evaluation workflows. This position also includes support for product teams, the establishment of secure data isolation and access controls, and the transition of teams from offline evaluations to online evaluations, live experiments, and production environments.
Responsibilities
- Develop and maintain infrastructure for evaluation processes.
- Enhance the efficiency and reliability of large-scale evaluation workflows.
- Support product teams in their evaluation needs.
- Create secure frameworks for data isolation and access management.
- Facilitate the shift from offline to online evaluation methods and live experiments.
Qualifications
- Bachelor’s degree or equivalent practical experience.
- A minimum of five years of experience in Python programming.
- At least three years working with large-scale infrastructure or similar systems.
- Three years of experience with Generative AI agents.
- Two years of experience in multithreading.
- Preferred: Master’s degree or PhD.
- Experience with large-scale data processing or workflow orchestration.
- Expertise in machine learning and related systems.
- Familiarity with ML infrastructure, model evaluation, LLM workflows, Python, and C++ is advantageous.
Google provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
