Job Summary
The position involves developing and executing assessments of scientific AI models, agents, and workflows that are tool-assisted. The focus is on creating reproducible methodologies and instruments to analyze failure modes and evaluate the quality of automated assessments. The role requires close collaboration with risk analysts and mitigation engineers to enhance protective measures and establish wider safety automation related to monitoring, evidence assessment, regression testing, and risk management.
Responsibilities
- Conduct evaluations of AI models and agent systems.
- Develop reproducible methodologies to analyze failure modes.
- Assess and enhance the quality of automated evaluations.
- Collaborate with researchers to improve safety measures and automation.
- Engage in monitoring and risk response activities.
Qualifications
- Strong empirical judgment and practical programming skills.
- Experience with contemporary AI systems.
- Aptitude for investigating ambiguous failures.
- Proven ability to communicate limitations effectively.
- Research Scientists should excel in formulating research questions and employing solid experimental methods.
- Research Engineers are expected to have a background in developing reliable research software and evaluation tools.
