Job Summary
The position involves creating and sustaining evaluation harnesses designed to evaluate multimodal foundation models across a range of tasks such as text processing, coding, and mathematical computations. The insights gained from your analyses will play a critical role in shaping training methodologies, selecting checkpoints, and detecting regressions in model performance.
Responsibilities
- Develop and maintain evaluation systems for multimodal foundation models.
- Analyze experimental outcomes to guide training strategies and checkpoint choices.
- Identify any regressions in model performance through thorough evaluation.
Qualifications
- Extensive experience in constructing model evaluation frameworks.
- Strong statistical analysis skills for interpreting experimental data.
- Proficiency in Python programming.
- Demonstrated success in creating benchmarks to influence research direction.
