Job Summary
In this role, you will help evaluate how large language models perform when diagnosing and fixing software bugs. You will analyze and prioritize GitHub issues, prepare development environments, and assess the quality and effectiveness of unit tests. Working alongside researchers, you will identify challenging codebases and support evaluations based on real-world software projects.
Responsibilities
- Review, analyze, and triage issues reported on GitHub.
- Set up and configure development environments for target projects.
- Evaluate unit tests and their ability to verify bug fixes.
- Partner with researchers to select challenging codebases and assess LLM bug-fixing performance.
- Navigate complex repositories and run local tests on real-world projects.
Qualifications
- At least 3 years of relevant experience.
- Strong proficiency in Java, Git, and Docker.
- Ability to understand and work effectively within complex codebases.
- Experience performing local testing on real-world software projects.
Careerflow.ai provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.


