Job Summary
The incumbent will be responsible for creating and developing an evaluation harness specifically designed for agentic AI, focusing on aspects such as trustworthiness, faithfulness, and overall performance. This role will involve red-teaming activities, ensuring the integrity of data separation, and executing automated testing throughout the platform’s microservices and AI pipelines.
Responsibilities
- Design and develop an evaluation harness for agentic AI systems.
- Conduct red-teaming exercises to assess AI system vulnerabilities.
- Ensure data isolation and validate its effectiveness.
- Implement automated testing processes across microservices and AI pipelines.
- Evaluate and analyze non-deterministic outputs produced by AI systems.
Qualifications
- Strong programming skills in Go or Python.
- Experience with LLM-based or agentic AI systems.
- Proven history in software development and quality engineering.
- Capability to effectively assess and validate AI outputs.
Sofico provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
