Job Summary
The role entails taking charge of optimizing the performance and throughput of extensive pretraining operations through refinement of GPU kernels, enhancing communication methods, and advancing parallelism techniques. The position requires collaboration with researchers and infrastructure engineers to diagnose and eliminate bottlenecks, while also providing support for the development of new model architectures.
Responsibilities
- Enhance GPU kernel efficiency and optimize throughput for large-scale pretraining tasks.
- Improve communication strategies and parallelism in distributed training systems.
- Work in close partnership with researchers and infrastructure engineers to identify and address performance bottlenecks.
- Support the adaptation and implementation of new model architectures within the training framework.
Qualifications
- Extensive experience with distributed training frameworks, particularly Megatron-LM.
- Strong programming skills in C++/CUDA and Python.
- Proven ability to increase model FLOPs utilization significantly.
- In-depth knowledge of GPU profiling techniques and strategies for parallel computing.
Hark provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
