Machine Learning Engineer — GPU Kernel

About the Institute of Foundation Models We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy. *As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development.* You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers. -------- The Role The GPU Kernel Engineer will play a role at the forefront of optimizing performance for the machine learning software stacks, especially at training and inference, and support the team to develop new and cutting-edge systems. The ideal candidate will have a strong background in parallel computing, and hands-on experience in system level coding, debug methodologies, and large-scale machine learning experience. This role focuses on CUDA kernel development and optimization. Distributed training experience is a plus. Key Responsibilities -------------------- * Understand, analyze, profile, optimize, and provide guidance to the team on deep learning workloads on state-of-the-art hardware and software platforms to improve their efficiency with different levels of optimization * Design and implement performance benchmarks and testing methodologies to evaluate application performance * Build tools to automate workload analysis, workload optimization, and other critical workflows * Triage system issues and identify bottleneck and inefficiencies by analyzing the sources of issues and the impact on hardware, network and propose solutions to enhance GPU utilization * Support the team to develop appropriate kernels and systems for new model architectures and algorithms * Participate in, or lead design reviews with peers and stakeholders to decide amongst available technologies. * Review code developed by other developers and provide feedback to ensure best practices (e.g., style guidelines, checking code in, accuracy, testability, and efficiency). * Contribute to existing documentation or educational content and adapt content based on product/program updates and user feedback. * Represent MBZUAI at industry conferences and events, showcasing the institution’s cutting-edge HPC and deep learning capabilities and establishing MBZUAI as a global leader in AI research and innovation. * Perform all other duties as reasonably directed by the line manager that are commensurate with these functional objectives. * Validate CUDA kernel outputs and gradients against reference implementations, and benchmark representative shapes, dtypes, and model workloads. Technical Qualifications ------------------------ Must-Haves: ----------- * Strong C++ skills and hands-on CUDA kernel development and optimization for deep-learning workloads. * Understanding of GPU memory hierarchy, warp/block execution, and compute-memory trade-offs, with demonstrated profiling-driven optimization. * Strong Python skills and experience integrating kernels with PyTorch or an equivalent framework, including numerical and gradient validation where needed. Nice-to-Haves: -------------- * Experience with Triton, CUTLASS, or PTX/SASS analysis. * Experience with multi-node distributed training or inference systems. * Experience validating mixed-precision computations, such as BF16 or FP8. *Benefits Include* *Comprehensive medical, dental, and vision benefits *Bonus *401K Plan *Generous paid time off, sick leave and holidays *Paid Parental Leave *Employee Assistance Program *Life insurance and disability