Machine Learning Engineer — Reinforcement Learning

About the Institute of Foundation Models We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy. *As part of our team, you’ll have the opportunity to work on the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development.* You will participate in the development of groundbreaking AI solutions that have the potential to reshape entire industries. Strategic and innovative problem-solving skills will be instrumental in establishing MBZUAI as a global hub for high-performance computing in deep learning, driving impactful discoveries that inspire the next generation of AI pioneers. -------- The Role We’re looking for an RL infrastructure engineer to help extend and scale our end-to-end RL training systems. You’ll work side-by-side with world-class researchers and engineers to: * Extend distributed training frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod) * Integrate rollout generation, reward computation, trajectory processing, policy updates, and weight synchronization * Build robust config + launch systems across multi-node, multi-GPU clusters * Own experiment tracking, metrics logging, and job monitoring for external visibility * Improve training system reliability, maintainability, and performance Hands-on experience developing end-to-end RL infrastructure is required. Strong infrastructure and systems experience is what we value most. Key Responsibilities -------------------- * Distributed Framework Ownership – Extend or modify training frameworks (e.g., DeepSpeed, FSDP) to support new use cases and architectures. * Training & Inference Integration – Connect training workers with rollout generation engines (e.g., vLLM, SGLang), including trajectory exchange and policy-weight synchronization. * Launch Config & Debugging – Create and debug multi-node launch scripts with flexible batch sizes, parallelism strategies, and hardware targets. * Metrics & Monitoring – Build systems for experiment tracking, job monitoring, and logging usable by collaborators and researchers. * Infra Engineering – Write production-quality code and tests for ML infra in PyTorch or JAX; ensure reliability and maintainability at scale. * RL Pipeline Development – Implement reward/verifier integration and trajectory processing, and validate log probabilities, token masks, and loss inputs with researchers. * RL Execution & Recovery – Coordinate rollout and training workers, including checkpoint/restart and failure recovery; track policy versions and sample staleness when using asynchronous execution. Qualifications -------------- Must-Haves: ----------- * 5+ years of experience in ML systems, infra, or distributed training * Experience modifying distributed ML frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod) * Strong software engineering fundamentals (Python, systems design, testing) * Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and debugging skills (e.g., NCCL/GLOO) * Ability to implement algorithms across GPUs/nodes based on mathematical specs * Experience working on an ML platform/ infrastructure, and/or distributed inference optimization team * Experience with large-scale machine learning workloads (strong ML fundamentals) * Hands-on experience developing an LLM RL pipeline, with ownership across rollout/inference and distributed training integration. * Working knowledge of policy optimization methods such as PPO or GRPO, sufficient to implement and debug sampling, log probabilities, and policy updates. Nice-to-Haves: * Exposure to mixed-precision training (e.g., bf16, fp8) with accuracy validation * Familiarity with performance profiling, kernel fusion, or memory optimization * Open-source contributions or published research (MLSys, ICML, NeurIPS) * CUDA or Triton kernel experience * Experience with large-scale pre-training * Experience building custom training pipelines at scale and modifying them for custom needs * Deep familiarity with training infrastructure and performance tuning * Experience with asynchronous RL, multi-turn or tool-using rollout environments, or custom reward/verifier systems. *Benefits Include* *Comprehensive medical, dental, and vision benefits *Bonus *401K Plan *Generous paid time off, sick leave and holidays *Paid Parental Leave *Employee Assistance Program *Life insurance and disability