Senior Software Engineer, AI Resiliency

Role responsibilities

Lead the development of AI software resiliency features for massive-scale supercomputers to minimize cluster downtime. Implement critical mechanisms such as fast checkpoint-recovery, error isolation, and proactive failure mitigation for AI workloads.

Requirements

Requires a degree in Computer Science or Electrical Engineering with over 6 years of experience in distributed systems and high-performance coding. Proficiency in C++, Python, and familiarity with AI frameworks like PyTorch or JAX is essential.

Key skills

C++, Python, Distributed Systems, Fault Tolerance, PyTorch, JAX/XLA, CUDA, NCCL, MPI, GPU Computing, Debugging, Profiling, CI/CD, HPC, Performance Tuning, Error Detection

Keywords

AI Resiliency, Supercomputers, GPU, C++, Python, Distributed Systems, Fault Tolerance, PyTorch, JAX/XLA, TensorFlow, CUDA, NCCL, MPI, HPC, Checkpoint-Recovery, Silent Data Corruption, Gdb, Perf, Valgrind, NVIDIA Nsight, CI/CD, Performance Tuning, AI Training, AI Inference, Software Engineering, Parallel Programming, Error Isolation, Straggler Detection, Cloud Computing, Scalability