Staff Software Engineer - AI Traffic & Inference Infrastructure

As a Staff Engineer on our Coupang intelligent Cloud Infrastructure team, you will design and scale the intelligent nervous system of our CIC Cloud AI platform. You won't just be moving packets; you’ll be building the orchestration and routing layers that ensure our LLMs and foundation models are highly available, low-latency, and cost-efficient. You will own the end-to-end lifecycle of traffic management from global load balancing to hardware-aware request routing across thousands of accelerators.

• →

  Intelligent Routing: Design and implement sophisticated load-balancing algorithms tailored for AI workloads (training, inference), optimizing request distribution based on model availability, and accelerator health. 

• →

  Inference Orchestration: Architect and evolve our inference infrastructure to support seamless model deployment, auto-scaling, and multi-AZ failover. 

• →

  Performance Engineering: Drive initiatives to minimize tail latency (P95 /P99) and maximize throughput using advanced batching, caching, and streaming token delivery techniques. 

• →

  Fleet Automation: Build robust infrastructure-as-code and CI/CD pipelines to manage dynamic compute fleets, ensuring they automatically scale to meet production and research demands. 

• →

  Observability & Optimization: Leverage deep telemetry data to tune system performance and hardware-agnostic scheduling across diverse GPU/TPU environments. 

• →

  Technical Leadership: Lead cross-functional initiatives across infrastructure and SW team, ML teams, providing mentorship and setting up the long-term technical roadmap for traffic management. 

• Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field. 

• AI/ML Domain Knowledge: Prior experience building infrastructure specifically for LLM inference or large-scale training clusters. 

• Low-Level Optimization: Familiarity with inference, including mixed precision, kernel tuning, or custom hardware accelerators. 

• Public/Private Cloud: Experience managing hybrid-cloud or multi-AZ deployments across AWS, Azure, or GCP. 

• Compliance: Experience operating in regulated environments with strict security and compliance requirements. 

Type of work model 

• Hybrid

Details to consider 

• Those eligible for employment protection (recipients of veteran’s benefits, the disabled, etc.) may receive preferential treatment for employment in accordance with applicable laws. 
   

Privacy Notice  

• Your personal information will be collected and managed by Coupang as stated in the Application Privacy Notice located below. https://privacy.coupang.com/en/land/jobs/