AI Operations Engineer

Overview

We're seeking a hands-on engineer to maintain and optimize our GPU-powered AI inference platform that serves large language models to internal applications. You'll own the full stack from bare-metal OS setup through production deployment, monitoring, and capacity planning to keep the infrastructure reliable and scalable. This role combines deep systems engineering with AI/ML operations, requiring a mix of automation skills and performance tuning.

What You'll Do6

• 1Provision and automate Linux servers: harden the OS, install GPU drivers and toolkits, set up Python environments, and deploy LLM inference engines using version-controlled automation (e.g., Ansible) for repeatable, auditable deployments.
• 2Deploy and manage LLM inference engines (like vLLM or TensorRT-LLM) across a fleet of enterprise GPUs, handling multi-GPU model sharding, quantization strategies, and GPU memory/cache sizing to maximize concurrency and throughput.
• 3Operate the API gateway layer for load balancing, model routing, API key management, and token counting, along with supporting services such as database backends, caching layers, and reverse proxies with proper access controls.
• 4Instrument and maintain the observability stack: collect GPU-level telemetry (utilization, memory, temperature), track key metrics (latency percentiles, tokens per second, concurrent users), and set alerting thresholds for capacity and hardware health.
• 5Collaborate closely with application development teams to onboard new workloads, review and optimize system prompts, and troubleshoot model behavior – distinguishing between model limitations, prompt issues, and infrastructure problems.
• 6Support model fine-tuning workflows by deploying LoRA/QLoRA adapters into the serving infrastructure and understanding how fine-tuned models differ from base models in terms of serving requirements.

Requirements6

• 1At least 4 years of hands-on Linux systems engineering experience, with no fewer than 2 years involving GPU infrastructure or ML/AI workloads in a production environment.
• 2Proven track record of deploying and operating LLM inference engines in production, including experience with model quantization, multi-GPU parallelism, and performance tuning.
• 3Strong working knowledge of the NVIDIA GPU software stack (drivers, toolkits, runtime libraries) and the ability to diagnose common failure modes and compatibility issues.
• 4Proficiency with infrastructure-as-code and configuration management tools (especially Ansible), plus solid understanding of Linux containerization (Docker, rootless operation, service management integration).
• 5Comfortable scripting in Python and shell for operational tooling, and disciplined use of Git for version control (branching, tagging, commit practices).
• 6Excellent troubleshooting and communication skills – able to document procedures, write runbooks, and explain infrastructure constraints to development teams, especially in environments with restricted internet access.

Who Should Apply