Senior Software Engineer – LLM Evaluation (US-based)

What You'll Do8

• Curate code examples and craft precise solutions to train and benchmark large language models
• Evaluate AI-generated code for efficiency, scalability, and reliability, then refine it as needed
• Build agents and automated verification tools in Python to assess code quality and detect error patterns
• Hypothesize steps across the software engineering cycle (prototyping, design, implementation, monitoring) and evaluate model capabilities on each
• Design verification mechanisms that automatically validate solutions to software engineering tasks
• Collaborate with researchers and cross-functional teams to improve AI-driven coding solutions against industry benchmarks
• Work primarily in Python, with supplementary tasks in JavaScript (ReactJS), C/C++, Java, Rust, and Go
• Correct and improve AI-generated code to meet production-grade standards

Requirements6

• At least 3 years of software engineering experience
• Strong Python expertise, including frameworks, tooling, and best practices for production software
• Background building full-stack applications and deploying scalable software with modern languages and tools
• Solid grasp of software architecture, design, debugging, and code quality assessment
• Excellent written and verbal communication skills, able to explain evaluation rationales clearly and structurally
• Must be based in the United States

Preferred Qualifications3

• Experience at high-scale engineering organizations like Google, Microsoft, Apple, Amazon, or Meta
• Graduation from top computer science programs such as Stanford, MIT, Carnegie Mellon, UC Berkeley, or Georgia Tech
• Familiarity with Go

Who Should Apply

Engineers with at least three years of production software experience and deep Python skills who enjoy analyzing code quality and building automated checks. The role suits someone who has worked on full-stack systems, can communicate evaluation findings clearly, and wants to shape how large language models handle software engineering tasks. Candidates must reside in the US.