Senior Python Engineer – LLM Evaluation (US-based)

What You'll Do8

• Craft and curate code examples in Python and other languages to build datasets for model training and benchmarking.
• Review AI-generated code and provide precise corrections to improve efficiency, scalability, and reliability.
• Develop automated verification tools and agents in Python that assess code quality and flag error patterns.
• Formulate hypotheses about stages of the software engineering lifecycle and test model capabilities against them.
• Design automated mechanisms that can validate solutions to software engineering problems.
• Work with cross-functional teams to refine AI-driven coding solutions and align them with industry benchmarks.
• Write clear, structured rationales for evaluation decisions.
• Contribute to model training initiatives by identifying and documenting code patterns.

Requirements6

• At least 3 years of software engineering experience building production systems.
• Strong expertise in Python, including frameworks, tooling, and best practices for production-grade software.
• Experience building full-stack applications and deploying scalable software using modern languages and tools.
• Deep understanding of software architecture, design, development, debugging, and code review assessment.
• Excellent written and verbal communication skills for explaining evaluation rationales.
• Must be based in the United States.

Preferred Qualifications2

• Experience at high-scale engineering organizations like Google, Microsoft, Apple, Amazon, or Meta.
• Graduation from top computer science programs such as Stanford, MIT, Carnegie Mellon, UC Berkeley, or Georgia Tech.

Who Should Apply

Candidates with 3+ years of production software engineering experience, deep Python expertise, and a strong grasp of software architecture. The role suits engineers from high-scale tech companies or top CS programs who can evaluate and improve AI-generated code. Must be based in the US.