Senior Software Engineer – LLM Evaluation (US-based)

What You'll Do7

• Curate code examples and develop precise solutions to train and benchmark LLMs, mainly in Python.
• Review and improve AI-generated code across languages such as JavaScript, ReactJS, C/C++, Java, Rust, and Go.
• Build automated verification tools and agents in Python to assess code quality and spot error patterns.
• Evaluate model performance on software engineering tasks like prototyping, architecture design, API design, and production implementation.
• Design verification mechanisms that automatically check solutions to engineering tasks.
• Work with cross-functional teams to refine AI-driven coding solutions against performance benchmarks.
• Provide structured evaluation rationales through clear written and oral communication.

Requirements6

• 3+ years of software engineering experience.
• Strong Python expertise with deep knowledge of frameworks, tooling, and best practices for production software.
• Experience building full-stack applications and deploying scalable software with modern languages and tools.
• Solid grasp of software architecture, design, development, debugging, and code quality assessment.
• Excellent written and verbal communication skills for structured evaluation rationales.
• Must be based in the United States.

Who Should Apply

Candidates with 3+ years of software engineering experience and deep Python skills, especially those who have worked at AI-frontier companies or hold strong CS degrees, will thrive in this role. The position suits engineers who enjoy evaluating code quality and collaborating with researchers on model training.