Member of Technical Staff, Coding Research

What You'll Do8

• Own the design of evaluation frameworks for coding agents, including benchmark specs, scoring rubrics, and quality standards.
• Lead end-to-end research initiatives that measure and boost coding model performance on a wide range of engineering challenges.
• Create high-quality datasets, golden examples, and evaluation protocols to ensure reliable assessment of frontier AI systems.
• Analyze model behavior and failure modes, pinpointing systematic weaknesses and turning findings into training and evaluation improvements.
• Build tooling and infrastructure to support large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Establish best practices for coding-agent assessment, ensuring methodological rigor, reproducibility, and measurement quality.
• Collaborate with researchers and applied AI teams to design experiments and evaluate emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research that communicate model performance insights.

Requirements8

• Strong software engineering background with proficiency in Python, C++, or similar languages.
• At least 3 years of experience in software engineering, machine learning, AI research, evaluation, or a related technical field.
• Proven experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.
• Ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills to investigate model behavior and derive insights from complex technical systems.
• Excellent written and verbal communication skills, with the ability to articulate technical findings to diverse audiences.
• Comfort operating in fast-moving research environments with significant ambiguity and evolving priorities.

Who Should Apply

You’re the ideal candidate if you love working at the intersection of AI research and software engineering, are passionate about measuring and improving coding model performance, and enjoy building robust evaluation systems from scratch. You thrive in ambiguous, fast-paced environments where you can independently drive projects from concept to execution. A strong track record in benchmark design, data systems, or agentic workflows will set you apart.

Salary Insight