Member of Technical Staff, Coding Agent Research & Evaluations

What You'll Do8

• Create and own evaluation frameworks for coding agents, covering benchmark specs, scoring methods, rubrics, and quality standards.
• Lead research projects from start to finish that measure and boost coding model performance on varied software engineering tasks.
• Build high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Examine model behavior and failure modes to spot systematic weaknesses and turn findings into concrete improvements for training and evaluation.
• Develop tooling and infrastructure for large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Set best practices for coding-agent assessment that ensure methodological rigor, reproducibility, and measurement quality.
• Collaborate with researchers, engineers, and applied AI teams to design experiments and evaluate emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research that communicates model performance and insights.

Requirements8

• Solid software engineering background with expertise in Python, C++, or similar programming languages.
• 3+ years of experience in software engineering, machine learning, AI research, evaluation, or related technical fields.
• Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.
• Proven ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills to investigate model behavior and derive insights from complex technical systems.
• Excellent written and verbal communication skills, including the ability to clearly articulate technical findings to diverse audiences.
• Comfortable operating in fast-moving research environments with significant ambiguity and evolving priorities.

Preferred Qualifications6

• Experience working on frontier AI systems, coding agents, or model evaluation research.
• Deep interest in understanding how data, evaluations, and feedback mechanisms influence model capabilities.
• Track record of independently driving ambiguous technical or research projects from conception to execution.
• Experience designing benchmarks or datasets for machine learning systems at scale.
• Familiarity with agentic workflows, tool use, reinforcement learning, or post-training methodologies.
• Publications, open-source contributions, or demonstrated technical leadership in AI, machine learning, or software engineering.

Who Should Apply

This role suits engineers with 3+ years in software engineering or AI research who have hands-on experience with Python or C++ and a strong interest in LLM evaluation. You should be comfortable designing benchmarks, analyzing model failures, and building tooling for large-scale experiments. If you thrive in ambiguous research settings and can communicate technical findings clearly, this is for you.

Salary Insight