Senior Software Engineer - AI Coding Agent Evaluation

What You'll Do10

• Audit AI-generated code and solutions from live software repositories for technical accuracy.
• Inspect agent actions, tool calls, and diffs to judge correctness and maintainability.
• Spot recurring failure modes, flawed design choices, and gaps in model reasoning.
• Rank competing model outputs and explain why one implementation outperforms another.
• Draft and refine scoring rubrics that define quality for coding benchmarks.
• Generate preference and evaluation data that feeds back into model training.
• Maintain pipelines and tooling for collecting, generating, and reviewing evaluation data.
• Distill findings into concise memos and status updates for research partners.
• Partner with researchers and engineers to scale qualitative review into repeatable processes.
• Report actionable results to AI researchers and engineering stakeholders.

Requirements8

• 5+ years of hands-on software engineering across production systems.
• Deep fluency in Python, TypeScript/JavaScript, Go, or a comparable production language.
• Track record working inside large, mature codebases rather than greenfield projects.
• Keen code review instincts and the technical judgment to defend your assessment.
• Ability to articulate why an implementation is correct, broken, or ripe for improvement.
• Clear written communication that stands on its own without follow-up questions.
• Regular use of modern LLMs or AI coding assistants in your workflow.
• Nice to have: exposure to LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training.

Who Should Apply

Engineers with at least five years of production coding experience who enjoy dissecting why a solution works, not just whether it compiles. The role rewards people comfortable in big codebases and fluent in Python, TypeScript/JavaScript, or Go, and who can write a crisp justification for their verdict. Candidates weak on written explanations tend to score low here, because the deliverable is often a clear rationale rather than a patch. Applying without real LLM or AI coding tool exposure also hurts fit, even though it is not strictly required. If you prefer building features all day over reviewing and critiquing code, this contract will feel like a poor match.

Salary Insight