Principal Coding Annotator / LLM Evaluation Engineer

What You'll Do7

• Design benchmark-style coding problems with reference solutions, in the spirit of SWE-Bench style tasks.
• Grade model output on code generation, refactoring, debugging, and implementation work.
• Test models on security tasks covering vulnerable code, exploit verification, and patch confirmation.
• Record failure modes, edge cases, and reasoning gaps in clear technical notes.
• Run side-by-side comparisons between internal Mistral-based models and leading external models.
• Set up or configure coding environments that support evaluation and reinforcement learning cycles.
• Apply annotation and evaluation guidelines with steady consistency across large task batches.

Requirements5

• At least 5 years of professional software development experience.
• Strong Python skills.
• Direct experience with vulnerability research, exploit reproduction or verification, or building, backporting, and validating security patches.
• Ability to apply structured evaluation criteria and write technical feedback others can act on.
• Fluent written and spoken English.

Preferred Qualifications4

• Working knowledge of a second programming language
• Prior code review, coding annotation, LLM evaluation, or benchmark design experience
• Team lead or mentoring background
• Based in Paris or London

Who Should Apply

Engineers with 5 or more years of hands-on development work who can read code like an attacker and grade it like a reviewer. Security background in exploit reproduction or patch validation matters here, alongside solid Python. The role suits someone who wants model quality decisions in their hands.

Salary Insight