Top GitHub Contributors – Agentic Review Project

What You'll Do12

• Create challenging, real-world problems that expose specific reasoning gaps in a target frontier model.
• Build detailed task instructions and an overview for each benchmark problem.
• Develop a golden solution that adheres to the task instructions and serves as a reference.
• Set up the required environment, including datasets, Python libraries, and metadata.
• Write a test notebook with unit tests that solutions must pass.
• Evaluate the model's performance on the designed tasks and identify failures.
• Classify failures as logical reasoning issues and extract patterns from the model's agent trajectory.
• Analyze agent steps to observe where core capabilities break down.
• Work closely with the research team to refine tasks and evaluation methods.
• Maintain high-quality documentation for tasks and solutions.
• Contribute to a growing benchmark suite that pushes model reasoning forward.
• Engage in regular syncs and provide feedback on task design and model behavior.

Requirements12

• Proven track record as a highly skilled GitHub contributor.
• Strong expertise in data science, machine learning, coding, or a deep quantitative background in frontier STEM.
• Ability to commit 30+ hours per week on weekdays.
• Proficiency in Python for building and testing task components.
• Experience with agentic development environments and model evaluation.
• Capacity to design complex, real-world problems that test model reasoning.
• Solid understanding of unit testing and test-driven development.
• Familiarity with benchmark creation and validation.
• Strong analytical skills to diagnose model failures and loss patterns.
• Excellent written communication for clear task instructions and documentation.
• Self-directed work ethic and reliability in a remote setting.
• Completed AI video interview as part of the application process.

Who Should Apply

Candidates with a strong GitHub portfolio and deep expertise in data science, machine learning, coding, or frontier STEM. The role suits someone who can commit 30+ hours per week on weekdays and enjoys designing challenging problems to probe AI reasoning. If you have experience building and evaluating agentic tasks, this project offers a chance to shape frontier model capabilities.