Member of Technical Staff, Realm Lab (frontier AI research)

What You'll Do7

• Take full ownership of research and evaluation projects, covering task design, dataset creation, benchmark validation, and performance analysis.
• Create and test capability evaluations, reinforcement learning environments, and benchmark suites for computer use, agentic workflows, and new model abilities.
• Produce high-quality datasets, task specifications, evaluation rubrics, and methodologies that raise post-training data quality and model assessment standards.
• Examine model behavior through failure analysis, error taxonomies, and capability gap identification to guide research priorities.
• Partner with researchers and domain experts to design experiments, calibrate evaluation quality, and boost model performance.
• Build reproducible evaluation pipelines and feedback loops that support iterative model development.
• Contribute to benchmarks, research papers, technical reports, and open research artifacts.

Requirements7

• Solid technical foundation in computer science, machine learning, AI, or a closely related discipline.
• Hands-on experience designing ML evaluations, benchmark suites, research datasets, or experimental frameworks.
• Proficiency in Python and a track record of building data pipelines, evaluation tools, or ML research infrastructure.
• Knowledge of LLMs, foundation models, post-training techniques, and evaluation methodologies.
• Skill at translating complex real-world workflows into structured tasks, measurable criteria, and research experiments.
• Strong analytical abilities in model failure analysis, experimental design, and evaluation quality.
• Capacity to work independently and collaborate with researchers, engineers, and domain experts.

Preferred Qualifications5

• Experience with reinforcement learning environments, reward modeling, simulators, or feedback-driven training systems.
• Experience evaluating computer use agents, tool use, browser automation, or multi-step agentic workflows.
• Familiarity with multimodal reasoning, long-context evaluation, instruction following, or deep research benchmarks.
• Experience with human-in-the-loop evaluations, expert-generated datasets, or post-training data.
• Contributions to AI research publications, open-source benchmarks, or evaluation frameworks.

Who Should Apply

This role suits someone with a strong ML or AI background who has built evaluations, benchmarks, or research datasets and is comfortable writing Python for data pipelines and evaluation tools. You should be able to break down real-world workflows into measurable tasks and enjoy analyzing model failures to guide research.

Salary Insight