Software Engineer – AI Code Evaluation & Benchmarking (US candidates only)
Listing checked August 25, 2026 · pay as published by Turing
Overview
This contract role places an experienced software engineer inside an evaluation workflow for frontier AI models. You will review AI-generated code for correctness, efficiency, and maintainability, validate solutions against real engineering tasks, and debug failures across different environments. You will also help build and refine evaluation datasets, benchmarks, and grading rubrics. The assignment runs for one month, requires at least 4 hours per day and 20 hours per week with a 4-hour overlap with PST, and is open only to candidates in the US. Work is fully remote.
What You'll Do8
- 1Review AI-generated code for correctness, efficiency, maintainability, and requirement adherence.
- 2Analyze software engineering tasks and judge whether proposed solutions meet expected outcomes.
- 3Debug code, reproduce reported issues, and verify fixes across different programming environments.
- 4Assess model-generated explanations and implementation approaches for technical accuracy.
- 5Create, refine, and maintain evaluation datasets, benchmarks, and grading rubrics.
- 6Identify edge cases, failure modes, and recurring weaknesses in AI coding systems.
- 7Document findings and give structured feedback to keep evaluation quality consistent.
- 8Work with project teams to define quality standards and evaluation methodologies.
Requirements10
- 1Bachelor's or Master's degree in Computer Science, Software Engineering, or a related technical field.
- 2At least 3 years of professional software engineering experience.
- 3Strong proficiency in one or more of these languages: Python, Java, C/C++, Go, Swift, Objective-C, PHP, or SQL.
- 4Solid understanding of data structures, algorithms, software design principles, and debugging methodologies.
- 5Proven experience performing code reviews and evaluating code quality in production or large-scale codebases.
- 6Ability to analyze complex technical problems and judge solution correctness with minimal supervision.
- 7Familiarity with Git and standard modern development workflows.
- 8Clear written communication and careful attention to detail.
- 9Bonus: experience with AI/ML data annotation, NLP, prompt engineering, model evaluation, or LLM-related projects.
- 10Highly preferred: prior experience evaluating AI-generated code, creating benchmarks, or assessing software quality.
Who Should Apply
The candidate who fits this role is a software engineer with at least 3 years of professional experience, strong code review skills, and real comfort in Python and Docker. They enjoy debugging, analyzing why a solution fails, and turning those findings into structured feedback for AI model improvement. This role is less suitable for engineers who want a long-term staff position with benefits, or who prefer building features over evaluating and benchmarking code. Common rejection reasons include failing the online automated coding challenge in Python and Docker, and not being available for the required 4-hour daily PST overlap or the one-month contract commitment.
Pay and demand for AI Training roles
AggregatedTypical pay
$71/hour
This role
Pay not disclosed
Most AI Training roles pay $42–$100 per hour.
Based on 1,080 similar roles that publish pay · 302 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 1,243
- Listed in last 30 days
- 479
- Remote
- 97%
Hiring most right now: micro1 (362) · SME Careers (253) · Mercor (156)
Most requested skills · share of roles
- ai training13%
- llm evaluation11%
- ai evaluation11%
- data annotation9%
Figures from AI Training roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
Undisclosed by Turing
Comparable roles pay
1,080 roles$42 – $100/ hr
Not this role’s pay. Turing has not published a range.
Confirmed during Turing profile screening.
Required Skills
Application Tip
Before applying, practice Python coding challenges that run inside Docker containers and study the RHLF-style evaluation format used for AI response ranking. In your application, highlight specific examples of code review and bug-finding work with measurable outcomes, such as error rates reduced or defects caught before release.
How NearSkill helps you apply
1. Check your job match
We compare your resume with the job requirements to help you understand how well your experience fits.
2. Continue to the employer’s website
We’ll direct you to the employer’s application portal, where you can complete the application.
3. Get updates by email
If NearSkill is handling follow-up for this role, we’ll email you when there’s an update on your screening.
Want to check your fit first?
See how your resume matches this role before applying.
Check my job matchSimilar open positions
Explore active roles that match your skills and interests.
Turing
VerifiedSenior Software Engineer – LLM Evaluation (US/Canada/WEU based)
Turing, a San Francisco-based research accelerator, is hiring a contract software engineer to evaluate AI-generated code and build datasets for large language models. You will curate code examples and write precise corrections in Python, JavaScript/ReactJS, C/C++, Rust, plus Java and Go. The role includes scoring model outputs for efficiency, scalability, and reliability, designing verification mechanisms, and partnering with researchers to strengthen enterprise coding solutions. This remote engagement runs one month, offers 10 to 40 flexible hours per week, and accepts candidates in the US, Canada, and Western Europe.
Handshake AI
VerifiedSoftware Engineer - AI Evaluation
Handshake AI seeks practicing software engineers for an ongoing part-time contract. You will create expert-level problems rooted in system design, software architecture, and real implementation work, then judge AI answers for technical accuracy and practical feasibility. The role runs remote and asynchronous, so you choose your own hours with no minimum. Your structured written feedback helps models reason more like engineers who build production systems.
Turing
VerifiedSenior Software Engineer - AI Coding Agent Evaluation
Turing seeks experienced software engineers to assess how well AI coding agents handle real repositories. You will treat the agent like a colleague whose pull requests you review: checking whether it read the task, picked a sound design, and produced working code. The work spans Python, TypeScript/JavaScript, and Go codebases, plus the evaluation rubrics and data pipelines that sharpen model behavior. Researchers rely on your written judgment to turn one-off observations into repeatable quality signals. This is a freelance contract, roughly three months long, with 40 hours per week preferred.
Micro1
VerifiedSenior Software Engineer for AI Code Evaluation and Model Review
micro1 is hiring senior software engineers to work with a leading AI lab. Your main tasks involve evaluating AI-generated code for correctness and design, writing challenging coding problems with reference solutions and test cases, and reviewing model reasoning on architecture, debugging, and system design. You will provide detailed technical feedback to help improve model performance on real-world software tasks. This is a remote contractor role with a pay range of $245 to $280 per hour.
Turing
VerifiedSenior Software Engineer – LLM Evaluation
In this contract role, you will build and refine training datasets that help large language models improve their coding skills. You will write, correct, and evaluate code in Python, JavaScript (ReactJS), C/C++, Java, Rust, and Go, collaborating with researchers and cross-functional teams. Your daily work includes assessing AI-generated code for efficiency, scalability, and reliability, plus building verification agents that catch error patterns. The one-month engagement runs 10-40 hours per week with partial PST overlap.
Terac
VerifiedSoftware Engineer - AI Agent Code Review
Terac pays software engineers to review coding tasks and evaluation harnesses used for benchmarking AI agents. You will inspect programming challenges, determine whether they mirror real software engineering problems, and verify logic, test cases, and environment structure. The remote session combines a guided conversation with screen-sharing exercises, so you explain your reasoning as you analyze code. Engineers in the USA, Canada, and France can take part.