Software Engineer - AI Evaluation
Listing checked October 1, 2026 · pay as published by Handshake AI
Overview
Handshake AI seeks practicing software engineers for an ongoing part-time contract. You will create expert-level problems rooted in system design, software architecture, and real implementation work, then judge AI answers for technical accuracy and practical feasibility. The role runs remote and asynchronous, so you choose your own hours with no minimum. Your structured written feedback helps models reason more like engineers who build production systems.
What You'll Do6
- 1Design software engineering problems that mirror real system design, architecture, and implementation tasks.
- 2Evaluate AI-generated answers for technical correctness, feasibility, and alignment with actual engineering practice.
- 3Write structured critiques that point out errors, gaps, and weak reasoning in model output.
- 4Provide feedback that helps AI models improve their software engineering judgment.
- 5Judge AI responses against how production software gets built and maintained.
- 6Set your own schedule and complete tasks at your own pace without a minimum hour requirement.
Requirements7
- 1At least one year of direct, professional software engineering work.
- 2Bachelor's degree in computer science or another technical field.
- 3Deep hands-on skill in one area: backend systems and distributed architecture, infrastructure and cloud platforms with DevOps, or systems programming and performance engineering.
- 4Located in the United States.
- 5Clear, detailed writing that explains complex engineering concepts to a general technical reader.
- 6A track record of working on your own in a remote, asynchronous setup.
- 7Experience at a top-tier technology company strengthens your application but does not make or break it.
Who Should Apply
The right candidate has at least a year of professional software engineering work, a CS or technical degree, and depth in one area such as backend systems, cloud infrastructure, or systems programming. You should enjoy writing careful, specific critiques of AI output and explaining why a design or code path fails. People who want a fixed schedule or a permanent full-time post will not fit this project-based contract. Applications often lose traction when a candidate lists broad engineering experience without a clear sub-domain, or when their writing samples stay vague instead of pointing to concrete system behavior.
Salary Insight
The listed rate is $120.00 per hour. For part-time AI training work that calls for experienced software engineering judgment, this sits above many general annotation rates and matches what platforms often pay engineers with direct industry experience.
Pay and demand for Machine Learning & AI roles
AggregatedTypical pay
$75/hour
This role
$120/hr
Most Machine Learning & AI roles pay $50–$100 per hour. This role's pay sits above that range.
Based on 587 similar roles that publish pay · 170 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 677
- Listed in last 30 days
- 468
- Remote
- 96%
Hiring most right now: micro1 (235) · SME Careers (143) · Mercor (84)
Most requested skills · share of roles
- python18%
- ai training17%
- llm evaluation17%
- trainer feedback12%
Figures from Machine Learning & AI roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
$120/hr
Required Skills
Application Tip
In your application, name the sub-domain where you have real depth, such as distributed backends, cloud infrastructure, or performance engineering, and pair it with one concrete production system you built or diagnosed. Quantify the scale or result, for example requests per second, latency reduction, or nodes managed, because vague claims score lower on fit for this role.
See NearSkill jobs more often in your search
Application & verification flow
1Instant rubric match
Your resume is scanned against this role’s requirements to check qualification fit.
2Screened before the employer sees it
Only profiles that clear screening are passed on.
3Outcome by email
We notify you at the address on your resume once the screening is reviewed.
Similar open positions
Explore active roles that match your skills and interests.
Handshake AI
VerifiedTechnical AI Evaluation Contributor (Remote US)
Handshake AI hires experienced software engineers, data scientists, and quants for part-time contract work that supports AI research. You will create expert-level technical problems and judge AI-generated answers for accuracy, feasibility, and alignment with real system design, modeling, or trading work. The project runs on an ongoing basis, so you can add it alongside a current role. No set schedule or minimum hour requirement applies.
Turing
VerifiedSoftware Engineer – AI Code Evaluation & Benchmarking (US candidates only)
This contract role places an experienced software engineer inside an evaluation workflow for frontier AI models. You will review AI-generated code for correctness, efficiency, and maintainability, validate solutions against real engineering tasks, and debug failures across different environments. You will also help build and refine evaluation datasets, benchmarks, and grading rubrics. The assignment runs for one month, requires at least 4 hours per day and 20 hours per week with a 4-hour overlap with PST, and is open only to candidates in the US. Work is fully remote.
Micro1
VerifiedSenior Software Engineer for AI Code Evaluation and Model Review
micro1 is hiring senior software engineers to work with a leading AI lab. Your main tasks involve evaluating AI-generated code for correctness and design, writing challenging coding problems with reference solutions and test cases, and reviewing model reasoning on architecture, debugging, and system design. You will provide detailed technical feedback to help improve model performance on real-world software tasks. This is a remote contractor role with a pay range of $245 to $280 per hour.
Micro1
VerifiedAI Evaluation Specialist
As an AI Evaluation Specialist, you'll help train next-generation AI systems by designing and executing hands-on evaluation tasks. Your insights will directly shape how models learn, reason, and perform on practical computer-based workflows. This is a fully remote contract role where meticulous observation and clear documentation are key.
Terac
VerifiedSoftware Engineer - AI Coding Evaluation (South Asia)
Terac wants experienced software engineers across South Asia to test the quality of environments used to score AI agents on coding work. You will join an AI-moderated session, inspect proposed programming tasks, and judge how well they match real engineering demands. Your notes will cover task structure, difficulty, and realism, plus where the coding evaluation harnesses need fixes. The project is remote, paid, and open to engineers in India and the Philippines.
Micro1
VerifiedJunior Software Engineer
micro1 is working with a top AI lab to bring on junior software engineers who will assess AI-generated code and craft engineering problems that test model capabilities. Your day-to-day involves reviewing code for correctness, quality, and design, then writing reference solutions and test cases that push models on real-world software tasks. You will also examine how models reason about system design, debugging, and algorithms, and give detailed technical feedback to help improve performance. This is a remote contractor role, and the team is looking for engineers who can explain their technical choices in writing.