SWE-Bench Task Auditor
Listing checked September 1, 2026 · pay as published by Mercor
Overview
This role puts you inside the evaluation pipeline for a frontier AI lab's models. You will audit SWE-Bench style repository tasks, checking reference patches, test harnesses, and Docker isolation for correctness and reproducibility. Your written, rubric-based feedback shapes which tasks get used for training and evaluation. The work is remote and hourly, paying $70-$90 per hour, and it demands strong open-source credentials plus fluency in Python and at least one of Java, Go, TypeScript, or C++.
What You'll Do5
- 1Audit repository-level benchmark tasks for correctness, reproducibility, and alignment with the intended grading criteria.
- 2Inspect reference patches to confirm they solve the stated problem and do not contain hidden shortcuts or answer leakage.
- 3Review test harnesses and Docker isolation setups so each task executes in a controlled environment and produces accurate pass/fail results.
- 4Detect reward hacking attempts, task contamination, or any behavior that inflates model scores without genuine problem-solving.
- 5Write structured, rubric-based feedback for each task, documenting issues and suggesting fixes.
Requirements8
- 13+ years of professional software engineering experience, with recent hands-on coding in production or OSS settings.
- 2Demonstrated open-source track record: merged PRs, committer status, or maintainer role on a real project.
- 3Ability to audit reference patches and test runners for correctness, and to evaluate Docker isolation in CI environments.
- 4Skill at spotting answer leakage, reward hacking, or other forms of task contamination that inflate benchmark scores.
- 5Professional fluency in Python, plus working knowledge of at least one of Java, Go, TypeScript, or C++.
- 6Familiarity with SWE-Bench, SWE-Bench Verified, or comparable repository-level coding benchmarks (preferred).
- 7Maintainer history on a major Python OSS project such as Django, Flask, scikit-learn, sympy, or pytest (preferred).
- 8Prior experience reviewing code or grading programming tasks (preferred).
Who Should Apply
The ideal candidate is a senior engineer with a public trail of merged PRs and maintainer experience who enjoys auditing code for subtle flaws. You should be comfortable reading patches from other developers and spotting the moment a test answer leaks into the training data. This role is less suitable for engineers who have never done formal code review or who have no open-source footprint. Applications often fail when the candidate lists no real merged contributions, or when they cannot articulate how reward hacking works in a repository-level benchmark. If you have triaged issues or reviewed PRs on a major Python project, that will separate you from most applicants.
Salary Insight
The rate is $70.00-$90.00 per hour. That rate sits at the senior end for contract software engineering work, reflecting the specialized nature of benchmark auditing and the need for maintainer-level judgment.
Pay and demand for Quality Assurance & Testing roles
AggregatedTypical pay
$70/hour
This role
$70–$90/hr
Most Quality Assurance & Testing roles pay $48–$90 per hour. This role's pay falls inside that range.
Based on 297 similar roles that publish pay · 113 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 355
- Listed in last 30 days
- 172
- Remote
- 94%
Hiring most right now: SME Careers (115) · micro1 (108) · AfterQuery (33)
Most requested skills · share of roles
- llm evaluation27%
- ai training25%
- trainer feedback23%
- python23%
Figures from Quality Assurance & Testing roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
$70–90/hr
Required Skills
Application Tip
In your application, include direct links to two or three merged PRs on a public repository and state your exact role in that project. If you have worked with SWE-Bench, show one instance where you verified a task or caught a leakage issue.
How NearSkill helps you apply
1. Check your job match
We compare your resume with the job requirements to help you understand how well your experience fits.
2. Continue to the employer’s website
We’ll direct you to the employer’s application portal, where you can complete the application.
3. Get updates by email
If NearSkill is handling follow-up for this role, we’ll email you when there’s an update on your screening.
Want to check your fit first?
See how your resume matches this role before applying.
Check my job matchSimilar open positions
Explore active roles that match your skills and interests.
Turing
VerifiedSWE Bench – Data Engineer/Data Scientist
Turing, a San Francisco-based research accelerator, is hiring experienced data engineers and data scientists for benchmark-driven evaluation of advanced AI systems. The role centers on SWE Bench-style tasks: building and validating data pipelines, processing structured and unstructured datasets, and preparing features for data science workflows. You will write Python code, run local experiments, and verify outputs for correctness and reproducibility. This is a fully remote contractor assignment with required overlap with PST hours.
Mercor
VerifiedAWS Serverless & Infrastructure-as-Code Task Auditor
This contract role puts you inside the evaluation loop for training data used by a frontier AI lab. You will audit AWS serverless and infrastructure-as-code tasks for architectural correctness, cross-service integration, and IaC fidelity. Expect to work through Lambda, Step Functions, DynamoDB, EventBridge, and related services, then deliver written feedback guided by a set rubric. The position is fully remote at an hourly rate.
Meridial
VerifiedTechnical Task Auditor
Independent contractors in this role audit tasks that train and evaluate AI systems. You pick one specialty from eight areas, such as GPU kernels, AWS Trainium, NKI, CVE and application security, machine learning, AI-assisted developer workflows, AWS Serverless and Infrastructure as Code, Kubernetes, or SWE-Bench. Your job is to judge whether each task is accurate, realistic, solvable, and reproducible, and whether reliable tests and evaluation criteria support it. When something falls short, you write feedback that other experts can act on. Not every specialty is required, and you complete a technical assessment for your chosen area during selection.
Turing
VerifiedSoftware Engineer – AI Code Evaluation & Benchmarking (US candidates only)
This contract role places an experienced software engineer inside an evaluation workflow for frontier AI models. You will review AI-generated code for correctness, efficiency, and maintainability, validate solutions against real engineering tasks, and debug failures across different environments. You will also help build and refine evaluation datasets, benchmarks, and grading rubrics. The assignment runs for one month, requires at least 4 hours per day and 20 hours per week with a 4-hour overlap with PST, and is open only to candidates in the US. Work is fully remote.
Mercor
VerifiedKubernetes Task Auditor
At Mercor, you will assess Kubernetes tasks that train a frontier AI lab's models. The work involves grading cluster-operation scenarios, checking manifest correctness, and judging whether failure-mode troubleshooting is realistic and complete. Each review ends with rubric-based written feedback that the AI team uses to improve model performance. This remote hourly role is open to candidates across the United States.
Micro1
VerifiedSoftware Engineer (GitHub Contributor)
This contract role puts your coding skills to work on AI training data. You will build reproducible Reinforcement Learning Environments that test how well an AI model can fix code, add features, refactor, and improve performance. Each task includes a golden reference solution. You do not need prior AI experience. Your Python, Java, Rust, Go, C++, or TypeScript background is what matters. Tasks take about 15 hours per run, and you can work from anywhere.