AI Developer Trace Task Auditor
Listing checked September 1, 2026 · pay as published by Mercor
Overview
Every coding trace you receive comes from AI-assisted developer sessions. You will assess each trace for code correctness, workflow soundness, and reasoning, then write clear feedback that follows a defined rubric. A frontier AI lab uses these evaluations to train and refine its models, so your judgments shape model behavior. The role is remote within the United States and pays $70 to $90 per hour.
What You'll Do6
- 1Review complete coding traces created by AI-assisted developer tools.
- 2Judge each session for code correctness, workflow soundness, and reasoning quality.
- 3Write structured feedback that maps to a specific grading rubric.
- 4Identify bugs, logic errors, and deviations from engineering best practices in multi-step coding workflows.
- 5Verify whether the AI tool followed the developer's specification and intent across the session.
- 6Document your findings to support model training and evaluation.
Requirements8
- 13+ years of professional software development experience.
- 2Daily use of AI-assisted coding tools such as Cursor, GitHub Copilot, Claude Code, or equivalent.
- 3Experience with agentic or spec-driven development workflows.
- 4Strong code-reading and debugging skills in full-stack or backend systems.
- 5Ability to evaluate long coding trajectories for correctness and engineering best practice.
- 6Familiarity with Kiro or Amazon CodeCatalyst (preferred).
- 7Prior experience grading or evaluating AI-generated code (preferred).
- 8Contributions to developer tooling (preferred).
Who Should Apply
Developers who use AI coding tools daily and enjoy reviewing code from others will fit this role well. You need at least 3 years of professional development experience plus a sharp eye for logic errors across full-stack or backend systems. This job is less suitable for developers who prefer writing new code over analyzing existing traces. A common rejection reason is a lack of hands-on agentic workflow experience. Another is providing feedback that is too subjective to fit a rubric.
Salary Insight
Pay is $70 to $90 per hour. This rate matches senior-level contract work and accounts for the specialized focus on AI-assisted coding evaluation and rubric-based feedback.
Pay and demand for AI Training roles
AggregatedTypical pay
$70/hour
This role
$70–$90/hr
Most AI Training roles pay $40–$98 per hour. This role's pay falls inside that range.
Based on 1,063 similar roles that publish pay · 302 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 1,217
- Listed in last 30 days
- 454
- Remote
- 98%
Hiring most right now: micro1 (356) · SME Careers (253) · Mercor (150)
Most requested skills · share of roles
- ai training13%
- ai evaluation11%
- llm evaluation11%
- data annotation9%
Figures from AI Training roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
$70–90/hr
Required Skills
Application Tip
In your application, list the AI coding tools you use daily and describe a specific session where you caught a critical bug in AI-generated code. If you have graded or evaluated code before, mention the rubric you used.
How NearSkill helps you apply
1. Check your job match
We compare your resume with the job requirements to help you understand how well your experience fits.
2. Continue to the employer’s website
We’ll direct you to the employer’s application portal, where you can complete the application.
3. Get updates by email
If NearSkill is handling follow-up for this role, we’ll email you when there’s an update on your screening.
Want to check your fit first?
See how your resume matches this role before applying.
Check my job matchSimilar open positions
Explore active roles that match your skills and interests.
Micro1
VerifiedSenior AI Trainer
micro1 seeks Senior AI Trainers to work on a customer-driven project that improves AI system quality through expert model training and evaluation. You will apply your generalist expertise and hands-on experience with AI products to help train next-generation systems. Your daily work involves evaluating AI chat and search tools, writing and refining prompts, and annotating model responses using detailed grading guidelines. This is a remote contractor role open to candidates in Northern America and Europe.
Terac
VerifiedSoftware Engineer - AI Coding Benchmarks & RL
Terac builds coding tasks that train and test frontier AI coding agents. Experienced software engineers design realistic full-stack engineering problems, write tests that judge correctness, and push beyond the UI into services, APIs, and backend systems. The codebase for these tasks is TypeScript with React, Node, SQLite, and Playwright, and it grows as each task builds on the last. Contractors work 10 to 20+ hours per week, paid each week at $45 to $75 per hour, with US and Canada timezone overlap preferred. You can use AI tools as much as you like while writing code and tests.
Micro1
VerifiedAI Evaluation Specialist
As an AI Evaluation Specialist, you'll help train next-generation AI systems by designing and executing hands-on evaluation tasks. Your insights will directly shape how models learn, reason, and perform on practical computer-based workflows. This is a fully remote contract role where meticulous observation and clear documentation are key.
Terac
VerifiedSoftware Engineer - AI Agent Code Review
Terac pays software engineers to review coding tasks and evaluation harnesses used for benchmarking AI agents. You will inspect programming challenges, determine whether they mirror real software engineering problems, and verify logic, test cases, and environment structure. The remote session combines a guided conversation with screen-sharing exercises, so you explain your reasoning as you analyze code. Engineers in the USA, Canada, and France can take part.
Turing
VerifiedSoftware Engineer – AI Code Evaluation & Benchmarking (US candidates only)
This contract role places an experienced software engineer inside an evaluation workflow for frontier AI models. You will review AI-generated code for correctness, efficiency, and maintainability, validate solutions against real engineering tasks, and debug failures across different environments. You will also help build and refine evaluation datasets, benchmarks, and grading rubrics. The assignment runs for one month, requires at least 4 hours per day and 20 hours per week with a 4-hour overlap with PST, and is open only to candidates in the US. Work is fully remote.
AfterQuery
VerifiedAdvanced Coding Expert
AfterQuery hires experienced engineers to create long-horizon coding tasks that test and train advanced coding agents. You will draw real bugs, features, regressions and migrations from a substantial codebase you own, then shape them into prompts that demand genuine investigation from an agent. Each task needs a Docker setup that builds and runs the same way every time. You also design test suites and review rules that separate correct work from shortcut solutions. The contract runs 30 hours per week, remote, with pay from $70.00 to $100.00 per hour.