ML Challenge Task Auditor
Listing checked September 1, 2026 · pay as published by Mercor
Overview
Contract reviewers with at least three years of hands-on applied ML experience will audit challenge tasks used to train and evaluate models for a frontier AI lab. The work focuses on experiment design, model-selection reasoning, and evaluation methodology, with feedback delivered through a defined rubric. Each review checks for data-quality problems such as leakage, metric gaming, and weak train/test/CV hygiene. This role is not an LLM-application-building or MLOps position; it demands reproducible critique of ML claims using frameworks like PyTorch, TensorFlow, scikit-learn, and XGBoost.
What You'll Do6
- 1Review applied ML challenge tasks for correctness, clarity, and methodological soundness before they are used in model training or evaluation.
- 2Assess the design of experiments, including choice of baselines, hyperparameter tuning procedures, and model-selection logic.
- 3Judge evaluation methodology for hidden biases, data leakage, and susceptibility to metric gaming.
- 4Write clear, rubric-based feedback that explains strengths and weaknesses and suggests concrete fixes.
- 5Reproduce or sanity-check reported results against provided code and data to verify claims.
- 6Flag any train/test/CV hygiene issues that could invalidate benchmark outcomes.
Requirements7
- 13+ years of hands-on applied or experimental ML work, covering experiment design, model selection, hyperparameter tuning, and evaluation methodology.
- 2Demonstrated data-quality rigor: ability to detect leakage, spot metric gaming, and enforce train/test/CV hygiene.
- 3Working proficiency in standard ML frameworks: PyTorch, TensorFlow, scikit-learn, and XGBoost.
- 4Ability to critique ML claims against evidence and reproduce results from code.
- 5Preferred: Kaggle competition or other benchmark experience.
- 6Preferred: graduate research or publications in applied ML.
- 7Preferred: prior experience grading tasks or performing peer review.
Who Should Apply
The ideal candidate has spent years designing and running applied ML experiments, knows how to spot data leakage and metric gaming from a mile away, and can write feedback that a researcher can act on. Reviewers thrive on adversarial evaluation: they enjoy poking holes in model-selection logic and reproducing results to see if claims hold up. This role is not a fit for builders who want to ship LLM applications or manage MLOps pipelines, because the entire job is auditing the rigor of existing ML tasks. Candidates often get rejected when they cannot provide concrete examples of catching leakage or gaming in past work, or when their experience is too narrow, such as only using AutoML without understanding the underlying methodology.
Salary Insight
The rate is $70.00 - $90.00 per hour, and the position is remote in the United States. That rate lines up with a senior applied ML practitioner who can audit experiments and write rigorous feedback, not a junior engineer.
Pay and demand for AI Training roles
AggregatedTypical pay
$70/hour
This role
$70–$90/hr
Most AI Training roles pay $40–$98 per hour. This role's pay falls inside that range.
Based on 1,034 similar roles that publish pay · 302 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 1,182
- Listed in last 30 days
- 441
- Remote
- 98%
Hiring most right now: micro1 (342) · SME Careers (253) · Mercor (143)
Most requested skills · share of roles
- ai training14%
- llm evaluation12%
- ai evaluation12%
- data annotation9%
Figures from AI Training roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
$70–90/hr
Required Skills
Application Tip
In your application, include a short write-up of a time you caught a data leakage problem, a gamed metric, or a flawed evaluation design, and explain how you proved it to the original authors. That kind of evidence separates strong auditors from candidates who just list ML tools.
How NearSkill helps you apply
1. Check your job match
We compare your resume with the job requirements to help you understand how well your experience fits.
2. Continue to the employer’s website
We’ll direct you to the employer’s application portal, where you can complete the application.
3. Get updates by email
If NearSkill is handling follow-up for this role, we’ll email you when there’s an update on your screening.
Want to check your fit first?
See how your resume matches this role before applying.
Check my job matchSimilar open positions
Explore active roles that match your skills and interests.
AfterQuery
VerifiedML Research Expert - Benchmark Authoring
AfterQuery seeks Master's and PhD-level ML researchers to design and validate benchmark tasks that measure what frontier models can do across machine learning. The work is remote, asynchronous, and flexible, with 5 to 40 hours a week available. Your domain expertise shapes the tasks, whether you focus on language models, reinforcement learning, robotics, ML systems, or another listed area. The role fits people who write ML code and publish results, not those with annotation or data-labelling backgrounds. Pay is $150 per hour.
SME Careers
VerifiedPython ML Quality Lead for Remote Contract Role
Remote, hourly contractor role overseeing quality for Python-driven machine learning training projects. You will review AI-generated Python code, ML pipelines, and model explanations, delivering precise feedback aligned to project rubrics. Assessments focus on code quality, data handling, reproducibility, and evaluation standards to guide contributors toward consistent results. SME Careers, a growth-focused AI data services company under SuperAnnotate, connects you with future expert opportunities within the expert network.
AfterQuery
VerifiedML Inference & GPU Kernel Research Engineer
AfterQuery builds a research cohort of senior ML inference and GPU kernel specialists to test how frontier AI models handle expert-level problems. You will not ship production services. Instead, you design hard scenarios, write reference answers and grading rubrics, then judge model outputs for technical accuracy and domain judgment. The work centers on ML inference, GPU kernel engineering, and large-model deployment, with schedules that stay asynchronous and remote. Candidates with 6 to 10 years in the field and a PhD, or a master's plus a strong research record, who send the requested materials get priority review.
Turing
VerifiedMLE Bench – ML Engineers
This contract role centers on benchmark-driven evaluation of real-world machine learning systems. You will work directly with production-grade codebases to build, run, and modify model training, evaluation, and inference pipelines. The work blends research and engineering, using frameworks such as PyTorch, TensorFlow, or JAX and collaborating with researchers to design challenging evaluation tasks. The position is fully remote and requires a minimum 20-hour week with a 4-hour daily overlap with PST.
Turing
VerifiedMLE Bench – Data Analyst
Turing runs benchmark-driven evaluation projects for frontier AI labs, and this role focuses on data analysis for MLE Bench. You will inspect real-world machine learning outputs, define and validate metrics, and write reproducible Python and SQL analysis scripts. The 3-month contractor engagement requires at least 20 hours per week and a 4-hour daily overlap with PST.
Meridial
VerifiedTechnical Task Auditor
Independent contractors in this role audit tasks that train and evaluate AI systems. You pick one specialty from eight areas, such as GPU kernels, AWS Trainium, NKI, CVE and application security, machine learning, AI-assisted developer workflows, AWS Serverless and Infrastructure as Code, Kubernetes, or SWE-Bench. Your job is to judge whether each task is accurate, realistic, solvable, and reproducible, and whether reliable tests and evaluation criteria support it. When something falls short, you write feedback that other experts can act on. Not every specialty is required, and you complete a technical assessment for your chosen area during selection.