AI Agent Evaluator
Listing checked October 10, 2026 · pay as published by Micro1
Overview
This contract role puts you on the front line of testing how well AI agents handle real errands. You will run two agents side by side on your own phone, using your personal accounts, and judge how they perform on tasks like paying bills, booking trips, and buying groceries. Each run takes about 1.5 hours, including evidence and scoring. The project kicks off Monday, October 12, with onboarding happening today through Sunday so you are ready to start. You will need an active personal AI account (Free, Go, or Plus, not Pro) and a smartphone that can record your screen.
What You'll Do7
- 1Run everyday errands with two AI agents, covering bills, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms.
- 2Approve or halt the agent at critical steps during each task.
- 3Record your screen for every attempt you make.
- 4Score each run on completion, quality, control issues, satisfaction, and time.
- 5Write clear feedback about what worked, what failed, and how the experience felt.
- 6Document each step of the process as you go.
- 7Use your own accounts on your phone to test the agents in real conditions.
Requirements9
- 1US-based with a Bachelor's degree.
- 22-3 years of experience, ideally in engineering, business management, or office/operations.
- 3Active personal AI account (Free, Go, or Plus; not Pro).
- 4Smartphone that can record the screen.
- 5Comfort using AI in daily life.
- 6Strong attention to detail and ability to follow instructions.
- 7Clear written communication skills.
- 8Generalists are welcome; domain experts are preferred.
- 9No coding or AI background required.
Who Should Apply
The ideal candidate is US-based, holds a Bachelor's degree, and has 2-3 years of work in engineering, business management, or office/operations. You already use AI tools in your daily routine and can document each step with care. This role suits generalists who are curious about how AI agents perform on real tasks. It is less suitable for people who dislike repetitive testing, struggle to follow detailed instructions, or cannot commit to screen recording every attempt. Candidates often get rejected when they lack an active personal AI account (Free, Go, or Plus) or cannot provide a smartphone that records the screen. Another common reason for low fit scores is weak written feedback that misses specific observations about the agent's behavior.
Salary Insight
The pay range is $25 to $45 per hour. No other compensation details are provided in the source.
Pay and demand for AI Training roles
AggregatedTypical pay
$70/hour
This role
$25–$45/hr
Most AI Training roles pay $38–$95 per hour. This role's pay sits below that range.
Based on 951 similar roles that publish pay · 300 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 1,080
- Listed in last 30 days
- 363
- Remote
- 98%
Hiring most right now: micro1 (284) · SME Careers (253) · Handshake AI (132)
Most requested skills · share of roles
- ai training15%
- ai evaluation13%
- llm evaluation12%
- data annotation10%
Figures from AI Training roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
$25–45/hr
Required Skills
Application Tip
Before you apply, confirm you have an active personal AI account (Free, Go, or Plus, not Pro) and a smartphone that can record the screen. In your screening answers, describe a specific time you used an AI tool for a real errand and what you observed. Mention your 2-3 years of experience in engineering, business management, or office/operations, and highlight your attention to detail with an example of documenting a process step by step.
How NearSkill helps you apply
1. Check your job match
We compare your resume with the job requirements to help you understand how well your experience fits.
2. Continue to the employer’s website
We’ll direct you to the employer’s application portal, where you can complete the application.
3. Get updates by email
If NearSkill is handling follow-up for this role, we’ll email you when there’s an update on your screening.
Want to check your fit first?
See how your resume matches this role before applying.
Check my job matchSimilar open positions
Explore active roles that match your skills and interests.
Micro1
VerifiedEveryday AI Agent Tester
This is a chance to put two AI agents through their paces on real errands like paying bills, booking trips, and handling groceries. You will use your own phone and personal AI accounts to run each scenario, approve or halt steps, and record the screen. After each run, you score the agent on completion, quality, control issues, satisfaction, and time. The project kicks off Monday, October 12, and onboarding happens today, Saturday, or Sunday so you are ready to start. Expect about 1.5 hours per task, including evidence and scoring.
Micro1
VerifiedAI Product Evaluator
micro1 is hiring AI Product Evaluators to support a customer's high-impact AI initiative. You will assess AI models and AI agents against detailed scenarios, then document what you find in structured reports. The work is operational and often runs against strict timeframes and usage limit resets. Your input helps shape how these systems learn, reason, and respond.
Micro1
VerifiedGmail & Google Calendar AI Evaluator (US Remote)
micro1 is hiring Gmail and Google Calendar AI Assistant Evaluators for a customer-driven project that improves how AI handles everyday digital tasks. You will draw on your own habits with email, scheduling, bookings, and coordination to test and grade assistant responses. The work is fully remote, open to US-based contractors, and pays $15 to $30 per hour. Your feedback will directly shape how AI models manage the tasks people delegate most often.
Turing
VerifiedAI Evaluation Specialist – Small Business & Operations
This project puts small business owners in the evaluator seat for AI chatbot output. You build realistic prompts from daily operational scenarios, then compare how multiple chatbots handle them over up to five turns. Judging comes down to clarity, usefulness, and accuracy, with written feedback required. The scope also includes marketing content, customer interactions, market research, and financial planning support. The engagement is project-based and runs for 16 weeks with a fixed set of evaluation tasks.
Micro1
VerifiedAI Evaluation Specialist
As an AI Evaluation Specialist, you'll help train next-generation AI systems by designing and executing hands-on evaluation tasks. Your insights will directly shape how models learn, reason, and perform on practical computer-based workflows. This is a fully remote contract role where meticulous observation and clear documentation are key.
Mercor
VerifiedAdministrative Services Manager - Remote AI Evaluation
Mercor needs Administrative Services Managers to assess AI-written office operations plans, facilities documents, and records procedures. You bring hands-on experience with facilities management, procurement, and records management to judge whether each response holds up in the real world. The project runs about 4 to 6 weeks and starts right away, with a 20-hour-per-week commitment. Your feedback helps shape how AI systems handle administrative workflows.