AI Quality Analyst - English
Listing checked August 25, 2026 · pay as published by Turing
Overview
This role puts you inside Gemini's personalization quality loop. You will design multi-turn prompts that draw on your own Google activity, including Gmail, Search, and YouTube history, then judge whether the model's responses are grounded, integrated, and helpful. The work involves side-by-side SxS evaluations, writing rationales that reference specific turns, and verifying debug info to confirm the model used your data correctly. This 3-month contractor engagement requires at least 4 hours per day with a 4-hour overlap with PST.
What You'll Do8
- 1Design and run multi-turn conversation prompts that span 1 to 5 turns and require the model to draw on your personal context.
- 2Assess each model response against the intent you set in the starting prompt to determine whether personalization was applied.
- 3Flag grounding failures when the model makes claims about you that lack supporting evidence or depend on flawed inferences.
- 4Evaluate integration quality to ensure personal details are woven into responses without robotic overnarrating.
- 5Compare two model responses side-by-side (SxS) and stack-rank them by helpfulness, ease of use, and enjoyment.
- 6Write concise, defensible rationales that reference exact turn numbers to justify your rankings.
- 7Inspect Debug Info to confirm the model used chat summaries and data sources as intended.
- 8Delete evaluation conversations after each session to prevent them from affecting future chat history.
Requirements12
- 1Native or near-native English reading and writing ability, since English is the project's focus language.
- 2Willingness to connect a primary personal Google account (not a test account) and enable personal data sources for real-world evaluation.
- 3Openness to work at least 4 hours per day, up to 40 hours per week, with 4 hours overlapping PST, plus full-time availability in your local time zone.
- 4Sharp analytical thinking to judge nuanced, ambiguous AI responses and spot personalization errors, poor inferences, and forced connections.
- 5Experience designing creative, multi-turn prompts that use personal context to challenge model capabilities.
- 6Keen attention to detail for reviewing side-by-side (SxS) responses and detecting subtle differences in naturalness and overnarrating.
- 7Strong written communication skills to produce structured rationales with explicit turn numbers.
- 8Ability to provide constructive feedback and detailed annotations to support model improvement.
- 9Self-motivation and comfort working on your own in a remote environment.
- 10Reliable desktop or laptop and a stable internet connection.
- 11BS/BA degree or equivalent experience in a related analytical field such as policy, law, ethics, linguistics, journalism, or computer science.
- 12Previous experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred.
Who Should Apply
The candidate who thrives here is comfortable letting Gemini read their personal Gmail and Search history, enjoys crafting creative multi-turn prompts, and can defend a side-by-side ranking with precise turn-level evidence. This role is not a fit for anyone who wants to keep their personal Google data separate from work, or who prefers repetitive labeling over open-ended evaluation and written rationales. Rejections often happen when a candidate misses the 24-hour assessment window or submits rationales that are too vague to show they caught grounding and integration issues.
Pay and demand for AI Training roles
AggregatedTypical pay
$70/hour
This role
Pay not disclosed
Most AI Training roles pay $40–$95 per hour.
Based on 975 similar roles that publish pay · 300 publish only a top rate; those count at the rate they gave
Rates shown per hour. Yearly and monthly pay converted; one-time fees and non-USD pay are not included.
- Live similar roles
- 1,110
- Listed in last 30 days
- 379
- Remote
- 98%
Hiring most right now: micro1 (294) · SME Careers (253) · Mercor (137)
Most requested skills · share of roles
- ai training14%
- ai evaluation13%
- llm evaluation12%
- data annotation10%
Figures from AI Training roles live on NearSkill when this page loaded. A role can close before you apply, so check the listing itself.
Compare your resume against these rolesLocation
Compensation
Undisclosed by Turing
Comparable roles pay
975 roles$40 – $95/ hr
Not this role’s pay. Turing has not published a range.
Confirmed during Turing profile screening.
Required Skills
Application Tip
When you complete the assessment, cite explicit turn numbers in your rationales and quote the exact signal that broke grounding or overnarrating. Showing that you can reference conversation details with precision is the strongest way to meet the written communication bar for this role.
How NearSkill helps you apply
1. Check your job match
We compare your resume with the job requirements to help you understand how well your experience fits.
2. Continue to the employer’s website
We’ll direct you to the employer’s application portal, where you can complete the application.
3. Get updates by email
If NearSkill is handling follow-up for this role, we’ll email you when there’s an update on your screening.
Want to check your fit first?
See how your resume matches this role before applying.
Check my job matchSimilar open positions
Explore active roles that match your skills and interests.
Turing
VerifiedAI Quality Analyst - English
You will judge how well Gemini draws on your personal history, including past chats, Gmail, Google Search, and YouTube activity, to craft relevant replies. The job centers on a new personalization feature, so your own Google account becomes the test bed. Each evaluation asks you to compare two model responses side by side and decide which one feels more natural and useful. You also classify weak Grounding, awkward Integration, and judge the overall Helpfulness of each output. Prompt design is a core part of the work: you create multi-turn scenarios from your own experiences to push the model.
Turing
VerifiedAI Quality Analyst (Personalization) - Turkish
In this contract role, you will evaluate a new personalization feature for Gemini. You will measure how well the model incorporates signals from your past conversations, Gmail, Google Search, and YouTube activity to make responses more relevant. You will design multi-turn prompts based on your own experiences and then judge the model's output on dimensions like Grounding, Integration, and Helpfulness. The work demands both creative prompt design and disciplined analytical review of model responses.
Turing
VerifiedAI Quality Analyst (Personalization) - Polish
Evaluate a new Gemini personalization feature by testing how well the model draws on past conversations, Gmail, Google Search, and YouTube activity to tailor responses. You will design multi-turn prompts from your own personal experiences, then score outputs on Grounding, Integration, and Helpfulness. The position is a contractor role that requires Polish reading and writing proficiency and at least 4 hours of daily overlap with Pacific Time. Work happens remotely on your own device, and every evaluation conversation must be deleted afterward to keep your personal history clean.
Turing
VerifiedAI Quality Analyst (Personalization) - Russian
You will evaluate how well Gemini uses personal data from past conversations, Gmail, Google Search, and YouTube activity to make responses more relevant. The work draws on your own experiences to craft multi-turn prompts, then rates the model on grounding, integration, and helpfulness. You will compare side-by-side outputs, write structured rationales, and verify that the model pulled from the right data sources. This is a remote contractor role requiring Russian fluency and daily overlap with PST.
Turing
VerifiedAI Quality Analyst (Personalization) - Vietnamese
This role puts you inside the evaluation loop for a new personalization feature in Gemini. You will design multi-turn prompts (1-5 turns) that draw on your own Google account data, including past chats, Gmail, Search, and YouTube history. Your core job is to judge whether the model's personalized responses are relevant, grounded in real evidence, and natural, using dimensions like Grounding and Integration. Work is conducted in Vietnamese, comparing two model outputs side-by-side and writing rationales that cite exact conversation turns. The engagement lasts 3 months and pays $15 per hour, with a daily commitment that includes PST overlap.
Turing
VerifiedAI Quality Analyst (Personalization) - Korean
This role evaluates a new personalization feature for Gemini that draws on past conversations, Gmail, Google Search, and YouTube activity. You will design multi-turn prompts from your own personal context and then judge how well the model grounds responses in real evidence rather than inferences. The work includes side-by-side (SxS) ranking of two model outputs, writing structured rationales that cite specific turn numbers, and checking Debug Info to verify chat summaries and data sources. Korean is the focus language, so reading and writing fluency in Korean is required. The position is remote, full-time with at least 4 hours of daily overlap with PST, and runs as a 3-month contractor engagement.