Data Analysis Expert - Remote AI Evaluation

What You'll Do6

• Turn an analysis you shipped into a test problem, complete with the dataset, the core question, the constraint, and a reference answer for grading a model's response.
• Review model-written analyses and pipelines for logic, asking whether a query answers the actual question, whether a join fans out, and whether a metric means what the model claims.
• Spot results that look valid but would mislead a decision-maker because of selection effects, data leakage, or a metric that fails to track the outcome.
• Write clear reasoning for each judgment so project teams understand why a model response passes or fails.
• Flag ambiguous instructions or missing context before they corrupt a grading task.
• Build rubrics and answer keys that other experts can apply to similar model outputs.

Requirements6

• Professional experience in analytics, data engineering, statistics, or machine learning.
• A track record of shipped analyses that someone used to make a real decision.
• Strong written communication, because most tasks require you to explain your reasoning in detail.
• Comfort with ambiguous briefs and the judgment to ask questions when instructions lack clarity.
• Depth in at least one area: business intelligence, data pipelines, statistical modeling, or machine learning.
• Ability to identify selection effects, leakage, and metric-outcome mismatches in analytical work.

Who Should Apply

Data practitioners with a few years of professional analytics, data engineering, statistics, or machine learning work fit this pool best, including those who can point to analyses that shaped a business or product decision. Clear writers thrive here because grading model output depends on explaining why an answer succeeds or fails. The role suits people who handle vague prompts without freezing and who call out unclear instructions instead of guessing. Candidates who need tasks defined from the start, dislike writing rationales, or lack shipped analysis work tend to score low. A common rejection reason is a resume that lists tools but shows no analysis with a concrete decision attached. Another is treating the short AI interview as a formality rather than a chance to demonstrate reasoning.

Salary Insight