Rubrics Expert | $35-$50/hr Remote
Overview
This role involves applying expertise in rubric design and assessment to train next-generation AI systems. The rubrics expert will design, develop, and refine comprehensive rubrics, analyze task performance, and provide structured feedback. They will establish scoring methodologies, lead calibration sessions, and oversee quality control to ensure high-quality AI training outputs. No prior AI experience is required—domain knowledge in assessment is key.
What You'll Do5
- 1Design, develop, and refine comprehensive rubrics tailored to AI training and assessment needs.
- 2Analyze task performance and provide structured, actionable feedback to guide improvement and calibration.
- 3Establish and document effective scoring methodologies ensuring consistency, fairness, and high-quality outcomes.
- 4Lead calibration sessions to align evaluators and maintain rubric integrity across projects.
- 5Validate assessment outputs for accuracy, reliability, and alignment with project objectives.
Requirements5
- 1Proven expertise in rubric design, performance analysis, and feedback structuring for complex assessment environments.
- 2Strong background in scoring methodology development and calibration for AI or educational projects.
- 3Demonstrated success in assessment validation and quality control processes.
- 4Exceptional written and verbal communication skills, with ability to convey nuanced judgment and rationale.
- 5Practical, hands-on grocery shopping experience (via apps, delivery, in-store, or pickup) to inform realistic scenario assessments.
Who Should Apply
The ideal candidate has deep expertise in rubric design, scoring methodology, and assessment validation, possibly from educational or instructional design backgrounds. They should be detail-oriented, analytical, and comfortable working in a cross-functional team. Practical grocery shopping experience is a unique but required qualification. No AI experience is needed, but a willingness to apply domain knowledge to AI training is essential.
Salary Insight
Contract position paying $35-$50 per hour.
Required Skills
Application Tip
Highlight any specific experience you have with rubric development and calibration, and be sure to mention your practical grocery shopping experience, as it is explicitly requested in the job requirements.