Intelligence Analyst

What You'll Do7

• Map analytic cases to defined taxonomies across multiple evaluation areas to ensure full coverage of intelligence scenarios.
• Write complex test cases that assess model performance on source handling, analytic rigor, and nuanced uncertainty expression.
• Grade AI-generated responses on reasoning, source use, recognition of limitations, and practical utility.
• Flag instances where the model over-refuses legitimate analytic exploration and document clear feedback.
• Document edge cases and analytic disagreements to support ongoing improvement of evaluation methods.
• Work with project leads to refine procedures for source evaluation and confidence-level assignment.
• Provide insights to calibrate analytic standards and enhance the quality of AI system benchmarks.

Requirements7

• Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background.
• Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards.
• Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor.
• Exceptional written and verbal communication skills, with a focus on clarity and precision.
• Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.
• Previous participation in AI evaluation or model assessment initiatives is a plus.

Who Should Apply

The ideal candidate has a background in all-source intelligence analysis, ideally within the civilian IC, and a strong grasp of source evaluation and confidence assessment. Experience mentoring or leading analytic reviews is valuable, as is familiarity with structured analytic techniques. Those who can clearly articulate model limitations and over-refusal scenarios will thrive here.

Salary Insight