Content Specialist III | AI Evaluation & Prompting

This remote contract role offers the opportunity to directly influence the quality and behavior of advanced AI systems.
You’ll evaluate model responses across diverse topics, conversation types, and real-world use cases.
Your work will help identify strengths, uncover failure patterns, and improve how AI systems communicate with users.
You’ll combine expert writing, analytical judgment, prompt engineering, and structured quality assessment to shape model performance.
The role involves close, hands-on experimentation with evolving AI products in a fast-moving environment.
You’ll translate detailed observations into actionable feedback, stronger prompts, and more effective evaluation frameworks.
Success requires balancing quality, speed, consistency, and independent judgment while adapting to changing product priorities.

• 5+ years of professional experience in writing, editing, journalism, production, linguistics, STEM, coding, policy, or another relevant subject-matter discipline.

• Bachelor’s degree or equivalent professional experience.

• At least 1 year of hands-on AI experience is preferred, including prompting, annotation, model evaluation, red teaming, or related work.

• Strong experience evaluating AI-generated content, large language model outputs, or other complex digital content is highly desirable.

• Deep expertise in at least one subject area, with the ability to evaluate content accurately as a subject-matter expert.

• Strong writing, editing, fact-checking, and research capabilities.

• Experience writing and refining prompts and assessing how prompt changes influence AI behavior.

• Strong analytical judgment and exceptional attention to detail.

• Ability to consistently apply detailed rubrics, guidelines, and evaluation criteria across high-volume work.

• Ability to identify subtle differences in AI responses and determine why a response succeeds or fails.

• Strong documentation and communication skills, with the ability to clearly explain findings, questions, blockers, and recommendations.

• Ability to verify claims against original or authoritative sources.

• Experience conducting detailed failure investigations is preferred.

• Comfortable working independently while escalating important questions, risks, and blockers early.

• Highly adaptable and comfortable shifting priorities as AI products and business needs evolve.

• Professional fluency in English is required.

• Must be authorized to work in the United States without requiring current or future sponsorship.