AI Evaluation Rubric Auditor
Mercor
AI assistant
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: AI Evaluation Rubric Auditor
Type: Contract
Compensation: $60–$85/hour
Location: Remote
Role Responsibilities
- Audit complete tasks, including prompts and grading rubrics, to ensure rubric failures are fair.
- Evaluate if information was available and if rubrics accurately grade the prompt requirements.
- Determine if valid answers were incorrectly marked wrong and if rubric structures follow