Software / AI / IT / Data Evaluator — Code and Systems Review

Software / AI / IT / Data Evaluator — Code and Systems Review is a remote evaluation track for reviewing software / ai / it / data evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain. **Why this role matters** ------------------------- AI data reviewers help turn software / ai / it / data evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data. **Responsibilities** -------------------- * Evaluate software / ai / it / data evaluation model outputs against a versioned rubric and assign severity tags for Software / AI / IT / Data Evaluator — Code and Systems Review assignments. * Compare paired responses and pick the stronger answer with a written rationale. * Label hallucinations, instruction\-following failures, and unsafe content with structured tags. * Capture ambiguous prompts and route them back to the program team for rubric updates. * Maintain reviewer\-quality scores by calibrating against gold\-standard examples each week. * Document recurring failure modes so the modeling team can target them in the next training run. **Qualifications** ------------------ * Prior evaluation, annotation, or human\-rater experience on software / ai / it / data evaluation or adjacent content for Software / AI / IT / Data Evaluator — Code and Systems Review work. * Comfort applying multi\-page rubrics consistently across long batches. * Clear written reasoning that names the issue and the rubric clause being applied. * Strong attention to detail and the ability to flag when a prompt itself is the problem. * Reliable async availability for at least 10 hours per week. **Example tasks** ----------------- * Compare two software / ai / it / data evaluation model responses to the same prompt and pick the stronger one with rationale. * Tag an unsafe response with the correct policy category and severity. * Audit a 50\-row batch for rubric consistency and report drift to the program lead. * Propose a rubric clarification after spotting a recurring failure mode. **Nice to have** ---------------- * Background in linguistics, content moderation, or trust \& safety review. * Experience with inter\-rater agreement metrics and calibration cycles. * Domain expertise that lets you spot subject\-matter errors automated checks miss. **Skills** ---------- * Model output evaluation * Rubric\-based annotation * Severity tagging * Inter\-rater calibration * Software / AI / IT / data evaluation **Work model** -------------- Remote — US\-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US. **Compensation** ---------------- $80–$120 / hr **Application process** ----------------------- Apply through AuraOne's specialist intake for role\-specific routing and review. Final project scope, schedule, and con