AI Conversation Quality Analyst

Role summary / purpose Improve the quality of chatbot responses by creating a feedback loop where AI responses are judged by humans. Review chatbot conversations and related artifacts including individual turns, model responses, retrieved content, citations, and metadata within approved tools. Apply approved labeling guidelines, taxonomies, and decision rules, completing all required labels, rationales, confidence assessments, and ambiguity indicators. Key responsibilities • Evaluate chatbot performance across dimensions including intent, task completion, correctness, completeness, relevance, groundedness, source attribution, response quality, safety, policy compliance, escalation need, and failure category. • Identify and escalate content that is ambiguous, sensitive, safety-relevant, outside current guidelines, or requires subject-matter-expert review. • Participate in onboarding, calibration, refresher training, quality reviews, and adjudication sessions to maintain consistent labeling decisions. • Conduct first-level quality checks on assigned work and correct errors identified through quality reviews. • Maintain complete and accurate records of completed annotations, quality results, exceptions, escalations, and open issues in designated systems. • Provide weekly reports summarizing completed volume, productivity, quality results, backlog, open questions, risks, guideline gaps, and required actions. • Recommend and document improvements to annotation guidelines and taxonomy based on recurring ambiguities, labeling patterns, and adjudication outcomes. Requirements – education & experience • Experience labeling conversational AI data including text and audio. • Professional fluency in English with strong reading comprehension and written communication skills. • Demonstrated ability to interpret and consistently apply detailed guidelines, taxonomies, and decision rules. • Strong analytical judgment and attention to detail, including the ability to identify factual errors, incomplete responses, unsupported claims, policy concerns, and ambiguous cases. • Ability to document labeling decisions and provide clear, concise rationales. • Ability to work accurately on repetitive tasks while meeting quality, productivity, and turnaround expectations. • Ability to participate in calibration sessions, incorporate feedback, and adjust decisions as guidelines evolve. • Ability to prepare clear weekly reports summarizing labeling volume, productivity, quality results, progress, and open issues. Nice-to-have / preferred skills • Prior experience in the automotive industry. • Experience with major technology platforms or AI data annotation services.