Software Engineering, Data Science, and Systems Design Experts, C++

Summary: The role involves evaluating LLM-generated responses to software engineering queries, ensuring accuracy and clarity while conducting fact-checking and code validation. Candidates are expected to have extensive experience in software engineering, particularly with C++, and to apply consistent evaluation standards. The position requires strong attention to detail and the ability to assess code quality and reasoning. Ideal candidates will also have experience with LLMs and a background in competitive programming or data annotation.

Key Responsibilities:

• Evaluate LLM-generated responses for accuracy, reasoning, clarity, and completeness.
• Conduct fact-checking using trusted public sources and authoritative references.
• Perform accuracy testing by executing code and validating outputs.
• Annotate model responses by identifying strengths and areas for improvement.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Ensure model responses align with expected conversational behavior and system guidelines.
• Apply consistent evaluation standards by following clear taxonomies and guidelines.

Key Skills:

• BS, MS, or PhD in Computer Science or a closely related field.
• 5+ years of real-world experience in software engineering or related technical roles.
• Expertise in C++ programming language.
• Ability to solve HackerRank or LeetCode Medium and Hard-level problems independently.
• Experience contributing to well-known open-source projects.
• Significant experience using LLMs while coding.
• Strong attention to detail and comfort evaluating complex technical reasoning.
• Prior experience with RLHF, model evaluation, or data annotation work (nice to have).
• Experience reviewing code in production environments (nice to have).
• Familiarity with multiple programming paradigms or ecosystems (nice to have).
• Experience explaining complex technical concepts to non-expert audiences (nice to have).

Salary (Rate): £100.00/hour

City: undetermined

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT