Software Engineering, Data Science, and Systems Design Experts, Swift

Summary: The role involves evaluating LLM-generated responses to software engineering queries, ensuring accuracy and clarity while conducting fact-checking and code validation. Candidates should possess significant experience in software engineering, particularly with Swift, and demonstrate strong analytical skills. The position requires a deep understanding of coding challenges and the ability to assess model responses against established guidelines. Ideal candidates will have a background in computer science and experience with LLMs and open-source contributions.

Key Responsibilities:

• Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness
• Conduct fact-checking using trusted public sources and authoritative references
• Conduct accuracy testing by executing code and validating outputs using appropriate tools
• Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies
• Assess code quality, readability, algorithmic soundness, and explanation quality
• Ensure model responses align with expected conversational behavior and system guidelines
• Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines

Key Skills:

• BS, MS, or PhD in Computer Science or a closely related field
• Significant (5+ years) real-world experience in software engineering or related technical roles
• Expert in Swift
• Able to solve HackerRank or LeetCode Medium and Hard-level problems independently
• Experience contributing to well-known open-source projects, including merged pull requests
• Significant experience using LLMs while coding and understanding their strengths and failure modes
• Strong attention to detail and comfortable evaluating complex technical reasoning, identifying subtle bugs or logical flaws
• Prior experience with RLHF, model evaluation, or data annotation work (nice to have)
• Track record in competitive programming (nice to have)
• Experience reviewing code in production environments (nice to have)
• Familiarity with multiple programming paradigms or ecosystems (nice to have)
• Experience explaining complex technical concepts to non-expert audiences (nice to have)

Salary (Rate): £60.00/hr

City: undetermined

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT