About the Role
What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software.
This is high-impact, technically demanding work at the intersection of software engineering and AI research. You'll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do.
This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI — this is the role.
What You'll Do
- Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
- Build and maintain scalable data pipelines for AI evaluation workflows
- Analyze AI-generated code for correctness, reliability, and edge-case failures
- Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
- Work with large code repositories and multi-language environments
- Collaborate on systems that improve how AI models understand and generate software
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to the design of evaluation frameworks that set industry standards
Who You Are
- 4+ years of professional software engineering experience — this is non-negotiable
- Experience working at a high-growth tech company or top-tier software organization
- Expert proficiency in Python — you write clean, performant, well-tested Python code
- Hands-on experience with code repositories and working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and data pipelines
- Track record of working in high-performance engineering environments with large-scale products or platforms
- Strong command of version control systems (Git) and modern development workflows
- Bilingual or native English speaker with strong written communication skills
- Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
Candidates with these additional qualifications have the highest chance of success:
- Senior or Lead-level engineering profiles with a history of technical ownership
- Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience
- Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
- Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
- Background in security engineering or significant open-source contributions
- Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join
- Work on cutting-edge AI evaluation projects alongside world-class research teams
- Fully remote — work from anywhere with a reliable internet connection
- Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved
- Freelance autonomy with meaningful, high-stakes engineering work
- Collaborate with a global community of elite engineers and researchers
- Potential for contract extension and ongoing engagement as new evaluation challenges emerge
How to Pass the AI Interview
Interviews for roles like these are commonly run by an AI interviewer, so you will likely be talking to software rather than a person. An AI interviewer does not react the way a human does: it takes your answers literally, gives few cues, and follows up on anything vague. Clear, specific, well-paced answers win. Name the concrete thing you built, the numbers, the trade-off you made and what you would do differently.
For this role, expect questions about designing benchmarks and data pipelines, testing Python code, and judging whether AI-generated code is actually correct, including edge cases and failure modes.
Read the invitation instructions carefully and follow them during the interview: the format, timing, tools you may use and any rules on what is allowed.
The best preparation is rehearsing out loud before the real thing. InterviewAce, our interview-prep product, runs mock interviews with an AI interviewer, lets you practise follow-up questions about your own projects and gives you feedback before the real one.
InterviewAce guides to read first
- What to expect from an AI interviewer How an AI interviewer asks questions, follows up and scores answers.
- Technical interview prep for software engineers A practical plan for coding, system design and technical questions.
- Using AI tools to practise job interviews How to rehearse with AI mock interviews and get useful feedback.
- Remote interview tips for video calls Camera, audio, lighting and pacing for interviews over video.
- Top 5 mistakes in technical interviews The most common slip-ups and how to avoid them.
- How large language models work Tokens, attention and how models like ChatGPT work, useful for AI roles.
- Behavioral questions and the STAR method Structure your stories so they land with any interviewer.
Free tools for your application
Polish your application with these free tools from InterviewAce.
- All free tools
- Resume builder Build a clean, ATS-friendly resume.
- Cover letter generator Draft a tailored cover letter in minutes.
- Salary calculator Check what your skills are worth before you negotiate.
From Autonix Lab
- InterviewAce by Autonix Lab Our interview-prep product, plus where it is featured.
- What Are AI Agents? A plain-English primer on AI agents.
- AI Agent Loops Explained How autonomous, goal-driven coding loops work.
- Multi-Agent Systems Agents that work as a team.
- Open-Source AI Agent Projects Tools worth knowing for agentic work.
This is general preparation. We do not run these interviews and cannot say exactly how they will be conducted.
Sponsored listing: posted by a third-party hiring platform. We may earn a referral reward if you apply through our link.