AI Evaluation Engineer (Agentic Coding / Software Engineering)
About Us
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
Role overview
We are looking for an AI Evaluation Engineer specialized in software engineering workflows to evaluate and improve datasets used for agentic coding models.
In this role, you will work on realistic coding tasks — reviewing model trajectories, validating outputs, and producing high-quality evaluations. This is a hands-on engineering role, requiring strong debugging skills, attention to detail, and the ability to assess correctness in real code scenarios.
Commitments Required: 8 hours per day with an overlap of 4 hours with PST.
Employment type: Contractor assignment (no medical/paid leave)
Duration of contract: 5 weeks+
Location: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria,Turkey, Vietnam
Interview: take home assessment (60min)
Responsibilities
- Execute coding tasks within agentic coding environments, maintaining strict evaluation protocols
- Review and evaluate model-generated code trajectories for correctness and completeness
- Validate outputs by reading code, running tests, analyzing logs, and inspecting artifacts
- Perform targeted validation using scripts, tests, and manual checks
- Write clear, evidence-based rationales for evaluations and rankings
- Design realistic, multi-step coding tasks and workflows (offline work)
- Create and refine evaluation rubrics and scoring criteria
- Ensure consistency, quality, and compliance across evaluations
- Identify issues in environments, instructions, or workflows and report with clear evidence
Requirements
- 5+ years of experience in software engineering, QA, developer tooling, or similar code-heavy roles
- Strong proficiency in at least one programming ecosystem (e.g., Python, JavaScript/TypeScript, Java, C/C++, Rust, SQL)
- Ability to read and understand unfamiliar codebases and implement/debug changes
- Experience running and interpreting tests, scripts, and CLI tools
- Strong debugging and problem-solving skills, including handling edge cases
- Comfortable working in Linux/terminal environments
- Familiarity with Git workflows and standard development tooling
- Experience with AI coding tools or agentic coding environments (e.g., Cursor, Claude Code, or similar)
- Strong attention to detail and ability to produce consistent, high-quality evaluations