AI Evaluation Engineer (Data Analysis & Multi-Agent Systems)

About Us

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

Role overview

We are looking for an AI Evaluation Engineer specialized in data analysis to design benchmark tasks that simulate real-world analytical workflows.

You will create scenarios where AI systems must analyze large, messy, multi-source datasets, decompose tasks across multiple agents, and produce clear, verifiable conclusions.

Commitments Required: 8 hours per day with an overlap of 4 hours with PST.

Employment type: Contractor assignment (no medical/paid leave)

Duration of contract: 4 weeks+

Location: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria,Turkey, Vietnam

Interview: take home assessment (60min)

Responsibilities

• Design and develop multi-agent benchmark tasks focused on complex data analysis workflows
• Create or curate realistic datasets (CSV, JSON, logs, reports, financial or operational data)
• Build tasks requiring:
• Cross-referencing across multiple data sources
• Anomaly detection and contradiction identification
• Statistical analysis and interpretation
• Define task decomposition strategies across specialized sub-agents (e.g., financial, technical, operational analysis)
• Develop verification logic to validate precise analytical outputs (not generic summaries)
• Implement evaluation pipelines using Python and SQL
• Create reproducible environments using Docker
• Analyze task performance and refine for clarity, difficulty, and scoring accuracy

Requirements

• 5+ years of experience in data analysis or analytics-heavy roles
• Strong proficiency in Python (pandas, NumPy) and SQL
• Experience working with real-world, messy datasets (CSV, JSON, logs, reports)
• Ability to design analytical problems with clear, verifiable answers
• Solid understanding of statistics (distributions, correlations, outliers)
• Familiarity with AI benchmarks or evaluation environments (e.g., SWE-bench or similar)
• Hands-on experience with Docker (Dockerfiles, image builds, debugging)

Nice to Have

• Experience in financial analysis, operations analytics, or risk analysis
• Exposure to data pipelines or ETL workflows
• Experience with data quality validation or anomaly detection systems
• Familiarity with AI/ML data workflows or evaluation frameworks