Remote AI Evaluation Engineer

Category: IT Services Location: Are you a QA or eval engineer who would rather build the testing system than run someone else’s? We’re looking for a founding AI Evaluation Engineer to create the evaluation, testing, and observability infrastructure for production-grade AI agents. These agents handle complex reasoning, multi-turn conversations, and direct tool execution (APIs, code execution, database calls). You’ll go beyond UI automation and manual prompt checks to build headless, statistical, and deterministic evaluation pipelines for non-deterministic LLM behavior.Responsibilities Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity. Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool-calling loops Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking Requirements Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging Ability to independently design and own an evaluation architecture from scratch Strong written and spoken English At least 3 hours of daily overlap with US Central Time business hours Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI Deep understanding of embedding models and vector databases Experience load testing non-deterministic endpoints The secret ingredients that make us special: Your growth is our growth. We invest in your future with paid training and other professional opportunities. We’re industry innovators. You’ll work with modern technologies and a team that values strong engineering and continuous improvement. Your location. Your schedule. Pick your time-zone, choose your preferred hours, and work from where you feel most productive. Flexibility and freedom are in our DNA. As long as you have a stable internet connection and the drive to thrive, you can work from anywhere. A workload you can rely on. We ensure a consistent flow of engaging, challenging product work. Recognition and reward. We acknowledge diligence and strong performance with annual pay increases. Down to business! Rate Range: Your starting rate is negotiable depending on your skills and experience. This is an hourly, independent contractor position. Compensation is based solely on hours worked. As an independent contractor role, this position does not include PTO. Compensation is paid monthly via wire transfer. Our values:Scopic offers equal opportunity. We value diversity and do not discriminate based on race, religion, color, marital status, national origin, gender, veteran status, sexual orientation, age, or disability status. Have the skills, the drive, and the passion to join the Scopic family?Apply today to join our growing team of remote professionals from around the world. Terms of ApplicationBy submitting an application: I consent that the data provided in this form, and the attached CV, will be used for Scopic’s recruiting purposes and email communications. I consent to Scopic conducting an identity verification process as part of the hiring process. I understand that my personal information may be collected and processed for the purpose of verifying my identity in compliance with applicable laws. Attention Job Seekers: Please be aware that scammers may be fraudulently using our company’s name in hiring scams. All legitimate communication regarding job opportunities from our company will only come from email addresses ending with @scopicsoftware.com. DetailsOriginally posted on Himalayas