Senior AI Agent & Evaluations Engineer

**Join Vacatia and Help Build the Future of AI\-Powered Vacation Ownership** **Location:** Portland, OR (Hybrid – Three Days In Office) Remote considered for exceptional candidates. **About Vacatia** Vacatia is building the future of vacation ownership. We operate in a fragmented, operationally complex industry where AI has the potential to fundamentally transform how decisions are made, how customers are supported, and how businesses scale. We're developing AI agents that sit at the center of critical business workflows—helping owners, supporting operations, surfacing insights, and automating decisions that historically required significant human effort. These agents interact with real customers and influence real business outcomes, making reliability, safety, and performance essential. We're looking for a hands\-on Senior AI Agent \& Evals Engineer to own the intelligence layer behind these systems. You'll be responsible for designing agent behavior, building evaluation frameworks, creating guardrails, and continuously improving agent performance as our AI footprint expands across the organization. If you're passionate about prompt engineering, agent reliability, and creating measurable AI systems that solve meaningful business problems, we'd love to meet you. **Why You'll Love Working at Vacatia** **Build the Future of Applied AI** Design and improve AI agents that directly impact customer experiences, operational efficiency, and business outcomes across our organization. **Work on Problems That Matter** Your work will influence real\-world decisions involving customer communications, mortgage outcomes, rental operations, and owner experiences. **Own the Intelligence Layer** Take full ownership of prompt design, agent behavior, evaluation systems, guardrails, and continuous performance improvement. **Measure What Matters** Build sophisticated evaluation frameworks, golden datasets, and automated scoring systems that ensure our agents continually improve. **Partner Across the Business** Collaborate closely with engineers, operators, and subject matter experts to transform business knowledge into scalable AI systems. **Join a Small Team with Outsized Impact** Work alongside experienced engineers and leaders who believe AI can create meaningful competitive advantages in a traditionally underserved industry. **Your Impact** * Design, refine, and optimize prompts, tool definitions, routing logic, and decision\-making behavior across Vacatia's AI agent ecosystem * Build and maintain evaluation frameworks, golden datasets, grading systems, and regression testing pipelines that measure agent quality and reliability * Develop guardrails and safe\-failure mechanisms that ensure agents operate responsibly in customer\-facing and financially sensitive workflows * Monitor production performance, investigate failures, identify edge cases, and continuously improve agent outcomes through data\-driven iterati