Job Summary
The ERDM Sr. AI Operations Analyst is responsible for ensuring that Sales & Finance AI solutions operate with the highest levels of trust, accuracy, control, and compliance.
We are seeking a detail-obsessed, data driven expert to serve as the guardian of accuracy for our Sales & Finance AI chat assistants. You will design validation frameworks, partner with data engineering teams, and ensure every metric the AI surfaces - especially Bookings and Revenue - is accurate, consistent, and trustworthy. You will lead incident response on any data accuracy or hallucination even
The role also serves as the bridge between Finance users and technical development teams. This role is responsible for translating business requirements into AI-enabled solutions and ensuring that Finance AI capabilities align with business goals and leadership needs.
Core Responsibilities:
Requirements & Solution Definition
- Gather and document business requirements from Finance users and stakeholders through interviews, workshops, observation, and feedback analysis
- Write clear, detailed user stories with well-defined acceptance criteria that IT can act on without ambiguity
- Ensure requirements reflect metric integrity standards and executive-readiness expectations
AI Quality Assurance
- Develop and maintain AI-specific testing frameworks - define test scenarios, expected outputs, edge cases, and pass/fail criteria tailored to conversational AI behavior
- Establish and document quality standards for AI-generated responses - accuracy, completeness, formatting, tone, and contextual relevance
- Validate AI-generated responses against Snowflake source data using SQL queries to confirm metric accuracy and consistency
- Conduct thorough UAT, regression testing, and release testing for every new feature, enhancement, prompt update, data refresh, or model change before go-live
- Confirm metric integrity - verify that Bookings and Revenue figures surfaced by the AI match official Finance reporting
- Coordinate business sign-off activities - manage the approval workflow with Finance stakeholders before releases are promoted to production
Hallucination Prevention
- Design and execute validation test strategies specifically targeting hallucination risks - test for fabricated data, incorrect metric associations, misleading trends, and unsupported narratives
- Identify and catalog high-risk Finance use cases where hallucination potential is elevated (e.g., multi-step calculations, year-over-year comparisons, segment-level breakdowns, forecast vs. actuals)
- Monitor response quality on an ongoing basis - conduct periodic spot-checks and audits of AI outputs across different query types and user scenarios
- Escalate and resolve quality concerns immediately - flag suspected hallucinations to the Data Accuracy Specialist (IC4) and team lead; coordinate with IT/Engineering to implement fixes and guardrails
Core Responsibilities
Executive Use Cases
- Support in designing and testing executive-focused use cases - ensure AI responses meet the precision, tone, and contextual depth expected by C-suite and SVP users
- Test and validate executive prompt frameworks - verify that curated queries, guided flows, and personalized dashboards produce accurate, polished outputs
- Contribute to improving usability and decision-support capabilities - identify opportunities to make the executive experience more intuitive, actionable, and insightful
Adoption & Change Management
- Develop and execute adoption strategies - create structured plans to drive Finance user engagement, from initial awareness through active daily use
- Lead user onboarding - design and deliver onboarding experiences for new users, including guided walkthroughs, sandbox environments, and office hours
- Deliver training programs - conduct live demos, workshops, and drop-in sessions tailored to different user groups (analysts, managers, directors)
- Manage communications - draft and distribute release notes, feature announcements, tips & tricks, known issue updates, and adoption progress reports to Finance stakeholders
- Collect and synthesize user feedback through surveys, interviews, and usage analytics to continuously improve the experience
Required Qualifications:
- Bachelor's degree Required
- 7+ years in Business Analysis / Finance Analytics / FP&A Operations
- Experience with AI-powered tools, chatbots, or conversational AI interfaces
- Understanding of Finance metrics and reporting (Bookings, Revenue, or related areas)
- Working knowledge of SQL and Snowflake
- Understanding of AI/LLM fundamentals - how large language models retrieve, process, and present information; awareness of common failure modes (hallucination, data staleness, context window limitations)
- Prompt Engineering Certificate or related experience
Preferred Qualifications:
- Familiarity with Abacus or similar platforms
- Hands-on experience with prompt engineering — designing, testing, and refining prompts to improve AI response quality and accuracy
- Experience creating training materials, user guides, or knowledge base content
- Experience with AI testing frameworks or quality assurance for data-driven applications
- Familiarity with AI governance or responsible AI principles