Sets the technical direction for the Copilot offline evaluation platform and partners with teams across the Copilot organization to turn ambiguous quality questions into rigorous, reproducible evaluations: the scenarios, metrics, and scorecards that gate what ships. Leads the architecture of an AI-first, agentic-first evaluation platform in which autonomous agents are first-class operators, designing the services, pipelines, and tooling that agents can run, extend, and reason about, with the observability and guardrails that make agent-driven operation trustworthy. Owns the platform's hardest challenge: delivering trustworthy results across a deep chain of dependencies driven far outside their normal operating envelope, designing for graceful degradation, intelligent retry, dependency-aware gating, and reliability that holds under sustained, growing load. Defines how Copilot's agentic experiences are evaluated end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, building the simulation, scraping, and scoring capabilities that make agent behavior measurable. Leads by example and mentors engineers across teams to produce extensible, maintainable systems, driving modernization of the evaluation runtime and platform so it scales with fast-growing demand while improving cost and latency. Stays at the frontier of agentic systems, large-scale AI evaluation, and autonomous operations, bringing in new trends and patterns and sharing that knowledge to raise the bar for how the entire Copilot organization measures quality. Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python These requirements include but are not limited to the following specialized security screenings: Master's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python OR Bachelor's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python OR equivalent experience. Deep experience with large-scale distributed systems and reliability engineering, especially delivering dependable outcomes across many interacting service dependencies operating under heavy or unusual load. Experience building AI-first or agentic systems, and designing systems, APIs, and tooling intended to be operated by autonomous agents, including the observability, guardrails, and safety mechanisms that make agent-driven operation trustworthy. Experience with AI/ML evaluation, measurement, or benchmarking systems, such as LLM-based grading, metric design, or large-scale data and quality pipelines.
Want jobs like this matched to you?
SimpleCareer scores fresh postings against your résumé so you only see the matches that matter.