Product Manager, AI Evaluation Lead

Join the Team Shaping the Future of Industrial Technology! Ready to make a tangible impact on global industries using cutting-edge technology? Trimble is looking for a passionate problem-solver like you to help us transform the way the world works! At Document Crunch, the quality of our AI is the product — and we're looking for someone to own how we prove and improve it. ## ## About Us Trimble is a global technology company that connects the physical and digital worlds, transforming the ways work gets done. With relentless innovation in precise positioning, modeling and data analytics, Trimble enables essential industries including construction, geospatial and transportation. Whether it's helping customers build and maintain infrastructure, design and construct buildings, optimize global supply chains or map the world, Trimble is at the forefront, driving productivity and progress. ## ## About Document Crunch Document Crunch is a premier AI-powered contract intelligence platform for the construction industry. By simplifying complex construction documents, Document Crunch empowers project teams to understand their risks and manage contracts with confidence. As part of the Trimble ecosystem, this technology serves as "intelligent DNA," accelerating AI adoption to create a future of zero project disputes and increased transparency across the AECO industry. ## ## What Makes This Role Great As Document Crunch scales and agentic capabilities spread across the platform, the volume and variety of AI evaluation we need is growing far faster than a small team can supply. This role exists to close that gap by an order of magnitude — not by doing all the evaluation, but by building the system and the network that multiply our capacity. You will be the single accountable owner of our AI evaluation program: defining the methodology and tooling, standing up the data sets and ground truth we measure against, and building and motivating a flexible network of subject matter experts from across Document Crunch, the broader Trimble organization, and beyond. The coverage challenge is real: evaluation has to span an expanding set of construction workflows across project types, phases, and user personas. You won't need to be a construction expert yourself — you'll need to design a great evaluation process, build the systems and partnerships that scale it, and pull the right expertise in at the right time. Done well, this becomes a durable competitive advantage for Document Crunch — and a model for how evaluation could work across other Trimble product areas. ## ## Key Responsibilities * Own AI evaluation end to end as a dedicated program, with the mandate to create a defined, repeatable, and scalable system ("Eval Ops"). * Design the evaluation methodology — test sets and benchmarks spanning the dimensions that matter for our product (accuracy/factuality, instruction-following, robustness, multi-turn conversation behavior, and domain-specific edge cases), plus regression testing to catch when model or prompt changes break previously working behavior. * Build canonical evaluation data sets — reference projects spanning varied size, complexity, document volume, and project phase (preconstruction, execution, closeout), each with proportionate question/query banks. * Define rubrics and own ground truth — author clear, anchored scoring rubrics and calibration sets, and stand up the human-validation workflow that confirms whether answers are actually correct and keeps reviewers aligned. * Stand up a tiered, scalable eval pipeline — combine automated evaluation (e.g., LLM-as-a-judge) for fast iteration with expert human review for calibration and ground truth, so scarce SME time is spent where it matters most. * Design the eval workbench and contributor workflow — define the end-to-end tooling and process SMEs use to contribute (building on our existing internal eval tooling), so the system runs efficiently at scale. * Build...