About this session

I spent 500B tokens structuring & connecting the entire corpus of biopharma drug data for systems in use by 19/20 top pharmas. These systems perform reliably, without catastrophic errors, on PhD tasks at scale, in a rapidly evolving domain. Past a certain point, the shape of production error rates mattes much more than overall accuracy. For example: - A false positive due to name collisions in biology? For our users, this is a forgivable mistake, the kind a human would make, barely a second thought. - A false negative without near force majeure? Years of broken trust. Understanding what error shape delivers superhuman value requires product, domain expertise, and customer feedback. I'll review case examples from our experience, and highlight non-obvious wins - the cross-org meeting structure, taxonomy of errors, and org-wide eval management/triage strategies we used to know what and when to ship.

Topics

Speaker

Matthew Jewkes

Standard Cybernetics