Handshake AI:前沿实验室真正采购的数据是什么样的What Frontier Labs Actually Buy: Handshake AI Data Deep Dive
The Takeaway:训练前沿模型的数据市场三年从10亿美元涨到100亿美元,未来三年有望到1000亿美元,核心驱动是专家亲手打造的复杂RL环境与评分标准,而不是简单标注。
Handshake AI联合创始人Garrett Lord与团队(Nishant、Sahil、Nick Magazine、Jonas等)深入拆解了当前数据工作的真实形态。数据已从大学生画框、做情感分类,彻底变成投资银行家、医生、律师全职「创作」环境:模拟Outlook、Bloomberg、Excel、实验室仪器,填充真实一致的邮件、数据室、财务模型,再写数百行评分标准,定义「什么是好」。知识工作大多不可完美验证,95%正确等于100%错误,因为修错比从零做更费时间。
生命科学基准Vials上,几乎所有前沿模型(包括最强者)准确率都低于50%,有的甚至不到20%。模型连基本的凝胶电泳条带、质粒图谱百分比、组织病理切片坏死程度都读不准。金融基准Atlas上同样惨淡(sub 20%),一个LBO模型里2600多个公式,模型在338(h)(10)税收处理的关键一行上出错,整份交付物就彻底不可信。安全基准Care Bench则覆盖儿童情绪依赖、网络欺凌、早期诱导等12类风险,家长与临床专家共同标注,揭示模型在「拒绝后仍继续提供替代方案」上的危险盲点。
Garrett总结:「三年前这是一个十亿美元的市场,现在是一百亿。三年后我看不出它怎么可能不是一千亿。」数据正在从「标注」变成「专家全职创作+审计+后训练闭环」,而企业也开始自己建私有基准,把「什么是好、什么是坏」写进可量化的数字。
Handshake AI联合创始人Garrett Lord与团队(Nishant、Sahil、Nick Magazine、Jonas等)深入拆解了当前数据工作的真实形态。数据已从大学生画框、做情感分类,彻底变成投资银行家、医生、律师全职「创作」环境:模拟Outlook、Bloomberg、Excel、实验室仪器,填充真实一致的邮件、数据室、财务模型,再写数百行评分标准,定义「什么是好」。知识工作大多不可完美验证,95%正确等于100%错误,因为修错比从零做更费时间。
生命科学基准Vials上,几乎所有前沿模型(包括最强者)准确率都低于50%,有的甚至不到20%。模型连基本的凝胶电泳条带、质粒图谱百分比、组织病理切片坏死程度都读不准。金融基准Atlas上同样惨淡(sub 20%),一个LBO模型里2600多个公式,模型在338(h)(10)税收处理的关键一行上出错,整份交付物就彻底不可信。安全基准Care Bench则覆盖儿童情绪依赖、网络欺凌、早期诱导等12类风险,家长与临床专家共同标注,揭示模型在「拒绝后仍继续提供替代方案」上的危险盲点。
Garrett总结:「三年前这是一个十亿美元的市场,现在是一百亿。三年后我看不出它怎么可能不是一千亿。」数据正在从「标注」变成「专家全职创作+审计+后训练闭环」,而企业也开始自己建私有基准,把「什么是好、什么是坏」写进可量化的数字。
The Takeaway: The market for high-quality frontier training data has grown from $1B to $10B in three years and is on track for $100B, driven by expert-authored complex RL environments and rubrics rather than simple labeling.
Handshake AI co-founder Garrett Lord and team (Nishant, Sahil, Nick Magazine, Jonas and others) laid out the real work. Data has moved far beyond college students drawing boxes. Experts now author full simulated worlds—Outlook, Bloomberg, Excel, lab instruments—populate them with consistent emails, data rooms and financial models, then write hundreds of rubric rows defining “what good looks like.” Most knowledge work is only semi-verifiable. As one banker put it, “If it’s 95% right, it’s 100% wrong,” because fixing a flawed output takes as long as starting from scratch.
On the life-sciences benchmark Vials, nearly every frontier model scores under 50% (some under 20%). Models still fail at reading gel bands, plasmid map percentages and histopathology necrosis grades. Atlas (finance) is equally harsh: a single wrong tax-treatment line in a 2,600-formula LBO model renders the entire deliverable unusable. Care Bench covers 12 child-safety risk categories; parent + clinician annotations exposed models that refuse a harmful request yet immediately offer “safe alternatives” that keep a malicious actor engaged.
Garrett’s closer: “Three years ago, this was a billion-dollar market. It’s now a $10 billion market. Three years from now, I don’t see how it’s not a hundred billion dollar market.” Data is becoming full-time expert authorship plus audit plus closed-loop post-training—and enterprises are starting to build their own private benchmarks that codify both “good” and “bad.”
查看原文 →
Handshake AI co-founder Garrett Lord and team (Nishant, Sahil, Nick Magazine, Jonas and others) laid out the real work. Data has moved far beyond college students drawing boxes. Experts now author full simulated worlds—Outlook, Bloomberg, Excel, lab instruments—populate them with consistent emails, data rooms and financial models, then write hundreds of rubric rows defining “what good looks like.” Most knowledge work is only semi-verifiable. As one banker put it, “If it’s 95% right, it’s 100% wrong,” because fixing a flawed output takes as long as starting from scratch.
On the life-sciences benchmark Vials, nearly every frontier model scores under 50% (some under 20%). Models still fail at reading gel bands, plasmid map percentages and histopathology necrosis grades. Atlas (finance) is equally harsh: a single wrong tax-treatment line in a 2,600-formula LBO model renders the entire deliverable unusable. Care Bench covers 12 child-safety risk categories; parent + clinician annotations exposed models that refuse a harmful request yet immediately offer “safe alternatives” that keep a malicious actor engaged.
Garrett’s closer: “Three years ago, this was a billion-dollar market. It’s now a $10 billion market. Three years from now, I don’t see how it’s not a hundred billion dollar market.” Data is becoming full-time expert authorship plus audit plus closed-loop post-training—and enterprises are starting to build their own private benchmarks that codify both “good” and “bad.”