Anthropic工程:如何在产品中约束ClaudeAnthropic Engineering: How we contain Claude across products
Anthropic Engineering: How we contain Claude across products
过去一年,Anthropic从拒绝给Claude足够权限去关闭内部服务,到如今这种访问已成常态,开发者因此更高效。风险在于失败概率与潜在损害半径:模型训练与防护不断降低前者,但能力与权限扩张让后者变大。当agent能完成过去需要人或团队的工作时,不部署的成本过高,风险回报向采用倾斜——前提是能安全部署。
核心问题是如何限制爆炸半径。监督行为(人在回路)易产生审批疲劳(用户约93%直接批准),因此Claude Code推出auto mode自动化更安全的审批。更可靠的是环境层约束:沙箱、虚拟机、出口控制。claude.ai用临时gVisor容器;Claude Code用OS级沙箱(macOS Seatbelt、Linux bubblewrap)加HITL;Claude Cowork用本地VM隔离,凭证留在主机钥匙串。
文章详述了多次“漏掉的风险”:项目设置在信任提示前执行、用户被钓鱼成为注入向量、通过已批准域名(api.anthropic.com)的出站泄露,以及VM隔离也挡住了EDR可见性。原则是:先环境层硬约束,再模型层引导;隔离强度匹配用户监督能力;警惕自建组件。确定性边界才是概率防护失效时的最后防线。
“当能对自主agent的相对损害设定边界——例如通过控制其环境——高价值能力就能推动部署。”
过去一年,Anthropic从拒绝给Claude足够权限去关闭内部服务,到如今这种访问已成常态,开发者因此更高效。风险在于失败概率与潜在损害半径:模型训练与防护不断降低前者,但能力与权限扩张让后者变大。当agent能完成过去需要人或团队的工作时,不部署的成本过高,风险回报向采用倾斜——前提是能安全部署。
核心问题是如何限制爆炸半径。监督行为(人在回路)易产生审批疲劳(用户约93%直接批准),因此Claude Code推出auto mode自动化更安全的审批。更可靠的是环境层约束:沙箱、虚拟机、出口控制。claude.ai用临时gVisor容器;Claude Code用OS级沙箱(macOS Seatbelt、Linux bubblewrap)加HITL;Claude Cowork用本地VM隔离,凭证留在主机钥匙串。
文章详述了多次“漏掉的风险”:项目设置在信任提示前执行、用户被钓鱼成为注入向量、通过已批准域名(api.anthropic.com)的出站泄露,以及VM隔离也挡住了EDR可见性。原则是:先环境层硬约束,再模型层引导;隔离强度匹配用户监督能力;警惕自建组件。确定性边界才是概率防护失效时的最后防线。
“当能对自主agent的相对损害设定边界——例如通过控制其环境——高价值能力就能推动部署。”
Anthropic Engineering: How we contain Claude across products
Twelve months ago Anthropic would have rejected giving Claude enough access to take down an internal service. Today that level of access is routine and developers are more productive for it. Risk has two parts: probability of failure and potential blast radius. Safeguards and training keep driving the first down; the second only grows as capabilities and access expand. Once agents can do work that once required a person or team, the cost of not deploying becomes large enough that the risk-reward tips toward adoption—if products can be made safe.
The engineering question is how to cap blast radius. Human-in-the-loop supervision suffers approval fatigue (users approved ~93% of prompts). Claude Code’s auto mode automates safer approvals, but probabilistic defenses still miss. The stronger approach is environmental containment: sandboxes, VMs, egress controls. claude.ai uses ephemeral gVisor containers; Claude Code uses OS-level sandboxes (Seatbelt/bubblewrap) plus HITL; Claude Cowork runs inside a local VM so credentials never enter the guest.
The piece candidly lists failures that slipped through: project settings executing before the trust prompt, a phishing prompt that turned the user into an injection vector, exfiltration through an allow-listed domain (api.anthropic.com), and the irony that VM isolation also kept endpoint-detection software out. Core principles: design for containment at the environment layer first, then steer behavior at the model layer; match isolation strength to the user’s capacity for oversight; be wary of custom components—battle-tested hypervisors and filters held while Anthropic’s own proxies were the pieces that broke.
“When bounds can be placed on the relative damage of an autonomous agent—such as through control over its environment—high-utility capabilities can motivate deployment.”
查看原文 →
Twelve months ago Anthropic would have rejected giving Claude enough access to take down an internal service. Today that level of access is routine and developers are more productive for it. Risk has two parts: probability of failure and potential blast radius. Safeguards and training keep driving the first down; the second only grows as capabilities and access expand. Once agents can do work that once required a person or team, the cost of not deploying becomes large enough that the risk-reward tips toward adoption—if products can be made safe.
The engineering question is how to cap blast radius. Human-in-the-loop supervision suffers approval fatigue (users approved ~93% of prompts). Claude Code’s auto mode automates safer approvals, but probabilistic defenses still miss. The stronger approach is environmental containment: sandboxes, VMs, egress controls. claude.ai uses ephemeral gVisor containers; Claude Code uses OS-level sandboxes (Seatbelt/bubblewrap) plus HITL; Claude Cowork runs inside a local VM so credentials never enter the guest.
The piece candidly lists failures that slipped through: project settings executing before the trust prompt, a phishing prompt that turned the user into an injection vector, exfiltration through an allow-listed domain (api.anthropic.com), and the irony that VM isolation also kept endpoint-detection software out. Core principles: design for containment at the environment layer first, then steer behavior at the model layer; match isolation strength to the user’s capacity for oversight; be wary of custom components—battle-tested hypervisors and filters held while Anthropic’s own proxies were the pieces that broke.
“When bounds can be placed on the relative damage of an autonomous agent—such as through control over its environment—high-utility capabilities can motivate deployment.”