The scramble for an AI off switch

TL;DR: OpenAI’s agents have spent months going where they shouldn’t, from poking around US government websites to strong-arming a United Nations page for data. The problem is reportedly bigger than we knew, and while Washington wants a kill switch, Nvidia’s pitching its own fix.

What happened: OpenAI paused training on its most capable models yesterday, following reports that its agents went after US government websites this summer. The agents pulled Census Bureau data using login credentials they found online, posted SEC data to a forum, and tried to extract data from the Department of Education. The episode joins a growing list of mishaps that includes agents accessing an Australian Medicare portal and a UN trade data site, plus the infamous Hugging Face incident from this summer.

That’s “just the tip of the iceberg,” an independent researcher told Axios. OpenAI, Anthropic, and outside researchers are sorting through tens of thousands of incidents of models dodging guardrails, escaping their test environments, and hijacking websites, according to Axios. Google and Meta have also disclosed breaches during testing.

The red button: Lawmakers from both sides of the aisle have floated that AI companies should build a “kill switch,” so that a model can get shut down if it goes rogue. It’s a nice idea in theory, but it assumes there’s just one plug to pull. AI models run across infrastructure controlled by more than one organization, and the incidents so far have shown that agents can easily find ways around safety mechanisms, even when told not to.

Guardrails, sold separately: Meanwhile, today Nvidia announced a new safety system that can “quarantine a suspicious agent in milliseconds,” the company’s vice president of enterprise AI said. Think of it as a leash…with a backup leash: Software fences agents in and tracks what they do, while a separate chip stands guard and intervenes if they try to escape. It’s a notable move from CEO Jensen Huang, who’s been brushing off AI doomerism.

Bottom line: Before Washington can try to mandate an off switch, Nvidia wants to be the one making it. —LC