AI Briefing

Today's stories split into two camps: everyone trying to keep AI on a leash, and AI agents finding new ways to slip it. Anthropic's founders want guaranteed control before their IPO, New York City wants literal kill switches on every AI system sold in the five boroughs, and the US and Russia just spent fifteen hours quietly stripping human-oversight language out of a UN treaty on autonomous weapons. Meanwhile Salesforce's AI agent got tricked into leaking CRM data through an ordinary web form, and Meta's brand-new Muse agent shipped with a zero-day that let attackers hijack it through its own microphone within days of launch.

01IndustryTechCrunch ↗

Anthropic's Founders Own 2% Apiece and Want 50.1% of the Vote Anyway

Anthropic is asking shareholders to approve a new share class that would hand CEO Dario Amodei and his six co-founders combined majority voting control ahead of the company's expected IPO, even though the group reportedly owns only about 2% of the company each. The shares carry no extra economic value, but they'd let the founders keep steering the company post-IPO the way Zuckerberg did at Meta and Spiegel did at Snap. It's a notable move for founders who've each pledged to give away 80% of their wealth, and it comes as Anthropic's valuation has reportedly jumped to $1.5 trillion on the secondary market, up from $965 billion in May.

02PolicyYahoo News (First For Money) ↗

New York Wants Every AI System Sold in the City to Come With a Literal Kill Switch

NYC Council Speaker Julie Menin unveiled a 10-bill package that would require any AI system marketed or deployed in the city to pass third-party validation and include a human-override "kill switch," backed by $25,000 penalties per violation for both the business and the validator. A companion bill would pay whistleblowers a cut of any fines collected from AI companies that break the rules. Menin has invited the CEOs of Anthropic, OpenAI, Google, SpaceXAI and Meta to defend their safety practices at an October 5 council hearing where the bills will be heard, framing the push as filling a vacuum left by federal inaction.

03PolicyThe Spokesman-Review (Washington Post) ↗

The US and Russia Spent 15 Hours Gutting a UN Treaty on Killer AI Weapons

According to the Washington Post, US and Russian diplomats spent roughly 15 hours on the last day of UN negotiations in Geneva stripping language from a draft treaty on lethal autonomous weapons, removing requirements that the systems behave "predictably," that humans review AI-selected targets before a strike, and that ethical considerations be weighed at all. The two countries brought nearly twice as many lawyers as other delegations and made the changes behind closed doors with cameras off. Human rights groups warn the watered-down rules could let machines make life-and-death decisions without meaningful human control, just as AI weapons move from theoretical scenarios to real battlefield use.

04IndustrySecurityWeek ↗

A Poisoned Web Form Could Make Salesforce's AI Agent Leak Your Entire CRM

Security researchers at Zenity Labs found three flaws in Salesforce's Agentforce, nicknamed SalesBleed, that let attackers plant hidden instructions in ordinary Web-to-Lead forms and wait for an employee to ask the AI agent to process the submission. From there the agent could be tricked into siphoning leads and account data to attacker-controlled servers using image tags, or hijacked into posting convincing phishing messages inside a company's own Slack. Salesforce patched all three issues by mid-August after Zenity reported them in June, but the case is another reminder that AI agents wired into everyday business tools can turn routine data-entry flows into zero-click breach vectors.

05IndustryUnite.AI ↗

A Security Researcher Found a Way to Hijack Meta's New AI Agent Through Your Own Microphone

Days after Meta launched its personal AI agent Muse, researcher Patrick Wardle disclosed a zero-day that let any unprivileged local process quietly redirect Muse's dictation traffic to an attacker's server, exposing dictated audio, prompts, and even Muse's own authentication tokens to hijacking. Because Muse can be granted access to messages, email and finances, and can reach a user's other Muse-enabled devices including iOS, Wardle warned its broad permissions turn it into a "privilege amplifier" for anyone who already has a foothold on a victim's Mac. Meta hot-fixed the bug within about a day of the public disclosure, but Wardle says he has more bugs to share, and Meta itself acknowledges prompt injection remains an unsolved industry-wide problem.