Final month, I reported on the widening internet of connections between the efficient altruism (EA) motion and AI safety coverage circles — from prime AI startups like Anthropic to DC suppose tanks like RAND Company. These are linking EA, with its laser-focus on stopping what its adherents say are catastrophic dangers to humanity from future AGI, to a large swath of DC suppose tanks, authorities businesses and congressional workers.
Critics of the EA deal with this existential threat, or ‘x-risk,’ say it’s taking place to the detriment of a vital deal with present, measurable AI dangers — together with bias, misinformation, high-risk functions and conventional cybersecurity.
Since then, I’ve been interested in what different AI and coverage leaders exterior the efficient altruism motion — however who’re additionally not aligned with the polar reverse perception system, efficient accelerationism (e/acc) — actually take into consideration this. Do different LLM corporations really feel equally involved in regards to the threat of LLM mannequin weights entering into the unsuitable arms, for instance? Do DC coverage makers and watchers absolutely perceive EA affect on AI safety efforts?
At a second when Anthropic, well-known for its big selection of EA ties, is publishing new analysis about “sleeper agent” AI fashions that dupe security checks meant to catch dangerous conduct, and even Congress has expressed considerations a few potential AI analysis partnership between the Nationwide Institute of Requirements and Security (NIST) and RAND, this appears to me to be an vital query.
As well as, EA made worldwide headlines most not too long ago in reference to the firing of OpenAI CEO Sam Altman, as its non-employee nonprofit board members all had EA connections.
What I found in my newest interviews is an attention-grabbing mixture of deep concern about EA’s billionaire-funded ideological bent and its rising attain and affect over the AI safety debate in Washington DC, in addition to an acknowledgement by some that AI dangers that transcend the short-term are an vital a part of the DC coverage dialogue.
The EA motion, which started as an effort to ‘do good better,’ is now heavily-funded by tech billionaires who think about stopping an AI-related disaster its primary precedence, significantly via funding AI safety (which can also be described as AI ‘safety’) efforts — particularly within the biosecurity house.
In my December piece, I detailed the considerations of Anthropic CISO Jason Clinton and two researchers from RAND Company in regards to the safety of LLM mannequin weights within the face of threats from opportunistic criminals, terrorist teams or highly-resourced nation-state operations.
Clinton informed me that securing the mannequin weights for Claude, Anthropic’s LLM, is his primary precedence. The specter of opportunistic criminals, terrorist teams or highly-resourced nation-state operations accessing the weights of probably the most subtle and highly effective LLMs is alarming, he defined, as a result of “if an attacker got access to the entire file, that’s the entire neural network.”
RAND researcher Sella Nevo informed me that inside two years it was believable AI fashions may have vital nationwide safety significance, similar to the likelihood that malicious actors might misuse them for organic weapon improvement.
All three, I found, have shut ties to the EA group and the 2 corporations are additionally interconnected due to EA — for instance, Jason Matheny, RAND’s CEO, can also be a member of Anthropic’s Lengthy-Time period Profit Belief and has longtime ties to the EA motion.
My protection was prompted by Brendan Bordelon’s ongoing Politico reporting on this concern, together with a current article which quoted an nameless biosecurity researcher in Washington calling EA-linked funders “an epic infiltration” in coverage circles. As Washington grapples with the rise of AI, Bordelon wrote, “a small army of adherents to ‘effective altruism’ has descended on the nation’s capital and is dominating how the White House, Congress and think tanks approach the technology.”
Cohere pushes again on EA fears about LLM mannequin weights
First, I turned to Nick Frosst, co-founder of Cohere, an OpenAI and Anthropic competitor which focuses on growing LLMs for the enterprise, for his tackle these points. He informed me in a current interview that he doesn’t suppose massive language fashions pose an existential menace, and that whereas Cohere protects its mannequin weights, the corporate’s concern is the enterprise threat related to others having access to the weights, not an existential one.
“I do want to make the distinction…I’m talking about large language models,” he stated. “There’s lots of interesting things you could talk about that are philosophical, like I think one day we might have true artificial general intelligence. I don’t think it’s happening soon.”
Cohere has additionally criticized the efficient altruism motion prior to now. For instance, CEO Aidan Gomez reportedly criticized the “self righteousness” of the efficient altruism motion and people overly involved with the specter of an AI doomsday in a letter to his workers.
Frosst stated that EA “doesn’t seem to exist much beyond its AI focus these days” and pushed again on their perception system. “If you find yourself in a philosophical worldview that ultimately provides moral justification, indeed, moral righteousness, for the massive accumulation of personal wealth, you should probably question that worldview,” he stated.
A giant flaw in efficient altruism, he continued, is to “assume that you can look at the good you’re doing and assign a number and know exactly how effective this is. It ends up in weird places like, hey, we should make as much money as possible. And we should put it all [towards combating] the existential risk of AI.”
AI21 Labs co-founder says mannequin weights usually are not ‘key enabler’ of dangerous actors
In the meantime, Yoav Shoham, co-founder of one other Anthropic and OpenAI competitor, the Tel Aviv-based AI21 labs, additionally stated his firm has saved its mannequin weights secret for trade-secret causes.
“We are very sensitive to potential abuse of technology,” he stated. “That said, we tend to think that model weights aren’t necessarily the key enabler of bad actors.”
He identified that in an period of a geopolitical AI race, “only certain aspects can be dealt with via policy.” As an alternative, he defined, “we’re doing our bit with strict terms of use, focus on task-specific models which by their very nature are less prone to abuse, and close collaboration with our enterprise customers, who share our commitment to beneficial uses of AI.”
Shoham emphasised that he and AI21 usually are not members of the EA motion. “As outsiders, we see there is a combination of thoughtful attention to responsible use of AI, [along] with less grounded fear-mongering.”
RAND researcher says EA beliefs ‘not particularly helpful’
Whereas RAND Company has been within the crosshairs of criticism over its EA connections, there are additionally researchers at RAND pushing again.
Marek Posard, a RAND researcher and army sociologist, spoke out final month on the RAND weblog about how AI philosophical debates like efficient altruism and e/acc are a ‘distraction’ for AI coverage.
“This is a new technology and so there’s a lot of unknowns,” he informed me in a current interview. “There’s a lot of hype. There’s a lot of bullshit, I would argue there’s a lot of real, very real concerns in flux. There’s all of these beliefs and ideologies, philosophies, theories that are floating around, I think, essentially people are latching on to at all.”
However neither EA or e/acc are “particularly helpful,” he added. “They’re also assumptions of what a small group thinks the world is. The reality is we know there are very real problems today.”
Nonetheless, Posard didn’t say that EA voices weren’t valued at RAND. In actual fact, he maintained that RAND promotes range of thought, which he stated is the “secret sauce” of the nonprofit world coverage suppose tank.
“It’s about diversity of thought, of people’s backgrounds, disciplines and experiences,” he stated. “I invite anyone to try to push an ideological agenda — because it is not set up to do that.”
Conventional cybersecurity is concentrated on present-day dangers
Whereas many (together with myself) might conflate AI safety and conventional cybersecurity — and their strategies do overlap, as RAND’s current report on securing LLM mannequin weights makes clear — I ponder whether the normal cybersecurity group is absolutely conscious of the EA phenomenon and its influence on AI safety coverage, particularly because the trade tends to deal with present-day dangers versus existential ones.
For instance, I spoke to Dan deBeaubien, who leads AI analysis and chairs each the AI coverage and product working teams on the SANS Institute, a Rockville, MD-based firm specializing in cybersecurity coaching and certification. Whereas he knew of the EA motion and stated that “it’s definitely a force that’s out there,” deBeaubien didn’t appear to be absolutely conscious of the extent of efficient altruism’s deal with the existential catastrophic dangers of AI — and noticed it extra as an moral AI group.
“We don’t have a lot of effective altruism conversations per se,” he stated, stating that he was extra involved about understanding the present safety dangers associated to folks’s utilization of LLM chatbots inside organizations. “Do I lie awake worrying that somebody is going to pull a lever and AI is going to take over — I guess I don’t really think much about that.”
Some specialists appear to be coexisting with EA considerations
Different DC-focused coverage specialists, nevertheless, appeared effectively conscious of the EA affect on AI safety, however appeared centered on coexisting with the motion fairly than talking out strongly on the file.
For instance, I spoke to Mark Beall, former head of AI coverage on the U.S. Division of Protection, who’s now the co-founder and CEO at Gladstone AI, which presents AI schooling and AI take a look at and analysis options to authorities and trade entities. He emphasised that Gladstone has not accepted any enterprise capital or philanthropic funding.
Beall stated that the dangers of AI are clear — so the traditional tech strategy of ‘move fast and break things’ is reckless. As an alternative, DC requires widespread sense safeguards, pushed by technical realities, that bridge the policy-tech divide, he defined.
“I helped set up the Joint AI Center at the Pentagon, and the fact is, many of those charged with safeguarding American interests have been working on AI long before self-promoted ‘effective altruists’ stumbled into Washington policymaking circles,” he stated. “At DoD, we established responsible AI policy and invested heavily in AI safety. Our mission has always been to accelerate responsibly. And for those on the fringes who think that US officials haven’t been independently tracking AI risks — or that they are somehow being duped — are wrong.”
‘Ungoverned AI’ was named a prime geopolitical threat
I additionally reached out to Ian Bremmer, president and founding father of Eurasia Group, which final week printed its listing of the highest geopolitical dangers of 2024 — with ‘ungoverned AI’ within the quantity 4 spot.
Bremmer centered squarely on present-day dangers like election disinformation: “GPT-5 is going to come out ahead of the US elections, and “will be so powerful it will make GPT-4 look like a toy in comparison,” he predicted. “Not even its creators truly understand its full potential or capabilities.”
That stated, he maintained there’s a “legitimate debate” in regards to the worth of open vs closed supply, and the significance of securing mannequin weights. “I think it would be wrong to assume, as many do, that the push to secure model weights is motivated purely by cynical business calculations,” he stated.
Nevertheless, if efficient altruism’s focus is actually altruism, Bremmer added that “we need to make sure that AI isn’t aligning with business models that undermine civil society — that means testing models not just for misuse but also to see how normal expected use impacts social behavior (and the development of children—a particular concern).” Bremmer added that he has “seen very little of that from the EA movement to date.”
The issue with EA, he concluded, is that “when you start talking about the end of the world as a realistic possibility—logically every other kind of risk pales into insignificance.”
VentureBeat’s mission is to be a digital city sq. for technical decision-makers to realize data about transformative enterprise know-how and transact. Uncover our Briefings.