12.11.2023
There have been a few breakthroughs in AI technology in the past few years. There are algorithms running on consumer hardware that can generate realistic images and fake videos. Large Language Models (LLMs) are becoming so good that they are indistinguishable from humans. The progress has been so fast and way above expectations for many experts in the field, that there are strong calls for regulating the research in AI or stopping it altogether. There are also calls on prohibiting open source projects related to the Large Language Models. The proponents of the regulation are concerned that someone will create a “super dangerous AI” that will kill or seriously harm humanity, while the opponents are worried that the ban will completely stop the progress in AI. I’d like to weigh in on potential dangers of AI for humanity and, on the other side, on the dangers of regulation for the progress of our civilization.
Many experts in the field of neural networks, such as Geoffrey Hinton call for AI regulation. Sam Altman, the CEO of OpenAI, also wants to impose safety rules on the AI community. In March 2023 the Future of Life institute published a letter that called for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT4”. The letter collected more than 30,000 signatures at the time of writing this text, including such prominent scientists and business leaders as Yoshua Bengio, Stuart Russel, Elon Musk and many others.
However, the AI community is not uniform in this question. One of the active critics of a potential AI regulation is Yann Lecun who calls it “a new wave of obscurantism”. Indeed, one can draw parallels with the calls for a ban on the printing press (concerns about an uncontrollable spread of ideas), cars (safety concerns), telegraph (“too fast for truth”), Large Hadron Collider (concerns about Earth destruction) and many, many other breakthrough inventions that impacted humanity in a very positive way. Opponents of the ban are rightfully worried that an AI regulation will stop the progress in AI altogether. Being a developer of the OpenCV and a part of the computer vision community for the past 25 years, I had a privilege to observe first-hand how the openness of data and code streamlined the development of the tech from which humanity now benefits.
And you don’t have to take my word for it. Look at what happened in other areas that are regulated. Nuclear fission is regulated because a power plant can blow up, but an unintended consequence of that is that we are still using the power plant designs from the 60s. The world starves for power, and the Earth is heating because of fossil fuels, but we do not have any innovation in the fission industry, because the threshold is so absurdly high (see, for example, this publication). Banks are heavily regulated, and the progress in fintech is painfully slow. Sending a wire transfer still costs a lot and takes anywhere from a few hours to a few days. And why would it be different? The trend is clear: the capital is increasingly concentrated in a few large banks, and that doesn’t push them to compete through innovation.
So,we have do admid that it is probable that the regulation will hurt progress, not just scientifically, but also in terms of real macroeconomy consequences. Which, ironically, carries the extinction risk for humanity: the less efficient is our (as a species) scientific progress, the higher the probability that we will be eliminated by an extinction event (global warming, pandemic, cosmic event, alien invasion – pick your favorite). Of course, the risk of that seems pretty small. But how large is the risk of developing the AI that threatens the existence of humanity? Let us take a closer look into it.
Obviously, if AI really has a substantial chance of killing humanity, it is better to err on the side of caution and impose the regulation. But do we really understand how AI will kill us? You would think that given the intensity of the discussion, there is a consensus on what specific risks the development of Large Language Models (LLMs) bears. Surprisingly, this is very far from the truth. Hinton talks about various risks, from bias and unemployment to existential risks for humanity. Bengio (here is a very comprehensive yet succinct summary of his position) thinks that given the continuing rapid progress in AI someone will create a “superdangerous AI” that either through misalignment (see The AI Alignment Problem) or intentionally creates an actor that poses an existential threat to humanity.
Yet, there is very little specifics on how exactly the “superdangerous AI” will destroy humanity. A much cited writer on the topic, Elizier Yudkovsky gives an example of an AI sending DNA in electronic form to a biolab that will produce protein on demand, resulting in artificial life or a deadly pandemic. If we want to eliminate this risk, it seems that regulating biolabs is a way more robust option than banning AI. When someone sells heroin on the Internet, we do not ban the Internet, we have police arrest the seller. On the same note, anyone worried about the bio risk should first advocate for banning the gain of function research that, as opposed to AI, is being a probable cause of the COVID-19 pandemic that has already killed millions of people. A lot of the GoF research is still done in BSL2 like the Wuhan lab.
This example shows that analyzing specific risks is very important for developing AI regulation. When I asked Roman Yampolsky, a prominent AI safety researcher, what are the specific scenarios where there is an existential threat to humanity, he refrained from spelling it out, implying that a “superdangerous AI” will be way smarter than humans, and we won’t know what kills us. An analogy that is often mentioned by the proponents of this idea is a 10 years old kid playing chess against a grandmaster. The kid will not understand why and how the grandmaster wins. Similarly, a “superdangerous AI” can kill us in so many different ways, that we won’t know it till it is too late. The problem is, any regulation should assume specific, even if implausible, risks in order to address them. Nuclear states haven’t banned any research related to nuclear reactions, they created a Treaty on the Non-Proliferation of Nuclear Weapons. Had they banned all research that could potentially lead to a nuclear bomb, we wouldn’t have seen nuclear power plants and, possibly, X-ray and CT scans.
So, let’s work with what we have and assess the specific risks we can think of in order to better understand the effects of the regulation that could eliminate them.
AI in social networks. Here the risk is that a “superdangerous AI” will be employed by malicious actors to create a farm of bots that will be hard to distinguish from real people. So, a group of people will have to use AI to create a large set of bots that attack the network in a coordinated fashion. For instance, they will persuade a group of people to commit an act of genocide. This is, in my opinion, the strongest argument for banning large scale AI. There is no doubt that with the rise of GPT-like tools, the struggle between social networks and bots will rise to the next level. However, this risk can be addressed without banning either AI or social networks. There are lots of ways social nets can figure out a coordinated event and ban the accounts or deamplify them. Also, a significant bot effort (including getting past captcha, making friends and generating lots of human-like posts) will still require a costly development effort as well as a substantial amount of power. This means with some additional effort from the social networks the cost of running an AI bot farm will be more expensive than the cost of hiring human bots.
Cybersecurity threats from AI. LLMs are capable of writing software code. What if hackers use it to hack into government facilities and blow up stuff? First of all, if there’s a possibility of blowing up stuff by hacking, it is wrong, because there is always a chance that someone will break through the cyber defense. But also, is AI going to be that helpful in hacking? While it is still not clear how well LLMs can teach themselves to program with reinforcement learning, general purpose LLMs are notoriously bad at generating code by themselves. A study on robustness of LLMs used for programming in Javascript have found “62% of the generated code contains API misuses, which would cause unexpected consequences if the code is introduced into real-world software.” Why is that? Common sense and evidence suggest that LLMs are good in programming when they are trained on a large collection of correct code solving real problems. This implies that LLMs are good on languages and problems that have a lot of open source code, and bad otherwise. Are there a lot of open source code used for hacking? Not really. One can argue that this can change in future and the cost of cyber break-ins will be dramatically reduced in the next few years. I think this is definitely possible, but banning AI in specific countries wouldn’t help to solve it, similarly to banning guns in high crime areas without enforcing the ban. There will be countries that will host bad actors. So a more reasonable approach would be to develop countermeasures against AI bots that increase their operating costs for hacking. Also, if a “destroy humanity” switch is connected to the Internet and can be hacked into, this seems like a very bad idea regardless of the existence of a “superdangerous AI”, and we need to put it off the Internet ASAP, before working on the AI regulation!
AI will be used to create chemo/bioweapons. The concern here is that AI is capable of creating bioweapons much better than people, and will be used by malicious actors. People used papers like this one as an argument. It’s behind a paywall, but one of its author gave a lengthy interview, you can get the gist from there https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx. Contrary to what many think, this paper doesn’t state that AI has created toxins that are more dangerous than the existing ones. They state that AI suggested toxins that are estimated to be very dangerous by some commercially-available model: “we even found some that were generated from the model that were actual chemical warfare agents.” If, however, you read the paper, you find our that they: (a) developed a predictive model that figures out if a molecule is toxic or not, (b) fed this model as a cost function to a neural network, (c) as the paper states, “we chose to drive the generative model towards compounds such as the nerve agent VX”. And when AI found a good solution to the cost function, they were “concerned”. The authors have not actually synthesized any of the agents and tested its toxicity. If their predictive model doesn’t work, then AI just synthesized rubbish. If the predictive model is really good, then AI synthesized toxins, but this is not the problem with AI, this is the problem with the predictive model that – maybe – shouldn’t have been created in the first place. And given that it is a relatively small network – nothing to do with LLMs – you can’t even ban AI here, you have to ban the datasets themselves. And this might be a good idea, datasets of toxic molecules shouldn’t be out there in the open.
The same principle applies to other bioweapons. If there is enough data to train AI to predict lethal chemical or biological weapons, it will be trained, and it won’t be an LLM, it will be a small simple cost function optimization that no one can ban, it’s like banning all math. We can – and should – ban dangerous data, like we ban the nuclear bomb recipes.
AI Alignment. The alignment problem is setting the goals for AI that are aligned with the goals of the humans who use this AI. Oftentimes it is very hard to formalize what we want, and so AI will eventually work on a different problem. The concern of AI safety people is that AI will decide that it has to kill people in order to achieve a goal that has nothing to do with human extinction. One of the examples is the paperclip maximizer, a thought experiment by Nick Bostrom: an AI that has a goal to create as many paperclips as possible will decide that it has to kill all humans, as they may switch the AI off and thus prevent it from making as many paperclips as possible. Eliezer Yudkowsky suggests that a runaway AI may decide to kill all humans, design a deadly virus, order its manufacturing with an online service and set up a deadly pandemic. He goes as far as suggesting to use nuclear bombs on data centers that train LLMs, even at the cost of starting a nuclear war.
Science fiction aside, obviously the AI alignment problem is real and has implications on applying reinforcement learning systems to practical problems. However, we are very far from a runaway AI system that has capability to kill humanity. Anyone dealing with reinforcement learning knows that in addition to defining the goal function one has to define the set of policies that will be optimized. It means that the runaway AI can’t accidentally decide to kill humanity and send a virus to the lab, this has to be programmed. The AlphaGo is programmed to make Go moves, it can’t play chess or kill an opponent. And if an AI system has a capability of killing people, the problem lies in the specific method by which a computer can inflict harm, not with the algorithm itself.
OK, we’ve covered some specific scenarios (almost impossible to cover all of them) and found out that the threat from specifically LLMs is not as high as portrayed by the Future of Life institute letter signees. Now let’s look at the other side of the scales: what we will be giving up if AI is regulated. The details of the regulation are still vague, so for the sake of simplicity let’s assume that the development and use of machine learning models with more than 1 trillion parameters will require an approval from a federal agency (for the reference, GPT-4 has 1.76 trillion parameters). First, large companies that already have large language models will be the winners, and this regulation will be their moat. This means startups will stop innovating in this area, and then, gradually, large companies will also stop, given the absence of competition. Second, more generally, the progress in this area is so fast because a lot of the research is open, and a lot of models are open source. Stability AI released their stable diffusion models in open source, and Meta has opened their Llama-2 model. We have observed many times that scientific breakthroughs don’t come from large groups of people that are constantly in meetings, they come from small focused research groups that have enough tools to do something new and cool. Regulation will delay the innovation in this field or stop it altogether.
Some experts advocate for a 6 month moratorium imposed by the government. 6 months doesn’t look like a big deal, and it will give us all some time to think about AI safety. I think this is quite naive, as there are very few precedents of a government relaxing a regulation. Make no mistake: if it is a 6 months moratorium, it is going to be an indefinite ban.
It looks like we do not gain much from banning LLMs, the risks are usually connected with other security threats that should be addressed separately (instead of banning LLMs that can create a deadly virus, ban the labs that can print such a virus from an online order). Meanwhile, LLMs are expected to make the economy way more efficient, and a ban can hurt a lot of people.
Are we ready to turn this area of rapid progress into something like fission, where only a few large entities move with a glacial pace? Decide for yourself.
[UPD] This text was written before October 30, 2023 when the Executive Order on AI was published. Steven Sinofsky, who ran the Windows division at Microsoft, has written a blog post on this order that I encourage you to read, where he draws parallels between AI and the early times of personal computers, where similar regulations were contemplated but not implemented. While we still don’t know the impact of the EO on the progress in large-scale AI, one thing is clear: it has dug a huge moat for large companies that are already working on LLMs, making it way more expensive for startups to enter the field.