You do not have Javascript enabled. Some elements of this website may not work correctly.

September 25, 2018

The development of artificial intelligence is well-poised to massively change the world. It’s possible that AI could make life better for all of us, but many experts think there’s a non-negligible chance that the overall impact of AI could be extremely bad. In this talk from Effective Altruism Global 2018: San Francisco, Carrick Flynn lays out what we know from history about controlling powerful technologies, and what the tiny field of AI governance is doing to help AI go well. Below is a transcript of Carrick's talk, which we have lightly edited for readability.

EAG SF 2018 v1.1

In my previous career, I worked in global poverty eradication, mostly doing field work. This meant working on projects all around the world, trying to find a place where I felt I could do the most good and have the largest impact. A little over three years ago, while I was working in Ethiopia, I discovered the effective altruism movement. When I did, I discovered just how big the biggest impact could be.

Think for a second about a single grain of sand. Imagine rolling it between your pinched fingers. Picture how small it is. Now, think about how much sand there is on earth, in all the deserts: the Sahara, the Gobi, the Arabian. All the sand on the ocean floor, on all the beaches, in all of the dunes. If you press your palm to the sidewalk in most places, when you come back, there will be little bits of sand embedded in it. There are a lot of grains of sand on earth. In fact, there are approximately a sextillion grains of sand on earth. That's a billion trillion.

EAG SF 2018 v1.1 (1)

Take a second to imagine that grain of sand pinched between your fingers again. Now, try to imagine that grain of sand is everything of value on earth. All the people, all the animals, everything. And hold that thought.

There are 100 trillion reachable stars. We have approximately 10 trillion centuries before entropy makes the universe difficult to inhabit. That means we have over a trillion trillion equivalents of earth's resources within our reach. That means if we imagine everything on earth as a single grain of sand, there are more earth equivalents out there, but within our reach, than there are grains of sand on earth.

EAG SF 2018 v1.1 (2)

The human brain is not good at thinking clearly about issues of scale. When we think about doing good, we tend to think in terms of prototypes. I've always been most motivated by human wellbeing, so when I think about doing good, I tend to think of helping someone. Effective altruism is, at its core, about making sure we don't stop with this prototype and this emotion. It's about trying to harness the altruistic motivation to have an effect in the real world.

For many of us, there's something a bit disconcerting about interacting with far future considerations. The prototype's kind of abstract. It's sort of weird. It transects a lot of science fiction and transhumanism and things we may not really be that comfortable with or into. We care about human suffering. We care about animal suffering. And those things exist here, now, on earth. We can see them. I am very sympathetic to this situation. I am in this situation.

EAG SF 2018 v1.1 (3)

However, the reality is that time and space exist. They will be filled with something. Maybe they will be filled with dead matter, completely devoid of life, and maybe that seems like a tremendous waste. Maybe they will be filled with terrible, exploitative human societies, and full of factory farms, and that seems like a disaster. If what you altruistically value exists in time and in space, and you want to do the most good, you have to think big. I may have this grain of sand in my fingers, but if I care about doing the most good, I can't focus on just this one grain of sand. There is a lot of sand out there.

Humans have done three things in history which stand above the others in terms of their impact on our own wellbeing and the wellbeing of animals. The first was our developing behavioral modernity. This occurred about 70,000 years ago. The leading theory for behavioral modernity was that it was the result of very small cognitive changes. Behavioral modernity allowed us to outlive the neanderthals, who had been preventing us from gaining a bridgehead to leave Africa. With this, we were able to expand across the world. As we did so, we caused the extinction of most megafauna on earth, and of all other species of humans, including the neanderthals.

EAG SF 2018 v1.1 (4)

The second was the development of agriculture, which occurred around 12,000 years ago. This technological advance increased human population about a hundredfold. It resulted in dramatic ecological damage, the exploitation and extinction of animals, and a decrease in human health and wellbeing.

The third was the industrial revolution, which came with further ecological destruction and the advent of factory farming. But, it also created dramatic wealth and resulted in the first sustained improvements in human health.

EAG SF 2018 v1.1 (5)

I'll also give a quick honorable mention to atomic weapons, which haven't had much substantive impact on earth, but definitely have that capacity.

Our intelligence, and the technologies we invent, are how humans have our biggest impact. For this and for other reasons, it is safe to imagine that the technology of intelligence will be at least as impactful for humanity and for animals as the industrial revolution. AI in its current state is not that impressive. It recognizes images well and it's good at some games, but that's about it.

EAG SF 2018 v1.1 (6)

However, it's easy to get lost in the details at the beginning of a large change, and to miss the forest for the trees. The earliest stage of the development of behavioral modernity was extremely subtle and mostly looked like - most likely - slightly improved communication within human bands. This took place over the course of thousands of years.

The line between early agriculture and certain types of seed-gathering techniques is so subtle that modern anthropologists can't always agree about what's going on with modern bands of humans. Steam engines were adopted slowly and were very unimpressive for a long time. With agriculture and with industry, humans living at the early stages could not have conceptualized what they would develop into. Fortunately for us, with intelligence, we do have one model of what it looks like when it reaches an advanced stage.

The difference between humans and gorillas is a few million years of evolution. Gorillas now only exist in zoos and a few little reserves that we keep them on so that we can look at them. The difference in intelligence between humans and neanderthals was, as far as we can tell, very small, and as soon as behavioral modernity gave us a little edge on them, we wiped them off the face of the earth. For the first time, we can kind of see a dim outline of an enormous change on the horizon, and it is coming at us. Also, for the first time, this change is not just going to affect our one unimaginably valuable grain of sand. This is for all the sand.

EAG SF 2018 v1.1 (7)

The goal of AI governance is to help humanity best navigate the transition to a world with advanced AI systems. It is possible that the default course of development and deployment of advanced AI will be great. There is an optimistic case that can be made that this transition is very unlike the last three, and it will go quite smoothly. The last three were sort of mixed. But this one could be just good. If this is the case, then with AI governance as a cause area, the best we can hope for is maybe small marginal improvements, though, again, across a lot of sand.

Unfortunately, there are also good reasons to think that the default course might be fraught. In a survey of AI experts who publish in top journals, the majority assigned at least a 5% probability that superhuman AI would be extremely bad, on the order of human extinction. Even among researchers who are highly optimistic about AI, it is generally agreed that advanced AI will not necessarily be safe, or at least cannot be guaranteed to be safe without some work and testing.

EAG SF 2018 v1.1 (8)

So, we'll do the work and testing, right? Maybe. It depends on how high the cost is in terms of time, resources, and the performance of the system itself, and how competitive the environment is in which that system is being developed. This is sometimes called the "safety tax". For example, what if the safety tax is two years, and yet there are several companies at the forefront of development, with tens of billions or hundreds of billions of dollars at stake, who are only a few months apart? Now, worse, what if these are not companies that are a few months apart from one another, but rival militaries? During the Manhattan Project, there was a serious concern that detonation might ignite the earth's atmosphere and kill everyone. It didn't really slow the Manhattan Project down.

Now, let's imagine that AI can be made safe easily, and that part is just covered. This is not nearly sufficient to guarantee a good outcome. Some tech companies have impressively cosmopolitan values, but there are legal constraints based on fiduciary duties they have to their stockholders. We also probably do not want any one company to control a particularly large stake in the future of the universe. It's also unclear if a government would let this happen without somehow taking control of AI development or deployment.

EAG SF 2018 v1.1 (9)

So, what if this technology is developed by or is in some way captured or seized by a government? Realistically, it would be the US government or the Chinese government. How comfortable are we with either the US or China dictating the future of the world? Scarier question: How comfortable do we think other nuclear states are with this?

Now, let's imagine we've managed to thread the two previous needles. AI is safe, and it was deployed without igniting a great power war. How are the benefits distributed? Continental Europe does not have a viable AI industry, let alone Latin America, Africa, or most of Asia. Unlike during the industrial revolution, where human labor was a complement to machinery, it does seem as though AI mostly serves as a substitute for human labor.

EAG SF 2018 v1.1 (10)

Additionally, most AI services have characteristics of a natural monopoly or an oligopoly. There's not much room for competition, and there's not really much room for small businesses. Maybe we are so wealthy after the development, that wealth naturally sort of trickles down, that a rising tide is able to lift all boats. But, we are extremely wealthy now, and children still die of malnutrition, malaria, and diarrheal diseases by the millions. And so far, the wealthier we get as a world, the more animals we subject to factory farming.

For us working in AI governance, the current priority is to increase coordination and to decrease race dynamics. This is both to reduce safety risk, and the risk of greater geopolitical destabilization. It's also hopefully to increase the breadth of stakeholders who are able to have a say in how the future is developed, and in doing so, hopefully to increase the moral circle under consideration.

EAG SF 2018 v1.1 (11)

There are several routes for this that we're pursuing in parallel. These include technical work on the possibilities of verification of AI-related international agreements, which are really necessary for coordination between nation-states, and in some ways a prerequisite for that being possible. Also, in case we someday we might want something like the international control of advanced artificial intelligence - not saying that we do, but if we do potentially want that option - we've been looking at case studies of other failed efforts to control decisive weapons technologies in the past.

For example, after World War I, there were several serious proposals to develop an international air force. This sounds a little weird, but there was actually buy-in from the United States, Japan, France and even Winston Churchill, who was not a dove, for this as a plan. The idea was for the League of Nations to have a complete monopoly on military aviation, with all other states banned from getting it. This international air force, then, would have two functions. One was to prevent any state from then developing military aviation. The other was to attack any aggressors who then started a war, with the understanding being that this would actually secure world peace.

Similarly, after World War II, and the atomic bombing of Japan, the US proposed to give up all of its atomic technology, including all of its resources, to an international body, and to subject itself to intensive inspections, to show that it hadn't retained any and that it wasn't restarting, if the Soviet Union and other states agreed to do similarly. Understanding how these plans failed, and trying to learn lessons from their failures, might increase the chances that we're able in the future to gain international control, if that is necessary. Third time's the charm.

We've also been doing some work in science diplomacy. Science as a field is very cosmopolitan. Even quite adversarial nations are quite good at collaborating in science. ITER is probably the best case study for this. ITER is a research project looking at nuclear fusion, and is being funded for many, many billions of dollars. It's a joint venture by the US, China, Russia, the EU, and some other states. What's important about this is that nuclear fusion is an important strategic technology. This is why the vast majority of this research is funded actually by the military, or the department of energy, but under military guard and sometimes for military purposes.

The knowledge and results from this experiment are supposed to be spread out from all the participants. If it does someday make sense to try to develop powerful AI as a collaborative effort, with many stakeholders, I think this might prove to be the best current model we have.

EAG SF 2018 v1.1 (12)

We also do a lot of tech company and AI research community engagement. The OpenAI Charter is possibly the largest recent win in this category. In the charter, OpenAI commits to developing AI for the common benefit of all humanity. But more interestingly, it commits to dropping out of anything like an AI race if a rival gains a lead, and even actually to join the efforts, free, to push them further forward into a greater lead.

Anyone who's interested in this, I would actually really encourage you to read this charter. It is inspiring. It is also very short. If you have good vision, you could read it printed on a business card. Also, if you're interested more in our engagement work, I would encourage you to talk with Jade Leung, who does this for our governance group.

So, I won't go into too many more details here, through I will provide resources at the end for people who want to read up on more of what we're doing and some of the work. But to do a quick summary, we're trying to understand the AI scene better in China and increase our engagement there. We're also trying to better understand issues of geopolitical instability that can be created by changes in the balance of power and also in the offensive and defensive dynamics of weapons technologies. We are also modeling the dynamics of tech races, and in particular, how they can be averted or spun down if they do begin, and a lot more.

AI governance, as a cause area, is absolutely tiny. People talk a lot about AI at these conferences, which gives the impression that there's a lot going on, but there are fewer than a dozen people who work on this full-time. There are not that many more who work on this part-time. This being a tiny field is in some ways quite good for you. It means that your opportunity for a marginal impact is absolutely huge, but unfortunately, what it also means is there is not very much absorptive capacity within existing groups to bring more people onboard.

For the majority of people who are interested in advancing this as a cause area, it is probably mostly about preparing yourselves to be more involved in the future as the field grows and expands. That said, as far as our immediate needs, we are desperate for people who are good at getting things done. This is something we do need immediately and there are immediate job openings for. We need good project managers, and good operations folks of all sorts. As well as being currently our largest bottleneck, this might be one of the main reasons why our absorptive capacity to bring other people on is so low. So, there's a force-multiplicative aspect for project management and operations work as well.

EAG SF 2018 v1.1 (13)

I want to in particular highlight that there's a role at the Governance of AI program in Oxford, which is currently the leading group researching this, for a governance project manager. This is a Visa-eligible position, and this will put whoever gets this role at absolutely the center of this as a cause area. We also badly need senior researchers who can supervise more junior researchers. Our research pyramid is sort of imbalanced, with the bottom a little too big for the top, and this is another bottleneck. We also have positions for junior researchers, which I definitely encourage people to apply to, though I think this is maybe less urgent or less impactful in the margin within this cause area.

For the majority of those who are serious about getting involved, what I recommend is investing in your skills and your career capital. This is especially true in high value areas, including studying topics in high demand. Also, building a career in a position of influence, especially in government, global institutions, or for example doing policy advising at a tech firm. Additionally, helping to build this community, and our capacity, including having a strong and mutually reinforcing career network among people who are interested in getting involved in AI policy from an EA or an altruistic perspective.

EAG SF 2018 v1.1 (14)

In my personal opinion, this third point is more important than most people realize. Having a strong social network, and having people who understand what it is you care about and why it is you're doing the things you are doing, even if it in some ways looks quite tangential to what it is you care about, can be very helpful in allowing you to keep your eye on the prize, and to work towards your goal.

The reason this social support can be so important is, as I mentioned at the beginning of the talk, the human brain is not good at thinking clearly in terms of scale. This means it's very easy, and natural even, to have some value drift. I've always cared a lot about human wellbeing. This is why I pursued my education in economic development, and my career in poverty eradication. My personal experiences in field work around the world have left me with a very strong, very emotional sense and prototype that still draws me very strongly to global poverty eradication.

But, I don't want my altruism to be about what I feel. I want it to be about how much good I can actually do and how much of an impact I can have outside of me, in the real world. Even if I actually can't conceptualize just how much there is, I do know where all the sand is.

EAG SF 2018 v1.1 (15)

If you're interested in learning more about our work, please go to our website.