September 18, 2018
GiveWell’s top charities have to meet a very high standard in several ways, including ways that promising charities may not always prioritize. So in recent years, GiveWell has been providing grants to promising charities to help them, if all goes well, achieve top charity status. In this talk from Effective Altruism Global 2018: San Francisco, Catherine Hollander explains how GiveWell’s Incubation Grants function, and why they’re an exciting addition to GiveWell’s charity evaluation work. A transcript of Catherine's talk is below, including questions from the audience.
Hello, everyone. I'm Catherine Hollander, a research analyst focused on outreach at GiveWell, and I'll be talking about our Incubation Grants program today. When EA Global reached out about the possibility of providing a talk about GiveWell's work, and I learned that the theme was "Stay Curious", our incubation grants program felt like the perfect thing to talk about, since this is really work that is intended to expand the horizons that we're considering at GiveWell.
Just briefly, I'm going to talk about what GiveWell Incubation Grants are. They're a newer part of our work. I'll also talk about the different types of Incubation Grants that we've made, and use some of our past Incubation Grants as illustrative examples. I also want to briefly touch on where incubation grants fit into GiveWell's plans for the future.
GiveWell, as many of you might know, is a nonprofit that's dedicated to finding and recommending outstanding giving opportunities. We publish a short list of top charities every year that represent some of the opportunities that we think are extremely evidence-backed, cost effective, transparent and in need of additional funding. We think that our top charities are great opportunities for the many donors who are interested in using our research.
We started to think, are there things that we could be doing to help develop the pipeline of potential future top charities? One thought that we had was, as a recommender and as a funder, we're really stepping in at a fairly late stage in charities' life cycles. We're looking for things that are extremely strongly evidence-backed, which means that by the time they're getting to GiveWell's recommendation, someone has probably already funded them, such that they could develop the track record that we need to see in order to make our recommendation. Someone has already funded the program research that we rely on to decide that a particular program is very evidence-backed, and that therefore we're interested in finding charities that support that program.
The incubation grants program is us at GiveWell stepping in as that earlier stage funder and trying to specifically develop the types of charities and the types of program research that we think will ultimately make it to our top charities list, where we can recommend them to a large number of donors and direct a significant amount of funding to support their work.
These are the goals for our Incubation Grants program: developing the pipeline of future top charities, and also improving our understanding of our current top charities since, particularly as GiveWell has grown, we're directing a significant amount of funding to the groups that we recommend. And so, we want to make sure that we answer questions that we have about them, and the Incubation Grants program is one way that we can do that.
I mentioned us stepping in as a funder, and making grants. The funding that we are providing for the Incubation Grants program comes from Good Ventures, which is a large foundation that we work closely with. You might be familiar with them if you're familiar with the Open Philanthropy Project. They were started by Dustin Moskovitz, who's one of the co-founders of Facebook and Asana, and his wife Cari Tuna and they're significant supporters of GiveWell's top charity program as well. Right now they are the sole funder of GiveWell's Incubation Grants work.
We're not recommending Incubation Grants to the donors who use our research. We are instead hoping that the Incubation Grants program builds the list of things that we are ultimately publishing for recommendation to all donors.
The first is work to seed potential top charities. This is, perhaps, fairly straightforward. We're looking at organizations at an early stage of development that would not meet our current top charity criteria. They don't yet have the kind of track record that we'd like to see to point to them and say that we think that there is a very strong case for impact that we can make to donors, but we do think they have the potential with additional funding and scale to become charities that we would recommend to donors.
There are a few different grants that we've made in this space. One is to a group called Charity Science: Health which came out of the EA movement. Some of you here might be familiar with it. They're providing SMS reminders for routine immunizations in India, but they're a very new organization, and so, not one that has the kind of record that we'd want to see before we recommend them as a top charity, but something that we're interested in following as they grow in scale.
The second type of Incubation Grant is work to support monitoring. Monitoring is really, really important for GiveWell. We're looking at charities where we want to really feel that we can demonstrate that they are having an impact. We want to see very good information from those charities, that indicates that they are successfully reaching people with their programs, that the people who are being reached with the programs are the people that they're intending to reach and that we're tracking any longer term impacts that we might want to.
One example of this type of grant that we made was for work to identify cataract surgery organizations where we might be able to help them implement monitoring since, previously, we were interested in the intervention of providing cataract surgeries to improve visual acuity due to cataracts, but we hadn't found an organization that met our bar for having monitoring. We wanted an organization that was demonstrating who they were reaching with their surgeries, how much those surgeries were improving their vision, and also just sort of what the counterfactual story was, whether those surgeries would have otherwise happened in the absence of the relevant charity's work.
Those were the types of monitoring questions that we hadn't answered, and a type of Incubation Grant that we made was to support work to identify and help a charity build a monitoring system that would help us answer those questions. That project is actually paused right now. I'm happy to get more into that during the Q&A, but that's sort of an illustrative example of that type of Incubation Grant work.
The third type of Incubation Grant is to support program research. GiveWell's research process is sort of divided into two different steps. The first step for us is generally doing a survey of the academic research, the independent research that's been conducted on a particular program to help people. This is not research that's generally done by charities themselves. It's often done by academics, and that's the first step in our process, is looking at the academic research to see which programs we think might be the most cost effective and evidence-backed, such that we would be interested in finding a charity that's working on them.
There are two reasons we start with the academy. One is that charities don't often, themselves, do this type of research. Running randomized controlled trials, for example, is quite expensive and not necessarily valued by most charities. Two is that some of the biggest drivers in differences in program effectiveness and cost effectiveness just come down to which program a charity is implementing. Whether you're delivering insecticide treated nets or performing surgeries, that in and of itself will be a big driver in differences among charities' cost effectiveness, perhaps more so than looking at two charities that are doing the same program. So we're interested in that as a first step.
A type of Incubation Grant that we're excited to make is one that looks at a promising area of academic research, but maybe there's only been one study that's done or we have open questions. In a case like that, we're interested in funding further program research to support our understanding.
Now, a few examples of past Incubation Grants that we've made.
The first is a group called New Incentives, and I believe they were our first Incubation Grant recipient. If not first, second. Back when I mentioned that we were starting this program in 2014, or around then, I believe we made our first incubation grant to New Incentives in January, 2014. The program that they were implementing at the time that we were quite interested in was a conditional cash transfer program that was intended to incentivize pregnant, HIV positive women to deliver their child in a health facility, in order to prevent the transmission of HIV from the mother to the child.
This looked to us like a promising program, but the organization was brand new. We just didn't feel like we had enough information to say "we really feel solid about the case for this charity", such that we would recommend it as a Top Charity, but we thought, "this seems like something that we would be interested to learn more about, and seems like it could become something that we'd recommend".
We made our first grant to New Incentives back in 2014. In 2015, they ended up realizing that they weren't reaching enough HIV positive women with their program, so they decided to expand the program to just be to incentivize pregnant women to deliver in health facilities. So, they shifted a bit there, and then we spent the next year looking into the impact, the research that we could find on facility delivery on neonatal mortality. So, does delivering a baby in a health clinic mean that it has better neonatal mortality outcomes?
What we found at the end of 2016 was that we didn't think there was sufficient evidence to convince us that that was a program that we should recommend. Through this whole time we had been engaging with New Incentives and we felt really positively about them as an organization, so we decided to make another grant to them to support a shift that they were making into providing conditional cash transfers to incentivize routine immunization. They've kind of gone through a few different iterations through our Incubation Grants program. What we're planning to do with them now is to run a randomized controlled trial, which I mentioned earlier as a type of evidence that we're particularly interested in.
It's a type of study where you divide people into two halves, a control group and a treatment group. You randomly assign them and then you look at the differences in outcomes between those two groups. We're now working with another research partner that I'm going to talk about momentarily called IDinsight to run a randomized controlled trial of their work and are now looking at them as a potential top charity contender, something like in 2020, so a few years out.
It's kind of an iterative process. I think this example illustrates that. The timeline might be quite long from making an initial Incubation Grant into a charity joining the top charities list, but we've certainly learned a lot through the New Incentives process and are continuing to follow them through our Incubation Grants program.
The second group that I wanted to talk about is another Incubation Grant that we made very early on in the program. This was another early 2014 Incubation Grant, to a group called No Lean Season. We became interested in the program that No Lean Season is operating, which is a seasonal migration program where they are providing small subsidies to individuals in rural northern Bangladesh to enable them to temporarily, seasonally migrate during the time of year when seasonal poverty is a large issue and job opportunities are harder to come by.
The idea is that if you give people a small incentive, that enables them to do something like buy a bus ticket to travel somewhere else where they can work, which can in turn have beneficial income and consumption benefits for their household.
In 2008 - this crossed our radar a little later, but the study was done in 2008 - there was a randomized controlled trial that was done by a Yale economist named Mushfiq Mobarak. He ran a small trial of this program. I think it was about 1,900 households that were involved with the initial trial, that saw significant benefits for household income and consumption as well as continued migration the year following the program in the absence of the incentive. When we looked at that we said, this seems like it has the potential to be one of the most cost effective programs that we've come across in a long time, and we're interested in this. But only 1,900 households, from one study that was done in this very particular area, we're not sure we would want to recommend that quite yet as one of our top charities.
We made this initial Incubation Grant to Evidence Action, which is an organization that runs the Deworm the World Initiative (which is one of GiveWell's longtime top charities) to scale this program. Evidence Action had also been interested in this program, so we sort of came together at that moment and provided them funding to scale this work. In the subsequent years, they conducted a number of additional randomized controlled trials. They scaled from about that initial 1,900 households to, I think, around 100,000 households in 2017. Last year we decided we were at the point where we wanted to evaluate them under our traditional Top Charity criteria since we thought that they might now be at the point of having that track record that we'd want to see, and that there might be sufficient evidence there where we'd say, this is something we want to recommend to donors.
We sort of put it through our normal process last year. We spent a lot of time talking to them about how much funding they needed. We built a cost effectiveness model of their program, which is a really important input for us. We looked at their monitoring and we did a site visit to their work in Bangladesh. Coming through that whole process, we ultimately decided, at the end of 2017, that, in fact, No Lean Season did meet our top charity criteria and so we added it to our list of the nine groups that we currently recommend.
This was our first Incubation Grant from the beginning to the Top Charity list example that we have and we're hopeful that this is something we'll see again in the future. This is currently the first and only that has gone from that initial grant all the way through to meeting our criteria.
The final example of a grant that I wanted to talk about is a group that we're working with called IDinsight. Buddy Shah, who's the CEO of IDinsight, is giving a talk at EA Global tomorrow. I think it's at 3:30 and I would recommend it, because they're doing really, really interesting work.
I mentioned that one of the things that we're really interested in at GiveWell is academic research. That's a really important first step of our process. But what we've found, unfortunately, is that academics aren't always incentivized to provide decision-relevant information to folks who are implementing programs on the ground. What we're often curious about is how a program will look when it scales, how will it look in another area, and there's not always the academic incentive to conduct research that answers those questions. IDinsight, we think, fills a really unique role in the development space by specifically trying to provide that information. They'll run low cost RCTs, randomized controlled trials, and try to provide decision-relevant information to policy makers, or people implementing programs, to help them really decide where the most effective places are for them to work, how cost effective their programs are.
This seemed really exciting to us. We have a lot of those questions that we're hoping to answer. We've partnered with them through our Incubation Grants program to work on a number of different projects with us, where we're hopeful that they can help us better understand the spaces that we're interested in. I mentioned with New Incentives, that first Incubation Grant recipient example, that they were running a randomized controlled trial; IDinsight is working with us on running that randomized controlled trial to help give us that information.
They're also working with us on our project to help better understand one of our current top charities, the Against Malaria Foundation. The Against Malaria Foundation is a group that's been on our list of top charities for a very long time. I think they initially joined our list in 2011, and in 2016 we realized that we had some questions about the ways in which they were conducting the monitoring of their program. The basic way that it works is that AMF, Against Malaria Foundation, checks a percentage of households that received their nets every six months after implementation, to see whether they're hung and also what the state of the net is, since we believe that the nets degrade over time, like, develop holes and become less effective.
I should say, the nets are things that you hang over your bed. They're treated with insecticide and they both block mosquitoes from biting you and also kill them, from the insecticide, so we think they're very good for preventing malaria. In 2016, we wrote about some of the questions that we had about the way that they were conducting surveys in the Democratic Republic of Congo and in Malawi, and we partnered with IDinsight, to help us both better understand their current procedures for conducting those surveys by having IDinsight observe them doing that on the ground, and then also to make recommendations for the Against Malaria Foundation to improve how they're conducting those surveys.
This part of their work is really an example of a type of Incubation Grant that's geared toward improving our current list of top charities, rather than building a new organization to join it. But we think it's really important to have this work as well, since we're directing tens of millions of dollars to the organizations that we recommend.
The final project with IDinsight that I wanted to call out is another pretty different type of work than the type of work I mentioned before. I mentioned cost effectiveness as a really important factor for GiveWell in coming up with our list of recommendations at the end of the year. We're looking for things that we could make a strong case are extremely cost effective.
Where this gets particularly challenging is that a lot of the different programs that we're interested in, and that exist in the world, have different outcomes. I mentioned cash transfers, deworming and the distribution of insecticide treated nets as some of the programs that were on our list for a long time. We think the benefits of those programs are very different. Distributing insecticide treated nets to prevent malaria, we think reduces child mortality from malaria. Distributing deworming pills, we think, has the potential for increasing the incomes of children who receive deworming pills, later in life.
Those are really different outcomes, and so when you're trying to think, which of those two is more cost effective, you have to come up with some sort of comparison between the two that you can make. The way that we do this right now is that we have staff members who work on GiveWell's research team, informed by everything from their own philosophical values to the World Health Organization's assessments of the different burdens of different diseases, to other information that we use to make those trade-offs. But the piece of information that we think we are most lacking to make those trade-offs is how the recipients of the interventions of our top charities themselves would make those trade-offs. How they would decide, "How much do I value increasing income versus improving health?"
We have done reviews of the literature to see if we can find information to inform that, but we've generally concluded that that information doesn't really exist and hasn't been collected. A lot of the research that's done on how those trade-offs get made has been in rich countries, rather than in the poorest parts of the world where GiveWell's top charities tend to operate.
And so, the final project I wanted to mention that we're working on with IDinsight is a project to survey the beneficiaries of our top charities to ask them and try to better understand how they would make those trade-offs, to inform our cost effectiveness models. This would be something that, again, I think would fall under the category of improving our current recommendations and strengthening the information that we can provide to donors who are interested in supporting these organizations.
So, those are a few of the Incubation Grants we've made. This piece of our work has been ramping up in the last few years, so there's quite a few that I didn't mention but I'm happy to talk about in the Q&A.
I want to close with a quick thought on some of our plans for the future. Especially for this audience, I want to convey that GiveWell is moving toward… We've been around for 11 years. We're moving toward being open to some of the sort of riskier ideas and harder to prove cases for impact than I think we have been historically. Incubation Grants plays a role there in giving us the ability to try to answer some of the questions that maybe have been challenging for us in recommending things in certain spaces in the past.
One example of sort of the riskier, harder to prove types of areas that we're interested in now is work in policy advocacy interventions, where I think we're interested in finding things that might be significantly more cost effective than our current group of nine top charities focused on global health and development. One of our best guesses for where we might find that is interventions that are targeted toward getting policies to pass that might affect many, many people for a long amount of time, that that might be a really cost effective thing for us to do.
I think we'd be interested in supporting research through incubation grants that will help us better understand the causal impact of policy advocacy interventions, since I think one of the biggest challenges there is you're not just going to have a randomized controlled trial of, like, the ability to pass a policy. It's going to be a little harder to tell that story, and so we want to support things that will help us potentially consider those types of areas in the future, but this comes with the second bullet point which is that we're still very interested in maintaining our level of transparency. What I mean by transparency is really being able to make a case, publicly, that is vettable by donors who are using our research for why we think that something meets our criteria and is very cost effective.
We want to be able to point to our rationale. We want to be able to point to evidence and monitoring that really demonstrates why something has ended up on our list and why we are saying this is one of the best opportunities we're aware of, that meets our criteria. Incubation Grants work is a tool that we're using to help us do that, and particularly as we move toward some of these areas that maybe in the past have been harder for us to assess by our traditional criteria.
We're very excited about this work. Just to give you a scale of it, I think last year we made something like close to $10 million worth of Incubation Grants, and so it's a big part of our work. It's not the only part of our work. We're still looking for organizations that meet our traditional criteria, not through the Incubation Grants program, but we're optimistic about helping move toward building that pipeline and bringing more organizations into our consideration.
Question: If you're a person who's interested in doing program research or helping an organization with monitoring, where can you find the outstanding questions that you'd like to see answered, or find the organizations that you could tack onto?
Catherine: Yeah, so, where would you work if you were interested in incubation? We have a list of all the organizations that we're currently supporting through Incubation Grants on our website, and we also have a list of priority programs that we've identified as promising areas, but we don't yet recommend charities that are working in. That's kind of one signal of where there might be either opportunities to start organizations or talk to us about that. I think for thinking about where we might potentially make Incubation Grants in the future, the sort of core question that we have is, how likely is this to become a GiveWell top charity?
We actually publish forecasts along with our Incubation Grants so you can kind of see, like, okay, we think that this grant has a 30 percent chance of resulting in a Top Charity, and so we're trying to sort of come up with initial rough models, think about how likely something is to become a Top Charity, and then make our decisions about what to fund based on that.
Question: Let's say someone decides that they're going to run with a project, and now they want to get GiveWell's attention and see if maybe they can get an Incubation Grant and grow into a recommended charity. How do they get on your radar?
Catherine: I think just reaching out to us. You can email me at firstname.lastname@example.org or email email@example.com. I think we want to have early conversations before anyone spends a ton of time developing something if they're trying to get Incubation Grant funding. We'd definitely appreciate the chance to check in, talk about our interests and hear about what the potential program is. Emailing us is probably the best way to begin that conversation.