Catherine Hollander: GiveWell Incubation Grants and Learning Through Grantmaking

What makes a charity worthy of assessor GiveWell’s “top charity” recommendation? And what about organizations that aren’t quite ready for that designation, but could be with the right backing? Catherine Hollander, a research analyst at GiveWell, shares why the charity assessor decided to launch its Incubation Grants program, the main criteria organizations must meet to become grant recipients, and a few examples of model grantees.

A transcript of Catherine’s talk, which we have lightly edited for clarity, is below. You can also watch this talk on YouTube or discuss it on the EA Forum.

The Talk

Today I'm going to talk about a newer part of GiveWell's work that we're very excited about: our GiveWell Incubation Grants program.

I'll cover some of the goals we have for this program, some of the different types of incubation grants we've made, and some past grants that are good examples of the type of work we're hoping to do with this program. Finally, I’ll discuss the program in terms of how GiveWell is evolving and planning for our future.

To start out, I want to briefly introduce GiveWell.

We're a nonprofit focused on finding and recommending outstanding giving opportunities. We publish a short list of heavily vetted, top-recommended charities each year that meet our very strict criteria for being named a GiveWell top charity. These charities are listed publicly on our website. So, this research is available for anyone to use. We recommend charities that we think can have a very big impact with every dollar donated.

We use four strict criteria to identify this group of top charities. The first criterion we look for in a charity is evidence of its effectiveness. There are two different types of evidence of effectiveness that we consider. The first is evidence that the program they’re implementing is an effective way to help people. What I mean by “program” is the independent way that a charity is helping someone. An example of a program is distributing cash transfers. Any charity could potentially do that program, but we first look at the evidence that distributing cash transfers is itself an evidence-backed way to help.

Evidence of effectiveness, for us, also means the charity’s evidence that it is successfully reaching the people it intends to reach. This is often referred to as the charity's monitoring. In the case of a cash-transfer charity, how does the charity know that it's actually reaching people who need cash? How does its staff know that the cash is being received? And do they know how people are spending the cash, to have a better sense of what sort of impact the organization is having? That's the first criterion we look for — the program evidence and the charity's own evidence that it is reaching people.

The second criterion we look for is the cost effectiveness of the program. GiveWell is looking to recommend things that we think can do a lot of good with every dollar donated. We spend a lot of time building complicated cost-effectiveness models to think about how much we think each charity can do with a dollar.

The third criterion we look for is the charity's transparency. I mentioned that the end product we produce is a list of publicly available recommendations, with the full case for why we recommend them, on our website. It's very important to us that charities be transparent, both with GiveWell — so that we can understand their work and the impact that they're having — and the public.

The final criterion we look at is the charity's room for more funding. What I mean by that is we look for organizations that are operating at a scale where they can take in a significant amount of money and use it well. We care about that because GiveWell's recommendations are used by many donors, so the charities we recommend typically receive a significant amount of funding. An organization’s scale and ability to absorb funds is really important for us.

We currently have nine top charities that meet these very strict criteria. These charities work on a variety of programs in global health and development, which we found tends to be the area that meets the two criteria of evidence of effectiveness and cost effectiveness. But back in 2014, we had been recommending a shorter list of top charities that felt very similar from year to year. When going to look for new charities that met our four criteria, we weren't turning up as many as we had expected. We were somewhat consistently recommending insecticide-treated nets to prevent malaria, de-worming treatments, and cash transfers. So, we started to wonder whether there was anything that we could do to try to find new potential top charities. This was the genesis of the GiveWell Incubation Grants program, this desire to see whether we might be able to help grow the pipeline of potential top charities and improve GiveWell's understanding of our own top charities.

In each case that we've recommended a top charity, someone else has typically come in and funded the charity at an earlier stage. The funder might have supported that charity to grow from a very small organization that didn't have much of a track record — something that GiveWell couldn't access to see whether we thought the charity was impactful — or that wasn't operating at the kind of scale where it could absorb the amount of funding that we would expect to direct to it if we named it a GiveWell top charity. Or maybe the charity received funding to research their program’s effectiveness.

The goal of the GiveWell Incubation Grants program is to see whether we can step in earlier as that funder to help develop our pipeline of potential top charities, taking a promising opportunity and funding it via Good Ventures, which is a large foundation we work closely with.

There are three different types of incubation grants that we've made in the last few years.

The first is a scaling grant to help young organizations go build a track record and determine whether or not they're having an impact. We've supported a few different organizations in this way. A couple have come out of the effective altruism community. Charity Science Health and Fortify Health, for example, are both organizations we are interested in for the Incubation Grants program.

We are also interested in supporting, through a second type of grant, academic research that helps us better understand whether a given program is effective. An example of that type of research is a randomized control trial of an insecticide-treated net. We wouldn't fund that one specifically because there have been quite a few studies on it already, but we look for research along those lines — work that helps us get a sense of a program's effectiveness.

The final type of incubation grant that we're interested in supports high-quality monitoring. This is quite expensive for charities to do. It often involves things like going back to recipients of a charity program to check on how the program has been run. In the case of an organization like the Against Malaria Foundation, which distributes insecticide-treated nets and is one of GiveWell's longtime top charities, its staff goes back to a percentage of households that receive nets and checks to see whether the nets are hung and what condition they're in if they're being used. It's quite intensive. Many charities do not have the kind of monitoring that GiveWell wants to see in order to make a top charity recommendation.

I'm going to give a few examples of each of these types of grants. The first I’ll mention was to a group called New Incentives.

We connected with New Incentives in late 2013 and made our first incubation grant ever to support their staff’s work in January 2014. It was a very young organization implementing a conditional cash transfer program we were interested in: giving people money on the condition that they take certain health-promoting actions. Specifically, it was a conditional cash-transfer program to help prevent mother-to-child transmission of HIV.

Based on our preliminary research into this program, it seemed like it could plausibly meet GiveWell's top-charity criteria in the future. But New Incentives was a young organization, and we didn't have quite enough evidence to name them a top charity. Instead, we made an incubation grant to support their work scaling and researching the program that they were implementing.

In the course of the years after we made this incubation grant, New Incentives realized that there weren't quite enough pregnant, HIV-positive women in the populations that it was serving to make that intervention a good fit for them. They weren't reaching enough people with that particular set of criteria. So, they changed the focus of their program to serve all pregnant women. They went from focusing on pregnant, HIV-positive women to try to prevent the mother-to-child transmission of HIV, to encouraging women to deliver in a health facility — whether or not they were HIV-positive — with the goal of reducing neonatal mortality.

In late 2016, GiveWell decided to take a deep dive into the evidence for the impact of delivering in a health facility and think about whether that was something that met our criteria for recommendation. We ultimately concluded, after looking at the evidence then available in that space, that we did not believe that incentivizing facility delivery would meet GiveWell's criteria.

We had a very positive impression of New Incentives as an organization after working with them for a few years. So, we provided an additional incubation grant to support their work to switch into a new program — also a conditional cash-transfer program — to incentivize routine immunizations in Nigeria. GiveWell has long been interested in immunization programs, and we thought that this might be a way in which New Incentives could meet our top-charity criteria in the future.

We've also provided funding for a randomized control trial of this work, which New Incentives began in 2017. We are on track to potentially consider New Incentives for a top-charity recommendation in 2020. The timelines on these incubation grants can be quite long, particularly if they involve some twists and turns along the way. But we think it's a really exciting situation when we find an organization that we think is aligned with GiveWell, very interested in evidence, and working to implement effective programs. We can work with them to scale and learn from their programs. We're currently in the process of conducting a randomized control trial on their work, and we hope to come back to them in 2020 or so, depending on when we have sufficient evidence to consider them for a top-charity recommendation.

The next organization that I wanted to talk about is a much more recent incubation grant. We made this grant in July of this year, and we're very excited because we think it might help GiveWell solve a problem that we frequently have come up upon in our history. As I mentioned, the first step of our research process is looking at program research — academic evidence that a particular way of helping people is an effective one, independent of a charity.

The “independent of a charity” part is where we can sometimes get into tricky situations. We'll find a program that, based on the academic studies that have been done of that program, we think looks really positive. We’ll say, "Great. We are excited to consider moving forward with a charity that works in this program." Then, we find out that there isn't a charity that works in that program and we're stuck. Since GiveWell is a recommender of charities, we can't really move forward without a charity to recommend.

So, Evidence Action Beta is itself an incubator that is, with this grant, creating a GiveWell-focused portfolio of promising programs that they are going to test, prototype, and scale.

This means we’ll now be able to chat with Evidence Action Beta about programs that we've identified and would like to see charities working in. And hopefully, we will ultimately have new top charities as a result of Evidence Action Beta's work. We also think Evidence Action is well-positioned to run this type of program, as they are currently the parent organization of two of GiveWell's top charities and one of our standout charities.

The final grantee that I want to discuss is a group called IDinsight.

They perform a different function than the other two groups I've talked about so far. We think they will help us fill in some gaps holding us back when we conduct research.

As I mentioned, a lot of the research that GiveWell looks at comes from academia. That frequently means big, academic randomized control trials. Those are very expensive, and they don't always provide a lot of information about the specific context in which charities operate. Maybe the study was done in one country, but the charities we're considering work in very different countries.

IDinsight specializes in providing decision-relevant research information to policymakers and decision-makers, such that they can run a low-cost randomized control trial in a specific context to help obtain decision-relevant information. We think that's a unique role in the development space, since academic incentives aren't always perfectly aligned with the information that would be most helpful to GiveWell. So, we've made a grant to IDinsight to import an embedded GiveWell team to help us answer some of our biggest research questions. IDinsight has been working with us on a number of projects through the years.

I mentioned the group New Incentives earlier. IDinsight is running the randomized control trial of New Incentives' conditional cash transfer program to incentivize immunization.

IDinsight also helped us answer some questions that we had about one of our current top charities, the Against Malaria Foundation. We realized that we didn't fully understand how they were doing their monitoring. IDinsight performed some site visits on the ground, watched Against Malaria Foundation conduct their monitoring, and reported back to us. That was extremely valuable to GiveWell. We're a team that is primarily based in San Francisco and focuses on synthesizing existing research, so having a research partner that can collect information on the ground to help us answer important questions is an exciting development for us.

To that end, one of the projects that we completed with IDinsight this year could help us answer a particularly tricky question that GiveWell faces when we're trying to make charity recommendations. I mentioned a few of the different programs that our top charities work on. Distributing insecticide-treated nets and cash transfers are some examples. It's hard to know how we should value those things relative to one another. But we are coming up with a list of bottom-line recommendations. We need to think about which is the most cost-effective, which has the most pressing funding needs, and where we should direct dollars today. We need to make trade-offs. How do we value a year of increased income relative to a year of healthy life? That is very challenging to do.

One of the [most salient] pieces of information that we feel is missing and affecting our ability to do that concerns how people in the areas that GiveWell's top charities operate in — in other words, people living in the poorest parts of the world — would make those trade-offs. We've looked for that information, and we just don't believe there is a lot of research on it available.

We asked IDinsight to conduct a pilot survey of people living in populations similar to those that receive benefits from GiveWell's top charities. The survey asked those people [living in the poorest parts of the world] how they might evaluate those different things and make trade-offs to better inform the way we make funding decisions and recommendations. That project is still under way. We don't have a final conclusion yet, but we're hopeful that the information we receive will help us make better decisions.

Finally, I want to provide context around where we see GiveWell going in the future.

Many people know GiveWell as a rigorous, randomized-control-trial-focused organization that makes strongly vetted recommendations. But we're becoming more open to [interventions] that are a bit riskier, with cases for impact that might be harder to measure. We hope that GiveWell Incubation Grants can help us answer some of the challenging questions that come up as we expand the scope of intervention types that GiveWell explores.

We're planning to do all of this with our typical healthy dose of transparency. You can see the full list of incubation grants that we've made so far. There are about 30 of them, so the three that I mentioned [comprise] a very small fraction of the total number of incubation grants that we've made. We plan to write about the case for each grant, why we've decided that it's a good bet for us to take, what we think the probability of success is, and which outcomes we're going to [assess] to determine whether or not the grant was a success. We'll share all of that on our website so that we can continue to bring the knowledge that we have to the people who are using our research, and who are interested in effective giving and effective interventions in global health and development.

In addition, one of the goals that we have for this program, and GiveWell as a whole, is to build incentives for people to generate cost-effective charities. We think that we are somewhat uniquely positioned to provide funding through an organization’s lifespan. With the Incubation Grants program, we might be able to fund an early-stage organization, help it scale, study its program, help fund monitoring (if that is an obstacle to us recommending something), and then, ultimately, move it to the GiveWell “top charities” category. From there, the many people who use and rely on our recommendations could direct significant amounts of funding to it. We hope that this is a helpful incentive and gives us more of a role in the full lifespan of a charity and its funding.

Nathan Labenz [Moderator]: Thank you, Catherine. I've been following GiveWell for a number of years and have always admired how meticulous the research and write-ups are. Have there been many new slam dunks since the organization recommended that first wave of top charities?

Catherine: Yes. There was a period of time, from about 2011 to 2015, when we had three or four of the same top charities. That was the period when GiveWell’s Incubation Grants program was getting started. We had the Against Malaria Foundation distributing insecticide-treated nets, GiveDirectly distributing cash transfers, and two deworming charities (depending on which year): Deworm the World Initiative and Schistosomiasis Control Initiative.

We now have nine top charities. Two of them work on deworming as well, and the rest work on other interventions. We've added a vitamin A supplementation charity, a seasonal migration subsidy charity, and another malaria charity that works on a different intervention of distributing preventive antimalarial drugs. The list hasn't been quite as stagnant as it was during the period when our Incubation Grants program was just getting started, but we do think these have all been within a similar global health and development context. There are a lot of direct delivery interventions. And they've all been in a fairly similar band of cost effectiveness.

We generally consider our top charities to be roughly four to 10 times as cost-effective as GiveDirectly, which is the cash transfer charity that we use as a benchmark. So we are particularly interested in finding charities that could beat that degree of cost effectiveness by a significant amount. We're looking into a lot of different areas, even if they might be much harder to assess. One thing we're looking at is policy advocacy organizations, which we think have the potential to be quite cost-effective.

Nathan: I was going to ask about things that are less trackable and harder to assess. Rob Wiblin did a podcast with Tyler Cowen, in which he argues that maximizing economic growth (subject to certain human rights constraints) helps him determine what a good policy directive is. How does GiveWell think about things like growth, which brings great value but is hard to figure out?

Catherine: It isn't always clear which growth interventions to support. I think for things that may be much more nebulous, like systems change, the core challenge as a charity recommender and a funder is deciding where to put dollars to have an impact. While we are expanding into things like policy advocacy, we remain committed to publishing very vettable recommendations in which the full logic behind those recommendations is available.

We're not yet at the point where we are ready to say, "Just trust us — we're going to try this out-there [intervention even though] there isn't much evidence." We're still quite far away from that, and still very committed to evidence. But we're committed to looking at evidence bases that are more complex than those we’ve considered in the past.

Nathan: A question has come in on the topic of deworming. Would it be highly effective to fund randomized controlled trials on deworming, given that there seems to be so much uncertainty?

Catherine: One of the incubation grants that we made in the past was to support a follow-up study to an earlier deworming study. For those of you who might not be familiar with the case for deworming, GiveWell's recommendation rests on research suggesting that children who receive deworming treatments have much higher incomes later in life. There are a lot of questions and debates about this particular set of evidence.

There are two challenges with running a new randomized control trial today. One is the timespan. Since the effects that we're particularly interested in happen decades later, we wouldn't see results that could answer this specific question for quite a long time. The other challenge is just the size of the study that would be needed to have the statistical power to answer the question. It would be a very expensive study as well.

We've primarily been interested in exploring whether there are existing deworming studies that we could follow up on today — studies done far enough in the past to allow us to determine how those children are faring. That's quite challenging in and of itself: to track people down who participated in studies in the past and see how they are doing today. But I think that would be our best guess as to the way to go about this, given those other challenges with running a new study.

Nathan: There are a number of questions coming in; we won’t have time to get to them all. Here's a good one: Should we donate now or later? Should we expect that better opportunities will be just around the corner given all the work that GiveWell is doing?

Catherine: That’s a good question, because we are trying to find things that are significantly more cost-effective. That being said, we're pursuing a lot of new areas, so we are not yet sure what we will find. We could uncover opportunities that are much more cost-effective or similarly cost-effective. Or we could find things that are not more cost-effective, and determine that today's opportunities are the best.

We do think that the opportunities we have today represent very evidence-backed, excellent ways to do good. We are extremely excited for people to support them and believe that they have very large funding needs, on the order of several hundreds of millions of dollars that we don't expect to fill with our current donor base. So, as excited as we are that we moved $117 million to our top charities in 2017, we think that their needs are even greater. We certainly see a strong case for giving today.

But we may well find things that are more cost-effective in the future. That is certainly one of our goals. I hope that level of uncertainty isn't too dissatisfying, but I think it represents the truth

Nathan: What is the hit rate that you think you will find on these incubation grants, and how speculative should a [grant recipient] be?

Catherine: We don't yet know the hit rate. Because of the timelines that I mentioned, we just haven't seen many incubation grants go through their lifecycle to the point where we would end up with a top charity recommendation, since it's a multi-year process. We started making incubation grants in 2014, but have really ramped it up in recent years, so there aren't that many older grants.

In terms of how we think about whether an opportunity is a good one to pursue, we really try to ask ourselves: What is the likelihood this becomes a top charity? How cost-effective does it seem now, and what is the likelihood we can obtain information about the questions that we care most about? We're trying to find things for which there is a plausible case that they will be as cost-effective, or much more cost-effective, than our current top charities. We're taking a lot of similar steps that we take to arrive at our top charity reviews. We look at cost-effectiveness calculations to make our best guess. We spend a lot of time investigating each grant and have plans for following up and publishing what we find on our website, as well as our predictions for the success of each grant and when we might see that success.

For example, I believe on the New Incentives grant page it says, “We think there's an X percentage chance that New Incentives becomes a top charity by 2020 or 2022.” You'll see predictions along those lines for the other grants we’ve made that lay out where we think the organization is and how it fits in.