The Trouble with Nonprofits
In the 1990s, a group of psychologists began studying what made experts expert. Their first task was to see whether experts really were expert — whether they were particularly good at their jobs.
What they found was that some were and some weren’t. Champion chess players, obviously, are much better at playing chess than you and I. But political pundits, it turns out, aren’t that much better at making predictions than a random guy off the street.
What distinguishes people who are great at what they do from those who are just mediocre? The answer, it seems, is feedback. If you lose a chess game, it’s pretty obvious you lost. You know right away, you feel bad, and you start thinking about what you did wrong and how you can improve.
Making a bad prediction isn’t like that. First, it’s months or years before your prediction is proven wrong. And then, you make yourself feel better by coming up with some explanation for why you were wrong: well, nobody expected that to happen; it threw everything else off! And so you keep on making predictions in the same way — which means you never get good at it.
The difference between chess and predictions is a lot like the difference between companies and nonprofits. If your company is losing money, it’s pretty obvious. You know right away, you feel bad, and you start thinking about how to fix it. (And if you don’t fix it, you go bankrupt.) But if your nonprofit isn’t accomplishing its goals, it’s much less obvious. You can point to various measurable signs of success (look at all the members we have, look at all the articles we’ve been quoted in) and come up with all sorts of explanations for why it’s not your fault.
This isn’t to say that we should have companies replace nonprofits, any more than we should have chess games replace predictions. The two serve completely different goals — nonprofits aim at improving the world, not making money. But it does mean that if you’re involved in nonprofits (or predictions), you need to be much more careful about making sure you’re doing a good job.
Unfortunately, few nonprofits do that. Take, for example, the Center for American Progress, widely believed to be one of the most effective political nonprofits. They say their goal is “improving the lives of Americans through ideas and action.” But their “marketing brochure,” while filled with glossy photos, doesn’t even attempt to see whether they’re accomplishing this goal. It touts that they’ve released “an economic strategy for the next administration,” “convened a task force … to develop policy,” and “developed a plan for the bulk transfer auction of at-risk mortgages.” There’s not a single attempt to demonstrate that any of these things has approved the lives of Americans, let alone estimate how much.
Measuring things is hard and expensive, even in the simplest cases. Measuring the effect of loaning money to Africans seems a lot easier than measuring the impact of of a think tank report. But when Peter Singer asked Oxfam to measure the effectiveness of giving microcredit to villages in West Africa, they declined, on the grounds that it would have taken up half the budget.
But not measuring is even more expensive. Imagine that Oxfam experimented with two microcredit programs and found that one did 10% better than the other. Even with this very modest improvement, it would only take helping five villages before the experiment paid for itself.
And, as anyone who’s done these sorts of experiments knows, you often see improvements well in excess of 10%. To take a silly example, Dustin Curtis experimented with getting more readers of his weblog to follow him on Twitter. After four experiments, he’d achieved a 173% improvement. And even this is probably underestimating things. I expect many nonprofits are not accomplishing their goals at all. Even if they made a little bit of progress, their improvement would be mathematically infinite. (It’s also quite possible that many nonprofits are actually being counter-productive. After all, before we started measuring the effects of medical treatment, we were bleeding people with leeches.)
What can be done about this? I think that everyone who donates to a nonprofit should demand an accounting of results — not just the number of times they’ve been cited in the media or the number of policy discussions they’ve held, but an actual attempt to measure how much they’re improving people’s lives. For most nonprofits, I expect these numbers will be depressingly small. But that’s much better than having no numbers at all. For feeling bad about failing is the first step to doing better next time.
You should follow me on twitter here.
September 7, 2009
I’ve seen measurement also lead to its own set of dysfunctions. Among educational nonprofits, measurement is constant and the results are reflected quickly in funding. But the result doesn’t look very good to me — there’s more obsession with the form of the measurement and how it can be presented, than with the goals themselves. It is not creating a set of professionals thoughtfully experimenting, reflecting, and using that feedback to adjust their process. They could do this, measurement need not create this effect, but it isn’t right now.
posted by Ian Bicking
on September 8, 2009 #
You can read lots about measuring the impact of charities here: http://www.givewell.net/
posted by Brian Slesinsky
on September 8, 2009 #
To my mind, this isn’t news, but that might be in part because I’ve worked in my family’s grant writing firm for so long. The real issue you’re discussing is one of incentives: nonprofits don’t have much incentive to measure how well they’re doing because funders don’t demand that they do, or at least don’t as a large group.
If the funder is federal, state, or local, the funder doesn’t have the resources (or the brains) to really measure effectiveness; if the funder is a foundation, the funder probably cares more about the appearance of doing good than effectiveness. I’ve addressed this point in some detail here: http://blog.seliger.com/2008/01/04/more-on-charities and here: http://blog.seliger.com/2008/01/23/foundations-and-the-future . The blow, which I contribute to, deals with the issues you raise on a regular basis, and if you want to learn more about why things are the way they are at the moment, take a look at the archives.
posted by Jake
on September 8, 2009 #
And of course, governments are the biggest nonprofits of all.
posted by Gordon Mohr
on September 8, 2009 #
Having both founded a non-profit and helped foundations give away money to non-profits, I think the situation is a bit more complicated and nuanced than portrayed here.
“Non-profits” and “companies” are huge categories. Some non-profits have very specific goals (let’s say improving handicapped accessibility to public facilities in a limited geographic region) which are at least as measurable as those of your average for-profit companies.
And some companies have very different metrics of “success” than others. Over the past 15 years, many larger companies have been deemed to be underperforming if they weren’t growing exponentially; but a Mom-and-Pop store on a main street does not need to do more than survive and provide a living to its handful of employees.
Some companies rate themselves based solely on relatively concrete measures such as profitability and market share, while others include softer criteria (such as image, customer and employee satisfaction).
Taking the Twitter example, one could easily poke holes in even a touted 173% “improvement” in the number of people following a blogger on Twitter. Did the increase in Twitter followers lead to more people actually visiting the blog, or did it become a subsitute for reading the full post? Did readers spend less time on the site as a result of being able to access the Twitter summary? Over time, did the increase sustain itself, or did people un-follow the blogger after the initial spate of interest? Did the push by the blogger merely hasten the acquisition of Twitter followers which would have happened without the big push, increasing the blog’s number of followers to the natural number it was going to achieve anyway, or did it get people to follow the blogger who otherwise wouldn’t have?
Without a doubt, some non-profits — like many businesses — do not actually reach their goals. But I would argue from experience that non-profits do get held accountable over the long haul to much the same extent as commercial enterprises, maybe even moreso. Donors and members have to be rationally convinced to part with their dollars for charity, and when the “product” is less tangible (e.g. improving liteacy in a distant country) that case can be hard to sustain year-after-year even when the organization is doing great work.
And so on. I appreciate the basic point of this post, but I do believe the profit/non-profit divide is less black-and-white than portrayed.
posted by Sam Pratt
on September 8, 2009 #
P.S. I meant to add that in my experience, serious grantmakers giving out more than small sums do in fact tend to demand that applicants for funding include metrics for evaluation, and that future largesse is dependent on a credible demonstration that these goals are met. And some funders are reluctant to fund the same projects year-after-year even when the project is demonstrating success, just on general principle. I think those outside of the non-profit sector tend to overestimate the challenge of funding such entities.
posted by Sam Pratt
on September 8, 2009 #
Typo: overestimate >>> underestimate
posted by Sam Pratt
on September 8, 2009 #
Lots of people are working on this:
http://en.wikipedia.org/wiki/Esther_Duflo
http://en.wikipedia.org/wiki/William_Easterly
Of course most political pressure groups can’t measure the effects of their actions, and even if they did have direct policy changes they could point to, it’s a delicate matter to claim credit and keep their relationships with the political class. Success has many fathers.
posted by Gabe
on September 8, 2009 #
This is the sole reason why I chose to donate to Kiva.org over some other options. The repayment of funds is direct feedback that my donation actually worked.
posted by haig
on September 8, 2009 #
Spot on. One shouldn’t lump together all non-profits, however, as some have been speaking out about this for quite some time. For example, Acumen Fund’s Jacqueline Novogratz: http://en.wikipedia.org/wiki/Jacqueline_Novogratz
posted by Andrey Fedorov
on September 8, 2009 #
Steve Jobs talked in an interview about that, The fact that he can measure if one of his ideas works or not just trying it, and some doesn’t work(like the Apple Newton).
He just let the people decide. With non profits, he said, you can’t know if you are expending good the money.
posted by Jose
on September 8, 2009 #
Some of my colleagues who worked for educational non-profits in downtown DC noticed that when they turned to measured metrics (and this reminded me of my own experience in the UK with research funding) the organization became too cautious with high risk initiatives. Instead they tried more incremental improvements or didn’t attempt to try something interesting at all.
I’m a great believer in measuring what can be measured but there’s also a reason why the MacArthur Genius Grants arrive with non-strings attached. No metrics, high risk, but the payoff can be big. This is why I think a lot of non-profits are reluctant to do metrics, they worry it will limit their opportunities to do something high risk.
posted by Paul Guinnessy
on September 8, 2009 #
“But not measuring is even more expensive. Imagine that Oxfam experimented with two microcredit programs and found that one did 10% better than the other. Even with this very modest improvement, it would only take helping five villages before the experiment paid for itself.”
Using the assumptions from this paragraph and the previous and Oxfam’s actual expenses shows that it would take more than seven years in the most optimistic calculation for Oxfam to come out ahead by deciding to measure. And if the new microcredit program actually turns out to be worse than the standard, Oxfam has lost half a year’s budget for no gain.
“And, as anyone who’s done these sorts of experiments knows, you often see improvements well in excess of 10%.”
How many people do you know who have done experiments in microcredit? or if you meant just experiments in general, I think it would be much more effective for you to take a a year or two off from your current diversions to (easily) develop a strategy to generate 10% excess returns on Wall Street and then pump your financial windfall back into your pet causes.
posted by Mark
on September 9, 2009 #
This post conjures up a good double bill: Adam Curtis’ THE TRAP, and THE WIRE: measuring, targets, non-profit institutions, gaming the system.
I’m still mourning the loss of the community centre at the end of my street. It seems someone decided it should be run like a private fitness club so that it’s efficiency could be measured. No one seemed to have closely interrogated the business model of private fitness clubs, or considered the centre might have immeasurable social value. The centre failed to be ‘profitable’ and closed down.
I live in a mostly lower income, black neighbourhood. We have undoubtedly spent the money we saved on the centre on healthcare and police. I believe the plan for the future is to turn it into luxury condos. Perhaps after that we can turn the housing estate into a fitness centre ;-)
posted by Brad Bell
on September 10, 2009 #
A similarity I’ve noticed among top administrators in D.C.-based nonprofits is that they are unable to refrain from moralizing, which should be unnecessary, as a good plan is quantifiably robust.
posted by Pablo
on September 11, 2009 #
You can also send comments by email.
Comments
I’ve seen measurement also lead to its own set of dysfunctions. Among educational nonprofits, measurement is constant and the results are reflected quickly in funding. But the result doesn’t look very good to me — there’s more obsession with the form of the measurement and how it can be presented, than with the goals themselves. It is not creating a set of professionals thoughtfully experimenting, reflecting, and using that feedback to adjust their process. They could do this, measurement need not create this effect, but it isn’t right now.
posted by Ian Bicking on September 8, 2009 #
You can read lots about measuring the impact of charities here: http://www.givewell.net/
posted by Brian Slesinsky on September 8, 2009 #
To my mind, this isn’t news, but that might be in part because I’ve worked in my family’s grant writing firm for so long. The real issue you’re discussing is one of incentives: nonprofits don’t have much incentive to measure how well they’re doing because funders don’t demand that they do, or at least don’t as a large group.
If the funder is federal, state, or local, the funder doesn’t have the resources (or the brains) to really measure effectiveness; if the funder is a foundation, the funder probably cares more about the appearance of doing good than effectiveness. I’ve addressed this point in some detail here: http://blog.seliger.com/2008/01/04/more-on-charities and here: http://blog.seliger.com/2008/01/23/foundations-and-the-future . The blow, which I contribute to, deals with the issues you raise on a regular basis, and if you want to learn more about why things are the way they are at the moment, take a look at the archives.
posted by Jake on September 8, 2009 #
And of course, governments are the biggest nonprofits of all.
posted by Gordon Mohr on September 8, 2009 #
Having both founded a non-profit and helped foundations give away money to non-profits, I think the situation is a bit more complicated and nuanced than portrayed here.
“Non-profits” and “companies” are huge categories. Some non-profits have very specific goals (let’s say improving handicapped accessibility to public facilities in a limited geographic region) which are at least as measurable as those of your average for-profit companies.
And some companies have very different metrics of “success” than others. Over the past 15 years, many larger companies have been deemed to be underperforming if they weren’t growing exponentially; but a Mom-and-Pop store on a main street does not need to do more than survive and provide a living to its handful of employees.
Some companies rate themselves based solely on relatively concrete measures such as profitability and market share, while others include softer criteria (such as image, customer and employee satisfaction).
Taking the Twitter example, one could easily poke holes in even a touted 173% “improvement” in the number of people following a blogger on Twitter. Did the increase in Twitter followers lead to more people actually visiting the blog, or did it become a subsitute for reading the full post? Did readers spend less time on the site as a result of being able to access the Twitter summary? Over time, did the increase sustain itself, or did people un-follow the blogger after the initial spate of interest? Did the push by the blogger merely hasten the acquisition of Twitter followers which would have happened without the big push, increasing the blog’s number of followers to the natural number it was going to achieve anyway, or did it get people to follow the blogger who otherwise wouldn’t have?
Without a doubt, some non-profits — like many businesses — do not actually reach their goals. But I would argue from experience that non-profits do get held accountable over the long haul to much the same extent as commercial enterprises, maybe even moreso. Donors and members have to be rationally convinced to part with their dollars for charity, and when the “product” is less tangible (e.g. improving liteacy in a distant country) that case can be hard to sustain year-after-year even when the organization is doing great work.
And so on. I appreciate the basic point of this post, but I do believe the profit/non-profit divide is less black-and-white than portrayed.
posted by Sam Pratt on September 8, 2009 #
P.S. I meant to add that in my experience, serious grantmakers giving out more than small sums do in fact tend to demand that applicants for funding include metrics for evaluation, and that future largesse is dependent on a credible demonstration that these goals are met. And some funders are reluctant to fund the same projects year-after-year even when the project is demonstrating success, just on general principle. I think those outside of the non-profit sector tend to overestimate the challenge of funding such entities.
posted by Sam Pratt on September 8, 2009 #
Typo: overestimate >>> underestimate
posted by Sam Pratt on September 8, 2009 #
Lots of people are working on this:
http://en.wikipedia.org/wiki/Esther_Duflo
http://en.wikipedia.org/wiki/William_Easterly
Of course most political pressure groups can’t measure the effects of their actions, and even if they did have direct policy changes they could point to, it’s a delicate matter to claim credit and keep their relationships with the political class. Success has many fathers.
posted by Gabe on September 8, 2009 #
This is the sole reason why I chose to donate to Kiva.org over some other options. The repayment of funds is direct feedback that my donation actually worked.
posted by haig on September 8, 2009 #
Spot on. One shouldn’t lump together all non-profits, however, as some have been speaking out about this for quite some time. For example, Acumen Fund’s Jacqueline Novogratz: http://en.wikipedia.org/wiki/Jacqueline_Novogratz
posted by Andrey Fedorov on September 8, 2009 #
Steve Jobs talked in an interview about that, The fact that he can measure if one of his ideas works or not just trying it, and some doesn’t work(like the Apple Newton).
He just let the people decide. With non profits, he said, you can’t know if you are expending good the money.
posted by Jose on September 8, 2009 #
Some of my colleagues who worked for educational non-profits in downtown DC noticed that when they turned to measured metrics (and this reminded me of my own experience in the UK with research funding) the organization became too cautious with high risk initiatives. Instead they tried more incremental improvements or didn’t attempt to try something interesting at all.
I’m a great believer in measuring what can be measured but there’s also a reason why the MacArthur Genius Grants arrive with non-strings attached. No metrics, high risk, but the payoff can be big. This is why I think a lot of non-profits are reluctant to do metrics, they worry it will limit their opportunities to do something high risk.
posted by Paul Guinnessy on September 8, 2009 #
“But not measuring is even more expensive. Imagine that Oxfam experimented with two microcredit programs and found that one did 10% better than the other. Even with this very modest improvement, it would only take helping five villages before the experiment paid for itself.”
Using the assumptions from this paragraph and the previous and Oxfam’s actual expenses shows that it would take more than seven years in the most optimistic calculation for Oxfam to come out ahead by deciding to measure. And if the new microcredit program actually turns out to be worse than the standard, Oxfam has lost half a year’s budget for no gain.
“And, as anyone who’s done these sorts of experiments knows, you often see improvements well in excess of 10%.”
How many people do you know who have done experiments in microcredit? or if you meant just experiments in general, I think it would be much more effective for you to take a a year or two off from your current diversions to (easily) develop a strategy to generate 10% excess returns on Wall Street and then pump your financial windfall back into your pet causes.
posted by Mark on September 9, 2009 #
This post conjures up a good double bill: Adam Curtis’ THE TRAP, and THE WIRE: measuring, targets, non-profit institutions, gaming the system.
I’m still mourning the loss of the community centre at the end of my street. It seems someone decided it should be run like a private fitness club so that it’s efficiency could be measured. No one seemed to have closely interrogated the business model of private fitness clubs, or considered the centre might have immeasurable social value. The centre failed to be ‘profitable’ and closed down.
I live in a mostly lower income, black neighbourhood. We have undoubtedly spent the money we saved on the centre on healthcare and police. I believe the plan for the future is to turn it into luxury condos. Perhaps after that we can turn the housing estate into a fitness centre ;-)
posted by Brad Bell on September 10, 2009 #
A similarity I’ve noticed among top administrators in D.C.-based nonprofits is that they are unable to refrain from moralizing, which should be unnecessary, as a good plan is quantifiably robust.
posted by Pablo on September 11, 2009 #
You can also send comments by email.