Comments on: Charity Evaluation Debate

By: Holden

Holden — Fri, 30 Mar 2007 02:17:35 +0000

Yes, existing metrics are terrible, because not enough thought (good thought incorporating emotion, not just mathematical thought) has been put into them. That’s exactly why we need more focus on metrics - we need to do a better job with them.

I think I now understand what you mean by a formal qualitative system. You’re saying that reviewers ought to publicly publish ratings, that are based on their subjective take on the whole picture rather than a formula. As you say, you should use metrics in forming ratings, but both are valuable.

FWIW, the current GiveWell site uses exactly the type of rating you describe (1-5 subjective ratings for “Strategies and Activities,” “Relevance of your dollar,” etc.)

As with movies, I assert that these numbers would be worthless without the ability to see the rationale behind them. We provide that as well.

I am not saying “problem solved.” Our scope and the quality of our information are a shadow of what they can and should be. And I’d love to see others - foundations, advisors, donors - taking the same approach.

(By the way, “professional” seems irrelevant - again, as with movies. What’s important is that the ratings be understandable and publicly available. I find IMDB movie reviews more useful than NYT ones …)

By: Sean Stannard-Stockton

Sean Stannard-Stockton — Thu, 29 Mar 2007 19:07:54 +0000

Actually there are existing quantitative rating systems (although, as we have agreed, they use poor metrics). But there are no formal qualitative systems. I’m calling for a professional qualitative system similar to equity research. Not for individual donors to do their own evaluation.

These analysts would use metrics of course, but they would get to know the “story” as well.

By: Holden

Holden — Thu, 29 Mar 2007 16:10:10 +0000

Your point about movies is clearer now, and I agree with you in content - just not in emphasis. There are certainly many people who are tilted toward over-measurement, and it is worthwhile to caution against this, as you do. But MANY more people are tilted toward under-measurement. The status quo for 99.9% of donors is to go entirely by the story, with no metrics.

I agree that both the story and the metrics matter. But the latter are what desperately need more thought and focus, for this particular sector at this particular time.