But I think you’re missing my point with DALYs. Movie ratings are not just qualitative, they rate the movie as a whole rather than aggregating sub-metrics which are suppose to be elements of what make a movie good.

The foundations you cite are not trying to impact DALYs, they are trying to improve quality of life. Your nonprofit improves quality of life without impacting DALYs. That’s the problem.

A good movie can be great with no award winning actors. Yet potential movie goers sometimes will see a movie because it has a great actor in it. The existence of the great actor can be an indicator of a good movie. But there are LOTS of great movies without great actors.

My point is that focusing on a metric like DALYs can make a funder miss the whole point. They are trying to fund great nonprofits that further their mission, but impact metrics.

My wish is that we focused on rating nonprofits and funneling money towards nonprofits which generate impact, but not get lost in quantitative metrics.

The difference between Rotten Tomatoes rating movies and nonprofits calculating DALYs - and one of the points I’m trying to make - is that Rotten Tomatoes does its aggregation and calculation in a standardized way, the same every time. And this enables consumers to choose between movies that are nearly as good, purely based on a number - realizing that there’s going to be some error, and knowing where the errors are from. But DALYs are subject to variations in methods at each stage of calculation, which is what makes the upper tier of cost-effective applicants so interesting - and potentially so misleading. That’s why I’d prefer foundations avoid the pitfall of pretending DALYs are a catch-all solution to the measurement problem.

A movie like Saving Private Ryan is “good” in a different way than Knocked Up is “good” (both critically acclaimed). Just like saving the whales is good in a different way from teaching children to read. Yet we still as a society have a generally agreement on which movies are good and which aren’t. We don’t even have that basic level of agreement with nonprofits. Ask any person on the street to name 5 great nonprofits and they can only name nonprofits that they have personal experience with or big well known (but not alway “good” nonprofits). Try the same question with for-profit companies and you’ll get names like Starbucks, Apple, and Google again and again).

No one claims that the best selling movies are the “best” because we generally do not equate popularity with quality.

So what about the metric for winning this contest. The point of this contest was not to find the “best idea”, it was to “encourage the philanthropy blog conversation” and to “engage in conversation”. I wanted to use an objective, transparent metric and so I decided that the best way to measure the extent to which a post encouraged a conversation was to measure the number of participants in that conversation.

Not a perfect metric, but an OK one. Think about some sort of rally (like a protest rally), the number of participants might not be the best metric for how much impact the rally had, but it is a decent one (and often used). That at least was my thought process.

Sean, the movie analogy confuses me. There are plenty of X’s (some completely arbitrary) that will separate the very best from the very worst, at least in terms of movies. Even if you throw out things like aggregated scores from Metacritic and Rotten Tomatoes for being essentially collections of opinions, how about tickets sold divided by seats available (or screens available times median theater size, if you don’t want to count the actual seats)? Sure, that reflects popularity, not merit (although the metric for this competition claims that those things are synonymous), but you get the idea. Where metrics for subjective subjects get really interesting is at the very top of the continuum - ask a Brando fan to decide between Streetcar and On the Waterfront, or a comic book geek to decide between Spider-Man II and Sin City. Or, as it applies to nonprofits, ask a foundation to pick between saving a child from contracting malaria or saving two adults from having their legs amputated. This is the point where foundations should be asking about the precision and accuracy of the metrics they ask for, and looking into the uncertainty that creeps in at each step. When they don’t, it becomes a game of which methods a grantee can use to inflate their numerical results, and the ethical applicants lose out.

Am I wrong? I might be. I’m new to this field. Please point out the errors in my thought process; I’m willing to learn.

I don’t have it in front of me, so my recollection may be faulty, but I think Amartya Sen, the Nobel laureate economist, made a compelling argument that DALYs are the wrong measure entirely, that instead the focus should be on what kind of real capabilities people in a society have, in his book “Development and Freedom.” Sounds like the issue with fractures goes right to the capabilities of the person who is treated, and the impact it has on their abilities and life chances, rather than morbidity. If you haven’t read it, his arguments might be helpful.

Thanks for your post.

What is the “X” that lets you know that Casablanca is a better movie than Look Who’s Talking, Too? Or that Star Wars is better than Bio-Dome with Pauly Shore?

I wrote a long post about this issue back in March that you can find here.

I think the knee-jerk response is a gentic tic activated by too many boring philanthropy conferences.

Last week, for example, I had to wipe spittle off my brow when a funder colleague ranted about metrics and impact of a particular program supported by a particular funder collaborative.

“We don’t have any data to show X so nobody will want to fund us.”

She assumed that the “X” was the be all and end all of defining program outcomes.

Before her self-satisfied glow could cause further irritation to other members, we reminded her that there were many other measurable outcomes of the program.

We learned from the experience that we needed to do a better job of communicating the complexity of the program the challenges of its evaluation and the importance of many positive outcomes beyond the magic X.

Philanthropoids often only see what they are looking for.

As a side note, I’m slightly curious what “a fundraiser” is going to do now that Pride at Work has effectively won his or her mini-competition, since they’re ineligible for Network for Good donations. It’ll be interesting to see. He or she may be out $500 of their own money.