Holden is outraged. He believes strongly that metrics are critical. In response to Phil Cubeta telling him “Only a madman would dream of measuring the good, all good, on a single universal scale.” Holden responds:

Will you admit that there are good things about modernity? … Forget about a metric to measure "GRAND TOTAL GOOD ACCOMPLISHED" - how about one that measures "Number of indebted people who were able to become self-sustaining thanks to a loan?", and another that measures "Number of diarrhea-ridden children whose lives were saved by medication? … Measuring good is incredibly hard, but more viable if you keep in mind why the numbers are there and what their limitations are. It can be human, and more importantly it allows a form and scope of teamwork that would be impossible without it.

Albert Ruesga responds to Holden:

I think some metrics accomplish what you say for some projects. When a youth development program starts costing $25,000 per young person per year, might it not be better to sock that money away into college accounts for the participants? I don’t know. So many axes of value to compare — quality of instruction, intensity of intervention, paucity of other opportunities: how do we assign weights to these? We can never compare two youth development programs the way we compare two cans of soda: they differ in too many respects.

And Holden disagrees with my movie critic analogy:

The fundamental difference between charities and consumer products such as movies and restaurants is that the mere fact of being a charity’s customer doesn’t mean you know ANYTHING about it. If we want to separate the good charities from the bad, I simply see no way around doing difficult, costly, painful, formal evaluations.

I think that Holden is right, that as a customer/donor of a nonprofit, you can’t simply judge their effectiveness based on your experience. But that’s not what I’m suggesting. You can “experience” every aspect of a movie, simply by watching it. To judge a nonprofit, you need to do much more proactive analysis. Talk to the board, to staff members, the people served, and competitors in both the nonprofit and for profit space. Part of this qualitative analysis is going to include researching certain metrics relevant to the nonprofit in question. I think examining how cost effective a nonprofit behaves is critical, but the evaluation needs to qualitatively take into account the industry the nonprofit is operating in.

Laura Quinn, the founder of Idealware (you can read a background interview with her on Beth Kanter’s blog), left me a comment saying:

I have deeply conflicted thoughts about this. One the one hand I agree and would go much further than you have about administrative expense ratios. These are not only ineffective, they’re *damaging* to the sector…

On the other hand, if you can’t measure in a way the good you’re doing in the world, I would question what good you’re actually creating. This doesn’t mean sum it up into one tidy little figure – evaluation is a difficult science, but far under-used. Many in the nonprofit sector have a tendency to operate based on an instinct that they’re doing good, and these instincts can definitely be wrong…

My point is that it’s tempting to rely on a “gut check” for how useful something is, but guts can be wrong, and they’re certainly drawn to flashier programs rather than things that are just quietly working…

My training is as a Chartered Financial Analyst charterholder. Every day I analyze for profit companies as investment prospects for my clients. We examine an untold number of metrics, but we also learn the “story” of the company. Best Buy isn’t just a company with a 5.48% operating margin, it’s a company that believes deeply that selling consumer electronics is becoming a service business, not just a retail business. Costco isn’t just a company that took in $62.42 billion in revenue over the past four quarters, it is a company that searches high and low for the very best quality/value propositions to offer to their customers and believes that paying their employees much more than their competitors is critical to the company’s profitability.

The metrics are important, but you can’t find the best nonprofits (or for profits) by simply measuring metrics. You need to know the story, the core beliefs that are driving the company’s behavior. Just as it can be far too tempting to buy the “hottest” stocks, it can be tempting to assume that the nonprofit that throws a great charity ball or that has the cutest pictures of kids on its website is a good place to donate. This is Laura’s point when she warns against “gut checks”. But on the other hand, you can’t get lost in the metrics, you’ll never find a magic number or set of numbers that will tell you what you need to know.