You can find the new thread here. I’d love to have your input.

Now what I’d really like to have everyone comment on is what information you would like to see available on Google Finance. As noted in this thread, Google aggregates and finds information, they do not tend to create their own. So what data should they display on the google finance page?

Rather than continuing to comment on this thread, let’s move this conversation to my new post from today.

I hope to see you all there!

I think that implicit to the conversation about NPO evaluation is a difference of opinion about what constitutes an adequate representation of an entity, whether it’s an individual, a group, a culture and that in order to proceed it’s important to flesh out the idea of identity. In this case, the NPO’s.

For example, I visited a homeless shelter near my house that’s been around for over 10 years and gives homeless people free breakfast and some housing. I asked the director, a Unitarian minister, how he measures his effectiveness. I had expected him to say something about the number of people he’s helped find jobs, or the number of lunches they hand out, etc. Instead, he replied simply, “Last week one of our (homeless) regulars died. And 10 people, who he’d gotten to know here, showed up for his funeral. That’s how I measure our effectiveness.” That sobering encounter really made me think hard. Many nonprofits are trying to make a difference in people’s lives and people are not products. We are complicated – changing attitudes, ideas and behaviors could take years and it’s hard to isolate which one factor contributed to any specific result. It’s much more complicated than running a business, where it comes down to profits, units sold, or in the case of google’s culture, milliseconds and bytes.

The challenge of evaluating a nonprofit using metrics, is similar to asking a parent – what kind of metric do you evaluate yourself on whether you are a good parent? Whether your kid eats green vegetables? Whether your kid makes it onto the basketball team? Whether your kid gets into an Ivy League college? It seems intuitive that none of these metrics adequately reflect adequately whether we have been good parents or not. We may look to some of these things, but then we also use our judgement – is the kid happy and well-adjusted? And even then, we may not know for years whether we succeeded as a parent until we see our kids grown up.

Sociologists struggled with the evaluation question in the 1960’s when evaluating government social programs and most gave up on idea that you could feasibly use scientific methods to evaluate nonprofit programs. MRDC does some great evaluations using control groups and random selection, but there’s very very few programs that would qualify for the huge expense and time involved (plus overcome the moral dilemma to assign certain people to control category where they don’t get treatment/the program).

A good balance I think is that suggested by Phil Buchanan – every organization should set and state its goals, strategies and indicators. Indicators, in my opinion, can embrace narrative stories of impact, as well some sensible numerical indicators. Stories are just as valuable – sometimes a lot more so because they are able to capture nuances – in information as metrics. And in terms of galvanizing donors and volunteers, the research consistently show that photos and stories are more powerful than just numbers.

I’m enjoying this discussion. Thanks everyone.

Perla Ni
GreatNonprofits

>>>Measurement is information; information is needed in order to establish the reputation of the prespect you want to invest resources (attention, ideas, time, money, contacts) into, whatever your motives, for profit or not. You have to establish their reputation to determine to what extent you can trust them – that is, what the risk involved is; what’s the probability that your investment will not have the desired effect (through incompetence, poor luck, or fraud)?

So the general call to arms here is correct – we need more information; measurement is quite a scientific (and thus,usually, more reliable) method of obtaining, displaying and weighing that information, and fairly and reliably converting into a metric for reputation (though other factors come into play – like the brand of the organisation, and the overlap of their values with your own – e.g. caring about the same issues you do).

So yes, I feel we SHOULD get obsessive about hard metrics. Other means of determining reputation – expert’s analysis, or the aggregation of popular consensus, are faulty. Only then do we get a relatively trustworthy estimate of the expected return on investment. Note, however, that talk of overhead, financial ratios, etc, is a misleading obsession. At best, these tell you about the company’s fundamentals – i.e. all you learn is that it is/isn’t likely to go bust, taking your investment with it.

It’s more important for an organisation to develop its own KPIs (key performance indicators) – which it should be doing anyway! – and then make their expected and actual performance against these KPIs public. These do NOT have to be the same metric as a neighbouring organisation, but it helps, since it allows us to make sure we’re being fair in converting these to reputation (since they don’t need interconverting and thus weighing)

HOWEVER, if, as some here have argued, the institutional model of philanthropy cannot effectively deliver these metrics, then the alternative is to cut out this middle man, get the donor (investor) closer to the point of impact – i.e. directly involved in how the money is spent, so he can judge for himself, first hand, how effective his investment is. This requires a huge shakeup in the organisation – even the basic philosophy – of how we do things – of a similar magnitude to the revolution UGC (in the form of youtube etc) is delivering to traditional media. it COULD happen. It has its obvious downsides, as does Youtube-style media production. But it is a very important couterpart to the traditional way of doing things, if done correctly.

but non profits get subjected to intrusive measurement for placement of money all the time with no assurance that if they pass they really will get something. the process for a for profit investment seems much more fair. our analysis has a cost. we are sure there can be a materially positive outcome if you pass this test. and we won’t change the test, though we will sample the product and the team and the process and the social impact, and the scalability, etc. on all levels that we find, from the quantitative to the qualitative, subjective. we turned down one recent investment prospect that made financial sense when, after a three hour meeting with the team, i asked our folks, so who wants to come up here and spend a minimum of a day and a half with these guys during a board meeting and talk to them 10 to 15 times a week, send dozens of emails a week on their behalf and work to open doors for them to reach their potential? no one wanted to invest our time in them. it was not a good fit. and that was a good reason to say no, validated by other, more quantifiable things.

metrics has a cost. realize the cost before you impose them.

IMO, an organization that has been effective in the past, per dollar spent, is likely to be effective with additional donations.

but that’s not the right frame for this. metrics as conceived here on this thread is in that small sphere of activity where the equations are linear; inputs equal outputs. that works if you constrain the problem in a way that eliminates other dimensions of impact, unintended consequences, etc. this is game where the imposition of metrics does not cause more money to flow, it only hampers operations at the behest or whim of imperialistic, they know best funders. any valuable metrics in this space would be a dialogue not a reporting to an all powerful funder.
development as freedom by Sen and Sabina deals with the power dynamic implicit behind all talks of metrics. it is easier to see in international development, where the power relationship has a colonial political base, but it is an underlying assumption behind nearly all of the comments on metrics on this debate, i think.

look at the power issues behind your demand for metrics. think how it would be to be on other side of that intrusive tool. would it help you work better or with more commitment? or would it be a braying loudspeaker from some big brother funder that made you stop doing what you were doing and report to big daddy, the company store?

Additionally, investments in a public corporation are instantly and daily vetted by the price action in the stock market, which means you also instantly know what everybody else is thinking. I wonder if there would be a way to create something similar for non-profits? Rather than an annual review of their 990, or other such slow-moving analysis.

As for metrics, I think you could compare metrics within categories. Such as homeless served per dollar, assuming you can also compare the quality of the care.

Most people are aware of the numerous ways in which schools class children and the role that this sorting plays in the reproduction of society (Bowles and Gintis, Illich, etc). Many smart and dedicated people who are concerned with the problem have spent a lot of time and energy designing and experimenting with alternatives. Although there have been some noteworthy reforms (i.e. the introduction of state-wide standards such as the Vermont Curriculum Standards, the creation of new tools such as rubrics and the student portfolio and the National School Reform’s retraining of teachers as “critical friends”), the general consensus among educators is the following: there is no silver bullet for school reform.

There is no single metric which can be applied to a student, because there is no model person and no given path to any particular end. That is not to say that standardized tests are of no use, but rather that there is no single tool or collection of tools that can tell us everything that is important about an individual. Likewise, there is no standardized collection which tells us all we need to know about an NGO. Because NGO’s are as diverse as people and there a innumerable ways in which they can be effective.

Evaluation is best considered as a process, not a score, a process of give and take between the evaluator and the evaluated. The tools that comprise the process need to be multiple, varied, frequent and responsive in order to accommodate an organization’s difference and growth organizations and the ever-changing values of society.

Furthermore — and Kevin makes this point beautifully — it is critically important that the evaluating process not harm the evaluated. On the contrary, in my mind, a useful test is one that causes the test taker to learn something — a prompt towards self-evaluation and self-directed growth.

NGO’s, like schools, are in the people business, which requires a combination of clarity of mission, rigorous self-evaluation and reporting and constant revision and retelling of the story to garner support. The evaluation process needs to respect the organic nature of the work and support, rather than hinder, its development.

We shouldn’t let pressure (public or market) force us to provide a product that the sector doesn’t call for. Instead, we should try to teach the public how to conduct due diligence and encourage NPO’s to turn the public’s need to know into an opportunity for improved governance.

my main point on which sabina are totally in synch is that metrics need to be multidimensional to be valid. the linear metrics being expressed on this thread really bore the heck out of me. this kind of thinking has been to little effect for decades. check out early jed emerson.

it’s an epistemologically impoverished frame to impose a manufacturing metaphor, which is what most of this thread does, on something that has many more than input and out dimensions.

the first thing metrics have to prove is that their imposition does no harm. stop and think before you stick some crazy metric dashboard into an enterprise. most of the time metric dashboards are killed by “friendly fire” from the people on whom they are imposed; people don’t cooperate with the measurement imposed or they feed it garbage or find a way to work around it.

I agree with the idea of measuring a nonprofit by its stated mission, strategies and outcomes.

Additionally, I would look to see how a nonprofit implements its mission internally and externally. Basically, does an organization practice what it preaches? If the issue is ending poverty, does the nonprofit pay living wages to its employees? Does it support economic development projects that hire local residents and pay decent wages? If the issue is access to health care, does it provide health care benefits to its employees, does it have paid sick leave?

Looking at the mission and values held and implemented by a nonprofit can go a long way toward interpreting the “cost per unit of service” figure.

Sean, thanks for taking this on and good luck with Google.

My gut feeling is that a lot of the things that charities do, with good intentions, don’t necessarily have much impact. Even if we say that charity X ran their program for Y people with Z dollars of funding, those metrics are moot if the program in question doesn’t work.

I assume that these issues are discussed in somewhat more rigorous form (randomized trials, or at least multi-factor outcome studies), and hopefully there is at least a degree of consensus among experts about what works and what doesn’t. But where can I, a non-academic, read about what really works and doesn’t?

I’ve read/skimmed a few books on aid, but they tend to be focused on government aid, rather than what private charities do, and/or their authors are such strong advocates of relatively extreme positions that I question their summaries of existing research.