A lively debate about nonprofit evaluation and metrics has been raging in response to my request for input on my meeting later this week with Google.org. However, the conversation has splintered into a debate over whether a systematic, “metric” driven process of scientific measurement is needed, or whether the frame of scientific measurement is “an epistemologically impoverished frame” through which to understand nonprofit evaluation.
I personally believe evaluating nonprofits is mostly about evaluating their output (the social good they produce). Since it is difficult (impossible?) to quantify this output, I think the focus on metrics as a framework for evaluation is misplaced. Metrics can be used, but they should be designed on a case-by-case basis for each situation. That being said, I think the conversation has fallen into the trap of being constrained by historical frames of reference.
I want to have a different conversation.
I’m interested in what information is available to donors who want to evaluate a nonprofit and which of this information is useful. Google.com is mostly a resource that points to information; they don’t tend to create a lot of their own content. So if we imagine a future version of the nonprofit data inside of Google Finance, I don’t imagine it will be some new metric that we design. Instead, it will point to existing information on the web. When I first wrote about nonprofit info in Google Finance, I said I hoped they would not display Charity Navigator ratings (although I would support them noting if a nonprofit had a zero or one star rating since I do believe that a Charity Navigator rating at this level is a significant red flag)
So the conversation I want to have is what information do readers think that donors should consider when evaluating a nonprofit? Then secondly, where or how can this information be captured online so that it can be displayed in Google Finance?
Open Invitation to Foundation Employees
I realize that if you work at a foundation, you may not want to jump into a conversation that involves telling another foundation what to do. However, the conversation we’re having here is really important and would not be complete without the input of the army of program officers (ie. Nonprofit evaluators) that read this blog. So please consider commenting anonymously (just let us know you’re a program officer) or comment publicly and realize that we’re having a broad conversation about nonprofit evaluation that goes beyond Google.org and Google Finance
Open Invitation to Nonprofit Employees
A conversation about nonprofit evaluation would not be complete without the input of the nonprofits being evaluated. What information do you, as nonprofits, what donors looking at when they evaluate you? It could be that someday the Google Finance website about your organization becomes the top ranked search result on google for your nonprofit. What do you want on that page?
70 Comments
It’s currently difficult to get at information that charities provide on 990s.
You can sometimes find them at a charity’s own web site, but going to the site of each charity you’re interested in and hunting for this info is cumbersome.
There are, I think, other ways to find this info online, but none of them that I know of are particularly easy. (If anyone *does* know of an easy way to get this info, please chime in.)
Anyways, 990 info seems to fit Google’s paradigm - indexing/collecting/pointing to existing info, rather than creating new info. Yes, I know that there are problems with 990s, but they’re better than nothing, IMO.
I can’t see how the Google Finance approach to presenting this data can work, given how heterogeneous the data is.
Some people want to donate to an organisation that’s very efficient at reuniting refugee families, others, to organisations that are effective at driving cancer research.
Unlike stock prices and fundamentals, there is so much diversity to the important data you want access to when evaluating NGOs, and none of that data is (at the moment) being output by these organisations as XML (with all the necessary metadata), so Google Finance is going to have a horrible time just bringing the important info onto their pages, let alone deciding how to present these different metrics, LET ALONE allowing sorting and searching by metric.
It just doesn’t seem like the right approach to presenting useful information about nonprofits. Something along the lines of Metacritic.com is much more appropriate in this day and age, but as you point out, Google doesn’t like curating information using humans, not algorithms (though with Google Knol, maybe that attitude is slowly changing).
Phil, 990 data is clearly important. But it only addresses the cost side of the equation, it says nothing about the social good being produced. It is like an income statement for a for-profit company that leaves out revenue. Certainly the cost data is useful (and easy for Google or others to aggregate and display), but it I think it is a bit like saying that when you design a new car, “you must have wheels”. Well, yes… but what else?
Guidestar makes this information available already. Maybe Google should buy Guidestar?
Philippe, why couldn’t Google display Great Nonprofit (MetaCritic for nonprofits) rankings? (Perla Ni, who’s commenting in this thread is founder of Great Nonprofits).
Google Knol does sound like a promising part of this equation. Let’s not constrain this conversation too much by Google’s historical choices. Also, let’s not assume that every nonprofit needs to be listed in the Google database. Currently, the Hoover’s data they’re using only includes larger nonprofits. In the for-profit markets we only expect data to be available for public companies. Most for-profit companies are private and it is almost impossible to find much info on them. Maybe nonprofits of a certain size should be included? Or maybe nonprofits should be allowed to elect to be “public”? In this case then they would be expected to provide certain data just as public companies are required to.
Sean, believe me, I understand the limitations of 990 type data.
But you’re asking what can Google contribute to the field with minimal human intervention (i.e. primarily using algorithms). While 990s are not good for all purposes, they are good for *some* purposes - there is useful information to be gleaned from them, IMO.
Basically, I agree with Philippe that the problem of measuring charities doesn’t lend itself particularly well to the type of solution that Google usually excels at.
I’ve been a lurker on this blog and this conversation is enticing me out in the open.
As much as I’d like to rejoin the evaluation discussion, I’m going to frame the question as, “How can Google help donors choose organizations to support?”
Seems to me the first step is to help donors themselves be clear about what it is they want to support — what they are trying to accomplish with their philanthropic dollars. Are they looking for organizations that are going to contribute to large-scale systems change — like reducing racism — where their dollars may be only a fraction of the budget? Do they want to have a deep impact in a particular geographic community (like the Ninth Ward in NO) or with a particular population group (third grade children)? Or do they want to be able to see the direct impact of their own dollars on individuals — like funding a Habitat for Humanity house? Phil B. talked about the need for NPO’s to be clear about goals; ditto for funders. If someone doing a search starts with putting some paramaters around what they want to accomplish (scale and locus of impact), Google could help identify who is on that playing field. This should be (and might actually be) readily retrievable from the nonprofits’ own websites.
Next, how could Google help donors choose among those doing the work they care about? 990’s are out there and Lucy listed many other sources of data. I also think that most investors put their money where they do because they have confidence in the people involved — whether it be for-profit or non-profit. How about links to biographies of key personnel, like the CEO and board members? Most organizations list these names on their websites and it would be simple to link to other information on the web about these individuals.
I also think it would be helpful to link to any media coverage of the nonprofits.
Some (not many yet, but some) organizations are starting to put their logic models / program theories on their websites, either explicitly in the form of a diagram or in narrative explaining how and why they do their work. Pulling this, where it is there, should be doable and useful.
It is good that your comments have matured away from the persistent bashing of Charity Navigator. Is this due in part to the unraveling of Givewell, which you have so favorably compared to Charity Navigator?
I will be fascinated if anyone can come up something to be displayed on Google that is not related to the 990. Do you have any ideas? You did not mention any.
By the way, are you even aware of the changes being made to the 990? It will contain a trove of new qualitative information. Did you ever engage in that debate? I feel grateful for groups such as Charity Navigator for pushing the issue for years, and bringing about results.
Cheers!
Re. Google.com is mostly a resource that points to information; they don’t tend to create a lot of their own content.
To expand on Sean and Teri’s comments, I think Google could provide great value by offering a live snapshot of web content available on nonprofits. Google could aggregate, organize and summarizing existing information: the good news as well as the bad news. Such a snapshot should help anyone–I don’t see why Google should tend to donors only–who wants to learn more about a given organization. It could also help compare organizations working on the same issues and/or the same region.
Some suggestions (by no means a complete list):
- Org’s website and/or blog
- Thematic focus [who else works on this issue?]
- Geographic focus [who else works in this county?]
- List and bios of key personnel [where else are they mentioned?]
- Financial info publicly available (990s, etc)
- Current and past funders [which other orgs do they support?]
- Evaluations publicly available of the org and/or its programs
- Media coverage
- International and in-country media coverage for orgs working overseas (VERY important and often neglected point, if you ask me)
- Blog mentions
- Online videos
- Presence in Second Life, Facebook and other communities
A caveat is that a list like this would likely favor orgs that are Internet-savvy and who have prominent spokespeople. Yet it could help collect a diversity of sources, opinions and data about any given organization. For instance, if an org has recently received negative coverage, this coverage is not likely to be found on its website (with notable exceptions).
Teri,
Making nonprofits searchable by the criteria you mention is a great idea. It would be interesting to see this evolve as a social tagging concept where donors could label a nonprofit based on causes, geography, etc.
I also like the idea of somehow providing more complete (objective) bios than the “official” bios supplied by the nonprofit. It seems this is something that could be done via the Wiki format that I suggested.
The media coverage element is already in the current Google Finance page (both news reports and blog posts).
If you look at the Google Finance for-profit pages, you find quantitative info (which in the nonprofits’ case would come from the 990). You also find links to more qualitative info. Maybe it is simply too early for an aggregater like Google to add value. Maybe we need more qualitative research being provided first.
Teri, if Google provided a platform for nonprofit evaluation to be made available to the public, do you think that large foundations would find it attractive to make their existing evaluations available through this platform?
Edith, you make a good point that everything we’ve suggested tends to favor internet-savvy orgs. That is because we’re assuming the nonprofit would be the one supplying/generating the content. It seems to me that with public for-profit companies (the other entities in the Google Finance database) you have legally required information disclosed by the for-profits (under SEC rules) and third party generated analysis.
I’m beginning to think that until we have third party generated evaluation and/or more extensive disclosure information, that the only real option is a more free-form wiki format rather the data driven model that Google Finance is designed to accommodate.
Erich,
I’ll be commenting more completely on GiveWell in a full post shortly. I strongly favor GiveWell style analysis over Charity Navigator. That being said, I swore off bashing CN when Trent Stamp announced his retirement. I haven’t been offering a lot of my own ideas because I’m trying to solicit input from the field. Although I have provided a number ideas over the past couple weeks of blogging on this topic.
Sean, I seriously doubt you have done much study of Givewell’s evaluations, your comments are so very superficial. If Givewell actually has such a wonderful evaluation system, why would that not be the obvious basis for any sort of Google data? Because it is an unwieldly yet superfical spreadsheet data dump?
I don’t think you or any of the cacophony of nonprofit bloggers have improved the discussion of nonprofit effectiveness at all. Essentially, you all come back to the “life saved” or “meal distributed” or “surgery performed” per dollar spent metric, but even that is only moderately helpful. Deciding whether a nonprofit is creating long term change is largely qualitative and requires personal experience. It fascinates me it took Givewell a year to conclude it costs less to save a life in Africa than to provide employment assistance in one of the most expensive cities in the world, but I guess that is where common sense comes into play.
Your inability to decide whether expense ratios are worthless, helpful sometimes, or should not even be made available should give anyone pause as to the depth of your understanding of the evaulation of nonprofits.
Great 2008!
Erich
Sean,
I do think that Google could greatly help philanthropies and nonprofits become more prepared for and accustomed with providing info and data to shed light into their goals, strategies and results. The data wouldn’t be provided only by the nonprofits, but would be collected through multiple online sources (media, blogs, etc.). And as an incentive, organizations who share their third-party evaluations would have more ‘complete’ profiles than the ones who don’t.
Oh, by the way, I never did understand why you decided to stop bashing Charity Navigator when Trent Stamp retired.
Is it because Trent Stamp had been hired by the Eisner Foundation, a $150 million dollar foundation, and you realized that if Eisner trusts the man who built Charity Navigator, it might look foolish / naive / ignorant for you to contine bashing it?
Cheers!
Erich
Edith,
If Google could figure out a good way to “permission” nonprofits to upload information in a way that was secure, maybe the database could be a way for donors and nonprofits to communicate about mission, effectiveness, etc, in a transparent way that would provide the resulting information to everyone. That would seem to reduce the cost to the nonprofit since once they answered a question, the answer would be available in the future.
And of course they could also link to third party data.
Erich,
I stopped bashing CN because the agreement from people in the field that CN’s model is deeply flaw is very broad based. I’ve done my part by trying to explain this to donors. But I think that for any real change to occur, a strong alternative must be presented rather than just pointing out the flaws of the leading evaluator. That’s why I’m so interested in the potential of Google doing something in this area.
I’m sorry you don’t feel that nonprofit bloggers have improved the discussion of nonprofit effectiveness.
Sean -
In your Dec 20 post you wrote: My Charity Navigator bashing ends today. I told Trent that he has all of my support in his new job.
Now you write: I stopped bashing CN because the agreement from people in the field that CN’s model is deeply flaw is very broad based.
Why can’t you ever stick to an answer? Are you writing that on Dec 20, the same day Trent resigned, you realized your work was done, that the world had stepped back from the brink of caring whether a nonprofit paid its professional fundraiser 75% of donations?
Your circular comments are a pathetic waste of time because no criticism you have ever made of Charity Navigator’s system are anything different from what Charity Navigator states on its own web site, such as here: http://charitynavigator.org/index.cfm?bay=content.view&cpid=484#15
Please don’t insult donors and everyone who has worked hard to bring accountability to the nonprofit sector by making such the broad, simple minded claim that there is general consensus that nonprofit expense ratios are of no value. In the past 60 days alone have been all over the map with your broad generalizations, as I have quoted numerous times.
Also, please don’t feel sorry that I don’t think nonprofit bloggers have improved the discussion of the nonprofit industry. You should apologize to the many people who have come long before you and Holden and who will be here long after people get tired of reading your ramblings.
Finally, it is great you are preparing a response to the controversy surrounding Holden and Givewell. Holden originally wrote about it Dec 31. You must be thinking very deeply about it. We await your findings!
Joy in 20078!
Erich
I had something typed out but in fact, I’ve just come across this, and it describes almost perfectly what I was advocating - much more eloquently, of course. http://triptronix.net/ishbadiddle/archives/2007/02/15/15.20.40/
THIS is what we need, and it’s not something Google.com (especially not Google Finance!) can do for us. Goofle Finance should stick to the 990 info (plus the forums, the links to news coverage and to blog posts)
Perhaps, however, Google.org could seek to stimulate progress in this area by putting up a prize/challenge. Not that this area should need incentivisation - the drive to open up to potential donors, and build strong relationships based on trust and reputation, should already be a high priority for the philanthropic sector.
It is difficult to create such metrics across the board without heavy investment of funds, but the metrics that we use at Central Dallas Ministries to cultivate our donors is “social return on investment.” In other words, the cost savings to the community generated by our work.
For example, we have determined that our health clinic saves nearby Baylor University Medical Center (BUMC)approximately $203 for every patient that we see (through decreased utilization of the emergency department). This sort of metric helps us to prove the worth of our clinic to BUMC.
Similarly, we provide a program for housing the chroncially homeless that saves the community $4 for each $1 invested in our program (i.e. the city typically spends $40K per chronically homeless person living on the street, but our program costs around $10K per person).
These metrics are fairly specific to our organization, and also to these particular programs. We do not have the capacity to determine comparable metrics for our after-school programs, law firm, employment training programs or food pantries. In general, all of these programs yield significant long-term benefits to society that outweigh the initial investment, but proving that requires a significant amount of funding in itself.
I’m a non-profit employee and I have found that more than anything else donors want to know what you are doing. They want stories about what they are doing and they want to know what the organization as a whole is doing with their money.
I am continually frustrated when donors ask for administrative rates as I think that it is a very subjective number. That would be my biggest concern regarding metrics is that I’m afraid that if it is all metrics non-profits will just record what they need to for their metrics to sound good.
Evaluations of NPO’s should include quantitative and qualitative evaluations on an even par with each other.
As argued very articulately above, there are outcomes that can be measured, metrics by which to measure them, and tools for reporting the findings, including the information contained in the revised 990. Google, could contribute to the evaluation of NPO’s by making this quantitative information more easily available to the public, and have the additional effect of reminding NPO’s that it’s our duty to set goals, evaluate outcomes, report findings and respond to outside review.
This quantitative evaluation process, however, is only half the equation. Unless Google succeeds in communicating this fact and steers the public adequately towards a qualitative evaluation, it should not develop any kind of evaluation aggregator.
As was articulated very clearly above, there are ways in which to make qualitative evaluations (i.e. links to CEO’s bio’s, narratives by reviewers on sites like Great non-profits, project profiles on websites, links to the NPO objectives and corresponding evaluations, open discussions linked to the NPO’s “portfolio” — a collection of all the evaluation tools being deployed.) I might add a new tool — peer review — a process resembling the tenure process in academia, but with the flat and constructive structure of the National School Reform’s “critical friends” groups for teachers.
Perhaps I’m stating the obvious or restating opinions already expressed above, but I recommend Google devise an approach that is equally quantitative and half qualitative or stay away from the enterprise altogether. The last thing Google — as a search engine whose great contribution to humanity is more and diverse locus of control — is to participate in any kind of activity that smacks of centralization and homogenization.
Correction: I recommend Google devise an approach that is equally quantitative and qualitative or stay away from the enterprise altogether.
“This quantitative evaluation process, however, is only half the equation. Unless Google succeeds in communicating this fact and steers the public adequately towards a qualitative evaluation, it should not develop any kind of evaluation aggregator.”
Maureen, I’ll definitely pass this on. I agree.
Sean, you wrote: Phil, 990 data is clearly important. But it only addresses the cost side of the equation, it says nothing about the social good being produced.
Still curious if you are even aware of the new 990?
To quote http://www.charitygovernance.com/charity_governance/2007/12/part-iii-mrs-ir.html
“As for Karnofsky, Hassenfeld, they may want to rethink their view of the Form 990 based on Parts I and III of the Core Form, which are front and center—pages 1 and 2 of the package. Here organizations are given the opportunity to describe their mission and significant activities. Part III is particularly important, requiring a description of the organization’s three largest services, with a focus on achievements. Don’t fault the Form 990 if organizations don’t tell you enough about outcomes. The IRS certainly has asked about them and given organizations the opportunity to tell their stories. For Karnofsky, Hassenfeld, and others, the solution is actually quite simple: Start with the Form 990 and if you don’t see what you want, move onto the next organization. If everybody did this, organizations would become more forthcoming in providing the sort of information that so many want. In fact, Karnofsky and Hassenfeld would be doing everybody a favor if after each unsatisfactory review, they sent a form letter explaining why they moved to the next candidate.”
Best of luck to donors and anyone who actually cares about the nonprofit sector.
Erich,
Yes I am aware of the new 990. Although I am not aware of any authority on the subject of nonprofit evaluation who thinks that the new 990 is a game changer. Describing your mission and your accomplishments is not the same as adequately demonstrating the social good that you produce. I don’t see why we should even expect this kind of info in a tax form.
Ah, you really are arrogant Sean. It is remarkable that you would dismiss so many as irrelevant and not being an “authority.”
Oh, and, if groups like Charity Navigator are not authorities, what does that make you?
Good luck, you obviously need it.
Erich
Trent Stamp is clearly an authority. He would be the most natural person to celebrate the new form. However, Trent said that the biggest difference between the old form and the new form is that the new form is longer.
Erich, I’m going to withdraw from this conversation now. As you point out, even Charity Navigator thinks that their ratings are not enough to evaluation a charity. My beef with them is that they’ve successfully marketed themselves to the press to the extent that many donors think that expense ratios are an effective way to evaluate a nonprofit.
I’ve said my piece on this topic. You’ve said yours. You can feel free to continue leaving comments, but I’m going to take a break from answering your sarcastic and condescending questions.
Oh Sean,
It is truly sad you know so little about the nonprofit sector, and are unwilling to engage in any discussion which does not meet your preconceived biases. Instead, you repeat yourself, change your answers and divert attention.
You are clearly wrong about Trent’s comment on the new form. Trent has not commented on the new 990, which came out in December. The link you cite is from June.
Yes, in the post you cite he states he would like to see the IRS take strict enforcement action, rather than simply revise the 990, but I won’t debate that here, as it is well beyond your scope of knowledge, and there is no need for me to speak for him.
I merely feel the need to point out that for you to imply Trent Stamp is critical of the new 990 is without any basis, period, for the benefit of others. He may indeed be critical, but you have no basis to make that claim.
You may also be language deficient, if you truly think that where the form calls for nonprofits to list their achievements, you somehow think that is different from a nonprofit describing its “social good.” Any donor would expect a nonprofit to describe its measurements of success. The 990 is the only standard form most nonprofits must file, and is the obvious source to turn to for information. You should be happy about its siginficant improvements, but instead, because it does not meet your preconceived bias, and you clearly are ignorant of the new form, you choose to pan it.
The only reason I post here is because you post with such authority, and I think so much of what you post is plain wrong. You need to be held to account, in case you mislead readers.
Best!
Erich
RE: Sean’s question about foundations sharing evaluation reports via Google.
I don’t see foundations making all the reports of evaluations they commission generally available on the web. This may happen eventually, but I don’t think the culture of most of the big foundations is there yet. Even if they are made available, the purpose of those evaluations is often to help foundations improve their own funding strategies (which vary widely in how well they are articulated). Their utility to potential donors to individual organizations is questionable — most foundation evaluations are looking at groups or clusters of grantees to see the overall impact, rather than focusing on individual grantees. They COULD be useful to donors who want to understand in general what change strategies are being used and what is being learned about their effectiveness.
Anyone other views on this?
Mike Everett-Lane from DonorsChoose.org here — also the author of the above-referenced blog post on microformats and microphilanthropy.
I think there are a few things that could be helpful re: Google and the nonprofit marketplace.
The first, of course, is helping implement my brilliant idea about microformats
[See Paul Ford’s essay on Google and microformats and marketplaces for a good intro.] Google can help set and propagate these markup standards.
Of course, microformats are good for “microphilanthropy” which is good for orgs like Kiva and Global Giving and DonorsChoose.org. But I also think that it will be good for the sector in general. In general, I think that making nonprofit data more searchable in an intelligent way would be a Very Good Thing. Teri’s idea above is a good start. Let’s make it easy for a donor (or client! or potential partner!) to find organizations by services offered, geography (think the hyperlocal web, but for social services not just for restaurants), hours of operation, volunteer opportunities, need for materials, etc.
Secondly, let’s take this idea a step further: make it easy for nonprofits to find funding. Let’s get all that data in the Foundation Directory out of the book and out of the library, so we can easily find a potential funder. The grants market is terribly inefficient, and perhaps Google could do something about aggregating the information off of foundations’ 990s as well as nonprofits’ 990s.
Just a few thoughts…
Oh
Teri, it seems to me that the Google interface would only really be useful if it is able to link to qualitative analysis of nonprofits. Program officers across the nation are working on this kind of evaluation every day. While I agree that most foundations do not have the “culture” that would be open to this idea, do you think that this information could possible not be a net positive to donors?
I know that in the investment management business, quantitative information is critical, but it is qualitative information that drives most decision making.
Mike, I like your idea of integrating foundation grantmaking info with the nonprofit data. Some foundations are already in Google Finance, like this page for the Hewlett Foundation (note my comment on the discussion board asking for information on which nonprofits I should support as a donor given my shared interests).
Also, simply making nonprofits more “searchable” would be a big help. Imagine if Google built something like Grantsfire that connected nonprofits and foundations.
@Erich: Can we return the conversation to one on nonprofit evaluation, assuming the Stannard-Stockton bash is over? I’d be interested in hearing your thoughts on evaluation…it’s a discussion that grew particularly heated, once upon a time, starting with a conversation at NetSquared last year.
Not sure I need to defend philanthropic blogging here; those that see the value in it will stay, and those that do not will leave, I assume.
@everyone: So, back to the topic at hand! I think successful nonprofit evaluation certainly lies far more in the qualitative than the quantitative. I think it’s also important to note that the very act of evaluation, to some degree, opposes the act of marketing (and I say this as someone with a background in internet marketing). Marketing is not necessarily obfuscation, of course, but it presupposes that your organization’s value needs to be presented in the best possible light, because that light *affects how you are seen* in the market. An attempt to evaluate is by definition an attempt to move beyond presentation, embellishments, cover-ups, and rhetoric of the salesman bringing his wares to market; this places the two irrevocably at odds. Evaluation, then, is at least partially concerned with penetrating the efforts of marketers. This becomes a chess game, instead of a one-sided consumption of marketed media…but its adversarial nature should be noted.
But there are more important questions lurking in that particular issue. For example: Why do some folks prefer a quantitative approach to a qualitative one?
I think the answer is that numbers are easier to verify, and we intuitively assume that verifiable facts do not lie. The problem there is that *every* nonprofit I’ve ever worked with has, more often than not, found itself trying to figure out the “right” numbers to put on a brochure, grant application, or press release. And for most organizations, it’s *easy* to come up with the numbers that represent them best. The problem is that those numbers are inherently the ones that least represent their problems, their struggles, or their learning experiences; they represent their perceived strengths. I’ve *written* them enough times; it’s what the powers-that-be think you should be putting out there to impress the rich folks.
That’s good business marketing, I suppose…or was, back in the stone ages. But it’s more interesting for me to know an organization’s challenges than its poster stories.
Highly flammable example: If Givewell had ten staff members and analyzed ten orgs this year (I’m making numbers up, for the sake of argument) and has 100 staff members analyzing 1000 orgs next year, their numbers will show that they have successfully gone to scale and created operating efficiencies to boot. I am interested in the Holden situation, and the organization’s reaction to it; where is that information in the numbers? Especially in the numbers most organization choose to make available to the market?
With a qualitative approach, on the other hand, we’re slightly more aware that narrative can easily become rhetoric. However, successful experiments in the social content creation space suggest that, in large numbers, we are capable of keeping each other honest (think Wikipedia). Allowing everyone to comment somehow turns the expected logical outcome on its head, and results in value-neutral media. This makes me think a wiki-based platform might work best, with live feeds in a sidebar with the basic numerical background on an organization, and a strong organizing ideology on the tone, values, and rhetorical direction of the content created.
As an exercise in forcing nonprofits to abandon traditional marketing for a more transparent approach, I think this would be highly effective. However, to succeed it would have to be implemented on a vast scale; otherwise, criticism is shaped into a targeting of one or more organizations who then become controversial. The entire *sector* needs to evolve; this might be a first step in that direction.
The problem, of course, is that this won’t let me analyze nonprofit scores on a Dave’s Nonprofit Index ticker; it’s a little less neat, and requires a little more reading, but it might provide a far richer analysis and research space.
To finish up this comment (apologies for the length), I’m not sure Google is the right home for this kind of project, though Google is certainly the right sponsor for the folks that do, eventually, build such a home.
Thanks for reading.
Dave.
David, grow up. Sean has been making nonsensical, spurious attacks and certainly should be called to account. If you or Sean can take a few moments to criticize my facts please spend your time doing that, until then I will take your criticism for what it is worth, more mindless chitchat.
If you enjoy a good bashing, how about this Holden-esqe remark Sean made on Dec 20, the day Trent Stam announced he would be leaving Charity Navigator to head the Eisner Foundation: But let me say this about Trent. He knows charities. He will likely make a great executive director. He won’t just use overhead expense ratios to make grant decisions for the Eisner Foundation.
So, here Trent Stamp is leaving Charity Navigator to begin a great adventure actually making grant decisions, and Sean feels the need to take one last swipe. How arrogant, and typical.
I think Sean deserves the condescension award, and I am happy to deliver it.
As an aside, it took Sean about an hour to comment on Trent Stamp’s departure. Holden’s implosion took place December 31, and Sean still hasn’t found time to comment.
What is it with nonprofit bloggers? You people don’t even hold people like Holden and Sean to account when they make contradictory, superficial comments. I realize Sean and presumably others on here feel the need to tell donors they are not capable of making decisions, but anyone who has read a single Givewell evaluation can easily see it is nothing more than a superficial data dump. You should care more about nonprofit effectiveness and less about media attention.
Best!
Erich
@Erich: Looks like Sean commented on the Holden thing as soon as the board decision came out (it was on the front page of this site before your previous comment).
My earlier comment to you was an invitation to comment on nonprofit metrics (the subject matter of this thread); your contribution here would be valuable. I don’t want to get into an off-topic discussion here; that just disrupts information architecture and derails conversations in the hope of making a splash (kind of what you were accusing Sean of).
I’m actually convinced you have valuable things to contribute, which is why I wish you’d written more about nonprofit metrics (other than the Holden sidestory, which is elsewhere). Put the anti-Holden stuff there, and the nonprofit evaluation stuff here?
Thanks,
Dave.
Gee David, thanks for the invite, but you are wrong to assume I have never commented on nonprofit evaluation here. You may want to peruse prior comments.
Trust me, I don’t plan to waste a lot of time on a pointless, self gratifying blog. It is simply that the exciting moves at Charity Navigator and the implosion at Givewell provided nice confirmation of comments I previously made here, which Sean had hastily dismissed, and this is a GREAT opportunity to call him on his hypocrisy and ignorance, and I am taking it.
I have made perhaps a dozen comments regarding nonprofit evaluation on this very blog, have carefully cited the maze of Sean’s circular arguments, and Sean constantly evades all the issues.
Pushing Sean on Givewell is entirely appropriate. Givewell was the first source other than a nonprofit fundraiser that I became aware of which claimed fundraising expenses should be ignored, and Sean repeatedly praised Givewell. I merely tried to find out why, and Sean’s only answer, as posted in this thread, is that people in the industry don’t find it useful. I have learned that when people refers to anonymous experts, they are generally making it up.
I think that any self professed nonprofit expert who claims paying attention to administrative and fundraising expense ratios is a waste of time is, in my opinion, ignorant, or worse. The fact that Sean is not even aware of the changes being made to the 990, and again dismisses them as being of no interest to the same anonymous “authorities” in the nonprofit sector, tells me not to waste my time trying to have a rational discussion with him.
My guess is Sean’s goal is to tell donors nonprofit evaluation is hard,so they will hire him.
It is obvious that aside from the simple financial metrics which provide so much basic information that everyone should be aware of, more complicated analysis requires examination of the individual charity. The 990 will provide information on the metrics the nonprofit itself uses and people will be able to judge that. It will flow back to Charity Navigator, Guidestar, and other aggregators. Beyond that, people will have to learn about a nonprofit directly, to see if it meets their values. It is pretty simple.
Sean isn’t even at the point of trying to reinvent the wheel, it is still trying to determine if there is a wheel.
Glorious 2008!
Erich
I know there are a lot of new readers here right now. So if you’re interested in my professional background and how I get paid you can visit the section about my firm. My firm Ensemble Capital helps donors structure the financial aspects of their philanthropic plans (we set up private foundations, charitable trusts, etc) and then get paid to manage the assets inside these vehicles and in many cases the donors personal assets. We do not offer services related to nonprofit evaluation.
Erich, might I venture a neutral observer’s opinion here? You’re coming off as a rabid beast. You’re trying to ram Charity Navigator down everyone’s throat and boring us with this incessant, completely irrational gloating about one/two unfortunate employees at GiveWell - a brush with which you perversely want to tar the whole organisation, even their entire philosophy and approach. I’ve had the misfortune to read you on this blog and others. It’s the same stuff again and again.
You attacked Sean’s explanation on why he feels even the new 990 is not sufficient to evaluate the suitability of a nonprofit for donation. Yet you did this without actually refuting his criticism, merely whipping out your belief in his ignorance of the matter at hand… again. This seems like double standards when you so clearly decry ‘circular arguments’.
You seem so hellbent on derailing this thread. Why? Why not email Sean and let others stick to the matter at hand?
I can’t stress enough that I feel you ought to re-examine your writing tone if you’re seeking to have any influence over the people reading your posts. I might suggest you’re somehow affiliated with Charity Navigator, but even they wouldn’t be so blatant in their self-righteousness and unquestioning, unassailable faith, and besides, their zealotry wouldn’t splash onto other people’s blogs like this. So I really don’t understand your behaviour.
So just give it a f*cking rest. It’s tiresome and it manages to miss not just THE point, but all the auxiliary rants you drag into the thread.
Hey, maybe it’s Holden again, exploring more advanced strategies of using fake identities in public arenas to put people off rival initiatives.
This time, rather than pretending to be a nice GiveWell fan, he’s pretending to be an asshole Charity Navigator zealot!
Phillipe, you are wrong, I am not trying to build up Charity Navigator, I merely called Sean a jerk for persistently bashing them.
What I am drawing attention to is the nonprofit expense ratios.
Have a great year!
Erich
Oh, and no Phillipe, reading the babble that goes on at this and similar nonprofit blogs, I am definitely not trying to influence anyone. I am convinced it is an incestuous group of nonprofit industry hacks who cross post.
Cheers!
This has gotten a bit sick (wish these personal sub-threads could disappear), but just in case somebody is still reading, a concept came out of a few non-profit meetings I had today:
What if Google could cross-reference 990’s with volunteer hours, and other volunteer stats like # of repeat volunteers. Obviously this would require self-reporting by the organization or by the volunteering individuals (might be a business there, we’re thinking about it).
Why, you ask?
Number of volunteers / volunteer hours, combined with financial info, shows a few things:
a) More volunteers generally means more leverage for your donated dollars. A 10-1 volunteer-staff hour ratio means each of your dollars should mean that your money goes further.
b) More volunteer hours, and especially things like repeat volunteers, should mean that the public finds this to be an acceptable and worthwhile charity. A whole bunch of community members wouldn’t be donating their time if nobody believed in it.
Hi David,
Sorry to butt in, but number of volunteer data is included in the new 990!
Read all about it. Seriously, people should read about the new 990. This is the best nonprofit governance blog I have run into.
http://www.charitygovernance.com/charity_governance/2007/12/part-i-hey-mrs.html
Cheers!
Thanks for the link! But just # of Volunteers is not sufficient (although better than nothing). I know plenty of charities that have a ton of people that come in for a couple hours on some sort of organized activity (like church or classroom helping at a food bank). While that certainly shows some belief in the mission, that doesn’t necessarily translate into monetary leverage (tons of volunteer coordination for low output). And repeat stats would show more.
Thanks.
Fundraising expense ratios are incredibly important…when you’re hiring or reviewing the performance of a fundraising employee. It’s an internal metric. It’s meaningless externally, as an evaluation tool for the organization. The harder you press for this number as an evaluation metric, the more you’ll find organizations classifying things creatively to paint a prettier “metric” for you; it doesn’t reflect the actual world nonprofits operate in.
@David: Correlating volunteer stats with narrative on volunteerism at the organization might create a very clear picture. I still like the wiki-based idea I suggested earlier, but perhaps it should be organized into tabs (such as “volunteerism”), with each tab correlating narrative and numerical data, but (key point) open to public editing and commenting, both anonymously and named?
Glad to see the thread back on course. The conversation’s too important to loose.
Dave.
I see volunteering stats as only marginally interesting.
I don’t think funders should essentially ‘take credit’ for volunteer time (i.e. my donations are leveraged because of all the effort of those poor sap volunteers).
And the fact that volunteers choose to work for an organization doesn’t necessarily make that organization terribly effective. I’ve volunteered for one organization that I thought was relatively ineffective in use of money (and volunteer time). I did so because I enjoyed the volunteer experience and the social interaction.
Dave, organizations will always put the best light they can on their numbers (publicly traded companies certainly do), but don’t you think that forward thinking nonprofits who really are effective can find a marketing advantage in being truthful and transparent in revealing their evaluation metrics?
In other words, I don’t see marketing and evaluation at odds (although there is certainly a natural tension). Instead I see transparent evaluation as one of the most effective marketing strategies (for effective nonprofits).
David, I really like your idea of volunteer hours. I continue to think that there is no one perfect metric, but the commitment level of an organization’s volunteers is part of the whole story.
The public goodwill that high volunteer hours suggest is an asset of the nonprofit and so it increases the total resources available to them. But I don’t think it says much about effectiveness, since volunteers are not the ones receiving the “product and services” of the nonprofit.
I don’t agree with Phil’s comment regarding his worry that funders would take advantage of the “poor saps”. Volunteers are a resources just like an endowment. Funders should also not fund an organization just because they are in strong financial shape.
Some more resources to add to the list Lucy created:
Issue Lab
Child Trends: What Works
Root Cause: Social Impact Research
Urban Institute: Outcome Indicators
You spoke too qucikly Sean - depending on the mission, the number of volunteer hours and the number of volunteers actually can be a direct sign of effectiveness. If, for example, community organizing or community building is a central activity the number of people who voluntarily contribute hours does demonstrate effectiveness. And think about “Forces for Good” and the emphasis that model of effectiveness puts on creating volunteers who are evangelists for an organizations and advocates for a cause. It’s not a perfect metric (I’d want to know volunteer turnover and event attendance too), but it is a metric of effectiveness for some organizations.
Good point Young Staffer. If community building is the mission, than clearly volunteer work is critical. I do think that volunteers are a key to being effective, but are not an indicator of effectiveness. In other words if you want to be effective, get evangelical volunteers are advocates (a la Forces for Good) is a very good idea. But if you look at an organization and see that they have lots of volunteers, it does not mean they are effective.
But as I said in my comment, “David, I really like your idea of volunteer hours. I continue to think that there is no one perfect metric, but the commitment level of an organization’s volunteers is part of the whole story.”
I’ll see if I can get the Forces for Good authors to weigh in here.
Expense ratios are useful for one thing: finding the most abusive charities. What I resent about Charity Navigator is that they use it to supposedly identify the best (the four star vs. the three star) and not just the worst – they claim to be an “evaluator,” not just a watchdog.
There are two problems with expense ratios and the broader Charity Navigator methodology: 1) accounting in the sector is not standardized and this invites gaming - a fixable problem, I admit and MORE IMPORTANTLY: 2) it perpetuates a giving market defined by one type of efficiency and not effectiveness – the result of which is ultimately inefficient.
It’s the difference between monetary/financial efficiency and economic efficiency, something far too often neglected in our discussions of spending in general.
A relevant comparison here is the question of whether the charitable tax deduction is efficient. In terms of treasury efficiency, this is the question of whether it helps charities raise more money that it takes away from the government, thereby having more money end up in “public” coffers (nonprofit and government combined) than would have without the deduction. That’s mostly an empirical question.
In terms of ECONOMIC efficiency, however, the question is whether the charities actually spend the lost tax revenue BETTER than the government would have. Thus, the question of whether or not the charitable tax deduction is actually efficient and effective policy turns not just on how much money its pumps into the social sector, but whether nonprofits spend it better than the government. If they don’t, then we’d be better off giving it to the government. How you make the calculation is much more complicated; it has to do with the values you hold, what the government would do with the extra money, what would happen to the giving in the sector without the deduction, etc. It ultimately has to do with how you evaluate the social benefits of nonprofit sector activities versus government activities.
That’s the kind of question expense ratios ignore about charities. A charity that can save 5 infants lives for $100 and spends 20% on fundraising is more economically efficient and more effective than one that saves 1 infant for the same $100 and spends 10% on fundraising. Sure, less of your money went to the programming in the first one, but it actually went farther – the social benefit of your dollar is empirically greater. They spent less on programming, but they did more with it. Granted, one would hope that the first charity would figure out how to do their marketing a little better and get larger gifts, but their failure to do so does not change the fact of their superior efficiency. Even as an inefficient fundraiser, your money was better spent on them. GiveWell, as opposed to Charity Navigator, attempted to address that question.
Knowing that an organization is spending money on programming and raising more (growing) is not at all the same as knowing whether they spend the money on programming well. The end result is an economically inefficient market – one that does not maximize giving to the organizations who are more efficient at turning dollars into social benefit.
Now, whether or not Charity Navigator has helped or hurt the efficiency of the nonprofit capital market on balance is a question I can’t help you with. I have no idea how it’s actually shaped donor behavior. To the extent that most charities that spend 50% of their budget on spending are probably not economically, monetarily or organizationally more efficient than others and that it has helped donors avoid them, the market is more efficient. But in allowing people to over-simplify the question of efficiency, it can hurt the market.
Very well said, Young Staffer.
I think Charity Navigator has its place in the world, but the problem is when folks start and stop their analysis with CN.
I want to see a focus on effectiveness, not just efficiency. That’s what GiveWell’s analysis tries to get at. But I think the GiveWell approach is difficult to implement. It requires a very detailed picture of a charity’s operations, one that was apparently difficult to coax from the charities, and this caused further issues.
When you look at the African Health report, you see that the three charities they rated highest differ most obviously in approach. One is focused on cosmetic surgery (albeit relatively cheap and potentially life-changing surgery like fixing disfigurations). One focuses broadly on diseases (like AIDS and Malaria). Another focuses on health infrastructure (clinics and the like in poor areas).
But to reach those conclusions, they had to rely on the data supplied by the charities (difficult to get, potentially unreliable), and their own estimates used in the analysis (they try to translate certain raw numbers, like # of bednets distributed, to impact values, like lives saved).
An alternative approach would be to look at what the academic community has to say about effective interventions in Africa, and then work down from there.
Basically, instead of looking at raw data from charities, and trying to figure out how effective they really are, one could look at what has been shown to be effective, and then find charities using those techniques.
This would omit the detailed spreadsheet analysis of each charity, and instead rely on the assumption that, within particular program areas, charities are likely to be roughly equal in effectiveness. i.e. If insecticide-treated bednets are highly effective, and two charities are both distributing them, then they are likely to be roughly as effective as each other. You’d still want some secondary analysis to help out here, including fundraising efficiency measures, and some subjective assessments of a charity’s experience and on-the-ground resources in a particular area. And conclusions would still be imperfect. But I think you’d have about the same degree of accuracy in your final conclusion as with GiveWell’s approach, albeit without the need for the hard-to-obtain detailed spreadsheet data.
There are positions on nonprofit metrics for every taste, yet we have found the menu to be so confusing and divisive that many lose their appetite. The discussion on metrics becomes productive only when the conversation shifts from one of being threatened, compared, or evaluated to one of using outcomes (and their values) as a tool. We feel the highest and best use of that tool is in ensuring sustainable funding.
The bottom line on this discussion is that there will never a universal metric to measure impact, effectiveness, or whatever term one prefers. Can metrics be used improperly? Yes. Can they be manipulated? Yes. Can they be difficult to quantify and communicate? Yes. These possibilities do not outweigh the benefits of a well designed and implemented program of demonstrating a nonprofit’s value.
Our position in this confusing arena does not come from a philosophical or academic perspective: it comes from the trenches of helping nonprofits get the funding they deserve. Those nonprofits that make it easier for funders to connect the dots to their respective outcomes deserve to have capital flow to them. Those that choose not to go down this path do so at their own financial risk.
Thank you for keeping this thread constructive. It’s very informative. Having reread it, I have some thoughts to share, many of which have already been stated above:
It is important to keep in mind that everyone is involved with the non-profit sector. In addition to roles we may choose to play as donors, administrators, volunteers, grant makers, etc., most of us are also involved with NPO’s as tax payers. By accepting the rules which accord non-profits a special tax status, we underwrite them and gain the right to an “adequate” evaluation.
Foundations are NPO’s, although they are not 501C3’s. As NPO’s they get a tax break and are open to public scrutiny.
Given our interest in NPO’s, we need a system that permits us to evaluate them in terms of their efficiency and effectiveness. Reports (such as, the revised 990) provide some critically important information, but the general consensus is that we need additional tools for gathering and reporting information from the broad spectrum of interested parties about the broad spectrum of NPO’s.
New technology provides more opportunities to create more adequate evaluation tools. My personal favorite from this thread is the wiki-based idea, with tabs correlating narrative and numerical data.
It’s good this thread has moved beyond metrics, since (a) input metrics are, for most nonprofits, remarkably confused with output and neighborhood benefits, (b) outcome measures are much, much more abstract and, paradoxically, seem so much more concrete than many might infer, and (c) indirect costs, the point of departure and usually the point of no return for such measures, have slipped into remarkable disuse and astounding inconsistency in the past 20 years.
I’m also sorry that this thread seems to have lost the fervor of the earlier argument on behalf of transparency, which ought to be a standard (and not a metric) for any expression. If we focus a little on other potential standards, then the questions of metrics and reliable, modestly consistent, reasonably comparable and timely reports makes much more sense.
Before that, however, let me note that I’m surprised no one has raised the MOST important innovation of 990 comparability: they can be filed electronically. That means they could - conceivably - be compared, over time, across agencies, across sectors. That all those data might be compared is much, much more interesting than that they exist idiosyncratically in each filing. Among other things, we could find out which data are useless for anyone, and thereby cut the form considerably; and, conversely, conduct a content analysis to compare narrative descriptions when present, which could, should, or might produce much more generalizable statements at least about sectors (health, education, etc.).
That said, the real question of metrics is what, if compared, would lead to better decisions - by philanthropists as well as by the NPOs themselves. In a painful discussion with a former politician, for example, I asked which elementary schools sent the highest ratio of failing students to 9th grade in the high school. The only response I got was, “you can’t blame the schools,” when the question I asked was how they could be helped to help their kids better.
This question of metrics is probably easier to explore in the context of education than many other NPO enterprises. Not only is there the blistering and ugly dialog about No Child Left Behind, but there really are some fairly simple metrics which, once explored, might lead to better reports and greater … transparency. In other words, it probably behooves this discussion to discuss nonprofit sectors before trying to generalize across sectors - a productive day care center is really hard to compare to a productive hospital or a productive Red Cross. And it won’t ever really compete in a 1:1 with either.
That said, I was struck by one of the Phil’s observations that it’s fairly easy to ask how many teachers teach, students learn, and so forth. That only showed that he didn’t know what schools do. When a school uses a volunteer, for example, the school gains both an input and an output, as well as a fund raiser and political advocate for public funding. As well as a satisfied parent, perhaps, or at least a neutralized nonparent. To measure such a “worker unit” in a single dimension ignores what might, and probably is, a much larger unit of institutional impact.
If its that hard to measure the simple number of teachers, it’s mind boggling to explore square feet (when or if they take field trips), books (when or if they’re on the net), equipment (when or if they have homework or off-site activities), or even meals.
It is much easier to track the unobtrusive measures - attendance, tardiness, staff turnover, hours of classtime, perhaps hours of homework, sometimes costs of materials - and to explore what measures might be needed to give a better profile of outcomes. Surely test scores are some of those measures, but many of the unobtrusives are as well - in their delta values, in the degree to which they may change with certain teachers, certain subjects, certain years, given certain conditions.
With those kinds of data tracked, it is quite conceivable to construct a profile. And, with profiles constructed, we could quite conceivably compare schools, teachers, classrooms, and kids. And we might then begin - just begin - to identify what OTHER measures, either qualitative or quantitative, ought to be collected to make those profiles more useful - to both funders and to the users at all levels.
Finally, I am really surprised that all this dialog about metrics ignores the largest of all funders - government. If an NGO is productive, it probably has government money as well. And if it has any government money, we might then begin to apply those metrics to government agencies - as you have elsewhere implied an evaluation of philanthropy and foundations. Now, is that to be political or “objective.”
Pfui.
Joe, you do a good job of pointing out how multi-dimensional some measures are (ie. that school volunteers are both inputs and outputs). Maybe “measure” is the wrong word all together. Maybe we should talk about judging nonprofits instead. In court, a judge or jury decides on a case, without pointing to a quantifiable set of measurements. Books and movies are judged without metrics as well.
But how might we judge nonprofits in this way?
One of the problems of developing metrics in a system which is not a system - but rather a “network” of many mini-systems - is that there are no realistic standards. “Cost-benefit” which works fine for lots of things is infinitely difficult to measure when, for just one example, the actual value of volunteers (one of the costs) is so hard to estimate (if they are big guns do they cost/value more? depends on what they’re doing and what you do about what they’re doing, and so forth).
A much easier way to derive a “system” which could have some resonance at least among similar agencies in similar fields is to identify “best practices” and suggest what “best outcomes” “best reflect” those practices. This begins to build “rubrics,” and those rubrics can eventually become metrics, once there is enough common understanding of what they mean and how they apply.
This need not be another library of cases (ala a B-school). It can be little snippets equivalent to the “knols” that the Google people talk about. For example, volunteers as promoters might be a “knol” with a 1 paragraph snippet to summarize what we mean; volunteers as “trainees,” as “assets,” as “Board candidates,” as “mentors,” etc. All of those could congregate in “volunteers as outcomes” which contrasts with “volunteers as teacher supplements” or volunteers as resources.
In court, a judge and jury dis-aggregates the evidence and synthesizes a judgment based on the sides presented by counsel. This tactic builds rubrics which correspond to the “rules of evidence.” At least it’s a hell of a lot better than a simplification that reflects “spin” more than substance, which is more or less the way it now is.
I think there is never going to be a metric, or even a series of metrics, that will a) fit all social enterprises, and b) fit all donors’ needs. What a museum does, what a soup kitchen does, what a clinic does, what a school does, what an advocacy group does, etc., are so fundamentally different that we simply cannot judge them by any common set of numbers. The important thing, methinks, is for the organizations themselves to say, here are our goals that we measure ourselves by, and here’s how we did.
The second aspect to this is that different donors (including foundations) will value different things. Perhaps A believes that %-to-admin is the most important, while B looks at volunteer hours leveraged, and C wants to know cost per service unit. XYZ Foundation wants to see new programs developed, and ABC Foundation has their own set of outcome criteria.
Just as investors in the equity markets will look at different metrics, depending on their strategies and beliefs, so do donors in the nonprofit market. The important thing, I believe, is to give donors access to good and verifiable and comparable data, and let them decide what is important.
Juries are asked to judge a case based on a single criteria — The Law. I think that private investment, where both institutional and individual investors are all pursuing different strategies, is a closer model to what happens with donors and foundations.
I want to point out to everyone who commented on this thread, that another version of this debate is forming in the comments to my podcast interview with Phil Buchanan of the Center for Effective Philanthropy.
You can find the new thread here. I’d love to have your input.
Just came across http://www.intelligentgiving.com - the info they put on profile pages is pretty good, and well presented. Could this approach be emulated?
Mike Everett-Lane makes an important point - there will never be an overarching set of metrics for all nonprofits. To add to his comment that organizations have to decide what to measure, I would suggest establishing a strong link to the mission statement that focuses on outcomes of programs instead of merely the short-term outputs. Metrics that point to changes in behavior and attitudes present a compelling case that programs are indeed created tangible community/societal benefit, which will attract not only more funding, but also more committed Board members, volunteers, and the like. You can find some instances (Nature Conservancy and Duke Children’s Hospital, to name a few) in our whitepaper at http://www.spm-nonprofit.com. Let’s keep this dialogue going!
So glad this conversation continues.
I liked the comparison to movie evaluations above, but the problem there is that we rely on a particular reviewer’s expertise as a guide to how good or bad a movie is; the narrative is meaningful and revealing, but the opinion at the core is one we must trust if we are to place any weight in the review. This would be impossible in the nonprofit world, unless we all recognize a core group of gurus who’s opinion will make or break any organization
@Sean (w/ apologies for the delayed response): Transparency could be a wonderful marketing tool, in an ideal world. The problem is that it requires critical mass before it’s effective, and a larger critical mass before the dinosaurs will pay attention. If *everyone* was transparent, it would change the rules of the game. Till then, no one wants to be first, because it could backlash…and in traditional marketing models, describing your problems, frustrations, and failures in narrative form is sales suicide. No one wants to be first; they’d much rather write happy blog articles about how amazing and uplifting their work is.
I like the idea of tightly tying an organization’s measurables to its mission statement, but the more I think about it the more I think a publicly malleable space for narrative text with an intentional community (perhaps slightly less fanatical than Wikipedia) is a fairly good solution. Let the world comment and critique, let the orgs write back, force a dialog…even for the orgs that would much prefer the comfort of their one-way websites.
Dave.
Met with the local 2-1-1 provider here today, and discussed an interesting wiki or otherwise approach: For any verbal recipients (obviously excludes animal rights and such), how hard to have an established, generic survey about how much the organization helped, sent to the actual recipients of the service? Even if people lie or inflate, the assumption could be that people lie across the board, which would still give you reasonable relative numbers. With an online survey, that can’t be that hard to collect - and I guarantee if it mattered to an NGO’s donors, they would be sure to tell their recipients to go rate.
Similar to the sites that try to “rate your doctor”, etc, so there are models out there.
Thanks for the good discussion.
–David
David,
I like the idea (though organizations working on Digital Divide issues would challenge the ubiquitousness of an online medium). I think the issue would be in finding a “generic” set of questions. Something truly generic to all nonprofit areas of work would be very, very abstract…effectively turning into a “confidence vote” by the org’s constituents, kind of like Amazon’s rating tally.
It’s also interesting to me (as a web developer) that building this kind of system would be relatively simple, technologically; promoting the platform to gain acceptance would be the difficult part. Thoughts?
- No way we will every compare 100% across categories. That’s like saying you could use just the P/E to compare an early stage biotech with an established brick-and-mortar retailer. Very hard to have a one-metric comparison in that fashion, but with enough metrics, or within categories, then they can be effective.
- Groups like US News use such self-reporting survey systems to evaluate colleges, for example, so I can’t believe it wouldn’t work for nonprofits. If somebody like Google said “fill out this extensive survey in order to be listed in our directory”, I’m sure many would comply, just like the colleges do. So it becomes more of a matter of establishing a good survey (or surveys for various categories).
- There have been attempts, such as http://greatnonprofits.org/
–David
Dave, regarding a core group of gurus. I don’t think that is so far fetched. Although the group might be quite large. In investment management there are a core set of analysts. People who work at investment firms and have certain credentials. However, you should note that for the most part, smart investors discount the importance of someones “credentials” and focus on the credibility of their argument. Also, Google’s Knol concept is based on the idea of “experts”.
I like the idea of an open wiki, but I don’t object to the longer term creation of a community of expert nonprofit analysts.
Dave, regarding “rate your doctor”. It seems to me that Yelp could be a good model. Great Nonprofits is already working on this. I think the “user review” model is part of the equation. But Holden Karnofsky has pointed out correctly that the question is not do donors, volunteers, recipients, or staff like a nonprofit, but how effective the nonprofit is. I’m sure a homeless person would give a 5 star rating to a nonprofit that handed out stacks of cash each week, but that doesn’t mean it is effective. Volunteers and donors can fall into the same trap by giving high ranks to a nonprofit that holds wonderful appreciation events and makes supporters feel all warm and fuzzy when they visit.
Dave, re: transparency. Your view assumes that if nonprofits are transparent, we’ll all see how bad things are inside and won’t want to give to them. I believe that many, many nonprofits are doing wonderful work and are full of dedicate, bright, passionate people. But all their “marketing department approved” messaging bores donors to tears.
I think the the first movers on transparency would naturally be some of the highest impact organizations (because these are the ones that would believe the most in themselves) and that they would find it to be a huge benefit.
For a controversial example look at GiveWell. I’m sure some people think GiveWell showed a failure of transparency, but that’s missing the real story. GiveWell was two young guys who consistently admitted that they had all sorts of flaws and regularly said that they didn’t know the answer to numerous issues. They regularly claimed to NOT be experts in their chosen field. Yet they attracted Lucy Bernholz to their board, massive mainstream media coverage and potential funding from the Hewlett Foundation (still under consideration as far as I know).
Showing people that they didn’t know a lot of things was not their downfall. Their problems came from anti-transparent actions.
(PS: Personally I’m sick of talking about whether GiveWell was good or not, but I’m happy to discuss what GiveWell’s experiment in transparency means to nonprofits. I think their experiment was an objective success if you measure it by how transparency helped their organization succeed.)