I think the the first movers on transparency would naturally be some of the highest impact organizations (because these are the ones that would believe the most in themselves) and that they would find it to be a huge benefit.

For a controversial example look at GiveWell. I’m sure some people think GiveWell showed a failure of transparency, but that’s missing the real story. GiveWell was two young guys who consistently admitted that they had all sorts of flaws and regularly said that they didn’t know the answer to numerous issues. They regularly claimed to NOT be experts in their chosen field. Yet they attracted Lucy Bernholz to their board, massive mainstream media coverage and potential funding from the Hewlett Foundation (still under consideration as far as I know).

Showing people that they didn’t know a lot of things was not their downfall. Their problems came from anti-transparent actions.

(PS: Personally I’m sick of talking about whether GiveWell was good or not, but I’m happy to discuss what GiveWell’s experiment in transparency means to nonprofits. I think their experiment was an objective success if you measure it by how transparency helped their organization succeed.)

I like the idea of an open wiki, but I don’t object to the longer term creation of a community of expert nonprofit analysts.

- Groups like US News use such self-reporting survey systems to evaluate colleges, for example, so I can’t believe it wouldn’t work for nonprofits. If somebody like Google said “fill out this extensive survey in order to be listed in our directory”, I’m sure many would comply, just like the colleges do. So it becomes more of a matter of establishing a good survey (or surveys for various categories).

- There have been attempts, such as http://greatnonprofits.org/

–David

I like the idea (though organizations working on Digital Divide issues would challenge the ubiquitousness of an online medium). I think the issue would be in finding a “generic” set of questions. Something truly generic to all nonprofit areas of work would be very, very abstract…effectively turning into a “confidence vote” by the org’s constituents, kind of like Amazon’s rating tally.

It’s also interesting to me (as a web developer) that building this kind of system would be relatively simple, technologically; promoting the platform to gain acceptance would be the difficult part. Thoughts?

Similar to the sites that try to “rate your doctor”, etc, so there are models out there.

Thanks for the good discussion.

–David

I liked the comparison to movie evaluations above, but the problem there is that we rely on a particular reviewer’s expertise as a guide to how good or bad a movie is; the narrative is meaningful and revealing, but the opinion at the core is one we must trust if we are to place any weight in the review. This would be impossible in the nonprofit world, unless we all recognize a core group of gurus who’s opinion will make or break any organization

@Sean (w/ apologies for the delayed response): Transparency could be a wonderful marketing tool, in an ideal world. The problem is that it requires critical mass before it’s effective, and a larger critical mass before the dinosaurs will pay attention. If *everyone* was transparent, it would change the rules of the game. Till then, no one wants to be first, because it could backlash…and in traditional marketing models, describing your problems, frustrations, and failures in narrative form is sales suicide. No one wants to be first; they’d much rather write happy blog articles about how amazing and uplifting their work is.

I like the idea of tightly tying an organization’s measurables to its mission statement, but the more I think about it the more I think a publicly malleable space for narrative text with an intentional community (perhaps slightly less fanatical than Wikipedia) is a fairly good solution. Let the world comment and critique, let the orgs write back, force a dialog…even for the orgs that would much prefer the comfort of their one-way websites.

Dave.

You can find the new thread here. I’d love to have your input.

The second aspect to this is that different donors (including foundations) will value different things. Perhaps A believes that %-to-admin is the most important, while B looks at volunteer hours leveraged, and C wants to know cost per service unit. XYZ Foundation wants to see new programs developed, and ABC Foundation has their own set of outcome criteria.

Just as investors in the equity markets will look at different metrics, depending on their strategies and beliefs, so do donors in the nonprofit market. The important thing, I believe, is to give donors access to good and verifiable and comparable data, and let them decide what is important.

Juries are asked to judge a case based on a single criteria — The Law. I think that private investment, where both institutional and individual investors are all pursuing different strategies, is a closer model to what happens with donors and foundations.

A much easier way to derive a “system” which could have some resonance at least among similar agencies in similar fields is to identify “best practices” and suggest what “best outcomes” “best reflect” those practices. This begins to build “rubrics,” and those rubrics can eventually become metrics, once there is enough common understanding of what they mean and how they apply.

This need not be another library of cases (ala a B-school). It can be little snippets equivalent to the “knols” that the Google people talk about. For example, volunteers as promoters might be a “knol” with a 1 paragraph snippet to summarize what we mean; volunteers as “trainees,” as “assets,” as “Board candidates,” as “mentors,” etc. All of those could congregate in “volunteers as outcomes” which contrasts with “volunteers as teacher supplements” or volunteers as resources.

In court, a judge and jury dis-aggregates the evidence and synthesizes a judgment based on the sides presented by counsel. This tactic builds rubrics which correspond to the “rules of evidence.” At least it’s a hell of a lot better than a simplification that reflects “spin” more than substance, which is more or less the way it now is.

But how might we judge nonprofits in this way?

I’m also sorry that this thread seems to have lost the fervor of the earlier argument on behalf of transparency, which ought to be a standard (and not a metric) for any expression. If we focus a little on other potential standards, then the questions of metrics and reliable, modestly consistent, reasonably comparable and timely reports makes much more sense.

Before that, however, let me note that I’m surprised no one has raised the MOST important innovation of 990 comparability: they can be filed electronically. That means they could - conceivably - be compared, over time, across agencies, across sectors. That all those data might be compared is much, much more interesting than that they exist idiosyncratically in each filing. Among other things, we could find out which data are useless for anyone, and thereby cut the form considerably; and, conversely, conduct a content analysis to compare narrative descriptions when present, which could, should, or might produce much more generalizable statements at least about sectors (health, education, etc.).

That said, the real question of metrics is what, if compared, would lead to better decisions - by philanthropists as well as by the NPOs themselves. In a painful discussion with a former politician, for example, I asked which elementary schools sent the highest ratio of failing students to 9th grade in the high school. The only response I got was, “you can’t blame the schools,” when the question I asked was how they could be helped to help their kids better.

This question of metrics is probably easier to explore in the context of education than many other NPO enterprises. Not only is there the blistering and ugly dialog about No Child Left Behind, but there really are some fairly simple metrics which, once explored, might lead to better reports and greater … transparency. In other words, it probably behooves this discussion to discuss nonprofit sectors before trying to generalize across sectors - a productive day care center is really hard to compare to a productive hospital or a productive Red Cross. And it won’t ever really compete in a 1:1 with either.

That said, I was struck by one of the Phil’s observations that it’s fairly easy to ask how many teachers teach, students learn, and so forth. That only showed that he didn’t know what schools do. When a school uses a volunteer, for example, the school gains both an input and an output, as well as a fund raiser and political advocate for public funding. As well as a satisfied parent, perhaps, or at least a neutralized nonparent. To measure such a “worker unit” in a single dimension ignores what might, and probably is, a much larger unit of institutional impact.

If its that hard to measure the simple number of teachers, it’s mind boggling to explore square feet (when or if they take field trips), books (when or if they’re on the net), equipment (when or if they have homework or off-site activities), or even meals.

It is much easier to track the unobtrusive measures - attendance, tardiness, staff turnover, hours of classtime, perhaps hours of homework, sometimes costs of materials - and to explore what measures might be needed to give a better profile of outcomes. Surely test scores are some of those measures, but many of the unobtrusives are as well - in their delta values, in the degree to which they may change with certain teachers, certain subjects, certain years, given certain conditions.

With those kinds of data tracked, it is quite conceivable to construct a profile. And, with profiles constructed, we could quite conceivably compare schools, teachers, classrooms, and kids. And we might then begin - just begin - to identify what OTHER measures, either qualitative or quantitative, ought to be collected to make those profiles more useful - to both funders and to the users at all levels.

Finally, I am really surprised that all this dialog about metrics ignores the largest of all funders - government. If an NGO is productive, it probably has government money as well. And if it has any government money, we might then begin to apply those metrics to government agencies - as you have elsewhere implied an evaluation of philanthropy and foundations. Now, is that to be political or “objective.”

Pfui.

It is important to keep in mind that everyone is involved with the non-profit sector. In addition to roles we may choose to play as donors, administrators, volunteers, grant makers, etc., most of us are also involved with NPO’s as tax payers. By accepting the rules which accord non-profits a special tax status, we underwrite them and gain the right to an “adequate” evaluation.

Foundations are NPO’s, although they are not 501C3’s. As NPO’s they get a tax break and are open to public scrutiny.

Given our interest in NPO’s, we need a system that permits us to evaluate them in terms of their efficiency and effectiveness. Reports (such as, the revised 990) provide some critically important information, but the general consensus is that we need additional tools for gathering and reporting information from the broad spectrum of interested parties about the broad spectrum of NPO’s.

New technology provides more opportunities to create more adequate evaluation tools. My personal favorite from this thread is the wiki-based idea, with tabs correlating narrative and numerical data.