David, I understand, rejects much of what I’ve put forward, but it seems he and I both find that we agree with much of what you’ve put forward in your comment.

I’ve laid out what I see as the link between high performance and high impact in a new post here. I hope your colleagues at Social Solutions find my use of the “efforts to outcomes” vocabulary in line with your own views.

Thank you for starting this important discussion about how we can make sure that funding goes to those organizations most likely to improve people’s lives.

There is no question that funding organizations that have proven impact, defined as having a randomized control trial done, is a pretty safe bet. As David Hunter points out, they most certainly do exist. But whether an organization has been proven to have an impact in the past, or never (since most organizations never have an expensive randomized control trial done), they have to be or become high performing in order for us to reasonably expect that people will benefit. Three characteristics strike me as absolutely critical if we are to have any confidence in an organization.

First, the program models have to be reasonable within the context of available research. Sounds obvious perhaps, but here’s a real life example: An organization working with high risk teens (serious behavior problems) decides to give the young people volunteer mentors for the duration of up to three months, and it has a flimsy program structure in place to support the matches.

Available research on mentoring makes clear, among other things, that 1) volunteer mentors can be successful with young people at lower risk, as opposed to this target population; 2) matches have to meet for a minimum of one year before outcomes appear, and those who fall apart after only a few months may not only be ineffective, they may cause harm; 3) it takes a great deal of training and support of both volunteer mentor and the young person in order to develop relationships to such levels of quality that they will benefit the teen. And even then, the benefits are modest. This program, in other words, would likely be doing harm. It is an absolute outrage that programs that completely ignore what is known, to the detriment of their clients, can possibly get even close to funding!

Second, an organization must ensure – with the use of real-time data – that the program is implemented in accordance with the program model. Unless the program is implemented in the same, sound way across staff, sites and over time (until lessons suggest that changes to the model should be made), the expectation cannot possibly be that outcomes will materialize.

The majority of organizations are – once they begin to track whether their programs are being implemented as intended – surprised to find that the difference between theory and practice is unacceptable and raises serious questions about whether the program can make a positive difference. High performing organizations then do what it takes to make implementation conform with the model. Most organizations don’t even keep track.

Given the potential consequences of poorly implemented programming (HIV, child abuse and other negative outcomes may not be prevented, for instance, when better programming could have done so), how is it that we can continue to fund these programs that don’t even attempt to manage performance in programming? How can we accept that?

Finally, while the above two characteristics may be enough during the initial phases of programming, once outcomes can reasonably be expected, they must be discussed. Again, David Hunter is right to point out that there are many levels of evaluative efforts that allow us to address the question: Does the program work? Just because a randomized control trial is out of the question does not mean the program should be incapable of saying something about how well it is achieving its outcomes. A high performing organization collects that data, learns and improves. Looking forward to your post on the topic.

WINGS for Kids is not the only organization that has experienced losing the funding battle to those who make unsubstantiated claims. That tide needs to turn.

I also appreciate your comment that “Sean suggests that figuring out how to achieve high impact is really an inherent characteristic of high-performing organizations.” This is of course my point. That high-impact organizations are the agents that figure out how to achieve impact. But it is clear from other comments that some of my posts have been interpreted to mean that I don’t think it is important for anyone to examine whether the efforts of high performance nonprofits are resulting in outcomes.

I believe that this examination is a critical element of high performance. Without it, the robust seafaring fleet described above turns into a Monty Python skit of a Navy rowing (very efficiently!) in circles or even off the end of the earth.

What concerns me is the commanding, self-confident tone of some other comments where the author clearly has a very incomplete grasp of the field or subject matter.

Two examples:

A number of people have asserted as fact that there are virtually no nonprofits with proven impacts. False. There aren’t hundreds, but there are quite a few if one looks. On my website I have links to about a dozen “evidence based” websites that publish evidence of effectiveness of specific programs and organizations delivering them in areas as diverse as juvenile justice, criminal justice, connecting kids to school, academic achievement and attainment, improving the prospects of kids in and aging out of foster care, preventing teenage pregnancy, workforce readiness and attachment to the workforce (of marginalized groups),etc.

Then there is the “straw man” argument about kinds of evaluations that prove social impacts. Various comments contain irrelevant references to “double blind” studies (utterly inapproriate for all social service impact evaluations) and suggest that in the absence of extremely expensive “gold standard” randomized control trials we can’t say anything about impact. Thus with sleight-of-hand they dismiss the topic of the acute relevance of program evaluation efforts. However, anybody who has more than a smidgeon of knowledge about program evaluations knows there is a spectrum of evaluative efforts, many of which have something to teach us if we are looking for effective programs. Child Trends has just published a nice report on this continuum…I recommend it to anybody who cares to really understand these matters.

These discussions – requiring some nod toward complexity – do not lend themselves to the “sound-bite” world of the blog. Hence I won’t try to elaborate on them. But I don’t think it’s wrong to ask those who are speaking so dispositively on these matters to take the time to get a solid grounding in them before attempting to persuade others of the worthiness of their views.

David Hunter
http://www.dekhconsulting.com

There ought to be a law against making unproven claims.

So often at WINGS, we find ourselves explaining why meaningful metrics are worth every penny we pay for our state-of-the-art performance management system. At times here in Charleston, S.C., we lose out in the competition for scarce funding dollars to other youth development programs who don’t measure performance, and are free to make vague and unproven claims about their impact

In answer to the oft-posed question, is it too onerous for small and lean nonprofits like ours to undergo systematic measurement and monitoring of how well targeted outcomes are being achieved, I say absolutely not – but be prepared to make your case and defend yourself. As Sean Stannard-Stockton says, overhead expense ratios are not a relevant evaluation metric, but all too frequently, that’s how we’re judged.

Laura, I agree with your points that build on mine about customer satisfaction done properly. My view is that we urgently need to improve and expand on constituency feedback practices in the sector. When you discuss these ideas with nonprofit sector leaders, they tend to agree (see our white paper on constituency voice and US human service organizations at
http://www.keystoneaccountability.org/node/241).

The next frontier for Constituency Voice is to develop comparative feedback data sets. And to get them published so that we can increase the incentives for organizations to take feedback seriously. For a for profit analogy, see http://www.jdpower.com/. We are plugging away at this through a number of pilot efforts and would welcome advice and partnership!

Everyone, I think this is a good discussion that is pushing hard in the right directions. It is clear that there is no simple answer here. A pluralistic narrative is emerging for me that carries at least three interlocking strands.

There is a strand about the need for better evidence. The discussion is pushing out the who, what, where, when and how.

There is a strand about structure and roles: how do / should key constituents feature in the structure of organizations? Board, staff, volunteers, beneficiaries, volunteers, champions, donors, competitors. The list goes on and the strand moves from the level of the organization to the larger ecosystem of organizations working on common problems. Parsing through the spectrum from ‘the organization’ to ‘the ecosystem’ is a critical task in this discussion.

And there is a strand, at this point mostly implicit, about the theory of change that underlies notions of performance and impact. I suspect that as tough as it might be to do in an open forum like this one, we need to push out and debate our main assumptions. We have tried to do this at Keystone. You can see the results at http://www.keystoneaccountability.org/about/toc and also http://www.keystoneaccountability.org/node/34.

CAnnot agree more on measuring from the viewpoint of the benefactor.

We do it FOR EVERY program as it is so critical to our ability to program effectively, measure results, make corrections, and know when to scale to the next steps.

I would also argue that doing an impact analysis up front is perhaps just as important as well.

R