Category Archives: Evaluation

How Much is Philanthropic Advice Worth?

August 28, 2008 – 9:41 am

(Note: The kind of advice I’m discussing in this post is grantmaking advice. In other words “which nonprofits should donors invest in”. My firm, Ensemble Capital Management, does not provide this sort of advice. We provide wealth management and philanthropic concierge services)

Recently I predicted that donors would move towards paying for grantmaking advice and that this would transform philanthropy. So how much and why will donors find it valuable to pay for this sort of service?

Let’s say there are five nonprofit organizations working on reducing homelessness in your city. Even without any quantitative impact statistics, it is completely commonsense to assume that some of those organizations are better than others. In fact, it is likely that one or two are significantly better than the others and that one or two are significantly worse. Just for the sake of argument let’s make some assumptions about the organizations.

Let’s assume that the five organizations have the following quantitative social return as measured by reduced cost to tax payers from reduced homelessness (ie. a return of 0% would mean that every dollar given to the organization reduced tax payer expenses by $1).

Org A: 35%, Org B: 25%, Org C: 10% Org D: 0% Org E: -20%

This example assumes that Org A and B are achieving great tax payer savings, Org C is producing a positive result. Org D is reducing tax payer costs on a dollar for dollar basis with donations (meaning the same results could be achieved by making donations to the government treasury). Org E is actually destroying value. If you gave an equal donation to each organization your average return would be 10%, so we can think of this as the “market return” or expected return.

Right now, I think that philanthropists are funding nonprofit organizations without focusing enough on which one is best. With this example we can see just how valuable advice that identifies the best nonprofits actually is.

If you gave $1 million to Org A, you would generate $1.35 million in tax savings. The same gift to Org C would save $1.1 million and a gift to Org E would save $800,000. If you assumed you couldn’t tell which one was best and gave $200,000 to each organization, you would save $1.1 million in taxes. This means that if you could pay an advisor to tell you which organization was the best, this advice would be worth $250,000 (the difference between the $1.35 million savings from funding Org A vs the $1.1 million savings from equally distributing your donation).

If you run the math, this means that an advisor who can correctly pick Org A out of the group can justify charging a fee equal to 18.5% of the donation. (Email me if you’d like to understand the math).

Most all donors would choke on the idea of paying an 18.5% fee on their donation. Donors don’t even like nonprofits have operating expenses, so most couldn’t stomach the idea of paying hefty “operating costs” just to decide which nonprofit to give to.

And yet by not paying and not identifying the best nonprofits, donors are destroying social value. They are starving great organizations of resources and subsidizing poorly performing nonprofits that should be going out of business.

The fact is, the spread of returns in my example is probably conservative. Those returns (from +35% to -20%) are more typical of the range you’d see in small to midsize businesses that are functioning in an imperfect market, but one that still rewards top performers and squeezes bad businesses until they go bankrupt. It seems to me that since in the nonprofit world, poorly performing nonprofits that have good fundraisers can stay in business, the spread between the best and worst performers is probable much larger than my example suggests (which means the advisor’s advice is worth more).

Now think about this: Americans gave $300 billion last year. 18.5% of that amount is $56 billion. It might seem ambitious, but that means that there is a potential $56 billion a year business opportunity for grantmaking consultants. That’s almost twice the size of the market of wine and beer. It is almost twice the size of the amount Americans gamble every year.

18.5% seems like a really high fee. $56 billion seems absurd. But think about it this way: If you endowed American giving, so that the $300 billion was a 5% annual payout (in line with most foundations) of our “community endowment”, then the fee would be equal to 0.93% of the endowment.

Guess what many for-profit investment advisors charge their clients? 1% of assets under management.

Being able to identify the best for-profit investments is a hugely valuable talent and a massive industry has grown up around it. I think that nonprofit analysis is just as valuable and we’re going to see quite a robust industry grow over time.

By Sean Stannard-Stockton | Also posted in Effective Giving, Impact Measurement, New Philanthropy, Philanthropy | Comments (3)

Philanthropy Evaluation: The Courtroom Approach

July 11, 2008 – 9:38 am

Steven Mayer is one of the people behind the Pathways to Progress website, dedicated to social justice philanthropy. Albert Ruesga wrote recently that “metrics based” philanthropy and “social justice” philanthropy are often viewed as two warring camps. But Albert suggested that in fact, “We fail to appreciate how closely united these two camps are in their rejection of philanthropy as usual.”

Today I want to highlight a recent essay by Steven Mayer that I think shows an approach to metrics that has the potential to bridge the divide between “metrics” and “social justice”. Because of the way Steven frames his approach to evaluation, I think he even presents a way to think about these issues that bridge the gap between the participants in the philanthrocapitalism debate.

Steve Mayer:

Our website JustPhilanthropy.org presents many productive avenues for pursuing social justice using the resources of philanthropy. Funders, nonprofits, and potential donors exploring these options frequently ask, “How can we evaluate these efforts?”

Needed: useful evaluation questions

“What is being achieved through this effort?” and “What kind of results are you getting?” are worthwhile questions that must be addressed to be fair to those who support this work. But demands for “measurable impact” and “outcome measures” are inappropriately placed on separate, local efforts; they apply more to the bigger picture, the picture indicated by disparities data. This is not to avoid the questions, but instead to find better ways of answering them. More satisfying data that inform next steps, stimulate innovation, engage participating stakeholders, and make better use of scarce philanthropic capital would come from asking for “evidence of progress” or even “early signs of impact.”

Think courtroom, not science

To appreciate these better questions, try this mental exercise: assume the program you support or operate has been accused of being trivial or ineffective, doing nothing to reduce disparities or improve social justice. What evidence could you provide in its defense? Think of a parade of witnesses testifying from their unique expertise, vantage point, experience, and vested interest. What “portfolio of evidence” could make a case good enough to persuade a jury of peers that this work, when considered in context, is useful and necessary for closing a key disparity?
No less rigorous or accountable

Asking for “evidence of progress” is by no means a diminished demand for rigor. Instead, it frames evaluation in more familiar and approachable terms. Data of all kinds can be considered — numbers, stories, graphs, pictures, records, opinions, artifacts, etc. There is no single “measure” that communicates effectiveness or truth, just as in a courtroom no single witness provides all the testimony. In a court, multiple lines of evidence are entered and judged on their merits, resulting in conclusions that stand tests of credibility and accountability.

I’ve seen lots of frameworks borrowed from different disciplines in an attempt to find a good way to evaluate the effectiveness of nonprofits and philanthropy. I even once suggested that we should turn to movie critics as a model of how evaluation should be performed. But I think that Steven’s court room framework is elegantly simple and captures the way I think of evaluation and impact analysis better than I have ever been able to describe.

The fact is, I think that the best way to evaluate the social sector is via a system similar to investment research on publicly traded stocks. But to most people, this is an alien system that they incorrectly believe is concerned only with quantitative evidence. In actuality, stock market analysis is much more like the idea that Steven presents, “Data of all kinds can be considered — numbers, stories, graphs,
pictures, records, opinions, artifacts, etc. There is no single
“measure” that communicates [the potential of an investment idea].” But most people do understand that in court rooms all sort of evidence is presented and evaluated as a composite whole.

Steven’s framework is brilliant. The one point I would make (I think Steven would agree although he doesn’t make this point explicitly) is that in the court room things musted be proved “beyond a reasonable doubt.” This makes sense because the ramifications for deciding incorrectly are very high. But in the nonprofit/philanthropy world, we simply need to get to a point where it can generally be agreed that some funding opportunities are better than others.

Bravo Steven!

By Sean Stannard-Stockton | Also posted in Cross-Disciplinary Conversations, Effective Giving, Impact Measurement, Philanthrocapitalism, Philanthropy | Comments (4)

Project Streamline

May 30, 2008 – 8:09 am

The Project Streamline report begins:

A national organization has dozens of foundation funders, each with a distinct application process, different requirements, and its own cycle for funding.

As part of their annual report to a funder, staff from a nonprofit service agency have to categorize their clients according to the funder’s specifications, even though the categories are not the same ones that the nonprofit uses.

Three times each year, a family foundation with broad funding guidelines receives 70-80 proposals in the mail. This overwhelms the single staff person, as well as the board members who serve as program officers.

Most grantmakers take their responsibilities to support nonprofit and other public-serving organizations seriously, and spend considerable time thinking about how they can be most effective. Stories of highly productive, warm, and mutually satisfying partnerships between organizations and their funders abound. Yet the grantmaking process is rife with inefficiencies such as those suggested in the above stories, and these inefficiencies mean that everyone is wasting time and money that could be devoted to accomplishing missions.

The Project Streamline report goes on to outline how incredibly inefficient the grantmaking process of foundations are and says:

Determined to address the great waste of time and energy caused by inconsistent and inefficient reporting and application procedures, eight organizations representing grantmakers and grantseekers came together to form Project Streamline. Project partners include the following organizations:

  • Grants Managers Network
  • Association of Fundraising Professionals
  • Association of Small Foundations
  • Council on Foundations
  • Forum of Regional Associations of Grantmakers
  • Foundation Center
  • Grantmakers for Effective Organizations
  • National Council of Nonprofit Associations

Together, these diverse partners commissioned a scan of grant application and reporting practices, their impact on grantseekers and grantmakers, and the implications for the field. This report is the result. Its goal is to spark thinking and dialogue on this topic across a wide range of grantmaking stakeholders of all shapes and sizes. (Emphasis from the original).

The report cites “Ten Flaws in the System” and three “Creative Approaches” to fixing the system. Given the readership of this blog includes a pretty even split of funders and grantees, I thought I’d run with the reports hope to “spark thinking and dialogue on this topic” and start discussing the report here. I’ll start Monday with “Flaw #1″. Some initial comments on the report can be found on the Project Streamline website (is comment #1 ironic? I sure hope so!)

One quote from the report caught my eye. It is one of those things that is both shocking as well as unsurprising to anyone who knows philanthropy.

“The administrative burden placed by funders on community nonprofit organizations is so heavy and so unrelenting, and places so many constraints on their ability to operate that it is a wonder they can deliver any services effectively.”

—Lynn Eakin, from We Can’t Afford to Do Business This Way

By Sean Stannard-Stockton | Also posted in Cross-Disciplinary Conversations, Effective Giving, Foundations, Grantmaking, Impact Measurement, Philanthropy, nonprofits | Comments (2)

The Tyranny of Metrics

April 30, 2008 – 2:53 pm

Reader Jeane Goforth writes:

Evaluation. My co-founder and I spent 2 hours standing on a street corner ‘after’ work discussing how to measure and convey the profound experiences we have daily. My stomach churns and my shoulders ache when our expert non-profit adviser talks about metrics. I struggle to add one more thing to my to-do list and I know metrics don’t say enough about why what we’re doing works and why it is important.

To me, this comment sums up everything that is wrong with metrics, both how they are dismissed by detractors and how they are misapplied by advocates.

Bad metrics are “one more things to my to-do list”. If the metrics don’t help the nonprofit run their organization better, than their relevance should be questioned. At Ensemble Capital we track a number of metrics about the performance of the firm. We don’t do this to show them to anyone else, we do it because they help us understand our organization better. But we also know that the metrics that are trackable do not capture everything (or even most) about our business.

One of the most important events in Ensemble’s history (in my mind) was when I was making a presentation to the CEO of a public company about why he should consider opening a private foundation. Half way through the conversation he cut me off and said, “Let’s do it. This makes all the sense in the world. A couple of years ago I asked my CPA if I should have a foundation and he gave me one answer. Then I asked my lawyer and he gave me another answer. But clearly you understand the reasons that I personally need a foundation, so let’s get one started today.” This individual had access to some of the best advisors around. Yet they failed him when it came to philanthropy. That was when I knew Ensemble was going to be a success. But that statement doesn’t show up in any metrics that we track.

Every nonprofit should be searching for relevant metrics to track that can help them run their business. If you don’t realize that total donations are going up only because two large donors have significantly increased their giving, then you will not see the increasing risk to your budget of losing those two donors. If you are working with high school students to help them get into college and the rate at which students are going to college is not budging, you have some examination of your program to do (although it doesn’t mean by itself that you’re failing).

But metrics that do not help the nonprofit are probably useless to donors. If a donor asks for a metric that you do not think is relevant, it is either because you are mismanaging your organization, or far more likely, the donor does not really understand what you do.

Quantifying “doing good” is tough if not impossible. But the idea that nonprofits don’t have time on their to do list to think about whether they are doing a good job is poison. Anyone running a for-profit or nonprofit organization should be thinking everyday about how they can do better and what tools can help them understand their organization. If “tracking metrics” is something that is just busy work, then it is useless work.

By Sean Stannard-Stockton | Also posted in Impact Measurement, Philanthropy | Comments (5)

Evaluation 2.0

April 25, 2008 – 10:21 am

I believe that in many aspects of life, a kind of pendulum effect exists. This effect describes the way in which people’s opinions tend to swing back and forth around reality. Rather than reflecting reality, people’s views vacillate in an arc around true reality. This creates a kind of boom/bust scenario that is very evident in the stock market (dot com stocks will make me rich! Ahh! Dot com stocks are poison!), but also shows up in politics, educations, pop culture, etc. I ran across a well put description this morning on Yahoo Answers:

I think the idea is that: somebody gets a good idea, and then a whole lot of people 1/2 understand it, and make it absolute, taking it to an extreme. Until somebody discovers “another” great idea, (the way things were originally done), and everybody jumps on that runaway train to hell.

The lesson: Moderation. A little common sense goes a long way. Don’t just “swing with the pendulum” of fashion in teaching/learning methods.

I think that there is a strong pendulum effect in philanthropy. I see it at work when we talk about metrics, evaluation, “philanthrocapitalism”, venture philanthropy, etc. Today I want to share with you an excellent article in the Financial Times by Gara LaMarche, the president of The Atlantic Philanthropies. I think LaMarche describes well the way in which approaches to evaluation have “swung too far” and his recommendations for a middle ground makes a ton of sense.

The philanthropic world, poked and prodded by a wave of new donors fresh from success in the business world, is grappling with the issue of evaluation. How do we know that grants – or, as they are now often called, reflecting the influence of the profit-making sector, “investments” – are making an impact?…

…Evaluation is a learning tool for the organisation and the funder, not a stick with which to beat grantees.

…Doing this correctly takes money… Funders should recognise and support their grantees in their efforts to learn what works.

…Evaluation should measure only what is important. Data should never be collected for the sake of it. The “metrics” obsession that has overtaken some funders has not always recognised this. Funders should never make grantees jump through hoops, distracting them from their core mission and costing valuable staff time, for reporting on trivial things. And there is nothing more demoralising, from the grantee’s perspective, than doing all this paperwork only to have it ignored.

…Both funders and the organisations they support need more humility about cause and effect. Organisations working for social or policy change should understand that no significant change was brought about by one organisation working alone.

…Finally, the most important thing: start with what you believe. If you have a passion about ending the death penalty or the isolation of older people – whatever it is – find a way to advance it first and worry about how to measure it second.

You can read the full, excellent article here.

By Sean Stannard-Stockton | Also posted in Cross-Disciplinary Conversations, Impact Measurement, Philanthropy | Comments (5)

Short-term vs. Long-term Focus in Philanthropy

April 4, 2008 – 8:03 am

In the Summer 2007 edition of the Stanford Social Innovation Review, Charles Conn, a senior advisor to the Gordon and Betty Moore Foundation and a high tech executive, wrote about the short-term focus of most foundations in an article titled Robbing the Grandchildren:

If future generations could vote on how foundations invest their money today, would they choose the current allocation? Byron Swift, chair and executive director of the World Land Trust, suggested this thought experiment to me, and I am disturbed to find that my answer is no……U.S. charitable foundations are better positioned than companies, governments, and universities to address these long-term, potentially catastrophic problems. One of the few sources of long-term risk capital, they control more than $500 billion in assets, generating funding that with other charitable giving totals almost 2 percent of GDP. With Warren Buffett’s gift, the Gates Foundation alone will control more than $60 billion in assets and $3 billion to $5 billion in annual spending. Other foundations closely associated with the digital revolution (such as Dell, Ellison, Packard, Hewlett, Moore, Omidyar, Page and Brin, Yang) could account for at least $50 billion to $70 billion more.

Perversely, though, many of these new tech entrepreneurs are worsening foundations’ shortsightedness by implementing businesslike metrics and controls in a way that reinforces short-term thinking and behavior. Other questionable management practices, such as low payout rates and lack of coordination with other organizations, further aggravate foundations’ myopia…

…A recent movement, sometimes called philanthrocapitalism or venture philanthropy, seeks to avoid complacency and lack of focus in foundation management by introducing rigorous success metrics and accountability practices. Many of these new-style foundations limit their scope to a few problem areas and, like corporations, intensely monitor outcome metrics, often with tight windows for review. To those of us who came to foundation work after a career in business, this sounds eminently sensible; after all, the foundation world is littered with fragmented, unfocused, and failed programs…

This short-term, metric-focused approach likewise hampers grantees. Foundations take the passionate and committed people in these institutions and harness them to near-term indices of progress. Grantees, in turn, stop playing the long-term game in order to keep the money flowing. They aim lower, too.

I agree completely with Conn’s thesis, but I want to elaborate because Conn fuses “short-term”, “metric-focus” and “businesslike” as if they automatically go hand in hand.

In the stock market, most people have become more and more short-term oriented. In the 1950’s, investors held stocks for an average of 7 years. Today the average is 11 months. Investors have gained access to vast amounts of information they never use to see and yet in many cases have used this information to change their mind more frequently rather than to gain more conviction in their decisions.

Almost all great investors make financial decisions based on a long-term outlook, not a prediction of what will happen in the next three months. At my firm Ensemble Capital Management, we talk about “arbitraging other investors’ time horizons”. In other words, we try to identify situations where short-term bad news about a company causes other investors to sell the stock so that we can buy it at lower prices. We use short-term good news that causes a stock to move higher to sell stock in companies whose longer-term outlook we think is deteriorating. Warren Buffett or most any great investor will tell you that Wall Street’s obsession with quarterly earnings reports is misplaced. Some companies (such as Coca-Cola, a company Buffett has owned a long time) have stopped issuing guidance to investors regarding what their next quarterly earnings might be.

It is human nature to want results as quickly as possible. But to achieve success, we must match our investment decisions to our time horizon. If we want to fix a local school because our child will be attending starting next year, then it might make sense to focus on short-term solutions. But most donors fund issues because they want to have a sustained impact on a situation. The techniques that might reduce crime in a bad neighborhood the most over the next month are unlikely to be the techniques that will have the largest, permanent impact on reducing crime rates over the next couple of decades.

Financial market participants are often short-term focused. They often focus on metrics which describe short-term conditions, but do little to illuminate long-term trends. But great investors and great philanthropists must focus on the information that matters to the long-term success of their projects.

Disclosure: Nothing in this post should be considered investment advice.

By Sean Stannard-Stockton | Also posted in Effective Giving, Grantmaking, Impact Measurement, Philanthropic Capital Markets | Comments (0)

Albert Ruesga on Metrics Mania

March 26, 2008 – 9:20 am

Albert Ruesga, who blogs at White Courtesy Telephone, was a panel member at the recent Metrics Mania debate at the Bradley Center for Philanthropy & Civic Renewal. Albert posted the text of his comments today and they are outstanding.

At a time when we are seeing a growing backlashing against “philanthrocapitalism”, it is interesting to look at what is being grouped under that term. For many people, “metrics” and the push for more “evaluation” of philanthropy is an unwelcome element of a “business-like” approach to giving. I believe that evaluating nonprofits and philanthropy in general is necessary for a the Third Sector to become a high-performing, high-impact driver of social good. But as I wrote last week, I think that much “evaluation” takes a scientific approach to measurement that is borrowed from the hard sciences, while the lessons of the liberal arts (under which investing and financial markets should be categorized) are more appropriate.

Albert, I think, would agree. He writes:

Measurement and evaluation, when done properly, are not just a bit of value-added for philanthropic or nonprofit work, they’re absolutely essential. Only a fool would disagree with that proposition.

But here I mean not just the kinds of formal evaluations described by Gary Walker in his essay, but informal evaluation as well: the kinds of course corrections we naturally make when we embark on a project, take a false step, and adjust what we do accordingly. Evaluation is not and should not be the sole province of the highly compensated consultant. We evaluate all the time; our own eyes and ears notice things the most astute consultant will never notice; and we’ll often be our own worst critics.

Now here’s where the metrics schmetrics comes in, perhaps: More nonsense has been spoken and written about evaluation than about any other subject in philanthropy. The number of people practicing evaluation without a license and without a proper scientific and philosophical grounding in the subject is, in my view, a scandal. Worries about evaluation, engendered in part by logic models the length of whale intestines, have become the math anxiety of the philanthropic world…

…I want to make clear that I’m not in the least anti-evaluation. As I’ve written elsewhere, I’m concerned that we tend to seek a kind of scientific or moral certainty from a formal evaluation where none exists. The questions that funders most often bring to an evaluator—Was this program worth our $25,000 investment? Should we continue funding it?—are questions only they, the funders, can answer. Say we measure a 25 percent drop in the truancy rate for a hundred kids in some program, and a 25 percent increase in their test scores. Is that worth $25,000 to you? Each donor needs to answer that question for him- or herself. As donors we will never be absolved of our responsibility to use our good judgment.

“Evaluation… the math anxiety of the philanthropic world.” What a great line. Evaluating nonprofits and social good in general is not about math. When I advocate for a financial markets-type approach to evaluation, I am not calling for number crunching. I wrote about this concept in January and will repost my comments from then rather than repeating myself:

Economics is often called the “dismal science”. I know that many people think that finance is boring. But the vision of financial markets as nothing but numbers and spreadsheets does not capture the reality. Do investors buy stock in Apple because they spent hours and hours processing spreadsheet calculations? No. While at the end of the day, buyers of Apple stock believe that the return on capital being generated by the company will make for a profitable investment, the information they use to determine that are not just numbers. The way in which Apple has captured the imagination of the consumer, (an intangible piece of data that cannot be added to a spreadsheet) is by far the most valuable asset that Apple has and it is a major reason why investors have flocked to the stock.

Have you ever watched CNBC, the news channel of the financial markets? It is far from some kind of spreadsheet crunching lecture. Every day, investors or all types come on the show and make passionate arguments for why certain companies are good investments. While numbers and calculations underlie much of their thinking, it is the story, the human story of the companies they discuss that take center stage.

Warren Buffet is widely considered the best for-profit investor of his generation. Does he sit in a corner office reading a spreadsheet the way that Phil suggests? The quote below is from noted investor Whitney Tilson (Tilson is a huge fan of Buffet and a fellow columnist of mine at the Financial Times):

If the future were predictable with any degree of precision, then valuation would be easy. But the future is inherently unpredictable, so valuation is hard — and it’s ambiguous. Good thinking about valuation is less about plugging numbers into a spreadsheet than weighing many competing factors and determining probabilities. It’s neither art nor science — it’s roughly equal amounts of both.

The lack of precision around valuation makes a lot of people uncomfortable. To deal with this discomfort, some people wrap themselves in the security blanket of complex discounted cash flow analyses. My view of these things is best summarized by this brief exchange at the 1996 Berkshire Hathaway annual meeting:

Charlie Munger (Berkshire Hathaway’s vice chairman) said, “Warren talks about these discounted cash flows. I’ve never seen him do one.”

“It’s true,” replied Buffett. “If (the value of a company) doesn’t just scream out at you, it’s too close.”

Taking liberties with Tilson’s quote, I would argue that donors should not “wrap themselves in the security blanket of metrics” because “the lack of precision around measuring the impact that nonprofits achieve makes them uncomfortable.”

World-class investors do not sit in their office crunching spreadsheets all day. Neither should world-class donors. But the underlying logic of both should be that of achieving the highest return on investment.

By Sean Stannard-Stockton | Also posted in Cross-Disciplinary Conversations, Effective Giving, Impact Measurement | Comments (6)

Do Nonprofits Want Funders to Be Critical?

March 5, 2008 – 9:25 am

A comment from an anonymous “director of development” was posted today on my Donors vs Investors III post (check out the growing conversation in the comment section of this thread):

Just to bring another perspective to this line of questions, here’s a fundraiser’s take. I’m sure there are lots of forward thinking, transparent non-profits out there who can speak candidly with anyone about mistakes and areas to improve, but my sense is that the vast majority are like my employer: they would never let any  information that might even suggest something less than sparkling about them be publicly revealed.

We have one foundation funder who is openly critical of us, and funds us with a contract and a set of concrete tasks the organization accomplish. I would call this funder a proactive investor. They didn’t just evaluate us, they made their findings known, and better yet continued to offer us money if we made an effort to clean up our act. Many staff are grateful for this funder, and believe our organization has improved with its participation.

So I guess my point is, perhaps a non-profit is best served by funders who can own their criticism, stand by it and use it as a tool. The many many non-profits out there who are less interested in critically evaluating themselves can benefit from proactive investors like the one I have described. And at least when I am in the room with this funder, I am more or less confident that what they say about us at conferences is what they say to my face.

Wow. I might be advocating for a more public dialog, but I’m surprised as anyone to hear a nonprofit employee say their organization has benefited from a major funder being openly critical.

By Sean Stannard-Stockton | Also posted in Cross-Disciplinary Conversations, Effective Giving, New Philanthropy, Philanthropic Capital Markets, Philanthropy, Transparency, nonprofits | Comments (7)

Investors vs. Donors

February 27, 2008 – 8:30 am

I have some questions and would love your feedback.

  1. Why do investors take credit for picking great investments (”look how smart I am, I bought XYZ stock!”), while philanthropists, especially foundations, claim that the credit goes to the nonprofits they fund (”the grantee did all the work”).
  2. Why is it acceptable for investors to talk about investments they think are bad (”Don’t buy ABC stock, their management is terrible!”), while philanthropists never badmouth nonprofits, even if they think they are ineffective?
  3. Related to #2: Why do public companies generally ignore all the talking heads who say negative things about them, while nonprofits find it intolerable to have a prominent person speak negatively about them in public?

If you’ve read this blog for awhile, you probably think these are leading questions and that I have a firm bias about which approach is better. But I’m truly asking these questions with an open mind. Recently I met with the director of philanthropy-focused grantmaking at a large foundation. I brought up the idea that publicly talking negatively about ineffective nonprofits (especially those that the foundation thought were not “fixable”, say because management was incompetent) would produce a positive social impact by directing other donors’ funds away from the bad nonprofit and towards more effective competitors. She told me that it was a primary value of the foundation to not harm grantees.

I think that is a very compelling counter argument and I’m interested in how readers view my three questions above and the idea that not harming grantees means never saying anything negative about a nonprofit.

By Sean Stannard-Stockton | Also posted in Effective Giving, Foundations, Grantmaking, Philanthropic Capital Markets, Philanthropy | Comments (9)

The Foundation Review

February 14, 2008 – 9:19 am

Tactical Philanthropy reader Teri Behrens (director of evaluation at W.K. Kellogg Foundation) is the inaugural editor of The Foundation Review, launching later this year:

I am very excited about this new venture because I believe the time is ripe for a peer-reviewed publication that will provide practical information about what works — and doesn’t work — in foundation-funded programs.  The target audience for the journal is foundation staff and boards, and those who work with them to implement and evaluate programs.

Anyone who works in philanthropy can point to stacks of technical reports about the field of philanthropy, and about the results accomplished by philanthropy — often published by foundations themselves.  However, there is no central place in which to publish these reports, making retrieval and use of the information challenging at best, and typically there has been no external review.

The Foundation Review will seek to fill this niche.  We are calling for articles written in an accessible style, emphasizing what was done and why, and describing the results and recommendations.  Articles should look more like the report a consultant would submit to a foundation or non-profit, rather than the typical heavy-on-theory academic discipline-based article. The topic for the first issue is comprehensive community change initiatives — see the attached for more information.

Please consider submitting an abstract by March 1, 2008

You can find a complete description of The Foundation Review with instructions for submitting papers here.

By Sean Stannard-Stockton | Also posted in Effective Giving, Philanthropy | Comments (0)

More on CalHospitalCompare.org

February 11, 2008 – 9:00 am

Last week I mentioned CalHospitalCompare.org, a website that shows comparative rankings for the effectiveness of California hospitals.

A reader emails today with some thoughts on problems with the measurements used by the site. He points out that the vaunted Cedars Sinai of Beverly Hills has only an “average” ICU mortality rate according to the site. The site assumes that a hospital should strive for a lower mortality rate (less patients dying is good right?):

Why are we pedantically told “lower is better”?  Why is lower better?  Is this not exactly, specifically, the dead-nuts center of the “metrics” conundrum?

If the Cedars ICU mortality rate was 50%, wouldn’t that mean Cedars must be where everyone in the know knows to go when intensively sick??????  Included among them, obviously, are the intensively sick at death’s door — thus inflating the ICU mortality rate with folks whose only hope is for more time in ICU — not escape from it.  “Mortality” measures nothing meaningful about the care and skill and hand-holding services provided AT DEATH’S DOOR in the Cedars ICU!

That 50% ICU croak rate might just mean Cedars is the place to go to be kept ticking long enough for all my family stragglers and long-losts to fly in and come kiss me goodbye?  I’d still add to the croak rate, but…  Wow!  What service!

What the hell kind of “metric” measures that PRICELESS service???!!!

What about the exact opposite “metric” finding:  Suppose Cedars ICU mortality rate was 1%.  This means, of course, if the “average” is 13%, that Cedars is either magical — or lying.  Let’s say they’re just wonderful.  And…  so…  via “metrics” getting into the hands of salesmen…  everyone discovers how wonderful Cedars ICU can be.  So business goes up on the word — logically, in the fullness of time.  Now…  More and more people at death’s door come to them for those precious last few hours or days…  But.  In the end, folks croak.  So?  Little by little, Cedars ICU mortality rate gets back in line with “average” and, like all intended measurements for goodness, is rendered meaningless.

Why, why, why, why so much time on this obsession with measuring The Good?

The emailer then takes me to task for focusing on measuring nonprofit outcomes. I found his argument above excellent. This is a great example of how important it is to measure the right things. We cannot depend too heavily on any one metric. There is no magic, simple way to determine how effective a nonprofit is.

Right now, any decent economist could give you a long list of statistics that show we are in a recession… and a list of statistics that suggest we are not. That doesn’t mean that metrics are worthless. It does not mean we should not strive to seek knowledge. It means that understanding our world is hard. But through understanding our world, we can create a better one.

By Sean Stannard-Stockton | Also posted in Cross-Disciplinary Conversations, Effective Giving, Impact Measurement, Philanthropy | Comments (0)

Do Nonprofits Trust People?

February 11, 2008 – 8:00 am

A new blog I’ve been following, LifeYears, asks an interesting question today:

Non-profits seem sceptical towards outcome-based indicators of how effective different initiatives and projects are. Is this because they are afraid of what “ordinary people” would think and do if they had this information? And if so, could such a worry be rational?

The author, Ole Rogeberg, poses the question very seriously without passing judgment:

Personally, I hope (and want to believe) that outcome-based indicators would make the connection between giving and accomplishing stronger, that people would not just give in order to “do a good deed” but also start giving to “have an impact on malnutrition” or “reduce malaria” etc. But other people might believe that the consequences would be different. But other people might believe that the consequences would be different. Two possible worries that immediately come to mind:
  • Myopic funding: The more concrete, short-term and certain a program was according to its outcome indicator, the more funding it might receive. This might make it easy to get funding for vaccines and malaria nets, but hard to get funding for investments and development in health infrastructure, education, human rights work with women, etc.
  • Over-focused funding: Some measures may catch the public’s interest far more than others (”go viral”), and this might lead to large shares of funds going to these areas - not because the need is the largest or the impact the greatest, but simply because the indicator is catchy in some way.

Ole ends:

Even if you personally don’t see the problem - how can we answer this fear and develop indicators that don’t have excessive (largely unintended) negative consequences?

Important questions. Change is difficult and often has unintended consequences. Many, many people are resistant to my calls for more measurement and analysis of nonprofit outcomes. How can we handle their concerns? Or might their concerns be right?

Powered by ScribeFire.

By Sean Stannard-Stockton | Also posted in Grantmaking, Impact Measurement, Philanthropy, Transparency | Comments (5)

United Way Responds to Google Finance Question

January 31, 2008 – 7:29 pm

Last month I posted a question on the Google Finance profile page of the United Way of America. I got an email today from Meg Plantz, director, Impact Design and Learning for United Way. You can catch up on the Google Finance for nonprofits discussion here.

United Way of America’s approach to program outcome measurement, reflected in its manual, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), encourages health and human service agencies to develop ways to measure outcomes quantitatively.  This often is a challenge because many human-service outcomes seem at first glance to be un-quantifiable, and many agencies are used to describing their successes with narrative vignettes rather than with numbers.  When they have used numbers, the numbers often have reflected outputs (e.g., number of classes offered, number of clients served) rather than outcomes (e.g., improvement in parenting skills).  However, for purposes of creating and tracking program improvement over time and demonstrating results to increasingly data-oriented funders, donors, and publics, numerical indications of program performance are important.

Our approach consciously responds to the challenges that quantitative measurement presents.  For example, in our approach, intended outcomes do not have to meet the “measurable” test.  In fact, when agencies are identifying what their outcomes are, we encourage them not to worry about how they will measure them, but instead to focus on what results are meaningful to the program and its clients.

Once the agency is comfortable that it has described the appropriate outcomes for its activities, then we advise them to identify measurable indicators of those outcomes.  Questions such as “What will tell you if clients achieve the intended outcome (e.g., improvement in parenting skills)?  What will you be able to see, count, or measure?” help agencies identify critical aspects of their outcomes (e.g., using age-appropriate discipline methods) and think about ways to quantify them (e.g., record observations of parents in role plays and tally entries that parents make in a journal).  Agencies are encouraged to pick indicators that will provide useful data – data that will help them with program improvement, in communications, and in other management tasks.

Qualitative information complements and can help with quantitative measurement.  For example, agencies can use their narrative success stories to identify intended outcomes and then use quantitative measurement as a way to learn whether the stories are unique or are representative of other clients.  Agencies can use their narrative stories to illustrate their outcome data rather than offering the stories as evidence of outcomes.  Qualitative information such as focus groups discussions of quantitative data can help agencies understand the meaning of the data and make appropriate program improvements.

Meg Plantz
Director, Impact Design and Learning for United Way

By Sean Stannard-Stockton | Also posted in Effective Giving, Impact Measurement, Philanthropic Technology, Philanthropy, Transparency, nptech | Comments (0)

Nonprofits Publishing Impact Analysis

January 29, 2008 – 10:14 pm

A nationally known nonprofit has asked me for examples of organizations that are “doing it right” in terms of publishing their impact analysis. In other words, which nonprofits are self publishing information that is useful to donors who are trying to examine if the organization is effective?

Any ideas? Leave a comment or send me an email.

By Sean Stannard-Stockton | Also posted in Impact Measurement, Philanthropy, nonprofits | Comments (7)

Efficient Markets in Philanthropy

January 24, 2008 – 9:25 am

In response to my post yesterday in which I discussed the value of information to philanthropy and why donors should desire efficient philanthropic markets, Phil Cubeta writes:

The logic here can become relentless and destructive. What this tends towards a lists, like league tables in a sport, with the best at the top. It leads then to managing a nonprofit by the numbers, to get the rating, and it leads to shutting down those that don’t rank high. We then have the tyranny of the metrics, however much arbitrariness is built into them…

The world you want - are you sitting in corner office reading a spreadsheet?

So are the philanthropic capital markets I envision boring and lifeless with endless spreadsheets and numbers to crunch? Not in the least.

Economics is often called the “dismal science”. I know that many people think that finance is boring. But the vision of financial markets as nothing but numbers and spreadsheets does not capture the reality. Do investors buy stock in Apple because they spent hours and hours processing spreadsheet calculations? No. While at the end of the day, buyers of Apple stock believe that the return on capital being generated by the company will make for a profitable investment, the information they use to determine that are not just numbers. The way in which Apple has captured the imagination of the consumer, (an intangible piece of data that cannot be added to a spreadsheet) is by far the most valuable asset that Apple has and it is a major reason why investors have flocked to the stock.

Have you ever watched CNBC, the news channel of the financial markets? It is far from some kind of spreadsheet crunching lecture. Every day, investors or all types come on the show and make passionate arguments for why certain companies are good investments. While numbers and calculations underlie much of their thinking, it is the story, the human story of the companies they discuss that take center stage.

Warren Buffet is widely considered the best for-profit investor of his generation. Does he sit in a corner office reading a spreadsheet the way that Phil suggests? The quote below is from noted investor Whitney Tilson (Tilson is a huge fan of Buffet and a fellow columnist of mine at the Financial Times):

If the future were predictable with any degree of precision, then valuation would be easy. But the future is inherently unpredictable, so valuation is hard — and it’s ambiguous. Good thinking about valuation is less about plugging numbers into a spreadsheet than weighing many competing factors and determining probabilities. It’s neither art nor science — it’s roughly equal amounts of both.

The lack of precision around valuation makes a lot of people uncomfortable. To deal with this discomfort, some people wrap themselves in the security blanket of complex discounted cash flow analyses. My view of these things is best summarized by this brief exchange at the 1996 Berkshire Hathaway annual meeting:

Charlie Munger (Berkshire Hathaway’s vice chairman) said, “Warren talks about these discounted cash flows. I’ve never seen him do one.”

“It’s true,” replied Buffett. “If (the value of a company) doesn’t just scream out at you, it’s too close.”

Taking liberties with Tilson’s quote, I would argue that donors should not “wrap themselves in the security blanket of metrics” because “the lack of precision around measuring the impact that nonprofits achieve makes them uncomfortable.”

World-class investors do not sit in their office crunching spreadsheets all day. Neither should world-class donors. But the underlying logic of both should be that of achieving the highest return on investment.

Recently Phil commented to Perla Ni regarding her site Great Nonprofits (which offers reviews of nonprofits written by volunteers, donors and the people served by the nonprofit):

Thank you so much, Perla, for setting the record straight. In fact, your site is the exact opposite of a metrics driven exercise. You are bringing together the voices of those who have been touched by a nonprofit. I finally “got” what you are doing.

An efficient philanthropic capital market does not only view numbers as valuable inputs to the decision making process. Sites like Great Nonprofits offer extremely valuable information to donors. This sort of qualitative information is critical to both donors and for-profit investors. Great Nonprofits is not the opposite of a metrics driven exercise. They are both part of the same process of determining where donors and investors should direct their capital.

Disclaimer: Nothing in this blog should be construed as investment, tax or legal advice. This blog is for informational use only.

By Sean Stannard-Stockton | Also posted in Effective Giving, Grantmaking, Impact Measurement, New Philanthropy, Philanthropic Capital Markets, Philanthropy, Transparency | Comments (7)