Category Archives: nonprofits

Measuring YOUR Nonprofit

During the conversations about what to measure in philanthropy, a dominate theme has been that no universal metric will ever work (although some participants do not agree). This idea is validated by measurement practices in the for-profit markets where different metrics are believed to be important for different companies.

So how should an individual nonprofit think about measurement?

I got the following email from a reader recently:

Your blog. I read it every day. It’s great. But frustrating.

How do WE measure success? We’re trying to implement a program like [deleted to protect privacy]. It will be difficult to quantify success, especially short term. We could have 5 students and really change their lives now–or maybe not be able to point to the impact for years. We could have 50 students and not connect at all. When we discuss this among the staff and with well-meaning supporters, everyone says to just make something up. That really grates on me. And we can’t be the only program with the same problem.

This was my answer:

For a minute, don’t think about numbers. Just tell me what you think your organization would look like in five years if it were successful. For instance, if you raised and spent $1 million and during that 5 years worked with 5 students. Would that be a success? What about 500 students? Or 5,000? If you had a choice between working with 500 students and feeling like you exposed them all to music, but didn’t really change their lives would that be better or worse than working with just 5 students and feeling that you totally changed all of their lives for the better?

After you have an idea of what success would look like, then we can think about ways to measure it.

Rebooting Nonprofit Evaluation Debate

A lively debate about nonprofit evaluation and metrics has been raging in response to my request for input on my meeting later this week with Google.org. However, the conversation has splintered into a debate over whether a systematic, “metric” driven process of scientific measurement is needed, or whether the frame of scientific measurement is “an epistemologically impoverished frame” through which to understand nonprofit evaluation.

I personally believe evaluating nonprofits is mostly about evaluating their output (the social good they produce). Since it is difficult (impossible?) to quantify this output, I think the focus on metrics as a framework for evaluation is misplaced. Metrics can be used, but they should be designed on a case-by-case basis for each situation. That being said, I think the conversation has fallen into the trap of being constrained by historical frames of reference.

I want to have a different conversation.

I’m interested in what information is available to donors who want to evaluate a nonprofit and which of this information is useful. Google.com is mostly a resource that points to information; they don’t tend to create a lot of their own content. So if we imagine a future version of the nonprofit data inside of Google Finance, I don’t imagine it will be some new metric that we design. Instead, it will point to existing information on the web. When I first wrote about nonprofit info in Google Finance, I said I hoped they would not display Charity Navigator ratings (although I would support them noting if a nonprofit had a zero or one star rating since I do believe that a Charity Navigator rating at this level is a significant red flag)

So the conversation I want to have is what information do readers think that donors should consider when evaluating a nonprofit? Then secondly, where or how can this information be captured online so that it can be displayed in Google Finance?

Open Invitation to Foundation Employees

I realize that if you work at a foundation, you may not want to jump into a conversation that involves telling another foundation what to do. However, the conversation we’re having here is really important and would not be complete without the input of the army of program officers (ie. Nonprofit evaluators) that read this blog. So please consider commenting anonymously (just let us know you’re a program officer) or comment publicly and realize that we’re having a broad conversation about nonprofit evaluation that goes beyond Google.org and Google Finance

Open Invitation to Nonprofit Employees

A conversation about nonprofit evaluation would not be complete without the input of the nonprofits being evaluated. What information do you, as nonprofits, what donors looking at when they evaluate you? It could be that someday the Google Finance website about your organization becomes the top ranked search result on google for your nonprofit. What do you want on that page?

What to Measure and Why in Philanthropy

I’m meeting with someone from Google.org next week to talk about what kind of information I think they should make available about nonprofits in Google Finance and other ways that Google.com’s mission statement to “organizing the world’s information” can be directed at the Third Sector.

In preparation, I’d like to spend some time speaking as a community about this issue. I encourage you to leave comments or email me your thoughts.

In response to the thread I started on the Google Finance Red Cross board about how effective they are, I got a comment from Leyla Farah of Cause + Effect public relations:

One item I’d offer: a measurement of “average cost of impact” - in other words, the organization’s total budget divided by the total number of people (or animals, or acres of land) it’s benefited within a specific time period. That metric would (1) force each organization to provide a definition of how it helps people (etc.) - and (2) force it to account for all the costs associated with providing that help.

While Phil Cubeta of Gift Hub scolded me for focusing on metrics:

Paradise Lost versus Gone with the Wind. What metrics do we use to determine which is better? Some subject matter requires judgment, taste, discernment, even wisdom. We have movie critics, book critics, educators to help us make more discriminating judgments. Before we cry ourselves hoarse over metrics, we have to ask whether philanthropy is more like art or more like business. The call for metrics can be a bullying move by the half educated to impose their MBA logic on a sector whose reason for being is that it stands in contrast to both government and business. As the old saying goes, “Do not attempt to cure what you do not understand.” Stressing metrics, Sean, is in terrible taste. You paint yourself as Barbarian.

Personally, I’d like to state that I don’t intend to stress metrics as being valuable unto themselves. However, I do think that all things in life can be judged, at least in each person’s personal view, as being bad, good, better and best (I’m sure there are some exceptions, but you get the point). I think it is critical that we find ways to judge nonprofits so that philanthropic dollars can flow to the organizations that do the most good in the world. To me, funding the best of what is available is far more important than trying to invent the next big thing. I think that information about nonprofits is what is needed and this is why I care about nonprofits being in the Google Finance portal.

As a professional investor in for-profit companies, I can tell you that there are very few (none) golden metrics that allow you to comprehensively judge one for-profit against others. Even very widely used metrics like “price to earnings ratios”, “dividend yields”, “profit margins”, and “earning growth rates”, have been show in practice to be very useful, but not in any way adequate to judging the superiority of one investment choice vs. another on their own.

In my Philanthropy Predictions for 2008 that I wrote for the Chronicle of Philanthropy, I made one reference to measurement:

A United Way-authored outcome-measurement template will be adopted by the sector as the standard format for nonprofit organizations to report on their effectiveness. The narrative-driven form will soon be available for download from the home pages of many nonprofits.

Note that I suggest a “narrative-driven form”. If you read analyst reports on for-profit investments, you’ll see a lot of numbers and metrics, but the heart of the report is a narrative about the company.

This brings me to an excellent comment from the thread mentioned above from an anonymous “young staffer”:

If I may carry the Paradise Lost vs. Gone with the Wind analogy a little further, I think it raises some interesting points.

The first is that there are plenty of potentially relevant metrics with which one could back up one’s a claim for each work’s superiority: their longevity in years, the number of universities that include them in introductory freshmen humanities courses (as a proxy measure of their centrality to our cultural canon), a RottenTomatoes.com-style survey of critics. I can even imagine poor grad students counting allusions to them in last year’s bestsellers.

Relying solely on any one of these potentially valid measures, however, would obviously leave you wide open to criticism for the flaws of your methodology and the limits of the analysis. To construct a strong argument for your preferred choice, one could use both the metrics and qualitative measures. Same goes for nonprofits - the measures are neither perfect nor complete, but that is not the same as nonexistent.

I think the other point is the difficulty of comparing apples and oranges. Let me reframe the question as “Paradise Lost” work of literature vs. “Gone with the Wind” work of film. Both are widely-considered seminal works in their mediums. It’s not hard to imagine metrics, like those above, that could easily distinguish each as a leader within its respective medium. It is much harder, however, to compare them very convincingly across mediums. An author and a film buff might reach very different conclusions about which one matters more in today’s culture. Their distinctive values and tastes will influence that decision.

The same, I think, is true for nonprofits. Too universal a measure like “average cost of impact” might not be helpful for identifying whether a great afterschool program in New York or a great community health program in Uganda is better. The costs and the measures of impact are on different scales. But metrics certainly might help you identify each within its field as the seminal nonprofit. From there, one’s values and tastes might be expected to guide your choice.

So there you have it, a good beginning to an important conversation. If there was a single webpage, like this one for the Red Cross, or this one for Cisco Systems, that contained all the information you would like to see when you wanted to examine a nonprofit for the first time and decide if you might want to support them, what information would you like there to be on the site?

Google.org owes me nothing and anything I tell them might be ignored. But on the other hand, I will deliver the message that we co-create over the next week in this discussion. Someone from one of the largest (and oldest) foundations has already asked me to pass on their offer of help to Google.org after reading my posts on the subject. I do think that any effort that you the reader put into this discussion will be heard by the powers that be at Google.org, even if they do not take action.

Google Finance for Nonprofits

I don’t know the inside story of how and why Google began including nonprofits in Google Finance. I almost wonder if it was an accident. Google pipes in data from Hoovers.com, which in turn has some limited info on nonprofits. Poking around Google Finance, I realized they also include profiles of cities (like this page for San Francisco).

The Google Finance nonprofit pages seem to not just be in beta, but appear to have not actually have been designed intentionally. For instance, the page on the Red Cross includes “Key Stats & Ratios” such as Net Profit Margin, which of course is not relevant for a nonprofit. The page also refers to “Related Companies”, instead of a more appropriate heading like “Related Organizations”.

So let’s assume for a moment that we have a bit of a blank slate to work with. If you were designing a template for the Google Finance nonprofit pages, what would you include? I was just cc’d on an email to Larry Brilliant asking him to consider some suggestions for what info might be made available on these pages and the sender is someone who is use to getting replies to his emails. So while we might have limited input into what Google eventually does, I don’t think this discussion is academic.

Here’s what I would like to see on the nonprofit pages:

Key Stats and Ratios: I would rename this “Key Stats” and not include any ratios. Displaying ratios imply that the ratio should be high or low, but very few ratios in the nonprofit world are all that relevant. In the for-profit world, most ratios include some sort of profitability number (not relevant to nonprofits), valuation metrics (not relevant to nonprofits) or are balance sheet ratios showing assets or debt (for nonprofits a big cash hoard can be viewed either positively or negatively). Instead, include info like: Fundraising Total, Total Budget, Total Employees, Endowment, etc. My strongest feeling is that the most important thing is for Google to avoid any mention of overhead expense ratios. Google has a chance to break the grip that overhead expense ratios have on donors and the media.

Overview: Right now, the Hoover’s profile is here. I’d like to see Google partner with someone more focused on the nonprofit sector than Hoover’s is.

Discussion: This is great. Don’t change a thing!

News: Recent headlines is a nice feature.

Blog Posts: Only some of the nonprofit pages include this section. This seems odd since I assume there must be only one template. But of course I would like to see this section maintained or expanded.

Related Companies: Calling this “Related Organizations” would make more sense. I think in this area Google should leverage their Map software and show me not only similar organizations, but local ones as well. If I’m looking at the Red Cross site from my home outside of San Francisco, I’d like to see disaster relief organizations that focus on the Bay Area.

Resources: This is a section I’d like to see added. Display links to GiveWell, Great Nonprofits, the nonprofit’s 990, the nonprofit’s website, a Wikipedia page, etc.

Video: Allow nonprofits to upload video content that donors can watch to get a better understanding of the organization.

Donate: Partner with Network for Good to allow donors to give directly to the nonprofit.

Contact info: Display contact info.

Blogging: Why not integrate with Blogger and offer a hosted blog to nonprofits to write their own blog?

Events: Include a list of upcoming events that the nonprofit is hosting or participating in.

Lastly, I’d like to see an area where the nonprofit can upload their own text about the organization as well as their answers to a set of predefined questions such as, “How does your organization track its effectiveness” as well as provide links to information such as mission statement, historical goals and what was actually achieved.

What would you like to see on the page? Leave a comment on this post and I’ll do what I can to get the suggestions into the right hands at Google.

Red Cross Replies

The Red Cross took less than 7 hours to respond to my question about their effectiveness on the new Google Finance for Nonprofits portal. You can read our back and forth here. I believe that this is the first discussion occurring on a Google Nonprofit page. Personally I’m glad the discussion is about effectiveness. The Red Cross gives a good reply that most donors will be happy with. I was impressed. But I’m sure that no foundation or someone like Holden Karnofsky would find the answer sufficient. No links to impact data. But that’s OK. All of that is coming a short way down the road. I think that open discussions between donors, nonprofits and others in a hosted forum like Google will only hasten the move towards transparency and demonstrated impact.

Google Launches Nonprofit Portal

This is a big deal. If you go to Google Finance, you can now search for charities by name and pull up data about them, news stories, blog posts and leave comments in a discussion forum (hat tip to “a fundraiser”). As far as I know this is brand new and as far as I know, I’m the first person to leave a comment in a discussion group.

On the Red Cross page, I wrote:

Is the Red Cross Effective? I don’t mean do they have low overhead expenses or some silly measure like that. I mean do they take donor dollars and use them to fund an organization that produces high levels of social impact? If the answer is yes, I’d love to know about any data that backs this claim up.

Thanks to anyone who can help.

Sean Stannard-Stockton
TacticalPhilanthropy.com

It was just last month that a One Post Challenge entry suggested that Google should buy GuideStar. Maybe Google thinks they can do it alone.

I think this is a game changer. If these Google pages resided at the top of the search results when people look up nonprofits, than these pages will become de facto home pages, but with blog posts, new stories and discussions that are both positive and negative. What if you’re a donor thinking about giving to the Red Cross and the first link you find is the Google Finance page? You take a look and find a question from someone asking if the Red Cross is effective… and no response from the Red Cross.

This is a big deal.

But it is obviously beta. Right now there is no silly overhead expense ratio analysis. But on the other hand Google bizarrely lists “Key Stats and Ratios” that are all blank, since the stats and ratios all refer to profitability measures. What data will Google choose to display? The choices they make will influence donors and the flow of charitable dollars in a big way.

What information do you think Google should list? I’ll do my best to get suggestions in front of Google.

If you want your voice heard, check out the Google Finance site, search for a nonprofit and leave your own comment in the discussion forum. You’ll be one of the first.

Great Nonprofits

I wrote yesterday about the Chronicle of Philanthropy article about Great Nonprofits. In the article, some people voice the criticism that the site has just positive reviews. So I emailed founder Perla Ni and put the criticism to her.

(Full Disclosure: Perla is a friend of mine. I know her professionally, but I also went to a party at her house earlier this month. Also, the photo of me that appears in the Chronicle of Philanthropy article is from when I volunteered at a Great Nonprofits event a few months ago. Clearly, I’m not the person to argue the merits of Great Nonprofits. So I’ll let Perla’s response to my question speak for itself. If readers have further questions, I’m sure Perla will respond in the comments section.)

Sean: “The criticism I know that you will hear and I will for writing about your project, will be that all the reviews are positive. The lowest rating on the site for a nonprofit with two or more reviews is 4+. How would you respond to someone who complains about the lack of negative reviews?

Perla’s response:

Great question. The answer is long - but I think your question deserves a full answer.

First, your readers shouldn’t look at the star ratings so much.  It’s a short-hand really and cannot fully reflect the nuances of someone’s opinion about a nonprofit.  Unlike product reviews, for instance, where the product is supposed to do a specific thing, nonprofits do a lot of different things sometimes and so it’s much more valuable to read the full review, especially the part on the “Great” things about the organization as well as “How to make it even better” which offers very concrete ideas for how the nonprofit can make its programs/services better.

Secondly, we have a selection bias right now in 2 ways because we’re just starting out.

1. As you’d expect, nonprofits that feel confident about their programs/services are more likely to participate on a website like ours. Though “consumer reviews” are standard now in just about every other sector - restaurant reviews, book reviews, car reviews, hotel reviews, movie reviews - it’s a new concept in the nonprofit world.

And so only the nonprofits that feel pretty confident about the quality of their services/programs are going to participate. My theory is that these probably are nonprofits that do indeed provide better than average programs/services.

2. Because since we’re starting out and the larger world of nonprofit stakeholders hasn’t heard of us, we rely on the nonprofits to get the word out to their clients, volunteers, donors and other stakeholders.

And the natural inclination is to ask one’s most trusted and closest stakeholders to write a review. But in time - as with anything on the internet - because the information is so readily available and transparent, other stakeholders become aware of the site, they will chime in and react to the reviews on the site. There’s a counterbalancing effect on the web because information is so transparent. People tend to feel more free, in fact, to disagree online than in person.

So in time, those two selection biases should correct itself in time.

Now, there’s a third bias that will continue even when the site gets more counterbalancing reviews. It’s the psychological tendency that people remember good experiences more strongly than negative experiences. This is well documented in psychology studies - and it’s the reason why the average review about consumer products is not 3 out of 5, but 3.4 out of 5. But this is still probably more accurate than the “Expert” reviews (ie: staff writers for consumer products magazines) where the average rating is 4 out of 5.

I hope that helps.

Forces For Good Podcast

Today’s podcast is with Heather McLeod Grant and Leslie Crutchfield, the authors of the new book Forces for Good: The Six Practices of High-Impact Nonprofits. Forces for Good examines the characteristics of nonprofits that are achieving high impact. Heather’s been an advisor to the Center for Social Innovation at Stanford, as well as to many nonprofits. She holds an MBA and worked at McKinsey, consulting with for-profit companies. Leslie is a managing director at Ashoka, a research grantee at the Aspen Institute, and a philanthropic advisor to foundations and high net worth individuals.

During the podcast Heather and Leslie discuss the importance of nonprofit groups engaging in political advocacy, the difficult in measuring impact (and the flaws in Charity Navigator’s system of measuring efficiency), the need for nonprofits to engage their volunteers, and the ways in which nonprofits can learn from Web 2.0 companies.

If you post comments and questions in the Comments section, Heather and Leslie will respond.

You can read some background about the book and both authors here.

Click on the link below to read the transcript…

Read More »