How to Live with – and Benefit from – Evaluation Requirements

By Elizabeth Duffrin
Published: September 10, 2010

If you’re skeptical of—and annoyed with—the program evaluation demanded by foundations, you’re not alone, said Tom Dewar, co-director of the Aspen Institute's Roundtable on Community Change, who once served as director of evaluation for the MacArthur Foundation in Chicago.

Funders can expect results too quickly, he said. They may ask for a more extensive evaluation than you can afford. And they tend to change their minds about what they want you to do. “They can start asking questions that have to do with where their heads are at rather than what was originally agreed to.”

But there’s a way to approach evaluation so that it serves your own purposes, he explained. Conducting evaluation on your own terms strengthens both your program and the relationship you're building with program officers at a foundation. Here's his advice:

Negotiate. Sit down with your program officers at the start and find out what they think is credible evidence of success. But make sure to educate them about the challenges you face so that their expectations match what you can accomplish.

For example, the goal of your project might be to reduce unemployment in the neighborhood. There are factors outside your control impacting that statistic, so you might suggest evaluating outcomes that you can influence more directly, such as the number of residents attending your job training program and meeting regularly with an employment counselor. “Figure out what you can do and then sell that as reasonable.”

Serve yourself. The ultimate purpose of evaluation should be to improve your own work. So even if a funder isn't interested in all of your goals, make sure the evaluation answers your questions, too. Above all, avoid leaving the direction of your project entirely up to your funders. Foundations frequently change priorities, “and you have no control over that,” said Dewar. “If you focus on your funders, you will lose your way.”

Start early. Evaluation can seem so daunting that it often gets left until the end of a project. “Evaluating your progress from the start allows you to make mid-course corrections—and that’s the real point of evaluation,” said Dewar.

If your goal is far off, look for short-term indicators that tell you if you're headed in the right direction. If you want to reduce crime in the neighborhood, how many residents are turning out for community meetings on the topic? How many are joining the Parent Patrol or the Neighborhood Watch? If the numbers are low, why and what can you do about it?
When data collection becomes a routine part of your job, evaluation is less of a burden. For example, neighborhood residents might not mind filling out a quick survey at the end of each community meeting. “That's better than an evaluation meeting at the end [of a project] that nobody shows up for,” Dewar remarked.

Don't leave it to the experts. Even if you hire outside evaluators, you will still need to supervise them to make sure your questions get answered. “Lots of times people hand [evaluation] off because they find the process annoying or time consuming,” said Dewar. But don't, he warned. “The consequences of it going badly or getting off track are real.”

Make sure the evaluators you hire are skilled both in research and in communicating that research to the outside world—whether to community residents, policy makers or foundations. “You want someone able to listen and explain to people rather than retreating into their expert world and language,” he explained. “If they do that, you’ve got to fire them.”

Involve the community. Community residents can be a huge resource for your evaluation. If your researcher only has time to interview 20 people, training residents to conduct interviews could significantly expand that sample size. “Lots of people find it interesting to be part of [the evaluation]—checking in with the neighbors or visiting the police station or interviewing the local businesses.”

Just make sure to keep the tasks manageable, such as conducting simple interviews with five neighbors. And make sure topics are appropriate for resident evaluators. Interviews about recreation, traffic or safety are fine, while questioning ex-offenders about arrest and probation is probably not. Leave questions that are too technical or sensitive to the experts.

Go beyond anecdotes. Telling human stories about your success is important, but don't forget to back them up with numbers, which tend to carry more weight in the foundation board room. You may already have more data than you realize, said Dewar, as anecdotes collected over time can become data. “Sometimes you're sitting on data and because you're a narrative-style person, you don't realize what evidence is [already] inside the story.”

If your project involves engaging different segments of the community in a development project, who is attending your meetings? How many residents, how many business owners? Who is speaking up? Does the nature of the comments change over time? For example, do you hear people specifically recognizing another interest while stating their own opinion? How often does that happen? Looking for things to count can provide a backbone to your narrative. Be creative.
Be honest. If your project falls short of its goals, be open about that with your funder. Scrutinize your work and describe what you learned and what you might attempt next. Describe the challenges you encountered.

The societal problems that non-profits tackle are extremely complex, and this is a chance to educate your funder in a way that can deepen your long-term working relationship, Dewar explained. Besides, foundations are so accustomed to “huge shortfalls between rhetoric about what you were going to do and what is produced,” that submitting a candid and thorough evaluation is unlikely to hurt you. In fact, he said, “it will make you stand out.”

Keywords: Aspen Institute, evaluation, MacArthur Foundation, tips