Sign up for our monthly Leap Update newsletter and announcements from the Leap Ambassadors Community:
Anne Goodman’s question to the Leap Ambassadors Community in February 2015, about how to structure a new Director of Outcomes & Evaluations position at the Saint Luke’s Foundation sparked insightful and substantive responses from five ambassadors based on experiences within their own organizations.
(FULL RESPONSES ORGANIZED BY THEME, BELOW)
Think of this as the first hire in building a team; they won’t be able to do it all on their own.
Benchmarking Evaluation in Foundations: Do We Know What We Are Doing?, The Foundation Review, January 2013 (recommended by Patti Patrizi who for 15 years ran the Evaluation Roundtable, which convenes evaluation directors from philanthropy and also benchmarks practice and is now running the Center for Evaluation Innovation in DC).
Necessary and Not Sufficient: the State of Evaluation Use in Foundations, which “highlights an important and interesting variation among those foundations where evaluation units report directly to the CEO,” and other relevant case studies, articles, and papers from the Evaluation Roundtable here (per Patti Patrizi).
Building Capacity to Measure and Manage Performance, Bridgespan (recommended by Isaac Castillo).
Important Note: As Patti Patrizi explained, “There is a huge range of emphasis in the job descriptions. The direct service providers (rightfully) need to emphasize data collection, data quality, and improvement of service. This is not necessarily the same job as that of an evaluation person in a foundation (of course depending on what kind of foundation it is, for instance an operating foundation would have some of the same needs as that of a service provider). And the person who is good at working internally within a foundation–that is doing theory of change and helping to identify strategy indicators etc.–is not likely to be the same person who can work in a hands-on way with grantees to help them improve their own data systems and use.”
Sample organizational structure: Also see Appendix for a detailed table describing how the Data & Evaluation department at Congreso is structured.
(ORGANIZED BY THEME)
THINK OF THIS AS THE FIRST HIRE
Isaac Castillo: “First, I will say that it will likely be difficult for a person to do both the internal work and the external/grantee work at the same time without any help. You would need a true superstar that has done both types of work already, and even then they would probably need help. And even when things are up and running, it will likely require multiple people to maintain the work (depending on the size of your org, and the number of grantees you would be working with). First piece of advice is to think of this as the first hire in building a team, and that this person would eventually hire one or more people to help with their work.
“When I was working in the Director of Data and Evaluation position, my time was split into thirds. One third of my time was spent working on internal cultural issues –working with staff to get them more oriented to outcomes measurement and performance management. Another third was working directly with grantees–helping them also understand outcome measurement and performance management, but also focusing on simple things like proper use of measurement tools and even just the benefits of simple technology (like Excel) to assist in the process. The last third of the work was focused on collecting community wide data and staying up on current program evaluation and program design research to help inform our work.
“As my position evolved, I hired someone to take over a large chunk of the external work–it was taking up a lot of time to do technical assistance and support for our 8 grantees, especially once the grantees left the paper/pencil stage and needed more support with their data systems. So that made us a staff of two. But really, we need a third person–someone to focus heavily on the internal work as we slowly expand the amount of programming we delivery ourselves.”
CONSIDER EQ (EMOTIONAL INTELLIGENCE) AS MUCH AS IQ—LOOK FOR PEOPLE WHO ARE GOOD WITH PEOPLE
Gabriel Rhoads (EMCF’s Director of Evaluation and Learning, recommended by Lissette Rodriguez): “When you’re hiring for an Evaluation Director, I’ve often been told you want to consider EQ as much as IQ. Evaluation and quality improvement can lead to change management, and when you’re hiring, be sure to keep a lookout for people that are good with people. It’s possible to bring on a team that’s fantastic with data, but the leader should be able to interpret, explain, and make all the information understandable.”
POSITION THE ROLE FOR MAXIMUM INFLUENCE
Alice Shobe: “Very quickly after hiring Mei Ling [our first measurement and learning position], I moved her to become a direct report to me so that I could increase her influence agency-wide. It was clear that I was prioritizing this measurement and learning culture shift and I helped positioned Mei Ling with the right technical knowledge to have the most influence/authority possible.”
START INTERNALLY WITH YOUR THEORY OF CHANGE, CULTURE, GOALS
Isaac Castillo: “I think that it is important that this person start by making sure there is a good theory of change and set of outcomes for your organization that everyone is in agreement upon.”
Alice Shobe: “My advice from our journey is to really focus internally first and make sure you’ve set the cultural and expertise foundation. We really spent a good couple of years doing that by focusing on sharpening our own organizational goals/outcomes and evaluation and personal goals/evaluation. From there we began to move from exclusively hiring external evaluators to more direct outcomes work on individual grants and work with our grantees. As we’ve moved into this current chapter, we are also working closely with our public sector partners to make sure we were focused on the learning/insights needed to advance our homeless work in our counties.”
FIRST HIRE IS DRIVING CULTURE CHANGE, SHOULD BE FOCUS OF KEY LEADERSHIP
Cynthia Figueroa: “The first person hired needs to know they are driving culture change in the organization. Discussions and planning with key leadership should be equally focused on change management. Need to establish what will be some small wins from the beginning.”
THEN HELP GRANTEES IDENTIFY WHICH INDICATORS THEY CAN SUPPORT IN BROADER SCOPE OF YOUR WORK
Isaac Castillo: “Once [your theory of change] is in place, then it should be presented to the grantees to help them understand what role they would play in the larger scope of Saint Luke’s work. At DCPNI, when we first start working with grantees (and at the beginning of each new grant cycle) we have them look at our theory of change and indicators and have them talk about which indicator they think they can help support. This way, right from the beginning, we get grantees to focus on a few indicators (we really don’t let them choose more than three) and we know pretty quickly where we have gaps in our service strategy. Ultimately, this helps get everyone oriented to a performance and outcome driven culture–internal and external people are all aligned to achieving the set of outcomes you have laid out. And it speaks to the importance of the data collection and outcome measurement work–if we can’t measure what we are doing, then how do we know if we are actually making a difference towards what we committed to?”
PRACTICAL, CAUTIONARY NOTE: GRANTEES ONLY HAVE SO MUCH TIME FOR DATA COLLECTION, DECIDE WHETHER TO EMPHASIZE NEEDS OF FOUNDATION OR GRANTEES
Patti Patrizi: “Grantees have only so much time and resources to put into data collection and monitoring. Foundations need to decide what they hope to emphasize in terms of data collection–the needs of grantees or the needs of the foundation. It is also important to be very clear about the different types of responsibility and accountability for a foundation or for its grantees. It is not the same.”
KNOW THE INTERNAL WORK WILL NEVER BE “DONE”
Isaac Castillo: I do agree with your strong internal push first–I think you need to do that to make sure your organization is in a good place. However, I do want to caution that the internal work will never be `done’–it will just evolve. Staff turnover will mean that there will always be basic culture work and training that will need to be done. And as existing staff become more bought into the ideas, they will have more sophisticated questions that will require more time from your evaluation staff to answer. At some point you will need to decide when it is time to do the external work as well–and that time may be signaled by hiring of a second person.”
Our thanks to Anne Goodman, Isaac Castillo, Cynthia Figueroa, Dominique Bernardo, Alice Shobe, Patti Patrizi, Lissette Rodriguez, and Gabriel Rhoads for sharing their insights.
This document, developed collaboratively by the Leap of Reason Ambassadors Community (LAC), is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License. We encourage and grant permission for the distribution and reproduction of copies of this material in its entirety (with original attribution). Please refer to the Creative Commons link for license terms for unmodified use of LAC documents.
Because we recognize, however that certain situations call for modified uses (adaptations or derivatives), we offer permissions beyond the scope of this license (the “CC Plus Permissions”). The CC Plus Permissions are defined as follows:
You may adapt or make derivatives (e.g., remixes, excerpts, or translations) of this document, so long as they do not, in the reasonable discretion of the Leap of Reason Ambassadors Community, alter or misconstrue the document’s meaning or intent. The adapted or derivative work is to be licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License, conveyed at no cost (or the cost of reproduction,) and used in a manner consistent with the purpose of the Leap of Reason Ambassadors Community, with the integrity and quality of the original material to be maintained, and its use to not adversely reflect on the reputation of the Leap of Reason Ambassadors Community.
Attribution is to be in the following formats:
“From ‘Structuring a Director of Outcomes & Evaluation Position,’ developed collaboratively by the Leap of Reason Ambassadors Community, licensed under CC BY ND https://creativecommons.org/licenses/by-nd/4.0/. For more information or to view the original product, visit https://leapambassadors.org/ambassador-insights/structuring-a-director-of-outcomes-evaluation-position/.”
The above is consistent with Creative Commons License restrictions that require “appropriate credit” be required and the “name of the creator and attribution parties, a copyright notice, a license notice, a disclaimer notice and a link to the original material” be included.
The Leap of Reason Ambassadors Community may revoke the additional permissions described above at any time. For questions about copyright issues or special requests for use beyond the scope of this license, please email us at email@example.com.