The role of the evaluator is much like that of Peter Brand in the movie _Moneyball_ (based on the book by Michael Lewis, a favorite author of mine). Peter Brands role as an economics whiz kid hired by the Oakland As was to help them figure out how to win. Using meaningful statistics, Peter and the General Manager Billy Beane, helped turn the game of baseball on its head by looking at data in a new way. Similarly, evaluators are charged with identifying and measuring the bottom line of non-profit services from a social impact perspective. We have seen an explosion of businesses and non-profits talk about the use of data (like they do in _Moneyball_).
But before anyone thinks about the analysis of data, we (as evaluators) have the responsibility to help organizations collect, use, and analyze data. I often feel like I need to wave the yellow (or red!) flag that you need to understand the data PROCESS before diving in to talk about the measurement of outcomes. The data that are produced and consumed by non-profits are only as good as the intent put into the process and the quality controls around that data.
My colleagues and I talked about this recently at a congressional briefing in Washington, DC that highlighted the need for a developmental approach to evaluation one we use on Elev8 (https://www.elev8kids.org/ ). Instead of diving directly into outcomes, we thought about how to approach data and data collection to get a grasp on the implementation of services FIRST. For the full description of how we did this, go to https://www.elev8kids.org/node/609.
In terms of process, whether you are a single agency measuring outcomes or you work collaboratively across agencies, there are key steps to get the data collection process off the ground (from engaging staff to training to sustaining the work). If you saw Moneyball, you know that some of the Oakland As staff (including the coach at the time) struggled with a new approach to evaluating players that relied on being data driven. This is no different than what we are seeing in the non-profit sector. In Collective Ideas to Collective Impact, a new guide published by nFocus Solutions and authored by me and Chelsea Farley, we discuss this as well as other key tenets of the work: https://www.nfocus.com/company/collective-ideas-to-collective-impact.
So my hope is that before getting caught up in the world and excitement of measuring impacts, you take a step back. As Michael Lewis, author of Moneyball says, People operate with beliefs and biases. To the extent you can eliminate both and replace them with data, you gain a clear advantage. Just make sure it is good and appropriate data!
About the author: Meridith Polin is a senior consultant with McClanahan Associates and previously served as program director at Public/Private Ventures. She is the evaluation project manager for Elev8 a multi-agency community schools model with 20 sites nationwide. Polin also provides management oversight, evaluation technical assistance and training to a variety of initiatives focused on education, out-of-school time and criminal justice. Polin presents across the country on performance management, including presentations at the University of Pennsylvania, the United States Department of Labor and the American Evaluation Association. Her previous experience included her tenure as the director of research and evaluation at Citizen Schools and also direct service work with youth. She is a co-author of Using Data in Multi-Agency Collaborations: Guiding Performance to Ensure Accountability and Improve Programs. Polin earned her B.S. in business administration from Boston University and her M.S. in community resources and development from Arizona State University. Contact Meridith at [email protected].
