logo of Dorothy Cotton Institute
Evaluation and Research

GET MORE INFO ON THIS TOPIC

DCI Research Fellows and our documentation team have been developing evaluation and research protocols to assess the impact of our programs.

 

While it is difficult, if not impossible, to draw straight causal lines between investments in community change and specific outcomes.[1] Community change is dynamic, complex and systemic, with multiple actors, initiatives and causal factors all operating in the same system, at the same time, over long periods of time. Attempts to definitely establish “causal proof” are typically ineffective and ill-advised.

However, this does not mean abandoning attempts to rigorously demonstrate the impact of particular change initiatives. Rather, it requires that we adopt multiple methods and use multiple sources of data to make a “compelling case” for the Dorothy Cotton Institute’s contributions to desired outcomes.

In doing so, we have selected evaluation methodologies and tools that are :

  • Capable of not just demonstrating the effectiveness of a particular program or intervention, but of deepening understanding about what works, why and how;
  • Inclusive of diverse worldviews and value systems in data gathering, synthesis and interpretation
  • Participatory and capacity-building, enabling diverse stakeholders to collectively monitor and enhance the change they are trying to produce while learning new skills and tools for critical reflection and evaluation;
  • Attend to real-time, “relevant” learning that can demonstrate paths to improving practice, impact and further change;
  • Support an open systems perspective, looking beyond impact on individuals to the arenas in which they interact.

 

DEFINING “SUCCESS”: INDICATORS OF SUSTAINABLE CHANGE

Including diverse stakeholders (e.g., teachers, administrators, community youth workers, parents, students, etc.) in helping define “success” and identify key indicators increases the likelihood that documentation efforts will be useful, relevant and credible – not only to DCI and its funders, but to the community as a whole.

 

Categories of indicators:

In collectively developing indicators, we will use the following six categories of indicatorsto guide this work:

1) A shift in definitions:  Is the issue of human rights defined differently in the schools or larger community?

2) A shift in identity and agency: Do people define themselves differently? Have their attitudes and/or sense of
efficacy changed?

3) A shift in relationships: Have people’s relationships and/or social networks changed? In what ways?

4) A shift in behavior and engagement: Are people behaving differently in schools, organizations, or the larger community? Are people in more engaged; has a critical mass of people been reached?

5) A shift in policy: Has an institutional, organizational or legislative policy or practice changed?

6) Maintaining gains: Have desired changes in policy or practice been maintained in the face of opposition (e.g., backlash)? Has additional progress been made?

 

RESEARCH METHODOLOGIES

We use several complementary methodologies to enhance the reliability of our data production, analysis and conclusions.

1)       Narrative Inquiry

This is a process of  extended interviews with people who have participated in workshops and other programs (i.e.
teachers, administrators, and community youth workers, young people etc.) to:

  • Explore the most significant changes in participants’ sense of themselves as educators & change agents), relationships, and educational practices as a result of being engaged with DCI’s human rights work.
  • Explore “community narratives,” counter-narratives, understandings and knowledge from which members of an emerging community of practice effect personal and collective change toward educational equity.
  • Consider how the knowledge, skills and strategies emanating from social movements and grassroots organizing may provide reformers with an expanded repertoire of change language and strategies
  • Consider strategies of empowerment, resistance and backlash, and grassroots organizing components such as developing relationships, common understanding and action.

2)      Cooperative Inquiry: Educator Learning Community

This is a process in which educator learning communities undertake a “cooperative inquiry” (Heron), a form of
participatory action research, to help explore the process of change, help identify indicators of success, help
identify the most significant changes, learn about challenges and how to overcome them, etc.

3)      Valuing Change: The Most Significant Change Technique

The Most Significant Change Technique, a narrative monitoring and evaluation tool, is especially useful in helping
identify unexpected changes and key values in a group/organization (through discussion of which changes are most
significant); offering a rich, rather than oversimplified, picture of complex outcomes; helping people learn rather
than merely “account”; helping focus on the effects of an intervention on people’s lives and/or work; and
encouraging participatory analysis (as people explain why they believe one change is more important than
another).

4)       Ethnography (Participant-Observation)

Members of DCI’s Research and Documentation Team function as participant-observers in workshops, learning
communities, and other gatherings, preparing extensive field notes and research memos.

5)       Surveys

We incorporate a variety of survey data – including historical benchmarks – into our analysis:

  • Workshop evaluations (including “critical incident” questions, as well as satisfaction data)
  • Human Rights Temperature surveys in schools, community programs, etc.
  • data such as test scores, other student assessments, attendance, disciplinary action, etc. that serve as proxies to measure the “academic performance gap” between white students and students of color, lower income students and students from wealthier families, and students with and without disabilities.

 


 

[1]  Anne C. Kubisch, Patricia Auspos, Prudence Brown and Tom Dewar (2010). Voices from the Field III: Lessons and Challenges from Two Decades of Community Change Efforts. Washington, D.C.: The Aspen Institute.