By Felix Fernandez
For 85 years, the Greater Washington Urban League (GWUL) has worked to financially empower the Black community within the DC Metro area. Through the GWUL Center for Financial Inclusion (CFI), the organization helps clients understand the underlying factors that influence their financial decisions, including past financial behaviors and trauma, institutional racism, and systematic discrimination in the housing, education, and employment sectors. GWUL innovative approach to adult financial education acknowledges the Black community’s historic struggles with institutional racism and discrimination that has result in long-term poverty and centers on connection between neuroscience, financial psychology, and lived experiences with current spending habits. In December 2020, the GWUL contacted FHI 360 to request evaluation support for CFI. Over the next two and half years, the GWUL and FHI 360 conducted a collaborative evaluation process that supported continuous improvement of the program, built the evaluation capacity of their staff, and documented clients’ experiences. In this post we share FHI 360’s collaborative approach toward evaluation design and capacity building, emphasizing the importance of collaboration.
Strategy 1: Collaborative Planning Sessions
From January to April 2021, we held three strategy sessions to review program offerings, design initial evaluation questions, and document desired goals and outcomes. In each two- to three-hour session, FHI 360 research and evaluation experts met with GWUL leadership and staff to review the program logic model and current data collection efforts, and brainstorm potential study designs. We also discussed several potential evaluation designs and the strengths, weaknesses, and related cost of each. Ultimately, GWUL selected a mixed-method evaluation approach that used existing survey data collection efforts and focused on developing an impact narrative that would emphasize the client journey/experience.
Strategy 2: Learning from existing efforts
When the evaluation team wasn’t meeting with GWUL, we were reviewing existing program materials, previous data collection efforts, logic models, and other related materials. These sources of data were used to revise evaluation questions and, more importantly, identify potential overlaps in information that could be used to support evaluation activities. This ongoing review directly informed the agenda for subsequent strategy sessions, identified potential gaps or areas in need of additional inquiry, and served as the first step in creation of a detailed activities and outcome mapping.
Strategy 3: Detailed Activities and Outcome Mapping
From our collaborative strategy sessions and review of existing documentation we were able to align program activities, goals, and outcomes, document current data collection efforts, and identify potential gaps. For each primary outcome we designed detailed matrix and tree diagrams (see example figure) that aligned specific service activities, short- mid- and long-term outcomes, and data collection efforts. This matrix was iteratively designed in collaboration with GWUL, ensuring the validity of the alignment, the appropriateness of the data, and the accuracy of the reported outcomes.
Strategy 4: Collaborative Design and Data Collection
We designed an 18-month evaluation process that included monthly program updates, ongoing review of program administrative records and documents, client surveys, focus groups, and serial interviews with clients. Thematic analysis and descriptive statistics would emphasize describing the client journey and experience with the CFI beginning with the January 2022 CFI cohort. Through shared data collection efforts, we supported GWUL capacity to collect evaluative data, provide strategic updates regarding program outcomes, and co-design an online dashboard that would provide aggregate feedback on short-term outcomes, client participation, and general feedback.
Strategy 5: Flexibility in Design and Approach
Throughout the year, GWUL revised their CFI program as they adapted to the needs of the clients. For example, they changed program requirements, entry points, and data collection. Originally, CFI designed the activities to be completed in a linear progression (see example figure for proposed client journey through CFI activities) beginning with an entry series of financial webinars. However, we found that clients entered at multiple points, participated as needed or desired, and left and returned to the program when they were able or interested. As are result, we revised our intended survey and interview plans to better document client experiences. We added component-specific focus groups (e.g., group financial therapy) and revised planned survey data collection timelines.
Conclusion and Implications
Our evaluation found that the key strength of the GWUL CFI program was its flexibility, adaptability, and ability to meet clients when and where they are. At the completion of our study, clients reported that the CFI had set them on a pathway to achieve future financial goals. The overwhelming majority of survey clients reported improvements in at least one financial area (94%), increased understanding of the role of financial stress and trauma in their financial habits (89%), and confidence in their ability to achieve one or more of their financial goals (87%).
Our collaborative engagement with CFI program staff resulted in a stronger evaluation design closely aligned to program activities and services, enhanced data collection and analysis through the addition of ongoing staff insights, and increased evaluation capacity of program staff to understand and use result to improve and scale up program efforts. For those interested in learning more about our evaluation findings, the full report can be found here.
Photo by: Canva