Learning Across Borders; A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program.

50 %
50 %
Information about Learning Across Borders; A Monitoring, Evaluation, and Learning...

Published on March 13, 2014

Author: MasterCardFoundation

Source: slideshare.net


The monitoring, evaluation and learning design of the MasterCard Foundation Scholars Program. Theory of Change, logic model and learning questions. Design challenges and solutions.

Learning Across Borders: A Monitoring, Evaluation, and Learning Framework for The MasterCard Foundation Scholars Program March 2014

Overview of Presentation  The MasterCard Foundation  The Scholars Program  Monitoring, evaluation, and learning (MEL) framework  MEL challenges, approaches, and lessons learned 2

The MasterCard Foundation Scholars Program 3

The MasterCard Foundation Scholars Program: Context and Programmatic Need Developing countries have achieved a remarkable expansion in educational enrolment, however:  Enrolment in secondary and tertiary education remains low, particularly among poor and rural populations  There are concerns about access to quality and relevant education, even as demand for an educated workforce increases  Many skilled graduates migrate to higher-income countries for better remuneration and working conditions  International scholarship programs too often contribute to brain drain 4

The MasterCard Foundation Scholars Program  $500 million, 10-year initiative to educate and support 15,000 young people – particularly from Africa – to act as “socially transformative leaders,” driving change and making a positive social impact in their communities.  11,000 secondary level Scholarships, 6,000 tertiary  Target population: youth who have demonstrated academic talent, social consciousness and an orientation toward leadership, and who come from highly disadvantaged socioeconomic backgrounds.  Implemented with and through a global network of 20 NGOs and secondary and tertiary education institutions, who select and directly support the Scholars. 5

The MasterCard Foundation Scholars Program: Program Components 6 Comprehensive scholarships Transition support 21st-century skill building Give-back component Alumni network

The MasterCard Foundation Scholars Program: Partners 7

The MasterCard Foundation Scholars Program MEL Framework 8

Evaluation and Learning in the Scholars Program Objectives:  Continuous program quality improvement  Evidence of outcomes, impact and best practices…  For our partners  For the Foundation’s strategic planning  For the global community of practitioners  As the basis of a proactive argument for propagating successful Program elements and the overall vision of the Program. 9

Evaluation and Learning Overview  Program-wide evaluation: primary end-point 2016  Implementation study  Impact study  Institutional change study  Longitudinal studies of outcomes and impact  Outcome study (subset of Scholar cohorts and comparison groups): continued past 5-year point  Outcome tracking (all cohorts)  Anticipated outcomes (Stay in / return to Africa, Give-back, career and leadership outcomes, citizenship outcomes)  Long-term qualitative work to understand Scholars’ roles in promoting social and economic development  Partner-initiated Research  Rapid studies on emergent issues, aimed at providing actionable intelligence. 10

The MEL Framework: The Process 11

The MEL Framework: Theory of Change 12

Approach and Inputs Activities Outputs ImpactsOutcomes Inputs Partner expertise To Serve Scholars Academic and other support Approach Provide Scholars with a wide range of opportunities for educational support and enrichment, give-back, and leadership development At Institutions Revise recruitment procedures Scholars using academic and other support services Partners disseminate lessons learned Increased academic performance Institutionalized changes foster best practices New cohort of leaders supports social transformation and economic growth, particularly in Africa 13 The MEL Framework: Logic Model Components

The MEL Framework: Learning Questions, Methods, and Data Sources Broad Learning Questions Main Design/Methods Data Sources Implementation How was the Scholars Program implemented? Mixed methods Qualitative data, complemented by quantitative data Impacts In particular, what is the impact of the Program on educational, employment, go-back, and give- back outcomes? Comparison group design • Random assignment • Regression discontinuity • Propensity score matching Survey data, complemented by qualitative data Scale-up Did the Scholars Program become a catalyst for broader change within and beyond partner institutions? Qualitative analysis Qualitative data, complemented by quantitative data 14

Challenges, Solutions, and Lessons Learned 15

Challenges, Solutions, and Lessons Learned: Designing a Unified MEL Framework Challenge  Develop a single unified MEL framework to evaluate The MasterCard Foundation Scholars Program Solutions  Develop the MEL framework from the ground up through a thorough understanding of partner projects and what they have in common  Use a flexible approach that allows for different methods for each partner: Partner-specific comparison group designs allow for a rigorous evaluation while using different methods to draw the comparison group  Random assignment  Regression discontinuity  Matched comparison group (via propensity score methods) 16

Challenges, Solutions, and Lessons Learned: Measurement Challenge  Locating applicants based on application data may be challenging  Several key outcomes are difficult to measure  Key outcomes, including educational attainment, career and service outcomes, giving back and going back, occur several years in the future Solutions  To locate hard-to-find applicants, use their own and their schools’ contact information  Draw survey items from existing instruments and pilot our own instruments  Establish a plan for tracking The MasterCard Foundation Scholars over time that is appropriate to the context 17

Challenges, Solutions, and Lessons Learned: Measurement—Baseline Data Collection in Uganda  Sample: 823 applicants spread around Uganda  Applicants were located by calling the phone number provided in their application  This is the first of three rounds of data collection planned 18 92% 8% Locating applicants Located Not located 82% 18% “Give-back” attitudes: Helping people in need is something one should do… for anyone for friends and family 2% 15% 83% Contact information No phone number 1 phone number 2 phone numbers

Challenge  Incorporate formative evaluation components that will generate relevant feedback for partners throughout the evaluation  Early feedback can then inform mid-course corrections as needed Solutions  Engage with partners while developing MEL activities  Collect quantitative and qualitative data at regular intervals  Provide partners with feedback from all data collection efforts 19 Challenges, Solutions, and Lessons Learned: Blending Formative, Summative Evaluation Approaches

Phone interviews with 18 graduating tertiary Scholars  Opportunity to gain early information on Scholar experiences  This early outcome information will support continuous quality improvement Scholars offer suggestions for Program improvement  Increase flexibility in resource allocation—for example, could leftover resources for textbooks be used for transportation?  Coordinate Scholar attendance at academic conferences to facilitate networking  Provide orientation to the Program in one intensive session early on, rather than spread over several sessions 20 Challenges, Solutions, and Lessons Learned: Regular, Relevant Feedback to Partners

Challenges, Solutions, and Lessons Learned: Regular, Relevant Feedback to Partners (continued) Scholars also say what is going well  The Program gives Scholars “the flexibility, time and resources to actually do the work you set out to do”  “[I’m] trying my best to really use all the knowledge they’ve given to me and trying to implement it in myself first and then try to introduce it to society”  The Masters in Development program is a “dream program” that fits well with career goal of reducing poverty in her home country 21

Conclusions  We have designed MEL activities that respond to the complex nature of The MasterCard Foundation Scholars Program with an approach that is • Flexible, yet rigorous • Capable of measuring key outcomes now and in the future • Formative and summative in nature • Collaborative and inclusive, responding to needs across the partnership 22

Contacts Barry Burciul  bburciul@mastercardfdn.org  http://mastercardfdnscholars.org/ Matt Sloan  msloan@mathematica-mpr.com  http://www.mathematica- mpr.com/CIPRE/Projects/MasterCard_Scholars_program.asp 23

Add a comment

Related presentations