advertisement

Zinman Randomized Experimentation for Program Mana

100 %
0 %
advertisement
Information about Zinman Randomized Experimentation for Program Mana
Product-Training-Manuals

Published on April 17, 2008

Author: Nellwyn

Source: authorstream.com

advertisement

Randomized Experimentation for the Program Manager: A Quick How-To Guide :  Randomized Experimentation for the Program Manager: A Quick How-To Guide Jonathan Zinman Assistant Professor of Economics, Dartmouth College Research Associate, Innovations for Poverty Action May 1, 2007 Presentation for IFC M&E Conference What’s Your Objective?:  What’s Your Objective? If it’s “beat or meet the market”…. Can’t afford to focus on evaluation per se What’s Your Strategy?:  What’s Your Strategy? Focus must be…. Using evaluation to feed innovation (Abhijit’s “dynamic learning”) Experimentation & the Learning Organization: A Virtuous Cycle:  Experimentation & the Learning Organization: A Virtuous Cycle Randomized Evaluation: A Quick How-to Guide:  Randomized Evaluation: A Quick How-to Guide What should I evaluate? How do I design and implement the evaluation? How do I interpret and apply the results? At each step will highlight how randomized-control trials (RCTs) can be part of an innovation strategy producing comparative advantage for: Client financial institutions (FIs) Wholesale investors like IFC What Should I Evaluate? :  What Should I Evaluate? Which interventions (“treatments”)? Which outcomes? Which Interventions?:  Which Interventions? Use RCTs to address critical questions: Business/programmatic/policy strategy Stakes should be high, given fixed costs of experimentation and related data collection Existing programs you’re funding (of course!) Also…. Which Interventions?:  Which Interventions? But also: “Microfinancial Engineering” Analogy to “high” finance “Design-build” partnerships between academics & financial institutions Input from funders, program managers, policymakers Don’t just “take opportunities” to do systematic experimentation Create opportunities. What’s a “program”? Innovation challenges Brokering Funding Which Interventions?:  Which Interventions? RCT interventions can be built into any critical business function Product development Pricing Marketing and targeting Monitoring and enforcement Risk assessment Which Interventions?:  Which Interventions? Product Development examples Savings products with new features: Commitment Goals Method and nature of communication with client Smoking cessation performance bond Smokers “bet themselves” they can quit (as measured by urine test in 6 months) Financial institution fills missing market: enforcement Micro-insurance Evaluate by randomizing Whether product offered Product features Which Interventions?:  Which Interventions? Pricing examples Loans: commercial, consumer Savings: various products Micro-insurance Evaluate by randomizing Initial offer Dynamic pricing (future offers) Which Interventions?:  Which Interventions? Marketing and targeting examples Marketing: what speaks to the client and why? Direct mail: randomize content Text messaging: randomize content Targeting: finding your market or intended beneficiaries Health insurance for the poor in the Philippines Working with government, its contractors to experiment with: Measurement techniques Incentives for using them Compensation Punishment (monitoring, auditing) Which Interventions?:  Which Interventions? Loan monitoring/enforcement examples: Require peer monitoring or liability? Incentivize peer monitoring (referrals)? How monitor and communicating with (delinquent) borrowers? What are the most (cost-) effective threats and penalties? Evaluate by randomizing: Mechanisms, incentives, protocols Which Interventions?:  Which Interventions? Risk assessment Possible to lend profitably to some rejected applicants? Innovations to test: Improve risk assessment model (credit scoring; other projects) Provide incentives to overcome loan officer conservatism (details to follow) Evaluate by randomizing: The approve/reject decision (within reasonable bounds) How-to Example From One Project:  How-to Example From One Project Experiment with risk assessment Objective: measure impacts of expanding access to consumer loans Popular product in South Africa Worked with a leading for-profit “microlender” “Traditional” micro(enterprise)credit largely absent in South Africa 4-month installment loans; 200% APR Example: Expanding Access:  Example: Expanding Access What we did. Overview of the experiment: Intervention: randomly approve loans for some close-to-creditworthy (“marginal”) applicants who would normally be rejected This is the “treatment group” Other marginal applicants remain rejected This is the “control group” Measure impacts Difference between treatment and control For the many outcomes of interest Lender: use its data to calculate profits Applicants: survey data on economic and well-being outcomes See “Expanding Access: Using Randomized Supply Decisions to Estimate the Impacts” (with Dean Karlan): http://www.dartmouth.edu/~jzinman/Papers/Derationing.Karlan-Zinman.pdf How Did This Evaluation Come About?:  How Did This Evaluation Come About? Remember Step 1: What do we evaluate? Which intervention(s)? Which outcomes? This project very much design-build Lender motivation Prior experiments with Lender Identified specific market failures (asymmetric information problems) Identified binding liquidity constraints for borrowers These reinforced Lender’s priors that loan officers being too conservative Profitable deals being left on the table Open to RCT as systematic way to control and evaluate risk of liberalizing criteria Motivation for this Evaluation: What do we Evaluate?:  Motivation for this Evaluation: What do we Evaluate? Researcher/policy/funder angles: Consumer credit controversial Policy tends to restrict rather than encourage access But why? Economic arguments for restricting tenuous But consumer (micro)credit markets growing Our methodology applicable to microenterprise credit as well Step 2. How do we do it? Design and Implementation:  Step 2. How do we do it? Design and Implementation 3 key issues in this case: Scope of study Implementing the intervention: how to randomize loan approvals decisions Tracking and measuring outcomes Design & Implementation: Scope:  Design & Implementation: Scope A. Scope: How big? Where? Required deal flow for a conclusive evaluation? How big a sample do we need to answer the questions of interest (statistical power) Researchers identify Best way to obtain the required deal flow? Researchers and Lender worked together to identify: A practical definition of “marginal” applicant Timeframe for the intervention (2 months) Participating branches. Chose 8 branches that would Produce required deal flow in the 2 month timeframe Be relatively easy to monitor and train Be representative enough to draw conclusions re: whether or not to scale up the intervention to other branches Design & Implementation: The Intervention:  Design & Implementation: The Intervention B. How actually randomize loan approvals? Insert research protocols into loan assessment process. In this case 2 additional steps: Loan officers rejected applicants into “marginal” and “egregious” New software randomizes “marginal” into “keep rejected” or “approve” (second look) New implementations streamline this with introduction of pure credit scoring model Train branch personnel Monitor (& incentivize) branch personnel to comply with protocols Design & Implementation: Measurement:  Design & Implementation: Measurement C. Tracking & measuring outcomes Lender data on sales, costs, & hence profitability Follow-up survey data on applicant outcomes: Economic (employment status, income, consumption) Subjective well-being (decision power, optimism, mental health) Researchers designed household survey instrument Contract survey administration to survey firm Close monitoring from pilot to final survey Results: Lender Outcomes:  Results: Lender Outcomes Lender made money: marginal loans were profitable Less profitable than loans above the bar But profitable nonetheless Even on initial loan Profits from acquiring new clients even bigger Did Lender scale up? That was the plan, but…. Then Lender was merged into a larger bank New senior mgmt hostile to “consultants” Old senior mgmt (our partners) banked knowledge and took to new firms Results: Applicant Outcomes:  Results: Applicant Outcomes Large, positive, statistically significant impacts on: Economic self-sufficiency (employment, income, above poverty line) Consumption (avoiding hunger, food quality) Outlook and control (decision power, optimism) No significant impacts on: Investment (education, housing, self-employment) Physical health Negative impact (90% significant) on mental health (depression, stress) Overall impact significant and positive If weight all outcomes equally Step 3. How Apply The Results?:  Step 3. How Apply The Results? The intervention itself Do (social) costs exceed benefits? In this case interpreting results simple: win-win Often there are tradeoffs: weighing costs and benefits requires some insight into the “whys” of impacts Here evidence of market failures from earlier experiments Prior and project evidence of binding liquidity constraints Opportunity cost of intervention(s)? In consumer credit key is ruling out negative effects: default policy/programmatic approach is to restrict access Unlike microenterprise credit, where default approach is to expand/subsidize access, and hence opportunity cost of subsidy matters How Apply The Results?:  How Apply The Results? Applying the results (external validity) Scalability Replicability Three complementary approaches: Design so that get answers re: why interventions do or don’t work Choose sites/markets/partners carefully Do lots of RCT experimentation Take-Aways:  Take-Aways RCTs deliver: Gold-standard measures of impacts Insights into the “why” questions that: Affect scalability Feed back into innovation RCTs are doable: Design-build partnerships with researchers for: Microfinancial Engineering Innovation that is scalable and replicable Experimentation & the Learning Organization: A Virtuous Cycle:  Experimentation & the Learning Organization: A Virtuous Cycle

Add a comment

Related presentations

Related pages

Zinman_Randomized Experimentation for Program Managers ...

Which Interventions?• Use RCTs to address critical questions:– Business/programmatic/policy strategy– Stakes should be high, given fixed costs of ...
Read more

Zinman_Randomized Experimentation for Program Managers ...

Randomized Experimentationfor the Program Manager:A Quick How-To GuideJonathan ZinmanAssistant Professor of Economics, Dartmouth CollegeResearch Associate ...
Read more

Randomized Experimentation for the Program Manager: A ...

Randomized Experimentation for the Program Manager: A Quick How-To Guide. Jonathan Zinman Assistant Professor of Economics, Dartmouth College Research ...
Read more

Optimizing Loan Contracting and Marketing Stategies Using ...

Optimizing Loan Contracting and Marketing Stategies Using Field Experimentation* ... randomized-control methodology that addresses MFI contracting and ...
Read more