Published on September 3, 2015
1. Not forgetting the Forests for the trees - The art and science of useful impact evaluations Presentation at ICRAF Nairobi, 3rd September, 2015 Dr. Jo (Jyotsna) Puri: Head of Evaluation and Deputy Executive Director, Diana Lopez: Evaluation Specialist Stuti Tripathi: Senior Policy and Evidence Uptake Officer
2. Who, Why, What and Why – 3ie • 2008 • Largest grant giving organization for impact evaluations • All areas • Raison d’etre
3. Deforestation – do protected areas help? Assessing trade-offs
4. Variable 1: Protected Areas
5. Variable 2: Road building
6. Predicting the location of Deforestation
7. Predicting Location of Impacts of the two variables • Where is deforestation most likely to occur? Which areas are vulnerable? • Are Protected Areas Effective? • Hypothesis: Plots devoted to the most lucrative use. So plot is cleared if clearing it is more profitable than letting it stay forested. 9
8. The Econometric model • The econometric model that we estimate is thus given by • Zi : Plot attributes (Slope, Elevation, Impedance weighted travel time, Soil Dummy, Population density) • Y1i*: Net profit from clearing • Y2i*: Net utility from protecting a plot otherwise0;0* 2 if1 222 * 2 otherwise0;0* 1 if1 1121 * 1 i Y i Y i e i WB i Z i Y i Y i Y i e i YB i Z i Y
9. Data – A spatially explicit GIS with many layers • Roads data (1982) • Land Use (1:1m, LDD) • District level census data on population (1990) • Physiographic: Elevation (DEM), Slope • Soil (FAO) • Socio-Economic: Population density • Cost of Travel to the market; Roads (DCW and LDD) • Protected Areas (IUCN) 12
10. 13 Roads (1982) and Forests Of North Thailand (1986) NORTH THAILAND Elevat.shp Elevat.shp 1 - 1000 feet 1000 - 3500 feet 3500 - 6000 feet 6000 - 7700 feet Elevat.shp
11. Sampling • Grid squares of 100ms, 28 Million data points • Sampled at 5 km (spatial autocorrelation): 6550 sampled pixels. • Impedance weighted cost of travel (Costdistance in Arc/Info)
12. NORTH THAILAND Elevat.shp Elevat.shp 1 - 1 000 fe et 100 0 - 3 500 fe et 350 0 - 6 000 fe et 600 0 - 7 700 fe et Elevat.shp
13. 16 Cleared Land (Y1 = 1) T- Stats Slope (degrees) -0.088 -10.652 Elevation (ms.) -0.001 -8.095 Population density1990 (people/km2) 0.003 4.532 Log(cost) (1982)** -0.191 -9.729 Soil and Province Dummies Not Shown Protected Area dummy (1986) -0.077 -0.332 Constant 1.295 8.870 Protected Area (Y2 = 1) Equation Slope (Degrees) 0.034 5.297 Elevation (ms.) 0.001 9.058 Population density1990 (people/km2) 0.001 2.297 Log(cost) (1982) 0.192 7.477 Soil and Province Dummies Not Shown Watershed dummy 0.188 3.543 Constant -4.098 -14.010 Log Likelihood -3714.7 No. of observations 4946
14. Results • In a static model, protected areas are not very successful in preventing deforestation; Agnostic about Wildlife Sanctuaries. • Roads hasten clearing. • Model useful in predicting vulnerable areas.
15. Accuracy of Predictions Actual Predicted Cleared Forested % of Predictions Correct Cleared 872 296 75% Forested 657 3133 83% % Correctly predicted 57% 91% 18
16. 19 Long way still....Khao Sanam Phriang Nam Nao Thung Salaeng Suang Predicted Threatened areas Wiang Kosai Mae Yom Chae Son Areas predicted to clear after cost is reduced by 100 units
17. Exploring the potential for using GIS/Big data: What affects agricultural expansion?
18. What affects Deforestation in the area? • Assessing impact of population and road bulding on the agricultural frontier Forest Reserves. • Uses panel data over 11 years (1986-1996) 24
19. Data Problems • Village level data • Unbalanced data set (Selection bias in villages?) • Ambiguous property rights • Groups of crops tracked. No consistent data on crop area. • Short run crop (soybean maize tobacco peanut) • Long run crop (upland rice, tea, tree crops) • Paddy rice • Self-reported data: Confounded zeros and missing values • Data on prices of crops
20. The Econometric Model • Log (Land devoted to crop i by village j in year t )= f(Pjt, Tjt , Wjit, Git, Nit, Lit , Ait, t) • Where • Pjt : Population of village j at time t • Tjt : Time taken to travel by most favored mode from village j to the market at time t • Ajt: Proportion of adults in village j at time t • Wjit : Availability of water in village j at time t for crop i • Gjt : Use of credit in village j at time t • Njt : Land productivity in village j at time t • Ljt: Status of property rights in village j at time t • t : time trend
21. Ag area Coefficient Intensity of cultivation Year 15.35** 0.0379 Area devoted to Paddy Rice 0.46** 0.01399** Area devoted to Upland Rice 0.21* 0.0011 Area devoted to Soybean -0.19+ 0.0037* Constant -641.17* 63.314** Sigma-u 1054.16 14.0611 Sigma-e 375.89 16.392 0.89 0.4239 Observations 1979 2174 R-square Within 0.042 Groups 622 629 R-square Between 0.054 R-square Overall 0.056 Gaussian Wald Statistic (chi2, 4df) 85.5 20.03 Prob > Chi2 0 0.0005 Results: Agricultural area & Intensity
22. Results •Overall, ease in access led to a substitution between upland rice and paddy rice •Upland rice area decreased. •Environmental AND livelihood benefits!
23. Conclusions • Magnitudes of policy instruments are clearly important to judge trade-offs. • The ‘where’ and the ‘how’ specially important. • Crop wise exploration is important. • Would have been excellent to have satellite and ground truthed data on • verifiable soil, • crop and • agricultural land data • Boundaries of forest reserves and location.
24. Promoting Agricultural Innovation in Sub-Saharan Africa and South Asia: an Impact Evaluation Program Challenges and Learnings ICRAF-Nairobi. September 3rd 2015
25. General Features of the Interventions • Target at the farmer association level • Target supply and demand: multiple intervention arm
26. Common challenges that can arise • Complex: Difficulties in identifying the outcomes of interest • Measuring the synergies among the different treatment arms • Very likely to have to deal with contamination • Eligibility and placement of geographical areas and farmers is not always clear cut • Women are usually underrepresented in the farmer associations
27. Learnings (1) • Doing qualitative work prior to first round of data collection it is key in order to : • Better understand of the the change the intervention can have on the population of interest • Better define the groups on which the intervention should have a differential impact • Better identify the possible sources of contamination and possible channels of spillover effects
28. Learnings (2) • Doing a pilot survey is also key to: • Better design of the data collection instruments • Identifying the constraints farmers may have to participate in the program
29. Learnings (3) • A good team work between researchers and implementing agencies is key for the project to move and implement successfully.
30. Evidence uptake and use
31. Generating evidence that is useful Helps to answer a specific policy question Rooted in the local context Opportunity exists for evidence to be used Message gives solutions (that can be drawn from the evidence) Affordable, logistically possible, politically feasible RELEVANT CONTEXTUAL DEMAND DRIVEN CLEAR FEASIBLE
32. Evidence uptake in 3ie-funded studies 0 2 4 6 8 10 12 14 16 18 Expand successful progs Close unsuccesful progs Change pol/prog design Inform discussions of pol/prog Inform design of other progs Inform global poldiscussions Improved culture of use of evidence
33. Expand successful programmes Wage subsidy programme in South Africa
34. 3ie’s ToC for evidence uptake 3ie funds high- quality policy relevant studies Researchers engage with key stakeholders Results in uptake of study findings We get applications - that ask policy relevant question - Comprises of a study team that has demonstrated experience in stakeholder engagement ASSUMPTIONS Researchers have the - Willingness and commitment - Understand how evidence uptake happens - Have the tools and resources to invest in stakeholder engagement Study makes policy relevant recommendations that are - Evidence based - Answer what works and why - Propose feasible solutions given context and cost ASSUMPTIONS ASSUMPTIONS Grant management is about making sure that the assumptions hold!!
35. 3ie’s ToC for evidence uptake 3ie funds high- quality policy relevant studies ASSUMPTIONS We get applications - that ask policy relevant question - Hold potential for policy impact - Comprise a study team that has demonstrated experience in stakeholder engagement Institution of preparation phase to facilitate context understanding and stakeholder engagement Increased weightage to policy relevance and impact How is this ensured
36. 3ie’s ToC for evidence uptake 3ie encourages researchers to engage with key stakeholders ASSUMPTIONS Researchers have Willingness and commitment - Understand how policy influence happens - Have the tools and resources to invest in stakeholder engagement Developing a plan and setting aside a budget for engagement Encouraging involvement of in country researchers Theory based evaluations– what works and why How is this ensured
37. 3ie’s ToC for evidence uptake Results in uptake of study findings Study makes policy relevant recommendations that are - Evidence based - Answer what works and why - Propose feasible solutions given context and cost ASSUMPTIONS Ongoing engagement with 3ie more dynamic model of interaction End of project interviews to learn and adapt processes How is this ensured
38. Ensuring quality and achieving relevance to users Use multiple prongs: integrated and regular use of many communication channels: social media, media, listservs, events, meetings, website Early Widely Often Establish champions
39. Thank you Jyotsna (Jo) Puri firstname.lastname@example.org
Impact Evaluations. Articles, experts, jobs, ... Program Manager, Impact Evaluation at Social Impact Past Analyst & Research Activity Manager, ...
... if not one of the first business courses, ... While management is both art and science, ... and useful principles of management.
... the authors state in their experimental evaluations that the methods are not ... impact of errors during ... the-art fusion based ...
Advertising Programmes Business Solutions +Google About Google Google.com © 2016 - Privacy - Terms. Search; Images; Maps; Play; YouTube; News; Gmail ...
To arrive at the edge of the world ... “See for yourself” is an unspoken credo of science. It is not enough to run an ... not very useful. ...
Eric Van Grasdorff is a Core Team Member of the Cameroonian NGO AfricAvenir ... olive trees , dates, and clean ... and practices the art and science of ...
... Simply Green Issue 6, 2013, Author ... as well as Food & Trees for Africa and ... assessment of human activity’s global impact and even the ...
So how should advisers raise the odds of having impact – and of being useful? ... 954 responses to “ The art and science of effective science ... not ...