How to Increase Extension Program Outcomes and Impacts (Slides for Educators)

33 %
67 %
Information about How to Increase Extension Program Outcomes and Impacts (Slides for...

Published on March 22, 2014




This video is meant for Extension educators to demonstrate the feasibility and usefulness of an Extension program and Evaluation Strategy that is based on specific goals. This presentation is a basic version and we have much more information that is part of continuous improvement in the slideshow. We will share other presentations with more information - so this is just the beginning! For program evaluation/monitoring questions, call 251-331-8416 or email For looking at some of my IPM Program evaluation publications, visit and click on 'IPM Evaluation Toolkit' in the menu. Thank you.

How to Improve Extension Program Outcomes using a Monitoring and Evaluation Strategy Dr. Ayanava Majumdar Extension Entomologist, SARE Coord. (AU) 111-A Duncan Hall, Auburn University 251-331-8416

Focus of this presentation • Structure & implementation of Alabama IPM Program • Provide some basics of evaluation – Extension perspective! • Encourage you to be consistent in your evaluation efforts • Develop capacity building within organization

About the Author (Ayanava) Ext. evaluations: • Memberships: American Eval. Assoc., Southeast Eval. Assoc. • AEA365 Blog Curator (2010-2011), Organization Capacity Building TIG Grant Panel, Ext. Eval. TIG member • Lead evaluator: USDA-NIFA, AL DoA, commodity grants, School IPM, AACAAS, NACAA Organizational capacity building: • Initiated ACES Program Evaluation Resource Committee, 2010… created the Evaluation Toolkit [online] • Workshops, webinars, publications: total ~7 hr, trained 161 Agents, County Coordinators, Specialists, Ext. Administrators

Alabama Small Farms • Produce fruits & vegetables that are consumed locally • Area = 6,000+ acres • Crop value = $48 million • Growth rate for industry = 14% • Vegetables: Tomatoes, sweet corn, watermelon, crucifers • Farmer markets: 135 (1100 farms) • Direct sales from AL farmers to consumers account for 0.2% of farm sales (Meter 2012)

Alabama Cooperative Extension System Ref.: Robinson, Dubois, Bailey (2005). Journal of Extension. Regional Extension Agents (REAs) Regional or statewide IPM events Outputs change annually Outcomes do not change! Impacts now underway. + CECs

ACES Commercial Horticulture Team – Vegetable IPM Team Members

ACES Home Grounds Team – Vegetable IPM Team Members Chris Becker, REA Willie Datcher, REA Mike McQueen, REA Alfred Jackson, Tuskegee Extension

Awards & Recognition • ‘Friends of IPM - Pulling Together Award’ from the Southern Region IPM Center, Raleigh, NC (2014) for IPM newsletter. • Friends of IPM: Future Leader Award (2012) By Southern Region IPM Center, Raleigh, NC Recognizes excellence in IPM program directly • Search for Excellence – Crops (2012) By National Association of County Agricultural Agents Recognizes a high impact IPM program • Achievement Award (2012) By the National Association of County Agricultural Agents • Communication Awards (6) By National Association of County Agricultural Agents

Structure of Alabama IPM Program: A Model Program for Study

Fund acquisition (external) • Funds needed to support REA travel & IPM demonstration plots • USDA/NIFA Collaborative Grants: $124,500 (part of two large grants worth $2.1 million) = 6% • SARE support: $10,000 + personnel for website • Alabama Department of Agriculture: $25,000 • Industry support: $45,000

Theory of Planned Behavior (TpB) Ajzen, I. 1991. The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes. 50: 179-211

Theory of Change: Ext. Logic Model

IPM Theory of Change Communities ImpactsModified from Kokate et al. 2009 Farmer-to-farmer Train-the-trainer Sustain change Key farmers Technology acceptance ExtensionResearch Project Management Capacity Building Need identified (context)

Ext. IPM Project Implementation Continuous needs & outcomes assessments 1 2 3 4 1 2 3 4 Model by Dr. A. Majumdar 2011 Based on Rockwell et al. 2003

CommunicationStrategy(IPM-CORE) No cost tools Technology identified & assessed Story developed Press release Extension Websites Social sharing (Slideshare, YouTube) Social networking (Facebook) Medium cost tools IPM Communicator Newsletter Extension regional meetings, IPM Web Conferences IPM Project flyers, bookmarks High cost tools Multiple partnerships (~20) IPM Exhibitions, Newspaper advertisements IPM on-farm demonstrations/ communications Small Farm IPM Field Guide IPM App?

Level 1: Regional IPM Meetings • Participants: Vegetable producers, certified organic farms, gardeners & homeowners, crop advisors, nonprofit organizations (partners) • Teaching techniques: Presentation, pest specimens • Typical length of talks: 46 min • Participants total: 1,535 • Number of events completed: 58 • Evaluation: Events are monitored for quality and impacts via multimodal surveys

Regional IPM Training Meetings (Level 1 training)

Level 2: IPM Workshops • Participants: Small vegetable producers, low resource farmers, Master Gardeners, nonprofit organizations, Extension Agents (in-service) • Teaching techniques: Case studies, simulations, specimens • Typical length of talks: 140 min • Participants total: 570 • Number of events completed: 14 • Evaluation: Qualitative & quantitative feedback, impact evaluations

Hands-on IPM Workshops (Level 2 training)

Level 3: IPM Field Days (Demonstration Sites) • Participants: Producers, low resource farmers, gardeners • Teaching techniques: Scouting, pest identification, decision-making system • Typical training time: 3-4 hours • Participants total: 308 • Number of events completed: 12 • Evaluation: Qualitative & quantitative feedback, impact evaluations, farm visits

IPM Field Days (Level 3 training)

Farmer-to-farmer training model is very influential for IPM project

Farmer-to-Farmer training during IPM Field Days Causes high information retention & impact on communities

Basics of Program Monitoring & Evaluation (PME)

Evaluation • Evaluation is the systematic collection of information about the merit, worth or significance of a program (Scriven 1999). • Evaluation is broadly defines as the systematic collection of data – both quantitative and qualitative – to aid users in developing knowledge about & managing a targeted set of activities (Scheirer 2012). • Ext. evaluation practice aims to find plausible conclusions, not cause & effect!

Rate your IPM program evaluation experience (useful exercise for audience)

Common myths! • My clients like me, so I do not need to evaluate. • Evaluation is hard & outside my job description. • Public does not respond to my surveys. • Nobody is going to look at the evaluations once they are done. • Only ‘experts’ can do evaluation. Why bother! PERCEPTION REALITY • Evaluation helps document your success. • 20% effort generates 80% information. • Provide time to respond, part of agenda, show improvements! • Share the results with all stakeholders. • Evaluation can be done by any trained individual committed to the standards.

Who can do the evaluation? Internal evaluators • Project leadership team directly guides evaluation (cost-effective) • Indicates organizational commitment /capacity building • Project monitoring possible (utilization-focus) • Fast assessment of reactive programs • Good participation External evaluators • Neutral third-party observers (expensive) • No internal capacity building • Good for impacts • Slow assessments & less useful • Poor participation (trust) Based on Patton (1997)

CDC Framework for Program Evaluation Standards Thomas Chapel, Chief Eval. Officer, CDC

Purpose of Evaluation Formative evaluation • Done when the program is active • Project monitoring • Continuous use • Flexible (mixed) designs Summative evaluation • Done at the end of a program • Measure impacts • One time use • Rigid design, expensive Patton (1997), Scriven (1972)

Utilization-focused Evaluations • Evaluation technique emphasizing USE (Patton 1993) • Improvement-oriented evaluation: includes formative evaluations, quality enhancement, Total Quality Management (TQM) • TQM includes using information systems to monitor program efforts & outcomes continuously • Use feedback as a monitoring system (like insect traps!). + Trouble= = Do something about it!

Evaluation techniques Reactive assessment: • Participants are aware of the assessment • Examples: surveys, interviews, tests Non-reactive assessment: • Participants are unaware of the assessment • Examples: observation Haas (2005)

Choose outcome ‘indicators’ carefully • Indicators are critical questions • Based on program objectives • Used for monitoring progress • Usually expressed as rates, percentages, efficiencies, etc. • There should be few trackable indicators (QUAL and QUAN)

Taylor-Powell & Henert 2008

Needs assessments Process evaluations (formative) Outcome Evaluations (formative) Impact evaluations (summative) Project monitoring Project improvement Modified strategy Modified outputs Evaluation Strategy in Extension IPM Program Graphic by Dr. Ayanava Majumdar, ACES

What do we measure? Hierarchy of Outcome Effects • Level 1: REACTION • Level 2: LEARNING • Level 3: BEHAVIOR • Level 4: RESULT Increasing complexity of evaluation Evaluate during program Kirkpatrick Four Levels, 1959, 1994 Post program evaluation QUAN QUAL

Hierarchy of Effects in IPM Program (In Pictures) • Level 1: REACTION • Level 2: LEARNING

Hierarchy of Effects in IPM Program (In Pictures) • Level 3: BEHAVIOR • Level 4: RESULTS

AL IPM Project OUTCOMES (Usefulness of Project Monitoring)

Project outcomes (2008-2012) • Developed a unique Theory of Change and Transformational Extension Education Model based on long-term outcomes • New producers reached 14% and rising (transitioning farms 7-10%). • General IPM adoption rate: over 90% • Adoption rate for unique IPM tactics: 63% (up from 38%) • Publication in print & electronic have significantly reduced barrier to IPM adoption • Funding for Small Farm IPM program has doubled since 2012. Much focus on Ext. Agent training.

• Rising interest of producers/Pull on project: – Over 10 times rise in participation in IPM events – 10 times rise in number of IPM events • Barriers to IPM adoption (crop advisors): – Lack of awareness reduced by 33% in 3 years – Difficulty in accessing information reduced by 33% • Barriers to IPM adoption (producers): – Lack of awareness reduced 16% in 3 years – Difficulty in accessing IPM information reduced 13% Project outcomes (2008-2012)

IPM Theory of Change Communities ImpactsModified from Kokate et al. 2009 Farmer-to-farmer Train-the-trainer Sustain change Key farmers Technology acceptance ExtensionResearch Project Management Capacity Building Need identified (context)

67 259 505 945 39 54 57 55 3 4 28 270 100 200 300 400 500 600 700 800 900 1000 2009 2010 2011 2012 Total participants Survey return rate (%) Major IPM Events GROWING INTEREST OF FARMERS IN IPM

1.5 6 6.9 73 36 26 5 8 10.18 34 29.1 11 7 20.9 0 10 20 30 40 50 60 70 80 2009 2010 2011 Beginning farmers (%) Conventional producers (%) LRFs (%) Transitioning farms (%) Gardeners (%) NATURE OF AUDIENCE AT IPM EVENTS

28 13.6 9.3 0 5 10 15 20 25 30 2009 2010 2011 Average land holding (acres) CHANGING NATURE OF EXTENSION CLIENTELE (reason for splitting IPM program into two campaigns to increase impacts) Organic/ Small Farms IPM Campaign Conventional farmers (with 15 to 50 acres) 2012 Farmers (<10 acres) Gardeners (<2 acres) Conventional/ Large Farms IPM Campaign

Conventional vegetable farmers = 15-50 acres Transitioning farmers = 10-14 acres Beginning farmers = <10 acres Low resource farmers = 2 to 4 acres CHANGING NATURE OF EXTENSION CLIENTELE (reason for splitting IPM program into two campaigns to increase impacts) Two IPM Campaigns launched in 2012 with separate strategies and outcomes. Rapid increase in small farm IPM funding and number of participants due to rising interest in local food systems.

45 29 10 23.2 30 2.2 20 11.4 2.5 5 0 5 10 15 20 25 30 35 40 45 50 2008 2011 Lack of awareness (%) High cost (%) Difficult to access information (%) Low availability of insecticides (%) Lack of time (%) BARRIERS TO IPM ADOPTION & CORRECTIONS Major concerns at the time

PROJECT IMPACTS (Watch 2013 Impact Video at

Impact Evaluation, 2010-2011 (n=58): • E-subscriptions growth rate = 12% • Current subscription base ~1350 with much higher readership. • 34% farmers, 28% company representatives, 38% others • 53% read for 15 minutes and 22% for 30+ minutes • 94% support continuation of the newsletter (2011 survey) • 70% use the information (2013 survey) • Six financial gains reports (2013): $3,550 in pesticide saving • Short-term impact of publication: $591 per case of adoption IPM Newsletter Impacts

Program Impacts: IPM Adoption Indicator 2011 2010 2009 Increase Adoption of insect monitoring/scouting practices 73.5% 41.0% +32% Adoption of insecticide recommendations 79.6% 46.5% 33% Use of biological insecticides 42.3% 26.5% 16% Overall IPM adoption 62.7% 38.5% 24% Economic & social impact assessments ongoing in 2012 and beyond. Received IRB Approval for Surveys.

Economic impacts • Yield loss in the absence of IPM: 44% or more • Vegetable producers gained $246 per acre by using IPM recommendations • Profits occur by judicious use of pesticides, adoption of appropriate control tactics after pest identification, and timely action. • Weekly IPM newsletter resulted in saving of over $500 per adoptive farmer

• Increased use of biological insecticides (42%) • Conservation of natural enemies (unknown) • Profits occur by judicious use of pesticides, adoption of appropriate control tactics after pest identification, and timely action. Environmental impacts

• Three new partnerships with influential commodity & consumer organizations • New partnerships with nonprofit agencies serving LRFs in Black Belt of Alabama • Impact assessments continue in 2012 & beyond. Social impacts

Charles Brannon, vegetable producer (Addison, AL): “This is the first year we used the IPM recommendation from the handbook and received training from Dr. A. In the hoop house alone, we sold about $5,500 worth of tomatoes which is double the output from last year. Fruitworms and stinkbugs used to do about 50% yield loss but not this year. With abundant production, we are taking our produce to large farmer markets in Birmingham and Decatur, not limiting ourselves just to the farm stand.” (Surveyed on July 5, 2012)

Danny Dickie, vegetable producer (Oneonta, AL): “We use the insecticide recommendations from the SE Vegetable Handbook and consult Mel Wade before we make a treatment decision. Without insecticides, we can potentially lose over 80% of our crops to worms and stink bugs that may result in over $20,000 loss per acre.” (Surveyed on May 30, 2012)

Albert Riddle, vegetable producer (Titus, AL): “We use the integrated pest management recommendations provided by Chip East – our Regional Extension Agent. I can lose over 50% of my tomato crop if I did not follow IPM for insect and disease management. That is about $10,000 loss per acre.” (Surveyed on May 23, 2012)

AND it all comes down to writing good reports… Questions for Dr. A? Thank you for your patience!

Add a comment

Related presentations

Related pages

Multiple Year Extension Program Outcomes & Impacts Through ...

Part of the challenge is that Extension educators ... program outcomes and impacts. The Extension ... program were designed to increase ...
Read more

Analyzing Program "Failure"

... documenting program impacts in recent years, Extension educators ... of outcomes/impacts ... Extension educators wait until the program is ...
Read more

How can the integration of evaluation within Extension ...

Evaluation Framework for Measuring Program Outcomes. Nav Ghimire. ... Judgment about how well your program objectives are being met…. What is Evaluation ?
Read more

Farm Transition & Estate Planning:

•Program Outcomes & Impacts ... •U of M Extension educators: two of which are IRS ... •Includes all PowerPoint slides.
Read more

Swash impacts outcomes - Education

Swash impacts outcomes. by irc. on May 22, 2015. Report Category: Education. Download: 0 Comment: 0. 169. views. Comments. Description.
Read more

PowerPoint Presentation - Education documents for farmers

Farm and Food tourism: Exploring opportunities in the West. ... Audience includes Extension educators, ... Program Impacts.
Read more

Guidelines for Documenting Short, Intermediate and Long ...

Guidelines for Documenting Short, Intermediate and ... help WSU Extension educators better ... language for describing outcomes and impacts in ...
Read more

“All My Money” Teaches Basic Financial Skills

“All My Money” Teaches Basic Financial Skills ... Extension educators and ... The objectives of the educational program are to increase the ability of ...
Read more