advertisement

Smart Evaluation Considerations

100 %
0 %
advertisement
Information about Smart Evaluation Considerations

Published on July 20, 2008

Author: jorgerestrepo

Source: slideshare.net

Description

considerations when designing evaluation of a marekting or public education campaign.
advertisement

Planning Campaigns with Evaluation in mind By Jorge Restrepo M.G.A Eureka Facts LLC

Design Clearly define the intended outcomes of a program and ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal. Educational, language and cultural barriers are necessary considerations in good evaluation design. Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.

Clearly define the intended outcomes of a program and ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal.

Educational, language and cultural barriers are necessary considerations in good evaluation design.

Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.

Design The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment. Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal. Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.

The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment.

Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal.

Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.

Methods - data collection Tips for in-person data collection Protocols and interviewer training are key Supervision and redundant verifications are important Dress-codes and interviewer profile Compensation/ Investment Controls Timing Response rate

Tips for in-person data collection

Protocols and interviewer training are key

Supervision and redundant verifications are important

Dress-codes and interviewer profile

Compensation/ Investment

Controls

Timing

Response rate

Methods - data collection Mail surveys Lists – sample frame Who is your respondent? Sponsor, package, timing, incentive Respondent burden Education/language Coding visibly, invisible coding Flexibility Timing Response rate - examine non-response closely

Mail surveys

Lists – sample frame

Who is your respondent?

Sponsor, package, timing, incentive

Respondent burden

Education/language

Coding visibly, invisible coding

Flexibility

Timing

Response rate - examine non-response closely

data collection Online Who is your respondent? - Your sample frame is everything Spam filters are a bigger concern than non-response Web links create disproportionate sampling Make web links and pop-ups as unobtrusive as possible Respondent fatigue CANSPAM COPA

Online

Who is your respondent? - Your sample frame is everything

Spam filters are a bigger concern than non-response

Web links create disproportionate sampling

Make web links and pop-ups as unobtrusive as possible

Respondent fatigue

CANSPAM

COPA

data collection Telephone Who is your respondent? - Your sample frame is everything RDD sampling – often no longer representative Combine methods No land line households Reach Incentives Cooperation Do not call registry

Telephone

Who is your respondent? - Your sample frame is everything

RDD sampling – often no longer representative

Combine methods

No land line households

Reach

Incentives

Cooperation

Do not call registry

Challenges in Analysis Most distributions – especially in public education efforts are not normal so common centrality statistics (mean, median) are not very meaningful. People often articulate categories (i.e. well, not well) better than continuum scales.

Most distributions – especially in public education efforts are not normal so common centrality statistics (mean, median) are not very meaningful.

People often articulate categories (i.e. well, not well) better than continuum scales.

Analysis Think outside the box – examine findings spatially Stars represent respondents to a campaign

Think outside the box – examine findings spatially

Analysis Think outside the box – examine findings spatially Flat findings examined by other means revealed strong differences by county

Think outside the box – examine findings spatially

Simplify constructs Build your assessment on existing frameworks and models, but create high level aggregates as well This evaluation measures knowledge and action

Build your assessment on existing frameworks and models, but create high level aggregates as well

High and Low Impact Groups A powerful tool is the use of classification trees to identify pockets of a population with high or low impact.

A powerful tool is the use of classification trees to identify pockets of a population with high or low impact.

Compare - Always Mirrored instruments are especially valuable in examining outcomes

Mirrored instruments are especially valuable in examining outcomes

Segment for Success Outcomes are rarely evenly spread out throughout a population

Outcomes are rarely evenly spread out throughout a population

Common cross-tabbing analysis Ethnicity Hispanic Origin Gender Age Education Urban locale Generational cohort Socioeconomic status Life stage, career stage Primary language Geography

Ethnicity

Hispanic Origin

Gender

Age

Education

Urban locale

Generational cohort

Socioeconomic status

Life stage, career stage

Primary language

Geography

Who we are Eureka Facts brings you the smart marketing Information you need to discover opportunities maximize results identify issues make decisions reduce market risks

Eureka Facts brings you the smart marketing Information you need to

discover opportunities

maximize results

identify issues

make decisions

reduce market risks

Smart Evaluation www.EurekaFacts.com EurekaFacts LLC 451 Hungerford Drive Suite 515 Rockville MD 20850 (301) 610-0590

Add a comment

Related pages

(considerations for) Assessment, Monitoring and Evaluation ...

(considerations for) Assessment, Monitoring and Evaluation for Results-based Adaptation Planning Hideki Kanamaru, Ph.D. Natural Resources Officer (Climate ...
Read more

Get Smart About Antibiotics | Program Development and ...

Evaluation Resources on program evaluation, including a Get Smart program manual... Related Links. Antibiotic/Antimicrobial Resistance; Get Smart ...
Read more

Evaluation Framework for Smart Grid Deployment Plans (pdf)

Evaluation Framework for Smart Grid Deployment Plans A Systematic Approach for Assessing Plans to Benefit Customers and the Environment
Read more

United Nations Development Programme - Evaluation

The Handbook on Planning, Monitoring and Evaluating for Development Results seeks to address new directions in planning, monitoring and evaluation in the ...
Read more

Designing Evaluation

2 Designing Evaluation for Education Projects Project evaluation serves two general purposes. Evaluation helps to determine the project’s merit
Read more

Federal Communications Commission

Federal Communications Commission . ... SAR Evaluation Considerations . ... and similar accessories are also discussed for recent generation smart phones.
Read more

Key Considerations when Evaluating an Enterprise ...

Key Considerations when Evaluating an Enterprise Collaboration Solution ... Technologies such as smart mobile devices, ... This makes the evaluation of a ...
Read more

North Carolina Smart Start Evaluation Notebook

North Carolina Smart Start Evaluation Notebook Smart Start Evaluation Assistance Team Frank Porter Graham Child Development Institute University of North ...
Read more

Writing SMART Learning Objectives - University of North ...

Writing SMART Learning Objectives To be useful, learning objectives should be SMART: Specific Measurable Attainable Results-Focused Time-Focused Learning ...
Read more