Class 5 designing so tl studies

50 %
50 %
Information about Class 5 designing so tl studies

Published on March 11, 2014

Author: tjcarter

Source: slideshare.net

Independent variable = the cause Dependent variable = the effect The researcher controls or manipulates the independent variable (the treatment) The dependent variable is what is measured, often called the assessment (knowledge, skills or attitudes). Testing Hypotheses

A Simple Hypothesis : The treatment (independent variable) improves students on the assessment (dependent variable). Three possible major problems related to causality: 1. The assessment was not measured well 2. The treatment was not manipulated well 3. Something other than the treatment caused change in the assessment (internal validity).

Construct and Internal Validity 0Construct Validity: Am I measuring what I think I am measuring? Am I implementing what I think I am implementing? 0Internal Validity: Did the treatment cause the outcome?

A Simple Hypothesis : The treatment (independent variable) improves students on the assessment (dependent variable). Three possible major problems related to causality: 1. The assessment was not measured well (reliability and construct validity) 2. The treatment was not manipulated well (construct validity) 3. Something other than the treatment caused change in the assessment (internal validity).

A study does not have absolute validity or absolutely no validity The level of validity relates to the confidence in the conclusions Construct and internal validity are measured on a continuum Construct validity does not imply internal validity (and vice versa) When a hypothesis is supported, it does not necessarily mean that the study has either construct or internal validity Some notes on evaluating construct and internal validity

What is meant by “construct”? 0 A concept, model, or schematic idea 0 A construct is the global notion of the measure, such as: 0 Student motivation 0Intelligence 0 Student learning 0 Student anxiety 0 The specific method of measuring a construct is called the operational definition 0 For any construct, researchers can choose many possible operational definitions

To Improve Construct Validity of Measures 0 Measure learning directly (clear operational definitions; learning is not the same as enjoyment or perceived learning) 0 Measure student learning through student learning objectives (ensure these are aligned with assessments) 0 Use Established Scales to Measure Student Attitudes and Personality (Don’t reinvent the wheel; Tests in Print)

Good Measurement is Important To Improve Construct Validity 0Know How To Score the Measure (make sure you’ve established this before data collection; know what is reasonable; IOTT; rubrics; training; IRR) 0Determine Whether to Use Graded or Ungraded Measures (pros and cons of both) 0Minimize Participant and Researcher Expectancies

To Improve Construct Validity 0Determine Whether to Use Multiple Operational Definitions (can use multiple measures) 0 Use a Retention Measure to Investigate Long-term Effects (but treat long term results with caution about other influences)

Good Differences between Conditions Improve Construct Validity The treatment (intervention) needs to be manipulated well to ensure construct validity The only difference between conditions should be the treatment Other variables that are different between conditions are confounds To determine construct validity, treatments need specific operational definitions Anything that can affect the results and cause a difference between students in treatment and control conditions needs to be documented

Potential problems in using different sections of a class Construct validity of the treatment is questionable in any design that compares one section of a class with another Classes are a social space, and the students and instructors are interdependent Students can ask different questions The class may have a different “tone” Splitting a class into two groups can minimize this concern; if students in a split class can be randomly assigned to a condition, internal validity will increase

Different Types of Comparison in Research Design Between Participants Within Participants: Multiple Treatments Within Participants: Multiple Measures How comparison works Students in one condition compared to students in another condition (control – Treatment; multiple T’s) All students in both control and treatment conditions Students receive both pre- test (control) and post-test (treatment) Strengths No carryover effects from multiple treatments; no instrumentation or testing effects from multiple assessments No selection bias; greater statistical power No selection bias; greater statistical power Weaknesses Selection bias without random assignment; many differences if groups are separate (e.g., two separate classes); lower statistical power Instrumentation and testing effects; carryover effects Instrumentation and testing effects; other confounds that occur between assessmens Improve Internal Validity by: Random assignment ; adding covariates Counterbalancing Increase number of assessments; add no treatment separate control condition; use alternative measures for assessment

External Validity Can the sample used in the study generalize to other groups or populations? Generally, it is impossible in classroom studies to get a sample that will generalize to all students. The researcher should report demographic chacteristics How realistic is the situation? In a classroom, if the treatment works, external validity is higher

Common Practical Problems in SoTL Research Researchers who think they need to measure everything Researchers who do not have many students: low statistical power Researchers who only have a single class; limits to type of design Difficulties in random assignment Difficulties in determining whether the treatment is potent enough to have an effect (see power above) Concerns about conducting an ethical study in a classroom or training situation

Don’t Use  Want to make statement about causality  Have low number of students Use  Have single group of students that cannot be divided  Have only one session in which to collect data Additional Options: Correlate many variables at the same time Simple Correlation

One-Group, Post Test Only Don’t Use  Want to make statement about causality  Want to make comparison to another group Use  Desired focus is on describing treatment and not assessment  Cannot have pre-test or control group  Want single group of students that cannot be divided

Two-Group, Post-Test Only Don’t Use  Have low number of students  Groups are very different  Have different assessments for each condition Use  Concerned about carryover effects  Concerned about testing and instrumentation effects  Have multiple groups  Have only one session to collect data Additional Options: • Use random assignment to improve internal validity • Add post-test to assess long-term change • Add additional conditions • Use covariates to improve internal validity and power

One Group, Pre-test, Post-test Don’t Use  Items other than treatment occur between assessments  First assessment affects second  Students likely to change between assessments with no treatment Use  Have low number of students  Have single group that cannot e divided  Cannot have control condition Additional Options: • Add post-test to assess long-term change • Use alternative measures to minimize testing and instrumentation effects

Two-Group, Pre-test/Post-test Don’t Use  Have single group of students that cannot be divided Use  Have multiple groups Additional Options: • Use random assignment to improve internal validity • Add post-test to assess long-term change • Use alternative measures to minimize testing and instrumentation effects • Add additional conditions • Use covariates to improve internal validity and power

Within Participants Design Don’t Use  Early treatments affect later treatments  Early assessments affect later assessments Use  Have low number of students  Have single group that cannot be divided Additional Options: • Add additional treatments • Counterbalance conditions to improve internal validity • Include pre-test to assess students before any treatment

Crossover Design Don’t Use  First assessment, by itself, affects second  Have single group of students that cannot be divided Use  Have low number of students  Have multiple groups Additional Options: • Include pre-test to assess before treatment • Add post-test to examine long-term change • Use random assignment to improve internal validity • Use alternative measures to minimize testing and instrumentation effects

Interrupted Time-Series Design Don’t Use  Have only one session to collect data  Early assessments affect later assessments Use  Have low number of students  Have single group that cannot be divided  Want to determine long- term effects Additional Options: • Add control condition to improve internal validity • Add additional treatment condition, with treatment at different time to improve internal validity

More Complex Designs 0Use Multiple Treatments to Investigate Interactions (Interactions) 0Use Moderators to Determine When Treatment Has Effect (Concept of ATI) 0Use Mediators to Investigate How Treatment Has Effect (Mixed Method?)

Remember! 0Each design has advantages and disadvantages 0Often, there is no clear right way, although some designs will be better than others 0There is no single ideal study that eliminates all potential problems and all alternative hypotheses 0No one study can answer all of your questions!

Add a comment

Related presentations

Related pages

Class 5 designing so tl studies - Documents

1.Independent variable = the cause Dependent variable = the effect The researcher controls or manipulates the independent variable (the treatment) The ...
Read more

Connecting the Variables: Designing Active Learning ...

Consistent with other studies, ... Connecting the Variables: Designing Active Learning Experiences That Work. ... 5:00 PM Mar 25th, ...
Read more

Amazon.com: The Multiplayer Classroom: Designing ...

... Designing Coursework as a Game on Amazon.com FREE SHIPPING on ... Designing a Class as a Game. 5. ... Case Studies. 9. Design for the Future ...
Read more

SoTL: Designing and Beginning - UCF Faculty Center for ...

Designing and Beginning ; What is SoTL? ... inductive followed by deductive studies; ... 5. Discuss ...
Read more

Scholarship of Teaching and Learning: Conducting SoTL in ...

The Scholarship of Teaching and Learning: ... 5. Make it public. Step 1: ... research (size of the class, ...
Read more

So da report class 5 - Documents - docslide.us

Download So da report class 5. Transcript. 1. SoDA report . 2. AgendaFigures Trends Platforms Evolution of digital strategy . 3. Figures Fuente: SoDA.
Read more

SoTL: Qualitative Research Types - UCF Faculty Center for ...

... (2002). Qualitative research in ... Biographical analysis takes the individual’s society and factors like gender and class ... 5(4 ), 405 ...
Read more

Design Research: An Analysis and Critique - Peabody College

Design Research: An Analysis and Critique ... focus on classroom design studies conducted to investigate ... designing supports for students’ or ...
Read more