advertisement

Open surveys and their analysis v3

57 %
43 %
advertisement
Information about Open surveys and their analysis v3
Business & Mgmt

Published on December 19, 2013

Author: davidalman

Source: slideshare.net

Description

The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments.

Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words. • In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents.
advertisement

Open Surveys and their analysis Open Surveys and their analysis 1. Open Surveys The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments. Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words. Open Surveys are very different to what are termed Statistical Surveys for, in a sense, they work in reverse. That is to say:  In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents. In comparison;  Statistical Surveys use a “forced” response approach to questions that are structured around a pre-agreed model. These models have factors (categories) that are used as the basis to develop questions around an area of interest. Respondents have little to no opportunity to explain their answers. The answers provide a means of statistically (quantitatively) confirming responses against a survey’s underlying model’s factors (categories). As Statistical Survey’s generally do not explore questions in detail or depth, this can affect how valid they are where issues are more complex and need respondents to explain reasons for believing something, as is available through Open Surveys. Open Surveys are also termed Open Question Surveys and use respondent comments that are sometimes described as Narrative data; Text data; or Qualitative data. 2. Advantages and disadvantages of Open Surveys Advantages:  Allows respondents to answer in their own words.  Allows for "richness" or "depth" of explanation in responses.  Can identify and explore issues not currently fully recognised or understood.  Can identify options for further action. Disadvantages:  Limited by the writing skills of respondents.  Analysis involves a more complex interpretation.  Stronger role of interpretation in analysing responses. David Alman October 2012 Version 3 Page 1

Open Surveys and their analysis 3. Designing Open Survey Questions Questions used in Open Surveys usually use a word such as how, what, when, where, and why, to allow respondents to express their thoughts and feelings such as their attitudes, opinions, understanding, likes, dislikes, suggestions, and ideas. Open (sometimes referred to as unstructured) questions are ones in which possible answers are not suggested in the design of the survey questions, and where respondents answer in the way they see the world. Open ended questions are framed to encourage self explanation of thoughts and feelings in a sentence, paragraph, or more. Open ended questions include some direction on which the feedback is sought, such as:  Please tell us what changes you would like to see?  Please tell us what in your view we do well?  Where could we improve?  Why do you use our services? 4. Analysing Open Survey Responses 4.1 Open Survey and Analysis Framework The purpose of an Open Survey design and analysis of responses is to provide a framework that: a) Accurately and as thoroughly as possible captures the intent of respondent comments; and b) Analyses these comments so that they construct, step by step, an understanding of issues raised in a way that is both meaningful and relevant: That “fit and work” [1]. An explanation of how respondent comments are analysed, developed, and tested is provided in 4.2 Steps in Analysing Comments. 4.2 Steps in Analysing Comments Step 1. Read through all comments to get a feeling for the responses, and themes that emerge from this overview. Step 2 Develop Categories 2.1 Create categories. Identify categories (sometimes referred to as labels) from the different themes that emerge from comments. When a response is allocated to a category this is based on what is actually said and also any underlying meaning that might be recognised. 2.2 Test to improve Category Analysis. Test to see if there are alternative categories that have a better fit. David Alman October 2012 Version 3 Page 2

Open Surveys and their analysis 2.3 Decide if there are sub categories. Think about what the categories are about. Once comments have been categorised look again at the responses in each category to identify what is being expressed by different viewpoints. Are these consistent with the category, and could they also identify sub categories that are of value in the analysis? 2.4 Quantify Categories. Assign comments to at least one category (or sub category) and group them so the quantity of responses in each category can be more easily counted. Refer to Diagram 1. Developing Categories from Respondent Comments. Respondent Comments ID All Survey Comments 54 My supervisor is not supportive and criticises me in front of others. There is a lot of rework causing delays. Teams and Divisions are unwilling to work together. Work processes are inefficient and bureaucratic. There are conflicting work objectives and targets. Work output is more important than people. 84 165 248 362 420 Category: Role performance ID Comments 84 Total 220 There is a lot of rework causing delays Category: Organisational objectives ID Comments 362 Total 63 There are conflicting objectives and targets Category: Work Relations ID Comments 165 Total 116 Teams and Divisions are unwilling to work together Category: Work processes ID Total 432 Comments 248 Work processes are inefficient and bureaucratic Total 260 Diagram 1. Developing Categories from Respondent Comments Step 3 Link Categories to develop Patterns and Model(s) Once comments have been studied and categorised (and sub categorised), the next step is to see how these categories link to form patterns and models:  Do some categories link in some way, and how do unrelated others link to form patterns?  Do these patterns, together, form to represent one or more models? In developing patterns check if there are exceptions to the rule that require the patterns to be broadened to include other categories, or changed into other patterns, or cast doubt over a pattern as a “rule”. Refer to Diagram 2. Developing Patterns from Categories. David Alman October 2012 Version 3 Page 3

Open Surveys and their analysis Pattern of Cultural Issues Pattern of Performance Issues Category: Organisational objectives Category: Work Processes Category: Organisational values Category: Work Relations Category: Job relationships Category: Job Performance Diagram 2. Developing Patterns from Categories In developing models these may be either: a) Drawn from the patterns of analysis; b) Brought to the analysis from a recognised published model that links to categories and their patterns. For example a recognised model relevant to the type of survey carried out such as a Hospitality, Health Service, Service Delivery, Management, Environmental Management, or Community Model. Either approach to developing a model offers advantages, such as an internally developed model can assist in gaining commitment from those involved in its development whereas the use of an external model offers validity through its credibility as a recognised benchmark. Whichever approach is used the selected model should be as inclusive as possible of all respondent comments. Refer to Diagram 3. Organisational Alignment Model based on fit to Patterns. David Alman October 2012 Version 3 Page 4

Open Surveys and their analysis Organisational Objectives Organisational Values Organisational Performance Organisational Culture Work processes Work relations Job performance Job relationships Diagram 3. Organisational Alignment Model based on fit to Patterns Step 4. Write up the analysis: Once all comments are categorised and analysed, and the patterns they form identified along with any model that fits those patterns, then the next step is to write up in a summary. The analysis can include:  An explanation of a model and the patterns that pull the analysis together.  Categories and sub categories identified (at the very least the more significant categories).  The number of comments covered by a category. This can be in either broad quantitative terms covering more than one category, or specific to each category.  An explanation about the categories and sub categories along with supporting (non identifiable) quotations drawn from respondent comments.  Recommendations based on the analysis. Recommendations for improvements can come directly from comments where there are several similar responses or from a single, different, comment. The survey design and analysis should demonstrate validity:  Face Validity: o How survey findings make sense in terms of credibility, relevance, and usefulness to respondents and those who decide to use survey recommendations; o Use respondents own words in providing credibility to survey recommendations. David Alman October 2012 Version 3 Page 5

Open Surveys and their analysis  Construct Validity: The extent to which the survey design and analysis minimise error and misinterpretation and is consistent in “fitting” with respondent comments. That is, where comments fit with constructs (or concepts) in terms of developing criteria; patterns; and models. Notation. [1] In references below, see Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey Responses, the term Concept Mapping is used. The term Construct Mapping is used here instead of the term Concept Mapping so that there can be a better and more logical alignment to Construct Validity. In this respect an Open Survey analysis could be viewed as a form of Construct Mapping in that comments are organised (i.e. constructed) first into categories; these categories are then organised (constructed) into patterns; and where these patterns may then be organised (constructed) into a model. When constructing such a map the outcomes (whether this relates to the construction of categories, patterns, or a model) should readily, not forcibly, “fit” with respondent comments and also “work” in the sense that categories, patterns, and any model selected can be seen to provide meaningful and relevant conclusions. “Fit” in this sense relates to Construct Validity, and “work” to Face Validity. References include: Outline explanations of Open Survey Analysis Frameworks Analysing open-ended questions. Website: http://intelligentmeasurement.wordpress.com/2007/12/18/analyzing-open-endedquestions/ Downloaded 8/8/12 Analysing Qualitative Data. Website: http://learningstore.uwex.edu/assets/pdfs/g365812.pdf Downloaded 8/8/12 A Brief Guide to the Analysis of Open-Ended Survey Questions. Website: http://cms.cerritos.edu/uploads/ResearchandPlanning/Brief_Guide_to_OpenEnded_Survey_Questions.pdf Downloaded 8/8/12 Concept Mapping Explained Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey Responses Website: http://www.socialresearchmethods.net/research/Concept%20Mapping%20as%20an%20Alt ernative%20Approach%20for%20the%20Analysis%20of%20OpenEnded%20Survey%20Responses.pdf Downloaded 8/8/12 Comprehensive References on Open Survey Design and Analysis Qualitative Data Analysis. Ian Dey. Website: http://www.drapuig.info/files/Qualitative_data_analysis.pdf Downloaded 8/8/12 Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park, California: SAGE Publications Inc. Organisational Alignment Model in Diagram 3 adapted from Figure 3.9 in: David Alman October 2012 Version 3 Page 6

Open Surveys and their analysis Addison, R., Haig. C., & Kearney. L. (2009). Performance architecture: The art and science of improving organizations. San Francisco, California: Pfeiffer. Key words Open surveys; Open survey analysis; Construct Mapping; Concept Mapping David Alman October 2012 Version 3 Page 7

Add a comment

Related presentations

Canvas Prints at Affordable Prices make you smile.Visit http://www.shopcanvasprint...

30 Días en Bici en Gijón organiza un recorrido por los comercios históricos de la ...

Con el fin de conocer mejor el rol que juega internet en el proceso de compra en E...

With three established projects across the country and seven more in the pipeline,...

Retailing is not a rocket science, neither it's walk-in-the-park. In this presenta...

What is research??

What is research??

April 2, 2014

Explanatory definitions of research in depth...

Related pages

Observational Surveys: Implementation and Analysis ...

document.write(adsense.get_banner_code('200x90')); Slide 1 Observational Surveys: Implementation and Analysis Slide 2 2 Observational Surveys ...
Read more

PatientDynamics GPAQ V3 Analysis Report

... (HX3 9DG) GPAQ V3 Analysis ... PatientDynamics is an approved supplier of NHS Patient Surveys ... The practice / doctor sent in their analysis ...
Read more

SO YOU WANT TO ADMINISTER A SURVEY…

SO YOU WANT TO ADMINISTER A SURVEY ... survey schedule to avoid competing with their administration windows ... o A data analysis plan, including open ...
Read more

Attitudes to Pensions: The 2012 survey (RR813 ...

Research and analysis. Attitudes to Pensions: The 2012 survey ... is the third in a series of surveys. ... under the Open Government Licence v3.0, ...
Read more

PatientDynamics GPAQ V3 Analysis Report

PatientDynamics is an approved supplier of NHS Patient Surveys and a licensed ... The practice / doctor sent in their analysis ... GPAQ V3 Analysis ...
Read more

Free Online Surveys | SurveyMonkey

Create Online Surveys for Free Find out what your customers or employees are thinking with a survey today. Sign Up FREE ...
Read more

ScottMadden Shared Services Customer Surveys v3

benefit of comparing their ... Conducting periodic customer satisfaction surveys should be ... ScottMadden_Shared_Services_Customer_Surveys v3 ...
Read more

Tell Them From Me student survey Frequently Asked ...

Tell Them From Me student survey Frequently Asked Questions (parents and carers) 1. ... The survey is open to all NSW ... Once the surveys are ...
Read more