Published on January 5, 2013
Evaluation and Revisionof Evolv On-Demand’sEnglish ProficiencyGrading Criteria San José State University i/o psychology internship Michael cilla
Agenda:Background info on the companyWhat is Selection Science?Details of the projectResultsRecommendations
The CompanyEvolv is the leader in science-based, on-demand talent matching andintelligence solutions, enabling enterprises around the world tosystematically improve their operations, brands and bottom line throughsuperior talent.Employees and employers enjoy a better job match, improving day-to-daysatisfaction.Clients benefit from identifying more productive talent that stays longer.Employees benefit from the satisfaction of excelling at jobs well suited tothem.Evolv delivers data-driven certainty and custom-configured solutions thateasily integrate with existing workforce solutions such as applicant trackingsystems.
Selection ScienceEvolv’s Selection Science collects and analyzesmillions of data points to provide a more objective Rigorously developed by a team of Ph.D. I/Oand data-driven decision engine. psychologists and market-proven with clients, theIt uses proprietary assessment algorithms to Selection Science decision engine starts smartevaluate existing employees and new job and gets smarter over time as it systematicallyapplicants for required traits, characteristics, correlates on-the-job performance to themotivations and skills specific to specific job selection criteria that placed them.roles.
Initial Assignment: The Project: Rating Job Applicants’ Proficiency in English for a Call/Contact Center in Puerto Rico The initial grading criteria were developed by the Evolv SelectionScientists in collaboration with linguists to define quantifiable gradingfor voice quality and English language proficiency in job applicantsfor a new client.
Questions used for voice auditions:Using a minimum of 5 complete sentences, please answer the following questions inEnglish. Call center agents often have to take initiative to reach their goals. Describe atime when you took some initiative or solved a problem. First describe the situation andthen describe the action you took.Using a minimum of 5 complete sentences, please answer the following questions inEnglish. Establishing good relationships with customers is very important as a call centeragent. Please describe how you establish relationships with people you’ve just met. Whatare some of the things you do or say? What challenges have you faced and how have youovercome them?Using your own words, please summarize what happened in the following situation. Johnwas already running late. John got out of his car even though he was running late. Johnwalked back to his garage to close the garage door. John’s garage door remote controldid not work.Using a minimum of 5 complete sentences, please answer the following questions inEnglish. Working in a call center can be a stressful job. Tell me about a time when youhad to deal with a very stressful situation at work or at school. First describe the situationin detail and then describe how you handled it and what you learned from it?Using your own words, please summarize what happened in the following situation: Valeriewas thinking about calling the customer service help line because her computer washaving problems, but after it started giving off smoke, she decided to call the firedepartment instead.
The Project: Rating Job Applicants’ Proficiency in English for a Call/Contact Center in Puerto Rico Preliminary Calibration Process: To establish consistency in scoring the applicants being selected, ateam of five employees uniformly calibrated assessments ofapplicants for approximately 30-60 trials. This consisted of the employees providing their individualassessments to the group, and then discussing why they decided toassign the scores that they chose. Based on these discussions, aconsensus was reached among the employees for applicant scoresbased on what qualities differentiated the variability in theirindividual assessments.
The Project: Rating Job Applicants’ Proficiency in English for a Call/Contact Center in Puerto Rico Scoring Applicants & Gaining Expertise: This calibration process continued for 60 days, however the teamof employees dropped to three, and finally down to one after 90days (>100 applicants). After scoring over 200 applicants, the primary applicant scorerpropositioned the supervisory staff of Selection Science to allow theemployee the opportunity to evaluate and possibly redesign thescoring system for school credit at the employee’s university.
The Project: Rating Job Applicants’ Proficiency in English for a Call/Contact Center in Puerto Rico Evaluation & Proposal for Revision The supervisory staff provided the approval and encouraged theemployee to utilize a rigorous statistical analysis of the availabledata, the conceptual subjective expertise the employee hadreceived while being the primary applicant scorer for this particularmodel, and additional research on previously developed speechassessment models developed by linguists.
Results Descriptive StatisticsDimension Correlation Matrix M Dimension 1 2 3 4 5 6 SD 1. Pronunciation --------Pronunciation 2.62 1.01 2. Semantic Content .71* --------Semantic Content 2.80 1.14 3. Fluency .70* .90* --------Fluency 2.76 1.19 4. Vocabulary .64* .86* .83* --------Vocabulary 2.96 1.14 5. Tone & Pace .59* .73* .71* .72* --------Tone & Pace 2.93 1.13 6. Total AggregateTotal Aggregated Score .79* .94* .92* .87* .80* -------- 2.93 Score 1.23 N = 212 * - Correlation is significant at .01 level.Of particular note are the strong relationships between semantic content, fluency, andvocabulary (r > .80), as well as the strong relationships between the applicants’ total scores andthe dimensions of semantic content, fluency, and vocabulary (r > .85).The Evolv English Proficiency Grading Criteria appears to have some redundancy due to theoverlap of the two dimensions: semantic content and fluency.
RecommendationsDelineate the performance component from the othercompetency-based components of the Grading Criteria byadopting the broader definition of fluency into the overallGrading Criteria.Define the phenomenon that the fluency dimension iscurrently assessing as encompassing the dimensions ofvocabulary and tone and pace.Adjust those dimensions (Vocabulary and Tone and Pace)to assess applicants’ lack of hesitation and lack ofstaggering their phrases as well.Replace the fluency dimension with a new dimensionassessing applicants’ comprehension.
RecommendationsProtocol for rating the dimension of comprehension:1) Include a new voice audition prompt to allow for the specific assessment of comprehension,possibly a “passing on a message” format.Some examples: • “Please tell Bill that this room is being used for a conference call this afternoon, and apologize for any inconvenience. Also,and ask if he would mind finding another room for today only. Finally, thank him sincerely for making this accommodation.”. • “Please tell John that I have to cancel our plans to have dinner tomorrow night. I could not find another babysitter to watch the children after my usual one cancelled. Please also tell him that I’m sorry for canceling at the last minute, and to call my cellular phone to reschedule for another time whenever he’d like”. • “Please explain to your customer, Sheila, that her wireless service is being discontinued for her lack of making her last monthly payment. Kindly ask if it is possible for her to make a payment today so that she does not experience an interruption in service. Finally, thank her for her time and wish her a pleasant day”.2) The criteria for assessment on a comprehension item should include checking for the applicants’use of their own words in the “message”, using the correct temporal sequence, and looking forquality over quantity.
Revised Dimensions & Detailed CriteriaPronunciation5 = Perfectly Clear1 = Garbled/Very Poor¬ How easily is he/she understood?¬ Special focus on ‘yuh’ vs. ‘juh’ as native Spanish Vocabulary speakers often substitute these phonemes. 5 = Extensive vocabulary¬ Special focus on /v/ and /p/ as both will sound like /b/ 1 = Very limited use of vocabulary/Use of non-English to English speakers words¬ Special focus on /s/ vs. /th/ as they are often ¬ Are speech, vocabulary, and grammar appropriate for interchanged an adult speaking on the phone? ¬ Is vocabulary noticeably limited (e.g., only simple,Semantic Content short words with a relatively small vocabulary)?5 = Perfectly sensible and complete ¬ Is there a significant amount of repetition of words?1 = Completely mixed ¬ Are words used appropriately?¬ Are responses coherent, sufficient, and conveying complete thoughts? Tone & Pace¬ Are thoughts and ideas linked sensibly? 5 = Very friendly, warm tone with a sense of energy and¬ Are responses flowing naturally, without any enthusiasm significant hesitation? 1 = Very slow or tired speech ¬ Is the speech too slow or rushed?Comprehension ¬ Is the tone of voice friendly?5 = Consistently demonstrates understanding of ¬ Does their voice have energy or sound sluggish?concepts ¬ Are the responses staggering and/or short and1 = Confused/responses off topic broken phrases?¬ Did the applicant use the majority of his or her own ¬ Does the applicant sound like a “natural” fluent words in the message? speaker?¬ Was the sequence of events temporally accurate?¬ Does the message qualitatively contain all information necessary?
Looking for a great internship at Learn From Within? Learn more about the Keynote Specialist position right now!
13 Pr Marketing Intern Keynote Group jobs. Find your next opportunity on Simply Hired. New jobs are posted every day.
Find Internships at the keynote group in the world's largest internship marketplace. Search internships by location, company and major on Internships.com.
keynote systems internship? Keynote Systems Net Worth is $121 M Sales. Keynote Systems (NASDAQ: KEYN) is a US-based Internet company that specializes in ...
Keynote Presentation Unless you have the same kind of computer that the school has, you will not be able to work on the Keynote at home. That ...
EMPLOYMENT ANNOUNCEMENT . Company: Morrison & Associates . Position: Internship - Keynote Presentation Developer . Salary: $20 hr. Hours: Flexible, short ...
KEYNOTES THAT INSPIRE ... Live Different’s keynote speakers share directly from their hearts. Their topics and stories are based on their own life ...
Toronto Journalism Internships, editorial intern jobs and video editor internships in Toronto at one of the hottest, most popular online publications
Summer Internship; Young Leaders Summit; ... keynote Tag. 2015 05 May Surgeon ... President Barack Obama to Keynote the APAICS 22nd Annual Awards Gala Dinner;