Hands and Speech in Space

50 %
50 %
Information about Hands and Speech in Space
Technology

Published on May 29, 2014

Author: marknb00

Source: slideshare.net

Description

Speech given by Mark Billinghurst at the AWE 2014 conference on how to use multimodal speech and gesture interaction with Augmented Reality applications. Talk given on May 28th, 2014.

Hands and Speech in Space Mark Billinghurst mark.billinghurst@hitlabnz.org The HIT Lab NZ, University of Canterbury May 28th 2014

2012 – Iron Man 2

To Make the Vision Real..   Hardware/software requirements  Contact lens displays  Free space hand/body tracking  Speech/gesture recognition  Etc..   Most importantly  Usability/User Experience

Natural Hand Interaction   Using bare hands to interact with AR content   MS Kinect depth sensing   Real time hand tracking   Physics based simulation model

Pros and Cons of Gesture Only Input   Gesture-only good for  Direct manipulation,  Selection, Motion  Rapid expressiveness   Limitations  Descriptions (eg Temporal information)  Operation on large numbers of objects  Indirect manipulation, delayed actions

Multimodal Interaction   Combined speech and gesture input   Gesture and Speech complimentary   Speech: modal commands, quantities   Gesture: selection, motion, qualities   Previous work found multimodal interfaces intuitive for 2D/3D graphics interaction   However, few multimodal AR interfaces

Wizard of Oz Study   What speech and gesture input would people like to use?   Wizard   Perform speech recognition   Command interpretation   Domain   3D object interaction/modelling Lee, M., & Billinghurst, M. (2008, October). A Wizard of Oz study for an AR multimodal interface. In Proceedings of the 10th international conference on Multimodal interfaces (pp. 249-256). ACM.

System Architecture

System Set Up

Key Results   Most commands multimodal   Multimodal (63%), Gesture (34%), Speech (4%)   Most spoken phrases short   74% phrases average 1.25 words long   Sentences (26%) average 3 words   Main gestures deictic (65%), metaphoric (35%)   In multimodal commands gesture issued first   94% time gesture begun before speech

Free Hand Multimodal Input   Use free hand to interact with AR content   Recognize simple gestures   Open hand, closed hand, pointing Point Move Pick/Drop Lee, M., Billinghurst, M., Baek, W., Green, R., & Woo, W. (2013). A usability study of multimodal input in an augmented reality environment. Virtual Reality, 17(4), 293-305.

Speech Input   MS Speech + MS SAPI (> 90% accuracy)   Single word speech commands

Multimodal Architecture

Multimodal Fusion

Hand Occlusion

Experimental Setup Change object shape and colour

User Evaluation   Change object shape, colour and position   Conditions   (1) Speech only, (2) gesture only, (3) multimodal   Measures   performance time, errors, subjective survey

Results - Performance   Average performance time   Gesture: 15.44s   Speech: 12.38s   Multimodal: 11.78s   Significant difference across conditions (p < 0.01)   Difference between gesture and speech/MMI

Subjective Results (Likert 1-7)   User subjective survey   Gesture significantly worse, MMI and Speech same   MMI perceived as most efficient   Preference   70% MMI, 25% speech only, 5% gesture only Gesture Speech MMI Naturalness 4.60 5.60 5.80 Ease of Use 4.00 5.90 6.00 Efficiency 4.45 5.15 6.05 Physical Effort 4.75 3.15 3.85

Observations   Significant difference in number of commands   Gesture (6.14), Speech (5.23), MMI (4.93)   MMI Simultaneous vs. Sequential commands   79% sequential, 21% simultaneous   Reaction to system errors   Almost always repeated same command   In MMI rarely changes modalities

Lessons Learned   Multimodal interaction significantly better than gesture alone in AR interfaces for 3D tasks   Shorter task time, more efficient   Multimodal input was more natural, easier, and more effective that gesture/speech only   Simultaneous input rarely used   More studies need to be conducted   What gesture/speech patterns? Richer input

3D Gesture Tracking   3 Gear Systems   Kinect/Primesense Sensor   Two hand tracking   http://www.threegear.com

Skeleton Interaction + AR   HMD AR View   Viewpoint tracking   Two hand input   Skeleton interaction, occlusion

AR Rift Display

Conclusions   AR experiences need new interaction methods   Combined speech and gesture more powerful   Complimentary input modalities   Natural user interfaces possible   Free hand gesture, speech, intelligence interfaces   Important research directions for the future   What gesture/speech commands should be used?   Relationship better speech and gesture?

More Information •  Mark Billinghurst –  Email: mark.billinghurst@hitlabnz.org –  Twitter: @marknb00 •  Website –  http://www.hitlabnz.org/

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

Hands and speech in space - dl.acm.org

Hands and speech in space: multimodal interaction with augmented reality interfaces
Read more

Hands and speech in space: Multimodal interaction with ...

Hands and speech in space: Multimodal interaction with augmented reality interfaces on ResearchGate, the professional network for scientists.
Read more

Hands and Speech in Space - Technology - documents.mx

Speech given by Mark Billinghurst at the AWE 2014 conference on how to use multimodal speech and gesture interaction with Augmented Reality applications.
Read more

Hands and Speech in Space: Multimodal Input for Augmented ...

1.Hands and Speech in Space: Multimodal Interaction for AR Mark Billinghurst mark.billinghurst@hitlabnz.org The HIT Lab NZ, University of Canterbury ...
Read more

Mark Billinghurst (HIT Lab NZ) - "Hands and Speech in ...

Mark Billinghurst (HIT Lab NZ) - "Hands and Speech in Space" at AWE2014 Augmented World Expo (AWE) ... Mark Billinghurst, HIT LAB NZ, ...
Read more

Mun and back in Kerbal Space Program with no hands ...

Using my speech recognition software 'CIDI' to get to the mun and back in Kerbal Space Program without using the keyboard or mouse at all, and ...
Read more

Prepared Remarks for JSC All Hands Mike Coats, Director ...

Prepared Remarks for JSC All Hands . Mike Coats, Director, Johnson Space Center . October 12, 2010 . Good morning: ... at the Johnson Space Center, ...
Read more

NASA

NASA.gov brings you the latest news, images and videos from America's space agency, pioneering the future in space exploration, ...
Read more