Developing Multi Touch Applications

40 %
60 %
Information about Developing Multi Touch Applications
Technology

Published on January 26, 2009

Author: bblanchard4

Source: slideshare.net

Description

Presentation at Microsoft's Saint Louis Day of Dot Net.

Developing Multi-touch Applications The future is here Brian Blanchard Chief Architect Essence Technology

Agenda Introduction Past, Future, & Present Touch scenarios, “Good, Better, Best” model, and developer roadmap Platform details: Native Win32 gesture and touch support Manipulations and Inertia WPF support UX guidelines for touch applications Where to get started - SDK, drivers, etc Wrap up

Introduction

Past, Future, & Present

Touch scenarios, “Good, Better, Best” model, and developer roadmap

Platform details:

Native Win32 gesture and touch support

Manipulations and Inertia

WPF support

UX guidelines for touch applications

Where to get started - SDK, drivers, etc

Wrap up

What is Multi-Touch? Multi-touch * denotes a set of interaction techniques which allow computer users to control graphical applications with several fingers. The concept of Multi-touch technology was created to incorporate collaboration into the computing experience. * Wikipedia definition

Multi-touch * denotes a set of interaction techniques which allow computer users to control graphical applications with several fingers.

The concept of Multi-touch technology was created to incorporate collaboration into the computing experience.

* Wikipedia definition

Multi-touch Past 26 years in the making Started in 82 by the university of Toronto First commercial attempt in 84 by bell labs Adopted by MS in 2001

26 years in the making

Started in 82 by the university of Toronto

First commercial attempt in 84 by bell labs

Adopted by MS in 2001

Multi-touch Future LucidTouch: Transparent handheld inputs Obscura Digital: HP - D5: Multi-Touch wall VisionAire: Holographic Multi-Touch

LucidTouch: Transparent handheld inputs

Obscura Digital:

HP - D5: Multi-Touch wall

VisionAire: Holographic Multi-Touch

Multi-touch Is Here! Hardware Multi-touch capable machines in market today, multi-touch appears in a broad set of form factors Desktops: $1200 - $2000 Laptops: $1800 - $3000 Software Windows 7, PDC partner demonstrations & Touch platform announcement, .Net 4.0 Consumers New scenarios, next wave of user experience, high “WOW” factor

Hardware

Multi-touch capable machines in market today, multi-touch appears in a broad set of form factors

Desktops: $1200 - $2000

Laptops: $1800 - $3000

Software

Windows 7, PDC partner demonstrations & Touch platform announcement, .Net 4.0

Consumers

New scenarios, next wave of user experience, high “WOW” factor

Touch Scenarios And Windows 7 We’ve identified several key scenarios for multi-touch: Navigating and consuming the web Reading & sorting email Viewing photos Playing casual games Consuming music and video Navigating files & arranging windows Using Office applications All focused on consumption That led to four areas of investment: Developer Platform : At the root is the touch developer platform that exposes touch APIs for any application UI Enhancements : Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences. Gestures : Multi-touch gestures have been added to enable consistent panning and zooming in most applications. Applications : A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7.

We’ve identified several key scenarios for multi-touch:

Navigating and consuming the web

Reading & sorting email

Viewing photos

Playing casual games

Consuming music and video

Navigating files & arranging windows

Using Office applications

All focused on consumption

That led to four areas of investment:

Developer Platform : At the root is the touch developer platform that exposes touch APIs for any application

UI Enhancements : Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences.

Gestures : Multi-touch gestures have been added to enable consistent panning and zooming in most applications.

Applications : A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7.

A Structured Approach To Adding Touch Support To Your Application Which application scenarios get the biggest benefit from touch? Content consumption, mobile, media and entertainment Decide which tier of investment is appropriate: Good – Free support, UX tweaks Better – Gesture support, touch-friendly UX Best – Touch-optimized experience

Which application scenarios get the biggest benefit from touch?

Content consumption, mobile, media and entertainment

Decide which tier of investment is appropriate:

Good – Free support, UX tweaks

Better – Gesture support, touch-friendly UX

Best – Touch-optimized experience

Touch Development Roadmap Windows 7 Release NET 4.0 / Surface 2.0 Release Multi-Touch Controls Multi-Touch API Surface Multi-Touch Controls & API Multi-Touch API Multi-Touch API and Controls

Touch Platform Overview Good Better Best APIs For Free! Panning/zoom gestures Right click gesture Gesture notifications Pan/zoom/rotate/etc Raw touch data Manipulation and Inertia processors Native Win32 Controls with standard scrollbars WM_GESTURE message WM_TOUCH COM based Manipulation and Inertia Processors WPF WPF 4.0 pan support in ScrollViewer Gesture events Inertia configuration Touch events Manipulation and Inertia Processors WinForms Controls with standard scrollbars WM_GESTURE message P/Invoke Manipulation and Inertia Processors in Microsoft.Ink.DLL Real-time Stylus or Ink Collector

For Free!

Panning/zoom gestures

Right click gesture

Gesture notifications

Pan/zoom/rotate/etc

Raw touch data

Manipulation and Inertia processors

Controls with standard scrollbars

WM_GESTURE message

WM_TOUCH

COM based Manipulation and Inertia Processors

WPF 4.0 pan support in ScrollViewer

Gesture events

Inertia configuration

Touch events

Manipulation and Inertia Processors

Controls with standard scrollbars

WM_GESTURE message

P/Invoke

Manipulation and Inertia Processors in Microsoft.Ink.DLL

Real-time Stylus or Ink Collector

Windows 7 Gestures & WM_GESTURECOMMAND Brian Blanchard Chief Architect Essence Technology

Brian Blanchard

Chief Architect

Essence Technology

WM_GESTURECOMMAND WM_GESTURECOMMAND Better : Good gesture support is key Window message conveying gesture notifications and information to applications Why use it? Richer than legacy notifications, smoother more responsive scrolling and zooming Compound gestures Coalescing Message bubbling and capture Legacy handling & SFP

WM_GESTURECOMMAND

Better : Good gesture support is key

Window message conveying gesture notifications and information to applications

Why use it? Richer than legacy notifications, smoother more responsive scrolling and zooming

Compound gestures

Coalescing

Message bubbling and capture

Legacy handling & SFP

WM_GESTURECOMMAND BOOL WINAPI GetGestureCommandInfo( __in UINT uMsg , __in WPARAM wParam , __in LPARAM lParam , __in LPARAM lExtraInfo , __inout PGESTUREINFO pGestureInfo ); typedef struct _GESTURECOMMANDINFO { UINT cbSize; DWORD dwFlags; DWORD dwCommand; DWORD dwArguments; POINTS ptsLocation; } GESTURECOMMANDINFO, *PGESTURECOMMANDINFO;

BOOL WINAPI GetGestureCommandInfo(

__in UINT uMsg ,

__in WPARAM wParam ,

__in LPARAM lParam ,

__in LPARAM lExtraInfo ,

__inout PGESTUREINFO pGestureInfo

);

typedef struct _GESTURECOMMANDINFO { UINT cbSize; DWORD dwFlags; DWORD dwCommand; DWORD dwArguments; POINTS ptsLocation;

} GESTURECOMMANDINFO, *PGESTURECOMMANDINFO;

WM_GESTURECOMMAND Interpreting gestures: dwCommand dwArguments ptsLocation Pan Distance between contacts Current center of gesture Zoom Distance between contacts Current center of gesture Rotate Absolute angle on rotate start, delta on updates Current center of gesture Two-finger tap NA Current center of gesture Right click gesture NA First finger down

Interpreting gestures:

WM_GESTURE Future Windows 7 Build Minor API surface changes: Naming: WM_GESTURE, GESTUREINFO, GetGestureInfo(), etc Additional fields in GESTUREINFO Added gesture configuration support: WM_GESTURENOTIFY GESTURECONFIG struct Set/GetGestureConfig()

Minor API surface changes:

Naming: WM_GESTURE, GESTUREINFO, GetGestureInfo(), etc

Additional fields in GESTUREINFO

Added gesture configuration support:

WM_GESTURENOTIFY

GESTURECONFIG struct

Set/GetGestureConfig()

Single-finger Panning Future Windows 7 Build Single-finger panning (SFP) provides an even easier way to pan where it is natural How to control SFP in your application: SetGestureConfig() for GC_PAN Specify GC_PAN_WITH_SINGLE_FINGER_VERTICALLY and/or GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY If you aren’t getting the SFP default: Consider standard scrollbars SBM_GETSCROLLINFO (remarks here are helpful) Implement WM_GESTURE support

Single-finger panning (SFP) provides an even easier way to pan where it is natural

How to control SFP in your application:

SetGestureConfig() for GC_PAN

Specify GC_PAN_WITH_SINGLE_FINGER_VERTICALLY and/or GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY

If you aren’t getting the SFP default:

Consider standard scrollbars

SBM_GETSCROLLINFO (remarks here are helpful)

Implement WM_GESTURE support

WM_TOUCH Brian Blanchard Chief Architect Essence Technology

Brian Blanchard

Chief Architect

Essence Technology

WM_TOUCH WM_TOUCH* family of messages Best: WM_TOUCH enables touch-optimized experiences WM_TOUCHDOWN, WM_TOUCHMOVE, WM_TOUCHUP Semantically similar to mouse messages Conveys raw touch data to Win32 apps Scenario examples: Finger painting, custom gestures, feeding higher level controls, etc Capture Coalescing

WM_TOUCH* family of messages

Best: WM_TOUCH enables touch-optimized experiences

WM_TOUCHDOWN, WM_TOUCHMOVE, WM_TOUCHUP

Semantically similar to mouse messages

Conveys raw touch data to Win32 apps

Scenario examples:

Finger painting, custom gestures, feeding higher level controls, etc

Capture

Coalescing

WM_TOUCH BOOL WINAPI RegisterTouchWindow( __in HWND hwnd, __in ULONG ulFlags); BOOL WINAPI GetTouchInputInfo( __in HANDLE hTouchInput, // input event handle __in UINT cInputs, // number of elts in the array __out_ecount(cInputs) PTOUCHINPUT pInputs, // array of touch inputs __in int cbSize); // sizeof(TOUCHINPUT) typedef struct tagTOUCHINPUT { LONG x; LONG y; HANDLE hSource; DWORD dwID; DWORD dwFlags; DWORD dwMask; DWORD dwTime; ULONG_PTR dwExtraInfo; DWORD cxContact; DWORD cyContact; } TOUCHINPUT, *PTOUCHINPUT; typedef TOUCHINPUT const * PCTOUCHINPUT; BOOL WINAPI CloseTouchInputHandle( __in HANDLE hTouchInput); // input event handle

BOOL WINAPI RegisterTouchWindow(

__in HWND hwnd,

__in ULONG ulFlags);

BOOL WINAPI GetTouchInputInfo(

__in HANDLE hTouchInput, // input event handle

__in UINT cInputs, // number of elts in the array

__out_ecount(cInputs) PTOUCHINPUT pInputs, // array of touch inputs

__in int cbSize); // sizeof(TOUCHINPUT)

typedef struct tagTOUCHINPUT {

LONG x;

LONG y;

HANDLE hSource;

DWORD dwID;

DWORD dwFlags;

DWORD dwMask;

DWORD dwTime;

ULONG_PTR dwExtraInfo;

DWORD cxContact;

DWORD cyContact;

} TOUCHINPUT, *PTOUCHINPUT;

typedef TOUCHINPUT const * PCTOUCHINPUT;

BOOL WINAPI CloseTouchInputHandle(

__in HANDLE hTouchInput); // input event handle

WM_TOUCH Future Windows 7 Build WM_TOUCH consolidated to a single message

WM_TOUCH consolidated to a single message

Manipulations Best: Manipulations are a great foundation for touch-optimized experiences Examples of higher level APIs/controls Manipulations are: 2D affine transformations (translate, scale, rotate) Superset of supported gestures Supports multiple concurrent manipulations Need a source of raw data: WM_TOUCH or RTS Similar to Surface APIs Interfaces: IManipulationProcessor IManipulationEvents

Best: Manipulations are a great foundation for touch-optimized experiences

Examples of higher level APIs/controls

Manipulations are:

2D affine transformations (translate, scale, rotate)

Superset of supported gestures

Supports multiple concurrent manipulations

Need a source of raw data: WM_TOUCH or RTS

Similar to Surface APIs

Interfaces:

IManipulationProcessor

IManipulationEvents

Inertia Provides basic physics Works hand in hand with Manipulations Interfaces: IInertiaProcessor IManipulationEvents – Same event interface as Manipulations

Provides basic physics

Works hand in hand with Manipulations

Interfaces:

IInertiaProcessor

IManipulationEvents – Same event interface as Manipulations

Multi-touch In .Net Framework For WPF: 3.5 update near Windows 7 launch Multi-touch, gesture APIs 4.0 release Multi-touch specific new controls For WinForms (Windows 7 launch): Interop to native Win32 APIs – see sample Use managed Manipulations/Inertia wrappers

For WPF:

3.5 update near Windows 7 launch

Multi-touch, gesture APIs

4.0 release

Multi-touch specific new controls

For WinForms (Windows 7 launch):

Interop to native Win32 APIs – see sample

Use managed Manipulations/Inertia wrappers

Multi-Touch In WPF UIElement and UIElement3D changes Gesture events (tracking) Touch system gesture events (single) Raw touch events Touch and touch digitizer devices Multi-touch support in controls: ScrollViewer update to accept pan gestures Base controls updated to be multi-touch aware Multi-capture support New multi-touch specific controls Compatible with Surface SDK 2.0

UIElement and UIElement3D changes

Gesture events (tracking)

Touch system gesture events (single)

Raw touch events

Touch and touch digitizer devices

Multi-touch support in controls:

ScrollViewer update to accept pan gestures

Base controls updated to be multi-touch aware

Multi-capture support

New multi-touch specific controls

Compatible with Surface SDK 2.0

Gestures In WPF Gesture events on UIElement & UIElement3D TouchGestureStarted, TouchGestureChanged, TouchGestureEnded NOT routed, to enable multi-capture scenarios Gesture in WPF based on Manipulations Enable Inertia at TouchGestureStarted Inertial movements as TouchGestureChanged Events Touch system gestures

Gesture events on UIElement & UIElement3D

TouchGestureStarted, TouchGestureChanged, TouchGestureEnded

NOT routed, to enable multi-capture scenarios

Gesture in WPF based on Manipulations

Enable Inertia at TouchGestureStarted

Inertial movements as TouchGestureChanged Events

Touch system gestures

Touch Events In WPF Touch Events on UIElement and UIElement3D TouchDown, TouchMove, TouchUp TouchEnter, TouchLeave Preview variants Promotion: Touch -> Stylus -> Mouse Multi-Capture CaptureTouchDevice(TouchDevice) ReleaseTouchDeviceCapture(TouchDevice) ReleaseAllTouchDeviceCaptures() TouchDeviceCollection TouchDevicesCaptured { get; } TouchDeviceCollection TouchDevicesCapturedWithin { get; } IsAnyTouchDeviceCaptured(); IsAnyTouchDeviceCapturedWithin();

Touch Events on UIElement and UIElement3D

TouchDown, TouchMove, TouchUp

TouchEnter, TouchLeave

Preview variants

Promotion: Touch -> Stylus -> Mouse

Multi-Capture

CaptureTouchDevice(TouchDevice)

ReleaseTouchDeviceCapture(TouchDevice)

ReleaseAllTouchDeviceCaptures()

TouchDeviceCollection TouchDevicesCaptured { get; }

TouchDeviceCollection TouchDevicesCapturedWithin { get; }

IsAnyTouchDeviceCaptured();

IsAnyTouchDeviceCapturedWithin();

Tablet Platform Extensions Real-time Stylus (RTS) RTS is an excellent raw touch data provider IRealTimeStylus3::MultiTouchEnabled() Getting multi-touch data via InkCollector is a little trickier. E.g.: ATOM m_atom = ::GlobalAddAtom(MICROSOFT_TABLETPENSERVICE_PROPERTY); m_dwProperty = TABLET_ENABLE_MULTITOUCHDATA ; ::SetProp(m_hwnd, (LPTSTR)m_atomPenService, (HANDLE)m_dwProperty);

Real-time Stylus (RTS)

RTS is an excellent raw touch data provider

IRealTimeStylus3::MultiTouchEnabled()

Getting multi-touch data via InkCollector is a little trickier. E.g.:

UX Considerations

ISVs are enthusiastic about multi-touch! We are working with several ISVs today on multi-touch applications, including: AutoDesk Avoco Secure Corel Cyberlink Fuel Industries Identity Mine Sonic

ISVs are enthusiastic about multi-touch!

We are working with several ISVs today on multi-touch applications, including:

AutoDesk

Avoco Secure

Corel

Cyberlink

Fuel Industries

Identity Mine

Sonic

Call To Action Decide which application scenarios make sense for touch Decide which tier of investment is appropriate Build something amazing!

Decide which application scenarios make sense for touch

Decide which tier of investment is appropriate

Build something amazing!

Getting Started What do you need to start coding? Hardware: Multi-touch beta drivers for in-market devices: HP TouchSmart 2 Dell Latitude XT Software: Window 7 M3 build – On the drive Touch Platform SDK and samples – On the PDC drive The Multi-point touch drivers are not yet a part of any OS. Your choice in hardware will impact the software needed. WPF Multi-touch .NET 4.0 Preview Releases in 2009 .NET 3.5 update to be announced

What do you need to start coding?

Hardware:

Multi-touch beta drivers for in-market devices:

HP TouchSmart 2 Dell Latitude XT

Software:

Window 7 M3 build – On the drive

Touch Platform SDK and samples – On the PDC drive

The Multi-point touch drivers are not yet a part of any OS. Your choice in hardware will impact the software needed.

WPF Multi-touch

.NET 4.0 Preview Releases in 2009

.NET 3.5 update to be announced

 

LucidTouch: http://research.microsoft.com/users/baudisch/projects/lucidtouch/index.html Obscura Digital: HP - D5: Multi-Touch wall http://channel9.msdn.com/forums/Coffeehouse/255503-Surface-Thats-not-a-multi-touch-screen-/ VisionAire: Holographic Multi-Touch http://makeitv.blogspot.com/2008/08/minority-report-holo-display-is-now.html

LucidTouch:

http://research.microsoft.com/users/baudisch/projects/lucidtouch/index.html

Obscura Digital:

HP - D5: Multi-Touch wall

http://channel9.msdn.com/forums/Coffeehouse/255503-Surface-Thats-not-a-multi-touch-screen-/

VisionAire: Holographic Multi-Touch

http://makeitv.blogspot.com/2008/08/minority-report-holo-display-is-now.html

 

Add a comment

Related presentations

Related pages

Windows 7: Developing Multi-touch Applications | pdc2008 ...

In Windows 7, innovative touch and gesture support will enable more direct and natural interaction in your applications. This session highlights ...
Read more

PC03 – Windows 7- Developing Multi-touch Applications

Multi-touch Is Here! Hardware. Multi-touch capable machines in market today, multi-touch appears in a broad set of form factors. Software. Windows 7, PDC ...
Read more

Developing for Multi-Touch Web Browsers - HTML5 Rocks

This is confusing in many multi-touch applications, ... a multi-touch application that involves complex multi-finger ... are developing for ...
Read more

Developing Multi-touch Software through Creative Destruction

Developing Multi-touch Software through Creative Destruction Ingmar S. Franke, Dietrich Kammer, Rainer Groh Technische Universität Dresden Fakultät ...
Read more

Developing a Multi-Touch Application for Medical Volume ...

Developing a Multi-Touch Application for Medical Volume Visualization Bachelorarbeit zur Erlangung des Grades eines Bachelor of Science (B.Sc.)
Read more

Windows 7: Developing Multi-touch Applications ...

of the PDC 2008 talk "Windows 7: Developing Multi-touch Applications" by Reed Townsend and Anson Tsao. At 81 minutes the video is probably a bit long for ...
Read more

MultiTouch SDK | Touch Software Development Kit & Tools

MultiTOUCH Software Development Kit (SDK) and Tools to develop professional Touch Screen Software for large interactive Digital Signage Displays.
Read more

Multi-Touch SDK » PQ Labs « Natural User Interface

There are 3 ways to write Multi-Touch applications in Windows: 1. Using PQ Labs Multi-Touch SDK & Gesture Library
Read more

MultiTOUCH Software | Digital Signage Touch Screen Apps

Interactive Multitouch Software for Digital Signage touchscreens. ... IR touch frames, ... multiple users control multimedia applications simultaneously ...
Read more