Published on February 23, 2014
Indian Institute of Technology, Guwahati Arushi Singh |MDes Kunal Drego|MDes
Problem Statement Designing Multimodal interactions in Cockpit Design Brief Think of pilots of the future, who control business jets and take their guests to destinations on time. Keyboards/Knobs and buttons are being replaced by touch screen and eventually will be replaced by voice and gesture. This requires a deeper understanding on how these multi modal interfaces seamlessly fit into the existing and future ecosystems with the target to reduce training cost, improve productivity and efficiency of the operator/pilot. We want you to design the future of aircraft control using multi modal interactions.
Flow Diagram Blue Sky Research Understanding the existing Systems Ground Reality Identifying the Problem Competitive Analysis Evaluation of Multimodal systems Solutions Comprehension of design parameters Design Considerations IDEATION Mind Mapping User Interviews Affinity Analysis Design Targets Potential Solutions Final Design Prototyping
Primary Flight Functions Aviate Communicate Navigate
Research 1. Studying the Avionics of Business Jet (specific study of Lear Jet 45) 2. Analyzing Existing Display Systems 3. Scope of multimodal systems in aviation 4. Cockpit Interactions 5. Competitive Analysis 6. User Interviews 7. Affinity Analysis 8. Design Considerations
Avionics of Business Jet (Lear Jet 45) LJ 45 Cockpit Layout
Avionic Systems Standby Instrument Group Primary Flight Display Radio Management Unit Multifunction Display Flight Management System Jeppesen Charts Landing Gear Yoke
Cockpit Interactions Pilot-ATC interaction Submit Flight Plan to ATC(online procedure these days) Clearance from Military Liaison Unit Permission For using Aerobridge CABIN CREW ATC PILOT CO-PILOT Request startup Clearance and departure sequence ATC gives the Flight Level and departure time Taxi Clearance Approach Clearance Typical Take-off procedure
Pilot- Co Pilot interaction CABIN CREW ATC Duties of the co-pilot PILOT Flight planning: mapping the flying route Monitoring Flight Instruments: Flying is a very demanding job as a large number of system need to be monitored. The co pilot assists the pilot is doing so at all times. Radio Communications: The co-pilot maintains the communication between the ATC and the aircraft during flying. Take Off to Arrival: The pilot must be aware of all systems in case he is given the position of pilot at any time during the flight. He will know the take off and landing procedures in case if he is in charge. CO-PILOT
Pilot-Cabin Crew interaction CABIN CREW ATC Improving cockpit and cabin crew communication Clarification of sterile cockpit rule: In order to reduce the misinterpretation of sterile cockpit rule to crews the training should reemphasize the importance of effective communication for the flight safety, rather than choke legitimate communication. PILOT CO-PILOT Requirement of joint Cockpit and Cabin training is recommended for improved communication and coordination. Providing aircraft technical training for flight attendants: The technical training can provide the flight attendant the opportunities to learn the basic knowledge of airplane terminology, and give them more confidence of communicating safety critical information to the cockpit, such as being able to describe mechanical parts or malfunctions of the aircraft and reporting the information in accurate and timely manner. Ref: http://aviationknowledge.wikidot.com/aviation:cockpit-cabin-communication:the-causes-of-the-ineff
Existing Display Systems OLED screens Paper thin display Wearable Technologies Projected Display DISPLAY Flexible screens 3D display Globe projection system (projection on solid surfaces) Holographic displays
System Modalities Speech Auditory Gestural Touch Haptic Visual Neural
Competitive Analysis THALES Odicis (One DIsplay for a Cockpit Interactive Solution) is a touchscreenbased display system concept enabling information tobe presented in new ways in all types of aircraft. Adaptable to both civil and military aircraft, as well as to helicopters, Odicis can be configured to provide more freedom in how the display space and touchscreen surface are used. BARCO The concept revolves around one composite, flush glass display, and two stand-alone control units with a multi-touch screen, to replace the multitude of displays present in cockpits today. In this way, Barco seeks to reduce the complexity pilots are currently facing, thus improving aviation safety. GARMIN First touchscreen glass integrated avionics system designed for light turbine aircraft. It uses a variety of 14.1 inch integrated cockpit displays for ease of viewing and operation and 5.7 inch touchscreen controllers for intuitive control. The G3000 is capable of running Garmin's Synthetic Vision Technology, a graphical 3D rendering of terrain.
User interviews User 1 Group Captain (Retd.) Charlie J Weir, Chief Flying Instructor, Gujarat Flying Club, Vadodara User 2 Captain Jeffery Lynn Moore, Pilot, Learjet 45 User 3 Captain Mankaran Singh Co-Pilot, Learjet 45
Affinity Analysis #User 1 Group Captain (Retd.) Charlie J Weir, CFI, Gujarat Flying Club, Vadodara User Statement Problem points Solution Opportunity Safety must not be compromised With new technology comes new challenges. Redundancy in systems can solve safety issues There is probability of false inputs in touchscreens Touch screens are so sensitive that there is a possibility of false inputs The systems must be so robust so as to prevent any erroneous inputs. It is difficult to switch from pilot to co-pilot avionics as it is required of me to fly and also instruct trainee pilots The systems are not ambidextrous. The central console and its systems like the FMS, Nav/Com displays must be such that they can be accessed by both hands easily Since the advent of fly by wire systems and sophisticated systems which assist flying(eg. The ILS CAT-IIIC allows absolute 00 approach) pilots become dependent too much on technology at times Technology creates a bias and complacency among pilots, as they tend to trust the systems too much. Pilots must be made as alert at all times during training. To get into cockpits you need to do some acrobatics Due to massive space crunch there is Revolving, sliding or foldable seats very little space to get inside the cockpit. can be designed.
#User 1 Group Captain (Retd.) Charlie J Weir, CFI, Gujarat Flying Club, Vadodara USER STATEMENT PROBLEM POINTS SOLUTION OPPORTUNITY It is imperative to have redundancy in the system to give backup in case of failure Electronic failures can lead to a black out of screens Sufficient multifunctional displays can solve the issue. HUDs and Wearable helmets are convenient when flying by visual reference Visual reference is required only while landing and taking off. Instrument flying is used otherwise Weather radars can be controlled using gestures like in military jets Terrains must be visible on the weather display Not all systems are used at all times There is difficulty to view terrains in case to low visibility. We can explore projected displays as they are convenient to use Systems can be brought into forefront as and when required Gestural interfaces can be used The weather radar display can give real time satellite images of the weather, terrain, nearest airport etc.
#User 2 Captain Jeffery Lynn Moore, Pilot, Learjet 45 USER STATEMENT PROBLEM POINT SOLUTION OPPORTUNITY Have an all-in-one screen Too much information overwhelm the user Systems can be called on screens as and when required In touch screen systems the user has to lock the information, so in case of an emergency unlocking can be difficult Pressure situations can change the user behavior In case of emergency the system should become intuitive and guide the user through emergency checklists Landing and Taking off must be independent of visibility Difficulty in runway visibility in case of bad weather. The aircraft is dependent on ground aids like the ILS to aid landing and take off. The aircraft can be made independent of the ground Smart systems will be in place in future There is no conversion mechanisms(for various parameters) in the international flights. If planes fly across international borders, conversion from metric system and imperial systems can be tedious With the intensive use of AI soon pilots will be replaced by specialized bots There must be a conversion add on in all avionics systems
#User 3 Captain Mankaran Singh , Co-Pilot, Learjet 45 USER STATEMENT PROBLEM POINT SOLUTION OPPORTUNITY There is a massive space crunch in the cockpit Due to a large number of systems, there is limitation of space in the cockpit The systems can be so arranged they reduce clutter. Gesture and touch screens can be used to reduce the space occupied by the systems. It looks like a jungle. Due to redundant systems and soft buttons there are space issues. The layout can be simplified by having all touch interface and removing soft buttons. It is very difficult to get in the cockpit The layout of the cockpit is such that it doesn’t have ample space for the pilot to get in. The seats can be made rotating, sliding or folding. It would be helpful if we could have smart checklists The pre flying and emergency checklists can Already been implemented in the be called out and analyzed by the system modern systems. The systems calls out the checklists It will help if the traditional flight manual is replaced The traditional manual is too bulky and it is A dedicated ,easy to use system can difficult to locate specific procedures in be of assistance to the pilot case of emergencies( in spite of the colour coding followed ) Backlit systems are helpful during night flying Too many screens can increase the intensity Projected displays cause less fatigue of the cockpit ambiance lights. on the eyes.
Based on the interviews we concluded the following: De-Cluttering Task Management Comfort Safety Affinity Analysis
Design Considerations Anthropometry Comfort Safety Psychological Cognitive Human behavioral
Ergonomic Consideration Operating areas of the stationary human body Ref: Ergonomic design of aircraft cockpit-Ionu CÎMPIAN Arm reach of average Pilot There is a definite access area around the pilot. This area is divided into static and dynamic region. There is an invisible shell which is easily reachable by the pilot. The idea behind this concept was to utilize that space so as to give maximum comfort to pilot.
Mind Mapping for identifying potential solutions Multi modal Systems Replacing yoke De Cluttering Easy Access Space Management Cockpit Layout Task Management
Potential Solutions Replacing the Yoke • 3D TrackBall • Screen cum yoke • Twist handle Space Management • Omnipresent Display • Foldable Screen • Fall Down Screen Easy Access Multimodal Systems • Moving the seat: rotate, slide, fold • Adjustable Seat • Gesture • Voice • Haptics De Cluttering Task Management • Replacing the central console • Making foldable screens to create space • Display system in use only • Secondary information :displayed in the peripheral vision area
Replacing the yoke 3D Trackball: The idea here is to have multi-axis analog control while retaining instinctive hand movement. The design is based on the 3D mice used for modeling that use its multi axis movement for manipulating objects in 3D space. The Twist handle: The intention is to have a pair of mirrored controllers on either side of the pilot giving the option to choose the control layout up to an extent. Concept #1
Screen cum yoke: This is another space saving system which converts into a yoke during take off and landings. The yoke handle close in otherwise to form a multi purpose display screen. The entire set up can fit snugly inside the central display and can be pulled out easily when it is to be used. Concept #1 Replacing the yoke
Space Management Omni Present Display The user sits in the center with display screens placed conveniently across the statically accessible areas Fall down Screens Screen falls from the body of the aircraft, goes back when not in use. Foldable Screens The Display cum input unit comes out from the armrest of the pilot’s chair and can be folded up when not in use Concept #2
Easy Access: Moving the Seat Three possible movements for the seat can be envisaged, namely rotate, slide and fold. A folding seat would have issues since the direction of access and the final orientation present no feasible solutions. Combinations of sliding and rotating seats can be considered considering the space availability and arrangement of equipment. A hybrid rotating and sliding mechanism is an implementable option. Considering the conventional layout of the cockpit, the pilots’ seats are sandwiched between the central console and a side panel/wall leaving them with very limited or no/uncomfortable entry to the seats. Two possible solutions for increasing accessibility would be giving a provision to move the seat or to modify the central console. Concept #3
Gestural Interfaces The space inside the cockpit can be divided into the static region(accessible while sitting) and dynamic region. By incorporating gestural interfaces, not only do we make a lot more cockpit area usable but also intuitive to use. With gesture recognition slowly creeping into the mainstream (eg kinect, nintendo etc) It will become an obvious choice in the future. It also it keep the pilot alert as he has to use gestures which will require simple arm movements. These systems can be combined with other modalities like speech, visual etc. Concept #4
Flying Bubble Considerations/drivers for the Flying Bubble • Movable and adjustable seat for ease of access • Flexible, collapsible and adaptable screens • Replace the yolk with a twist handle • Removal of clutter by using an omnipresent screen • Touch and gestural interface for efficient space usage and intuitive interface • Emphasis on pilot comfort and convenience • Personalized space • Safety Concept #5
Dedicated Troubleshooting systems The traditional flight manual is replaced by a smart system that runs self diagnosis. Alerts the crew using voice, visual or haptic outputs and gives a step-by step emergency averting solution. Concept #5 Flying Bubble
Pilot’s personal Bubble The control bubble: interface • The dome shape is built around the dynamic range of hand control making it both comfortable and intuitive • The screen layout is personalized and dynamically adjusts according to importance and situations • The control actuates on pilot authentication and provides personal space to the pilot • The interface is a hybrid of space gesturing and touch based Final Concept
Cruise mode • The bubble disappears during cruise mode • Only the essential monitored parameters are displayed in the peripheral vision • The screen becomes partially opaque to reduce the incoming excess sunlight Night mode • Simulation of the bubble in night mode Concept #5 Flying Bubble
Scenarios System give out warning. Pilot uses the print manual. It takes too much time to find the desired remedial action Pilot wants to take a washroom break. He gets all the critical information inside so that nothing fatal happens.
Pilot feels sleepy. Decides to take a nap. He tells the copilot to take charge. Co pilot too feels sleepy and falls asleep. Auto pilot is unmonitored.
F L E X I B I L I T Y
Meet “CADDY” Color morphing lights Flexible electronic paper Speech Synthesis Wearable Display Cylindrical revolving Display(viewable without distortion) Flexible Display
Functions • Flight Plans • CAS/System warnings • Flight Charts • Autopilot engage/disengage • Flight Manual • Computation • Conversion • Fatigue Alarms • Give airport information • Cockpit Surveillance • Daily logs (maintenance etc.) • Syncing with co pilots caddy
Flexible Electronic paper • Electrically-charged surface • Electrophoresis • Electronic Ink • Charged black and white particles • Encapsulated "ink“ • Sandwiched between laminated plastic films • Pixels controlled by a display driver.
System Modalities System is touch and voice activated Gestural Interface for intuitive viewing Visual stimuli in case in case of any emergency(CAS warnings/if pilot is sleepy etc. Haptic feedbacks in case of emergencies/if pilot is feeling sleepy Senses biological changes like heart beat Touch enabled input and display
Speech Synthesis Today’s pilot has become more of a “systems operator” and frequently spends more time manipulating his flight management system than he does actually manipulating the aircraft controls and looking out the window. Ref: The Benefits of a Speech Recognition Enabled Cockpit-Adacel Systems Inc
Advantages and Use Electronic Flight Bag (EFB) Interaction Voice is most intuitive Under stressful flight conditions : (e.g., abnormal or emergency flight situations, marginal VFR) Visual channel is maxed out; Manual channel is moderately to heavily loaded; Auditory channel is relatively lightly loaded. Correlation of Unfamiliar Local Data Level and/or Heading Bust Monitoring Direct Aircraft Systems Queries Glass Cockpit Configuration Data Entry for FMS, Autopilot, Radio Frequencies Ref: The Benefits of a Speech Recognition Enabled Cockpit-Adacel Systems Inc
High noise levels Changes in the speaker’s voice due to illnesses, air pressure, vibration Operator accents Inability to differentiate between words or phrases Difference between the printed word and the ways in which the word(route=root or rout) Train the system to recognize each operator’s voice patterns Memorize and speak a limited set of commands Need to speak slowly Limited command sets Ref: The Benefits of a Speech Recognition Enabled Cockpit-Adacel Systems Inc
Task Flow 1 START “ACTIVATE CADDY(or alias name)” “Display the chart for NDB to ILS runway 27 Left at Orlando International” “Good Morning Captain” “Chart found” Chart Displayed “DISENGAGE CADDY(or alias name)” STOP
Task Flow 2 START Emergency Captain: Left engine on fire Tell me remedial action Switch off engine power to cutoff fuel Check Turn on Fire Extinguisher Check DISENGAGE “CADDY” STOP
Usage : For Charts • • • • Refer Charts while visual flying Take notes on charts Mark VFR maps Calculations while flying
Portability • • • • Easy to carry Light weight Foldable Rollable
Wear-ability • • • • Gives essential information at all times Biological sensing On the go functions Personalized touches
View-ability • Easily viewed while comfortably sitting • Rotates to show all relevant information at all time • Rotation can be controlled via gestures • No distortion takes place • Predictive display
Prototyping: Display Screens Scale Down Mockup
3D Renders HUD Primary Display FMS Landing Gear Caddy Yoke Utility Tray
Integrated System warnings: Active Mode
Integrated System warnings: Caution Mode
Integrated System warnings: Warning Mode
References www.aerospace.honeywell.com Accessed on 15th Dec’2013 www.aviationknowledge.wikidot.com Accessed on 17th Dec’2013 www.aviationtoday.com Accessed on 17th Dec’2013 th www.ainonline.com Accessed on 19 Dec’2013 Human Centered Design Approach to Integrate Touch Screen in Future Aircraft Cockpits-Jérôme Barbé, Marion Wolff, and Régis Mollard Ergonomic design of aircraft cockpit - Ionu CÎMPIAN Accessed on 19th Dec’2013 Accessed on 25th Dec’2013 Projection Technology for Future Airplane CockpitsDieter Cuypers, Herbert De Smet, Xavier Hugel,Guilhem Dubroca Accessed on 4th Jan’2014
References The Benefits of a Speech Recognition Enabled Cockpit-Adacel Systems Inc Accessed on 8h Feb’2014 Cylindrical 3-D Video Display Observable from All Directions-Tomohiro Endo Accessed on 10h Feb’2014 Voice-Activated Cockpit for General Aviation-Robert B. Wesson, Ph.D. (The Wesson Group) & Gary M. Pearson (Adacel Systems Inc.) Accessed on 10h Feb’2014 Audience Behavior around Large Interactive Cylindrical Screens-Gilbert Beyer, Florian Alt Flight Safety Digest-Paperless Cockpits Vol. 24 No. 6 June 2005 Accessed on 12h Feb’2014 Accessed on 15h Feb’2014 Integrating voice recognition and automatic target cueing to improve aircrewsystem collaboration for air-to-ground attack-Mr. Greg Barbato, Air force research laboratory’ Accessed on 17h Feb’2014
AeroChallenge 2013. UI/UX; 460 17 0 Multiple Owners. Parag Gupta. Bangalore, India ... Awarded First Prize at the Honeywell AeroChallenge 2014.
Honeywell Aerospace innovates and integrates thousands of products and services to advance and deliver safe, efficient, productive and comfortable experiences.
Hello, I’m Parag. ... Honeywell AeroChallenge 2014: ... 2014. A colloquium paper on design for Behavioral Changes in Societies.
View Parishrut Bhatia’s professional profile on LinkedIn. ... Honeywell Aerochallenge 2014 Honeywell Technology Solutions, Aerospace, Bangalore.
2014. The winners: Team Hills Road, from Hills Road Sixth Form College, Cambridge with Chief of the Air Staff Sir Andrew Pulford, ...
View Kunal Drego’s professional profile on LinkedIn. LinkedIn is the world's largest business network, ... Second Runners Up Honeywell Aerochallenge 2014
Issuu http://issuu.com/ching_c Work Experience Design Intern