Social Interaction Innovation for Autism

2020-2021

Client

Arrows - Assistive Technology

Skills

HCI Innovation | UX Research | Branding

The Challenge

Empower Communication

High Functioning Autistic (HFA) individuals commonly suffer from impaired communications abilities causing social isolation, hence, lack of confidence and independency.
As a father of an HFA teenager, the difficulties are part of my daily routine. With the introduction to Human Technologies Interaction (HCI) and the exposure to the potential of technology to support and create solutions, I’ve initiated the Arrows project, harnessing augmented reality implemented on smart glasses to develop a tool aiming to support communication and in real time when in need.

I was exposed to numerous experiments and researches with various attempts (including touch screens and robots) towards the target of improving the wellbeing of these individuals but hardly any assisting in real time in real life situations.

Next, I've consulted with relevant expert stakeholders (Psychologists, Speech therapists and Augmented Reality experts) to articulate the design and the hypothesis that mediating social cues through AR will expand the conversation cycle and improve the social skills of the autistic individual.

Research

Consulting Sessions

Create & Design

Mapping the difficulties I came up with the concept of a virtual integrator to support conversations in real time.The original prototype included several overlaid animated icons (shaped as arrows) designed in Unity and implemented on Microsoft's Hololens 2 smart glasses to serve as basic cues for the interaction in a controlled environment seeking POC.

UX Aspects in AR

The challenge

Augmented Reality is yet to set rules or wireframe restrictions.

Some of the aspects dealt were with positional tracking (SLAM - Simultaneous Localization and Mapping, LIDAR - light detection and ranging) and mostly trial and error

Optimization
Testing augmentation in space - an objective to avoid distraction.
Shape of Icons - common arrows to enhance comprehension and maintain cross cultural engagement.

Size & locations
Icons were tested in different colors and locations that will stand out yet not interfere with the interaction. (Shown in figure below).

Photo by Sergio De Paula on Unsplash

Test were done with real HFA aiming to answer their needs and was overlooked by caretakers and psychologists adding more data to the equation including the issues of eye contact.
After presenting the options, the top left was chosen as it did not distract the user and maintained eye contact as well as realizing the cue and using it.

Currently, the projection of the icons is monitored and controlled by the operator of the system and the next phase is to create a remote controller app used by a caretaker that could either be implemented on a smartphone or a tablet or rather on the glasses itself being gesture sensitive. In parallel a development of the artificial intelligence of the system to perform the projection automatically being speech and expression sensitive.

User Testing

The 1st iterations of user testing was conducted with 16 teenagers in a controlled environment. The system was tested in an open conversation managed by an experienced therapist.
The evaluation was accomplished through qualitative analysis with a post experience questionnaire and an overview of conversations videos that led to system modifications.

Schema of the experiment design

Branding

In order to package the system, I've created a branding layer for the company. My vision for the logo was to combine the initials of the augmented reality technology within the name of the innovation as well as harnessing the shape of the arrows as part of the logo serving as the letter R.

Arrows Logo

www.ar-rows.com