Research Article

The Use of Virtual Reality and Eye-Tracking to Inform A Grocery Store Intervention

Hana S Alarifi^1,2*, Laura Barcelos Nomicos², Bshayer Jan^1,2, Mashael Alzamil^1,2, Meaad Daleel^1,2, Reem Alabdulaziz^1,2, Hesham Aldhalaan¹ and Linda J Hayes²

¹Center For Autism Research, King Faisal Specialists Hospital and Research Center-Riyadh, Saudi Arabia

²University of Nevada, USA

Submission: March 09, 2022; Published: March 16, 2022

*Corresponding author: Hana S Alarifi, Center for Autism Research, King Faisal Specialists Hospital and Research Center, Saudi Arabia

How to cite this article: Lu Zhao, Jialing Li. The Effect of User-Generated Image Pixel Quality on User Experience: A Scenario-Based Experiment in Social Media. Psychol Behav Sci Int J .2022; 17(5): 555971 DOI: 10.19080/PBSIJ.2022.17.555971.

Abstract

Virtual Reality (VR) is commonly used as a cost efficient and accessible tool for providing therapeutic services as an effective assistive technological gadget. VR has been used with individuals on the autism spectrum disorder (ASD) to overcome difficulties and develop new skills by providing virtual environments coupled with the ability to control and customize variables within the environment. This study aimed to examine the effectiveness of virtual environments and emerging technology devices of VR and Eye-tracking to teach children with ASD how to navigate through grocery store aisles and find specific items. A within subject design with a multiple baseline across four participants was used to collect data in natural and virtual environments. Wearable eye-tracking glasses and eye-tracking equipped VR head-mounted Display HMD was used for a validation of the eye gaze response. Results showed that all participants achieved acquisition and generalization to a criterion of 100%.

Keywords: Applied Behavior Analysis; ABA; Virtual Reality; VR; Intervention; Eye-tracking

Abbreviations: ABA: Applied Behavior Analysis; VRT: Virtual Reality Intervention, CFAR: Center for Autism Research; ADHD: attention deficit hyperactivity disorder

Introduction

Virtual Reality Technologies (VRT) allow for a person to experience the visual and auditory sensations of an environment from the comfort and safety of a clinic or their own home. Due to decreasing costs, VRT have become a rapidly growing paradigm as an assistive technology and interventional method for individuals with autism and other disabilities. VRT is an extended-term (?) with a number of underlying technologies, such as virtual environment simulations [1]. VR is a form of head-mounted display [2] that display programmed software that illustrate virtual environments for collaborative virtual interaction and immersive virtual environments [3]. The components of which include 3D graphics, avatars, autonomous controls, and 2-way input/output ports. VR can provide a realistic simulation of the real world that can be used for various therapeutic purposes.

Virtual Reality can be used by therapists and instructors to provide a safe environment for treatment of a variety of different disabilities and disorders, such as attention deficit hyperactivity disorder (ADHD), depression, bipolar disorder and ASD [4]. By incorporating VR into therapy with children on the autism spectrum, several of the barriers of teaching skills in the natural environment can be overcome. Behavioral treatment requires repeated practice of skills. When in-situ training is required for generalization, this means that the clinician, client, and caregiver must be available to go to community locations to teach skills on numerous occasions. Scheduling these types of outings can be difficult especially in single parent households or households where both parents work fulltime. There are also safety concerns with these types of outings. In these uncontrolled environments the client is exposed to variety of potential risks, particularly when learning safety skills such as crossing the street. There are also logistical issues with in-situ training. Frequently the clinician does not have access to the environment where the behavior needs to be taught/practiced. With some health insurances, for example, the clinician is barred from using insurance-funded treatment time in the school environment. VR may overcome these barriers by allowing the client to practice in a virtual environment nearly identical to the natural environment. The utility of this technology is assessed in how well skills learned in the artificial environment generalize to the natural environment. While this research is limited, there offers promise for future investigations and applications.

Several studies have demonstrated the utility of VR in autism treatment. Skills such as crossing the street, initiating a conversation, finding a seat, walking in public, ordering food, and responding to catastrophic situations have been taught using VR [5] (VR systems offer flexibility and control to the service providers and therapists to perform repetitive exercises that directly represent real-world controlled simulations to motivate, engage, and reinforce the learner with certain social settings. This is duplicative – said above) In the past, some of the concerns with VRT have been motion sickness and uncomfortable headsets.

Newbutt et al. [6] published a brief report that conveys the improvements of VRT in these areas and reduced costs associated with the equipment. The practicality of Newer VRT devices was tested with individuals with ASD by this author. Participants with IQ scores below 70 were recruited. A self-report questionnaire was used to measure the acceptance and readiness of a group of participants to the technology. Outcomes confirmed that the majority of participants reported a pleasant experience with willingness to repeat.

Malihi et al. [7], assessed and compared the usefulness and prosperity (prosperity?) using two different types of technologies for children with ASD. The first method was the Head-Mounted-Displays HMD, and the second method was the 360-degree view of conventional videos displayed on normal screens. The research involved a school bus simulation recording shown in 360-degrees via regular screen and compared with a virtual reality environment simulation via HMD of a school bus experience. The efficiency of both methods was collected by self-report questionnaire and follow up calls. The study confirmed immersion and reduced anxiety levels by using VRT and that it does perform better compared to regular screens due to the immersion provided. HMD immersion produced higher engagement levels and more positive outcomes.

Adjorlu et al. [8] provided an example of HMD with VR technology to provide an intervention for children with ASD. The research group focused on studying the shopping skills of school-age participants. Participants were tested in the natural environment prior to any VR interventions as well as after VR interventions. The VR training was held in a local school. Participants were trained on four items to be found in a grocery store with VR first, to be examined in real grocery store later. The success of the training was based on the speed and accuracy of the participants’ shopping. Results were unclear to interpret due to data collection methods where it was subjective to different grocery store conditions in each trip and the lack of measurements for each individual’s searching behavior. The lack of objective measurements caused highly variant results to prove the hypothesis with stable that results are not consistent to be reliably used. There is still much to be explored in terms of the therapeutic applications of VRT. Very few studies to date combine applied behavior analysis and VRT to teach new skills to individuals with ASD.

Stein et al. [9] is one of a few studies in which a high-end HMD VR was introduced by VARJO. The model of VR-3 released in the third quarter of 2020 it is announced as the most immersive and clear VR headset that was accredited to be used by the European Aviation Safety Agency (2403 2424 2487-EASA 0010011661) to increase safety training to their pilots indicating the quality and sensitivity that VARJO achieved among VR headsets [10].

The first aim of the current study was to examine the generalization of skills taught across the VR environment and the natural environment. The second aim was to assess the acceptability of VRT and wearable eye-tracking glasses for participants. The final was to evaluate how to best incorporate eye-tracking and VR with behaviorally oriented teaching methods.

Method

A written consent form was sent electronically and given in person to all participants’ parents explaining the purpose of the study, how the data will be collected and used for, the benefits and risks of the study, privacy, confidentiality, and devices used in the study. Also, it contained parents’ consent to the video recording of the all the sessions for data collection purposes. The parents were informed that their children's participation was voluntary, and they have the right to discontinue the participation at any point of the study.

Participants

The participants in this study were diagnosed with ASD. They were recruited through the Center for Autism Research (CFAR) patient waiting list. Participants were chosen based on a survey completed by parents. Participants with scanning skills already in their repertoire, participants with severe problem behaviors interfering with compliance, and deficits in expressive and/or receptive language were excluded from the study. Chosen participants caregivers were contacted in person, by phone, and by email prior to the introductory session to get the signed consent forms. The sample included both female and male participants between the ages of 10-17. Six participants were ultimately recruited for the study (Table 1).

Two of the recruited participants did not complete the study due to either schedule conflicts or physical disabilities that made engaging in the physical components of the study impossible. Participants were selected from different cities in Saudi Arabia with no consideration regarding a specific language, nationality, or gender. All participants were recruited from CFAR list who had previous diagnosis at the center.

Setting

Data collection took place in two different settings. Baseline and generalization probes were completed in grocery stores, while intervention sessions were conducted at the Center for Autism Research (CFAR) Riyadh, Saudi Arabia in the Human Behavior Lab (HBL). The HBL dimensions are 3.5 meters by 9 meters, which gave the participants some space to move around during the application of the VR intervention.

Materials

Advanced technologies of wearable devices were used. Pupil Invisible, clear eye-tracking glasses and VARJO VR-3 headset running a hyper-realistic simulation of a virtual supermarket were used. VRT device used as interventional tool is shown in (Figure 1).

Pupil Invisible wearable glasses. They are light weight clear glasses that collect eye-tracking data linked to the cloud database that work online and offline. The glasses where the participant was wearing them while they are equipped to record scenes recording and eye tracking data in synchronization. The information collected were corresponded with eye tracking data for each wearer. Pupil Invisible enabled therapist to monitor participants’ eye-gaze in real time using a linked Android companion device (cell phone) along with recording the entire session for re-use and inter-observer agreement verification.

The Pupil Invisible’s video recordings were used to collect interobserver agreement for all sessions. Observers collected the baseline and the generalization data in the natural environment. During the VR baseline and intervention, data were collected via VARJO-3. Participants’ performance in both VR and real-time were displayed to the therapists during the session to form decision making. A recording was obtained for later verification and inter-observer agreement. Based on the recordings, a calculation of the percentage of IOA agreement of responses for 100% of the trials for each participant.

Procedure

The participant was walked to the lab HBL accompanied with the ABA therapist to start a scheduled 30-minute session. Sessions started as planned based on the phase of the study. The study was conducted in stages (introductory session, baseline in the natural environment, VR baseline in the virtual environment, VR intervention, and generalization in the natural environment).

Pupil Invisible glasses were used only during the natural environment baseline and generalization probe session. Whereas VARJO VR-3 were used during the VR baseline and VR intervention phases. During all stages of the study, eye tracking data were collected for objective and informative benefits. A nonconcurrent multiple-baseline across participants design was utilized. Responses were manually collected by the therapists while VRT collected eye tracking data. Both data types collected by the technology gadgets and therapists were compared and corelated in this study. The advantages of eye-tracking data in both the glasses and VR headset were observed to determine how VRT can inform ABA therapists and increase the validity of the program outcome by verifying the result with the objectivity of eye-tracking measurement. Additionally, reducing the duration of intervention due to the informed and individualized method of decision making and continuous individualized modification of instructions. A simplified procedure flow chart including the task analysis steps are summarized in (Figure 2).

Introductory Session

The introductory session (IS) included building rapport with potential participants by introducing the therapist’s name and asking the participant about his/her name and age. Then the therapist engaged in small conversations with the participant and presented several toys. An interview of the participants was conducted to determine if the participants fit the criteria to participate which were the ability to follow specific and simple instructions such as: look right, look left, look up, look down, go forward, go backward, and bring item. In addition, confirming the ability of client to identify pictures of grocery items in an array of 3, the ability to complete the calibration process in four different directions. If the participants were unable to discriminate between potential items and distractors, therapists trained them using discrete trial teaching (DTT). After that, the therapist familiarized the participants with the VR headset by playing games. The physical requirements of the inclusion criteria were considering the tolerance of wearing the headset. A minimum weight of 40 Kilograms and Hight of 120 Centimeters were required in recruiting participants. Once it was confirmed participants met the criteria, they were scheduled for a baseline session.

Natural Environment Baseline

The natural environment baselined (NEB) was conducted in a local grocery store, where participants wore Pupil Invisible to record their eye gaze. When delivering the instruction to find a specific item, the client was standing at the end of the aisle. Specific items were chosen randomly for each client from the list of items confirmed to be in both the VR environment and the local grocery store. Data collected based on specific steps that were determined in the task analysis see chart 1. The video with eye gaze tracking was used to collect data and IOA for the session.

Virtual Reality Baseline

A single virtual reality baseline (VRB) session conducted to each participant to identify participants’ skills level in the virtual environment. Participant asked to wear the VR headset then therapist deliver the SD to find a specific item from the virtual grocery store aisle independently. Data from the VRB and NEB was used to identify which task analysis’ step(s) were not in the participant’s repertoire.

Virtual Reality Intervention

The virtual reality intervention (VRI) training sessions took place at Center for Autism Research (CFAR). Prompt levels determined based on a probe conducted for each participant. A prompt delay used during the intervention to provide opportunities for independent responses. The lead therapists (LT) delivered discriminative stimulus (SD) and provided prompt as needed. The Assistant therapists (AT) stood in front of the personal computer (PC) screen monitoring participant eye gaze to provide gesture prompts through the pointer as needed. The LT provided physical prompt as needed based on the AT’s feedback.

Before delivering the SD, the LT reminded the participants about the following instructions: look right, look left, look up and down. Praise was provided contingent upon correct responses. The intervention contained a 9-step task analysis used for data collection. If the participant independently performs the step, it was recorded as correct, while prompted responses were considered incorrect. Sessions lasted 30 minutes with a break given to participants halfway through.

Generalization Probes

Generalization probes (GP) took place at a local grocery store after the participants reached the mastery criteria in the VRI phase. The same procedure used in the NEB applied to the generalization probe, yet different grocery store was utilized.

Result

(Figure 3) shows, the percentage of correct responses in baseline, intervention, and generalization probe across four participants. During baseline in a natural environment, mean levels of correct responses are 54 %, 65 %, 70 % and 69 % for MM, AW, HM and QM respectively. During VRB, all participants had no correct responses. During the VRI, the mean levels of correct responses were: 93.42%, 92.75%, 96.11% and 94.88 % for MM, AW, HM and QM respectively.

During the GP, all participants got 100% of responses correct. With MM in the NEB condition, the correct responses started at 75% then decreased to 25%. During VRI condition, correct responses increased until they reached the criterion in the third trail. With AW, correct responses increased then decreased during NEB. During VRI, correct responses increased, then the data flatlined (?) for two sessions at 80% then increased again to reach the criterion in the fourth session. With HM, the data show a decrease in the trend to a low level 25% in the NEB. Correct responses accelerated during the VRI until reaching the mastery criteria in the third trail and maintaining. QM data points were unstable during NEB, therefore VRI was delayed for this participant until baseline became stable. At the beginning of the training the data points were unstable until the fourth trail, then the participant reached 100% of correct responses and remained consistent. A descriptive graph shows results below:

Discussion

The results of the current study suggested that VR is effective in teaching scanning and finding items in a grocery store for children diagnosed with ASD. Some of the benefits of the VR were: participants acquired the targeted skills in a few/training sessions. It took them 7 sessions, 30 minutes per session, to reach mastery. Also, it gives therapists the ability to easily control and manipulate the environment. Therapists were able to control both the physical and the virtual environments during the VRI. Sessions took place at a clinic which contributes to eliminate noises and crowds. Regarding the virtual environment, therapists were able to place clients in a specific place in the aisle. Also, there is a pointer that functioned as prompt to lead the client to look at a specific item. Therapist were able to choose items in the VR similar yet not identical to the natural environment to promote generalization

Additionally, it does not require the presence of all therapists to collect IOA because it has a recording option which helps therapists collecting accurate data. VR also helps therapists tracking and recording client’s eye gaze to monitor their scanning skill accurately during the intervention and ensure that the client found the correct item. Participants showed motivation for the VR by running towards it and mand for it vocally, so the VR was a teaching and reinforcing tool at the same time. In a timely manner, it saves both therapists and participants’ time because all the sessions can be conducted at the clinic using the VR headset instead of going back and forth to the grocery store. Although participants’ levels on the VR baseline were zero, lower than their performance in the natural environment, the intervention using the VR was successful. Participants reached the mastery criteria and their performance maintained during generalization probes in the natural environment.

The current study suggests that VR provides an efficient way to use technology to teach behaviors that can be cumbersome to teach in the natural environment. The customizability allows for more creative training sessions and to better way to collect accurate data. Recommendation for future studies is to compare conducting intervention using VR and more traditional methods without VR to compare the number of trials to mastery.

Limitations and Future works

The results of VRT with ABA intervention were very positive and promising. The ABA interventions were improved by the support of eye tracking data and the high control of variables that we can be manipulated due to providing the session of grocery store experience within the clinic. The cost of field tips pre and post and during intervention was saved, in addition to saving the time of the therapists. The ability of re-using the VR offers another financial advantage of this type of intervention.

However, the success of this experiment came with a few difficulties that are worth noting. One of the most common challenges is that Eye-tracking VR use requires a calibration of the participant where participants are asked to follow a moving dot that appears on the screen. The process of calibration is critical because the machine captures and learns the eye gaze and distancing of each participant. Another common challenge comes because if the wearer squints, the glass will not read the eye gaze. Additionally, technical support personnel must be presented in all sessions to troubleshoot technological issues regarding the VR. For example, if the client left the VR setup zone, the IT must reset the room setting. Moreover, grocery store items in the VR were unchangeable which resulted in participants memorizing the item’s places. Lastly, the limited grocery store category where there was only one aisle that contains all the items.

Future research can benefit by allocated larger VR enabled zone where participants are able to conduct larger verity of behaviors and more complex training. Another suggestion is using customized VR environments that recent days more technology companies are investing in creating content and giving grants to enrich the metaverse.

Funding

The author(s) disclosed receipt of the following financial support for the research and publication of this article: This work is funded by King Salman Disability Research Center. Grant number R-2020003.

References

1. Mitchell P, Parsons S, Leonard A (2007) Using virtual environments for teaching social understanding to 6 adolescents with autistic spectrum J Autism Dev Disord 37 (3): 589-600.
2. Strickland DC, Coles CD, Southern LB (2013) JobTIPS: A transition to employment program for individuals with autism spectrum J Autism Dev Disord 43 (10): 2472-2483.
3. Wallace S, Parsons S, Bailey A (2017) Self-reported sense of presence and responses to social stimuli by adolescents with autism spectrum disorder in a collaborative virtual reality environment. Journal of Intellectual & Developmental Disability 42(2): 131-141.
4. Hodges LF, Anderson P, Burdea G, Hoffmann H, Rothbaum BO, et al. (2001) Treating psychological and phsyical disorders with VR. IEEE Computer Graphics and Applications 21(6): 25-33.
5. Parsons S, Cobb S (2014) Reflections on the role of the ‘users’: challenges in a multi-disciplinary context of learner-centred design for children on the autism spectrum. International Journal of Research & Method in Education 37(4): 421-441.
6. Newbutt N, Sung C, Kuo H-J, Leahy MJ, Lin C-C, et al. (2016) Brief report: A pilot study of the use of a virtual reality headset in autism J Autism Dev Disord 46 (9): 3166–3176.
7. Malihi, M, Nguyen J, Cardy RE, Eldon S, Petta C, et al. (2020) Short report: Evaluating the safety and usability of head-mounted virtual reality compared to monitor-displayed video for children with autism spectrum Autism 24 (7): 1924–1929.
8. Adjorlu A, Hoeg ER, Mangano L, Serafin S (2017) Daily living skills training in virtual reality to help children with autism spectrum disorder in a real shopping 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) pp: 294-302.
9. Stein N, Niehorster DC, Watson T, Steinicke F, Rifai K, et al. (2021) A comparison of eye tracking latencies among several commercialhead-mounted i-Perception 12 (1).
10. Labedan P, Darodes-De-Tailly N, Dehais F, Peysakhovich V (2021) Virtual Reality for Pilot Training: Study of Cardiac Activity. In VISIGRAPP (2: HUCAPP) pp: 81-88.