Abstract

Upper limb injury often requires repetitive and long-term physical rehabilitation which can result in low adherence due to the repetitive and internally motivated nature of the exercises. Immersive Virtual Reality (iVR) systems enhanced with games can address these challenges. These systems provide a platform for adaptable sensing and analytical tools to track progress, personalize therapy, and increase long term engagement. This paper explores such a system, through an iVR-based experience for upper-extremity rehabilitation called “OpenButterfly,” where users follow movements to protect a virtual butterfly. OpenButterfly enables a dynamically controllable environment for individual exercise by utilizing motion capture, a biomechanical model of torque and angular momentum, and a biometric pipeline for brainwave, heartrate, and skin conductance analysis. We examine this experience for five adult users with varying degrees of injury over the course of eight weeks. Our results suggest that experiences like OpenButterfly provide strong platforms for long-term physical therapy engagement, analysis, and recovery. Lastly, this paper concludes with considerations for future research into adaptive iVR physio-rehabilitation.

1. Introduction

Shoulder pain affects 18-26% of the U.S. population ¹, which can make daily activities difficult. The shoulder girdle is a complex joint consisting of five large muscles and four rotator cuff muscles. It is capable of movements in all three planes resulting in at least 16,000 distinct arm positions ². Consequently, there are multiple factors that cause shoulder pain and a limited range of motion. This can include acute injury to muscles, tendons, or bones, as well as joint replacement surgery; a stroke; osteoarthritis, or an infection ¹. Regardless of the cause of shoulder injury, intensive and long-term rehabilitation is critical to restore the shoulder’s normal function ³. As with most physical therapy, shoulder rehabilitation begins with restoring the shoulder’s range of motion, then incorporating muscle-strengthening exercises ^{4, 5, 6, 7, 8}. This is typically under the supervision of a therapist who monitors progression and increases exercise difficulty when appropriate.

Eventually, the supervised rehabilitative sessions will end, and the therapist will recommend maintenance exercises that and strength is improved, patients begin to do exercises at home. While conventional in-person therapy has been proven effective, there is still often a problem of compliance with at-home physiotherapy. Non-compliance is difficult to measure as there is no gold standard for measuring adherence and usually relies on self-report, but evidence suggests that noncompliance to at-home exercises for therapy ranges from 30-50% ^{9, 10, 11}. Long-term, supervised therapy is commonly expensive, and home-based exercises can be repetitive, lacks positive feedback, which often leads to boredom and lack of compliance ^{11, 12, 13}. Without proper adherence to exercises the patient may never fully recover and possibly start to retrograde the progress previously made with a therapist ^{11, 14}.

Virtual environments and the increasingly recent use of games for health could yield the potential to address these issues. The ability to create stimulating and re-configurable virtual worlds has been shown to increase therapy compliance, accessibility, and performance analysis ^{12, 15, 16, 17}. Research has shown that engaging in a virtual environment during treatment can distract from pain and discomfort while motivating the user to achieve their therapy goals ^{18, 19}. Considerable success has been reported in using virtual environments for a broad range of therapeutic interventions of a psychological and a physiological nature ^{20, 21}, but the success of these environments has been constrained due to prior cost and hardware limitations ²².

The term VR was coined long before the advent of recent immersive virtual reality (iVR) systems. This has led to differences in how the term VR is applied, and these differences can be seen within the existing literature. For the purposes of this review, we define VR as non-immersive systems that utilize a monitor and allow user interaction through conventional means such as keyboards, mice, or custom controllers ²². VR systems that provide a head-mounted display (HMD) with a stereoscopic omni-orientation monitor, along with appropriate three-dimensional spatialized sound, are categorized as iVR. These systems are similar in how they present movement-based tasks with supplementary visual and auditory feedback but differ in their interaction methods ²³. Moreover, the past five years have seen explosive growth in the field of iVR systems, stemming from a projected 200 million head mounted displays systems sold on the consumer market since 2016 ²⁴. This mass adoption has been in part due to a decrease in hardware cost and a corresponding increase in ease of usability. From these observations, we argue that the integration of iVR with the physical rehabilitation process can offer a less expensive and more computationally adept option for long term therapy.

Immersive virtual environments can engage users and motivate them to overcome difficulty using virtual task goals in the context of rehabilitation. This leads to positive effects such as reduced discomfort, increased compliance, and flexibility ^{16, 19, 25, 26}. To maximize immersion, iVR Head Mounted Display systems (HMDs) may be a promising tool that can fully engross the user in a virtual world.

Other researchers, e.g., Lindner et al. demonstrated the efficaciousness of therapist-guided psychotherapy through a low-cost iVR HMD system ²⁷. The authors found that the use of iVR devices successfully provided practical benefits for self-led and therapist-led intervention ²⁷. In a review by Won et Al., iVR was found to be promising in assisting with the management of acute and procedural pain for adolescent patients by the process of neuromodulation through stimulating experiences ²⁸. In another survey, Li et al. explores and demonstrates the benefits of iVR applied to rehabilitation, disability management, surgical training, psychological disease treatment, and analgesic modality ²⁹. In Laver et al.’s review for VR therapy with stroke survivors, non-immersive VR therapy has been shown to improve arm function and activities of daily living for stroke survivors despite being less effective than conventional therapy ³⁰. Laver et al. also concluded that researchers designing VR rehabilitation programs should conduct pilot studies to evaluate usability and validity of the system and evaluate user’s motivation, engagement and enjoyment. In this paper, we evaluate our system, assess user experience, and highlight the analysis of musculoskeletal simulation and biometric response during the rehabilitation process.

Dynamic simulations can aid in analyzing performance as well as estimating the internal loading of the musculoskeletal system ³¹. These simulations are extremely valuable in the context of rehabilitation and health. It is critical to find a balance for efficient exercise and speed of recovery, as overexerting strength and ROM may injure muscles. Finding this balance can be assisted by use of a modelling software such as OpenSim.

OpenSim is an open-source software system for developing musculoskeletal models and creating dynamic simulations of various movements ³¹. The goal of OpenSim is to build a freely available library of movement simulations for the biomechanics community that has been validated and is ready for treating movement pathologies. The capabilities of OpenSim are vast and have been used to understand many applications such as human gait ^{32, 33, 34}, design of assistive devices ^{35, 36, 37, 38}, characterization of injuries ^{39, 40}, and animal movement analysis ^{41, 42}. Gait mechanics has been well explored with OpenSim, but as of now upper-body contributions are limited.

For our study, we desired to contribute multiple analysis techniques of various shoulder movements by utilizing the upper extremity model developed by Delp et al ³¹. We chose this specific model, as it includes all of the large muscle groups and the full ROM of the shoulder and elbow ³¹. This simulation may prove valuable as it provides shoulder joint torques that can be tracked over an extended period. Torque is important as it used to describe the movement and force produced by the muscles surrounding the joint ^{43, 44, 45, 46}. Prior research has examined the torque of upper-body exercise for more in-depth injury assessment; for example, Perrin et al demonstrated that bilateral torque enables clinicians to more accurately set guidelines in rehabilitation of varying athletic groups ⁴⁷. Another metric we focus on is angular momentum ⁴⁸; this provides a metric to monitor user movement performance over several exercises, ensuring safety and preventing overuse. Several other studies have explored the benefits of quantifying angular momentum for robotic assistance ⁴⁹, the severity of lower body gait impairment ^{50, 51}, and how it contributes to whole-body muscle movement ⁵². By examining average torque and angular momentum per session, we illustrate the average forces and amount of movement performed during gameplay for each user. In addition to describing the mechanics of the shoulder for rehabilitative purposes, we also examine the physiological response to the game.

Emotion is considered a critical element of recovery and healthcare ^{14, 21}. However, understanding the internal physiological activity generated by the emotional and physical response of a patient is often challenging or overlooked during the physical therapy process. In this study, we complement the data acquired by iVR HMDs with wearable sensors that can infer emotional states. Specifically, this study explores three commonly used biosignals: brain waves, heart rate, and galvanic skin response.

The collection of brain activity through Electroencephalography (EEG) has been previously used to infer cognitive state ^{53, 54, 55}. Heart Rate (HR) enables the quantification of physical intensity and has been used as such with VR exercise environments ⁵⁶. Galvanic Skin Response (GSR) or the change in the electrical impedance of the skin caused by the sweat gland has been correlated to physiological arousal ^{57, 58}. All of these biosignals have been extensively explored in the past in a range of studies from non-immersive trials to understanding the physiological response to gaming, television, and music ^{59, 60, 61, 62}. The ability to track and understand the physiological response of emotion during rehabilitation may be immensely insightful for therapists, and many researchers have recognized this potential for robotic assisted applications.

For example, Novak et al utilized psychophysiological responses during robotic stroke rehabilitation tasks to measure psychological state during rehabilitation ⁶³. The authors found that GSR offers the most potential as a psychological state indicator during physiotherapy, with HR and other measures as useful supplementary information. When applying these measures as a biocoorperative feedback loop for upper extremity rehabilitation, the Novak et al found that these biofeedback metrics were not reliable as a primary data source in motor rehabilitation for adjusting task difficulty for users with and without upper extremity impairment ⁶⁴. The authors argued that metrics such as GSR and HR are ideal for supplementary information to help contextualize the rehabilitation process ^{63, 64}. Similarly, many researchers have been examining these metrics for flow during robotic assisted gait training. Guerrero et al demonstrated that physical human-robot interaction has a positive impact on experience, challenge, and skill of human motor performance when biofeedback is utilized to estimate physical and mental states ⁶⁵. Koenig et al utilized biofeedback for real-time closed loop control of cognitive load in neurological patients during robotic assisted gait training ⁶⁶. The study demonstrated that a pre-trained adaptive classifier could be used to automatically adapt exercise difficulty for both healthy and post-stroke subjects. The authors argue that measuring physiological signals for treatment in the clinic require extensive effort from therapeutic staff and that future studies must consider the tradeoff between effort of staff with attachment of sensors ⁶⁶.

From our literature review, these studies have demonstrated that the degree of physiological activation can predict the degree of engagement in rehabilitation and that physiological responses can be used in a closed-loop context to dynamically adapt exercise difficulty during rehabilitation. Many of these studies suggest that biofeedback highly useful for supplementary information to consider rehabilitation performance and experience. However, there is an apparent need to factor in complexity of the system and analysis when utilizing biofeedback for rehabilitation systems. Considering these findings, our system illustrates the use of commercially available biofeedback sensors and automated analysis routines for outpatient recovery to address this challenge. Additionally, we present an immersive virtual experience that enables remote monitoring, multi-modal rehabilitation analysis, and gamification of upper extremity exercise for post-therapist intervention.

The goal of this study is to evaluate our iVR therapy system over the course of 8 weeks. Our target users are those recovering from musculo-skeletal injuries who have completed conventional therapy and need to continue therapy exercises without the monitoring of a therapist. We perform our study in a lab setting to evaluate user performance and experience through musculoskeletal simulation, game analytics, questionnaires, and biofeedback response. We coin this new system of rehabilitation as OpenButterfly. OpenButterfly is a heavily modified version of Project Butterfly by Elor et al ⁶⁷. We also developed a new game tool to easily record and implement custom exercise movements into the game, run repetitive personalized exercise sets with individual users, and have developed a pipeline for multi-modal analysis. Specifically, the goals of this study are to evaluate the following:

1. User performance using game play analytics.

2. Forces and total movement during gameplay sessions using biomechanical simulation.

3. User experience by measuring physiological response to gameplay and gather user feedback.

Through this work, we hope to highlight methodologies for other researchers interested in diving deeper into the rehabilitation process with immersive virtual environments, biomechanical analysis, and biofeedback.

2. Experimental Design

Our target user group consisted of outpatients recovering from shoulder injuries that were pre-cleared for participation. Additionally, these users were patients who failed to complete their at-home exercises, which, as explained prior, can lead to incomplete recovery and increase the risk for re-injury. To recruit study participants, a survey was emailed to interested college university students with general questions about their desire to participate in the study, if they had a relevant injury, if they participated in physical therapy, and what stage of recovery they were currently in. Follow up interviews were conducted with respondents to get more information about their injury and long-term recovery goals to determine if they met the user group criteria. After such screening, five students (one female, four males) with ages ranging from 21 to 28 were chosen, and each student provided informed written consent to participate in both studies. All participants were continuing normal daily living activities but claimed to have limited strength, and/or a limited range of motion. This study received IRB approval from the Office of Research Compliance at the University of California - Santa Cruz. To document user participation, we established a data-collection pipeline and methodology.

The effects of OpenButterfly were examined during an eight-week period through a multimodal analysis of biomechanical, biometric, and gameplay responses. To enable such an analysis, we designed the OpenButterfly pipeline, as shown in Figure 1, and applied the analysis after testing individual users per exercise session. Our user testing protocol can be seen in Figure 1. The sessions consisted of the following methodology stages:

1) Preparation: The study administrators sanitized the iVR equipment, made sure all biometric sensors were fully charged and ran a session of OpenButterfly with all sensors active to ensure data communication quality.

2) Baseline: All biometric sensors were placed on the user in the exercise area. The administrator instructed the user to remain still and relax. After a 15 second period of adjustment, a 30-second baseline was recorded to mark each users’ resting state for every session.

3) Rest: The user was instructed to relax for 90 seconds before performing the exercise with OpenButterfly. This was done before every new exercise was prescribed.

4) Exercise: Users completed 60 seconds of gameplay using OpenButterfly with the iVR headset and biofeedback/biomechanical data recording system. Upon completion of one set, the Rest stage was repeated.

5) Survey: After all exercises were completed, users filled out a brief survey indicating preference, pain, immersion, and self-reported advancement toward long term recovery goals. Such survey questions can be seen in Table 3.

Two researchers were always present to monitor user experience and followed a strict written protocol when interacting with users. This ensures a consistent method of tracking progression in the course of the eight weeks. Below is the list of equipment that we used in the study. Every sensor was chosen with accessibility and cost as a factor.

1) HTC Vive: Through Vive and the Unity Game Engine, motion capture and game data are recorded during runtime at 90 Hz using a data exportation method developed in previous studies by Elor et al ^{16, 67}.

2) OptiTrack: A motion tracking system of 10 reflective markers was recorded at 120 Hz using 8 OptiTrack 13W cameras ⁶⁸.

3) InteraXon Muse 2 - Brain Sensing Headband ⁶⁹: Muse is a commercially available headband that records EEG on the pre-frontal cortex (TP9-10, AF1-7) with dry contact electrodes. The headset has built-in internal noise suppression with 2uV RMS noise floor and a 50 or 60Hz regional notch filter. Muse was connected wirelessly to the Muse Monitor app on a mobile device and employed a Cooley-Tukey FFT ⁷⁰ to extract brainwave band power in bels. Muse has successfully been used in other studies to infer mental state, analyze event potentials, and record biofeedback ^{71, 72, 73}. Foreheads were sterilized with saline wipes before gameplay.

4) Neulog GSR logger sensor NUL-217 ⁷⁴: The NUL217 is a GSR logging sensor that measures the conductivity of the skin between the fingers. The logger connects to a USB-200 Module and records GSR in micro Siemens with a 10nS resolution at a max sample rate of 100Hz. The two finger electrodes were sterilized with saline wipes before gameplay.

5) Polar OH1 - optical heart rate sensor ⁷⁵: the Polar OH1 is a 6 LED optical heart rate sensor that is used with an armband to record beats per minute through Bluetooth at 1Hz sampling frequency.

These devices are easy to set up for a user at home and are a more affordable solution compared to clinical grade sensors; i.e. these biofeedback sensors do not utilize single use components, unlike more conventional systems that may use EEG gels or sticktrodes.

Our game, titled “OpenButterfly”, consists of a virtual butterfly that moves within reach of the participant to guide the user through their required movements. OpenButterfly gameplay can be seen in Figure 2. It is an adaption of Project Butterfly by Elor et al, previously designed for upper-extremity impairments for older adults while using a soft robotic wearable ⁶⁷. Specific new contributions to the OpenButterfly software includes a new system that records and prescribes custom exercise movements, runs automated repetitive personalized exercise sets with individual users, and provides increased stimuli for movement guidance. These contributions were designed through feedback sessions with collaborating physical therapists across Santa Cruz, California.

To guide movements, projectile crystals emanate from a 15m distance and move on a collision path with the butterfly. Users were informed that the goal of the game is to protect the butterfly from these crystals. The player holds an orb in their hand that they can place over the butterfly to protect it. The crystals explode when they hit the orb, letting the player know they successfully protected the butterfly and earned a point.

For prescribing custom movements, the path of the butterfly can be predetermined and set using a simple interface. The therapist can enter the ”Path Development” game mode, where they see the butterfly in their hand. When the trigger is pressed, they can move the butterfly in any path they desire at any speed. These movements can be saved and accessed through internal comma-separated value files. Through these movement files, the butterfly will follow recorded exercises that are automatically normalized to each user’s arm length, target arm, and prescribed speed of movement. These implementations were done through utilizing the Unity3D Game Engine’s Microsoft .NET File I/O C# Libraries ⁷⁷.

Our study examined this new game mode by recording and prescribing seven new exercises in collaboration with therapists. The distance of the butterfly from the user is scaled based on arm length, which was measured for each participant at the beginning of the study using the relative position of the game controllers to the headset. Thus, the game automatically scales exercise paths to the user’s arm length -a feature requested by our collaborating physical therapists. Such game paths can be seen in Figure 3. These changes were done as a means for therapists to adjust the game to fit their users’ needs and to enable dynamic customization and calibration. While some of the game’s stimuli have been explored by Elor et al through Project Butterfly, OpenButterfly will be one of the first studies to examine these new exercise features when applied to iVR therapy over the course of eight weeks.

At the end of each session, participants were asked several Likert scale questions about their iVR experience that day. These questions were taken from a Jennett et al. survey for immersion in games ⁷⁸ and was modified to focus more on user engagement. Such survey questions can be seen in Table 3. The surveys were utilized to track self-perception for users at the end of each session. These questions were used to evaluate if the users would remain engaged, entertained, and immersed over the eight-week period. Additionally, an exit interview was conducted at the end of the eight weeks to determine what modifications users would want to help improve rehabilitation. This enabled us to establish a mixed- method approach of gameplay, biomechanical, biometric, and survey responses for OpenButterfly.

Each of the biosensors, the OptiTrack motion capture system, and HTC Vive produced their own output files with their respective recording frequency. Approximately 1,200 data files were produced during the Pilot and Revised Study. Python ⁷⁹ scripts were written to structure the file management system and then sync all the files for each user session. OpenSim ³¹ simulations were ran using the motion tracking files from the OptiTrack system, generating approximately another 1,100 files. Statistical analysis was then conducted on the data files using MATLAB ⁷⁶. The full pipeline for collecting and processing our data can be seen in Figure 1.

3. Pilot Study

The goal of the pilot study was for participants to perform common daily movements with an incremental and gradual amount of weight increase on their arms. The movements chosen were Forward Arm Raise (FAR), Side Arm Raise (SAR), and Horizontal Abduction (HA). These movements are simple single plain movements. We were careful to start with low-intensity movements to ensure participant’s safety.

Additionally, we hypothesized that participants would initially be unable to have full ROM through each exercise but would be able to progress to full ROM with a small amount of wrist weight by the end of the pilot study.

The pilot study was performed for the first four weeks.

Each week consisted of two sessions where users performed 30-45 minutes of exercise (time includes rest). During each session, exercises were performed in the following order: FAR, SAR, and HA in order for a total of three rounds. The first round was played without weight for a warmup, and the subsequent two rounds were performed with the appropriate weight per user. Additionally, users had a 90-second rest between exercises, and each exercise was performed for 60 seconds at ten repetitions per minute. Aspects of this protocol are highlighted in Figure 1.

Full ROM for these movements was attained before adding weight to the user’s wrist. The weight was added in small increments to elicit a strength progression, and users’ average weight per session can be seen in Table 1. Weight was only increased when participants could comfortably perform two consecutive rounds of all three exercises for a given weight. To account for the participant’s responses being influenced by the novelty of the VR game and or headset, an initial session was performed to introduce the game mechanics and enable the participants to be familiar with the OpenButterfly environment and movements.

The averages of collected data can be seen in Table 1. The most prominent observation was that all users were able to complete the entire ROM of each exercise quickly, indicated by the high compliance rate, which allowed us to begin using weight early on in the study. To understand user engagement, effort, and immersion we employed a modified survey from Jennet et al. ⁷⁸. Table 3 shows the questionnaire asked at the end of each user testing session. Generally, users agreed that the game was engaging; they put a lot of effort into participating and felt immersed throughout gameplay sessions.

All elements of player behavior and biometric events (with the exception of user jaw clenches) were found to be significant, as seen in Table 1 A-s1 to A-s5. Users were able to acclimate to a 100% increase in weight resistance while moving their weak arm at a total average of 30m of distance per session. Additionally, users were able to successfully protect the butterfly at a compliance rate of a mean 96%, where compliance is defined as the time protecting the butterfly divided by the total time of the exercise session. In terms of both compliance and arm travel distance, these results held a low range of standard deviation, indicating that user performance was fairly constant between all users for these sessions. From the biometric data starting at a resting heart rate, the exercise sessions induced an average mean increase of 11.78 beats per minute, indicating increased physiological intensity from the exercises (shown in Table 2 [A]). Galvanic skin response was also found to be at a positive increase for all pilot protocol sessions, with a mean 1.47 micro Siemens resting-state change indicating arousal from gameplay, as shown in Table 2. For brainwave activity, the pilot protocol generally held a mean increase of all wavebands for alpha, beta, delta, theta, and gamma powers -this may confirm that users were well mentally stimulated and physiologically challenged during gameplay.

One thing that we learned from the pilot study is that the exercises of the Pilot Study were effective in increasing general strength, as can be seen in Table 1 where average weight between each session increases consistently. However, our users had a more substantial initial ROM than we anticipated. For our revised study, we needed to help our users progress more in ROM than the exercises in the Pilot Study required.

We performed ROM expansion by adding two common adduction/abduction movements: External Rotation (EXR) and Abducted Rotation (ABR), as well as two multiplanar movements: Mixed Press (MXDPR) and MixedCircles (MXDCR). These movements can be seen in Figure 4 in the green region. Since these new movements focused more on stretching, no weight was used while performing these four movements.

In the Pilot Study, the first round was always played without weight as a warmup. Since the new stretching games were played without weight, we decided to do two rounds first of the non-weighted movements followed by two rounds of the weighted movements. This is further explained in the Revised Protocol Section.

4. Revised Study

Learning from the pilot study results, we modified our game to have a more appropriate protocol for our users. To address insufficient ROM exercises, new exercises were created, as shown in Figure 4 (EXR, ABR, MXDPR, MXDCR). These movements require a greater ROM at different angles than the pilot games. FAR, SAR, and HA games were kept to specifically address increases in strength by still utilizing the wrist weight progression protocol.

The revised study lasted four weeks, with twice a week session consisting of 30-45 minutes of exercise. During each session, two rounds of EXR, ABR, MXDPR, and MXDCR were performed with a one-minute rest between each exercise. Each movement was performed for 60 seconds at a slow tempo to allow for stretching at the limit of each subject’s ROM. These exercises were always performed without any wrist weight as stretching was the goal, not strength.

Afterward, two rounds of FAR, SAR, and HA were performed with a one-minute rest between each exercise. This followed the same weight increase protocol as the pilot study to ensure a safe progression in strength exercises.

From the revised protocol, users engaged in greater weight resistance than the Pilot Study [A], and subsequently, there was far more variability between users. Arm travel distance was less in total, but the movements were far more complex and slower. HR and GSR were found to be less than the Pilot Study’s [A] mean resting state change but still elevated by nearly 10bpm and 1.13 micro Siemens, respectively. The lowered heart rate may be an artifact of the slower tempo in movement, and declining GSR may further indicate acclimation to the game with a lowering rate of arousal (however, it was still elevated far above resting state). Table 2 lists these results. As this table shows, the Revised Study [B] results were significantly different from the Pilot Study’s results [A] for all data sets. Specifically, the Pilot Results [A] had a greater compliance rate, weak arm movement, HR change, and GSR change. In contrast, the revised study’s results show higher levels of brain activity for all wavebands as well as Blinks and Jaw Clenches. This may indicate that the Revised Study was more mentally stimulating while both increasing weight resistance and game compliance, as shown in Table 1.

Using OpenSim with the motion tracking data from each session, we were able to determine the amount of torque placed on the shoulder for each exercise, as shown in Figure 5. We took the integration of torque with respect to time to determine the amount of angular momentum the shoulder generated from exercise.

While the average torque for each user dropped between the Pilot Study [A] and the Revised Study [B], the average angular momentum for each user increased between the studies, as shown in Table 2. This decrease in mean torques is a result of more exercises being performed without weight. For example, 8 of the 14 exercises in the Revised Study [B] were played without weight, whereas only 3 out of the 9 exercises in the Pilot Study [A] were without weight. The increase in average angular momentum between the Pilot Study [A] and the Revised Study [B] occurs because more games are played in the Revised Study [B]. Figure 5 shows each user’s average torque and angular momentum for every session.

We expected to see more of a steady increase for torque and angular momentum in each study since the users were lifting the same or more weight than their previous session. However, we observed that users’ average torque and angular momentum fluctuated a bit from session to session in the same protocol rather than continuously increasing. We believe this is important to show as we can see which sessions the users were not performing as expected. Users told us that some days they would come in more stiff or sore than other days. This data may be indicative of these cases and theoretically could be used to help adjust the exercise protocol. These results may be helpful for other researchers interested in expanding upon this work for a more personalized and reactive therapy regime. Understanding day to day fluctuation through this data could lead to new algorithms for a more customizable rehabilitation through adapting to users’ capabilities. Even if users decline or have a setback (i.e., sore from previous day’s activities such as yard work), these insights could be used to tailor the difficulty to the most important muscle groups for maximizing therapeutic benefit.

From this data, we can also see the most significant changes in torque and angular momentum come from the different amount of games played; in short, the more games played, the more angular momentum was gained. This is useful when deciding a rehabilitation plan as the variables to consider are weight resistance or total movement. So, by utilizing the angular momentum, we can determine the exact amount of motion of a specific game, no matter the movement path of the arm. Such an analysis of angular momentum could be used to build a more intelligent progression.

A significant difference between our two studies was found from Wilcoxon significance testing at 95% certainty for the qualitative user surveys, as shown in Table 3. Users reported the Revised Study [B] to be significantly more challenging ([Q7]), progressive ([Q10, Q11]), and liked ([Q1]) the protocol more than the Pilot Study [A]. Conversely, users reported that they felt the revised protocol provided significantly less distraction ([Q4]) from their real-life compared to the pilot protocol. These results may indicate that the protocol choices for the Revised Protocol [B] were successful in challenging each user from an engagement and effort perspective.

5. Discussion

From our multimodal analysis of our eight-week study, we show that OpenButterfly accomplishes our goals of increasing ROM and increasing strength. This was a multi-step process that required two stages to adjust and further customize the game to the users’ capabilities as rehabilitation necessitates. The Pilot Study [A] was useful to help determine the capabilities of our users and how to set achievable goals for them. This stage showed that the users achieved a full ROM for the first three exercises (FAR, SAR, HA) and were ready to start training with weights very quickly. Starting with simple movements was a safeguard against exercises that were too advanced for their state of recovery. The insights from the Pilot Study enabled us to create more complex movements and continue to work on strength.

For the Revised Study [B], we created four new movements (EXR, ABR, MXDPR, and MXDCR) that were performed without weight to target enhanced ROM. At the same time, the original exercises from the Pilot Study [A] were carried over to focus on strength building. We saw ROM increased to meet these challenging and further-reaching movements indicated by the increase in the rate of the compliance recovery during the Revised Study [B], as shown in Table 1. Also, in Table 1, results from average weight indicated a successful increase in strength. The Revised Study [B] displayed an improvement where more exercise difficulty was leveraged to safely challenge the users. Users also enjoyed the new exercises and stated it was similar to ”unlocking a new level in a game.” This is a consideration as we move forward: making levels that are of different movements so that the game remains challenging and does not become repetitive.

Additionally, the users’ physiological recordings and self-reported responses indicated that users were able to remain engaged with the game beyond the novelty effect period for the course of the eight weeks, as seen in Table 1 and Table 2. Consequently, technology like OpenButterfly may become a promising tool for addressing the problem of adherence to a rehabilitation program. The more enjoyable and engaging the program, the more likely users will continue the program. This adherence with OpenButterfly is particularly exciting, as other researchers may be able to utilize similar iVR physical therapy experiences for long-term treatment.

Through our study, we believe several insights can be useful to game developers and researchers. First, we learned that the ability to easily and quickly create custom paths for arm movements during gameplay allowed us to efficiently adjust our games between the Pilot Study [A] and the Revised Study [B]. This adjustment needed to occur because all users were able to complete a full ROM with added weight within a few sessions. The Pilot Study [A] exercises proved valuable as a baseline ROM and for improving strength. The Revised Study [B] had more complex movements targeting ROM and kept the now proven original exercises for targeting strength. These game modifications were guided by our collaborating physical therapists to increase the difficulty of an appropriate progression in strength and ROM. The ability to record custom motion paths and normalize movements to each user’s arm length and height proved to be a valuable tool.

Another useful tool was the biomechanical simulation, as it offered more in-depth analytics into user performance through analyzing performance during a session. In traditional PT, the therapist can monitor progression through measurements of ROM and strength, typically pounds lifted or level of a resistance band. Through our study with OpenSim, we are able to provide this data and, in the future hope to have everything streamlined so that no matter the movements performed, simple or complex, we can provide a thorough representation of the amount force placed on the working joint for a therapist to examine. With further user testing, perhaps researchers can build more sophisticated models for simulation that will provide individual muscle for training and rehabilitation for any of the movements performed in the game. This can help the therapist target specific muscles to aid in a focused recovery.

Additionally, biometric data from OpenButterfly may help researchers understand users’ physiological responses to the iVR experience. This helps with recovery as more enjoyable user experience is likely to lead to better adoption of a rehabilitation program. Our data indicates users had higher brain activity for the Revised Session [B], which should be further explored in considering rehabilitation monitoring and game adaption for future studies. These metrics also provide a possibility for determining how much effort the user is putting into the game on a physiological level. Since this isn’t a strenuous workout, we want to make sure users are working at an appropriate level. We learned that the levels we chose were enough to elicit a strength increase response, but not so much that user is at risk of injuring themselves.

We believe our study has shown this game’s feasibility for helping with the recovery process, and fellow researchers, developers, engineers, and therapists may find aspects of our research useful for their endeavors. Collecting HR, GSR, and EEG may provide a deeper understanding of a user’s engagement and physical effort with iVR exercises. This can help with game development in creating exciting experiences to help increase a user’s desire to play the game. Biomechanical simulation can provide valuable metrics to a monitoring therapist and also give a progress log over an extended period of time. OpenButterfly itself shows that other games can be created to aid with recovery, and we suggest from guidance with our collaborating therapists that in future games, there is a way for a therapist to dictate the movements of the game easily, so it is customizable to the user’s needs.

There are several limitations that may impact generalizability of the results. The study examined 5 users with OpenButterfly, but in the future iterations we plan to recruit more users. While the study lasted 8 weeks it would be helpful to understand more long-term effect by conducting the study for 12-16 weeks since our goal is adherence of users. Additionally, we limited the frequency to 2 sessions per week to ensure adequate time for recovery, but having the users progress to 3 and 4 times per week could yield better benefits. While our long-term goal is at-home use, we conducted our study in a lab to examine performance and user experience of our system. The next step would be applying what we learned from our system feasibility study and conduct an at-home user study.

6. Conclusions and Future Work

The purpose of OpenButterfly was to create an effective and feasible iVR physical therapy game to help users with shoulder injuries through multimodal rehabilitation analysis. This was accomplished by working with therapists and enabling game recordings to mimic movements found in physical therapy targeted at ROM and strength training. Through OpenButterfly, we present a novel study that is a long-term, customizable highly immersive virtual reality game for shoulder rehabilitation that analyzes physiological response and uses biomechanical simulation to identify the joint kinetics and dynamics. Working with therapists, we have identified useful tools and data sensors to aid in developing games targeted at recovery that we believe other serious game researchers will find helpful. This multimodal rehabilitation analysis will help with the next iteration of our game to ensure user engagement and that users are working at an appropriate threshold, not too intense to risk injury but difficult enough to elicit a physiological adaptation.

We explored two experimental studies: a Pilot Study consisting of three single everyday movements targeted at a basic ROM and strength, and a Revised Study that incorporated four new movements aimed at ROM from insights gained from the multimodal rehabilitation analysis of the Pilot Study. Our results indicate that users were able to overcome the novelty effect of iVR through extended exposure to gameplay over eight weeks. We were also able to measure heart rate, galvanic skin response, and electroencephalography while our users played the game, allowing us to understand their physical strain and emotional response while playing. With the motion capture data, we were able to determine the kinematics and dynamics of the shoulder during gameplay through biomechanical simulation. We believe this data would be useful for physical therapists as it helps quantify the forces of the joint for an entire session and would provide a method for remote therapists to quickly understand users’ exercise session.

In the future, we plan to explore the design of new levels within the game that contain more complex and less predictable movements to challenge users physically and mentally. Our long-term goal is to develop an at-home recovery game that is capable of providing meaningful game data remotely to the therapist. Subsequently, we plan to explore a more complex biomechanics model capable of identifying individual muscle force contribution to movements. The incorporation of runtime biomechanical models to identify muscle weaknesses may further aid in custom movements for an individual user to help maximize their recovery by ensuring the targeted muscles are being used for a given movement. We hope to deploy this system for at-home use to make OpenButterfly more accessible for users in need. To this end, there are more butterflies to protect as we continue working with therapists to modify the game and models.

Acknowledgements

The authors thank the student participants that helped make this study possible. The authors would like to thank Dana Seth, PT, DPT, for her guidance on developing progression movements for the game to ensure user safety, and Rebecca Friedman, DSC, for her advice and suggestions for therapist needs. Additionally, the following UC Santa Cruz students are acknowledged for their support: Teresa Faasolo, Evanjelin Mahmoodi, and Conrad Esch.

References