Article Versions
Export Article
Cite this article
  • Normal Style
  • MLA Style
  • APA Style
  • Chicago Style
Research Article
Open Access Peer-reviewed

OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy

Michael Ora Powell , Aviv Elor, Mircea Teodorescu, Sri Kurniawan
American Journal of Sports Science and Medicine. 2020, 8(1), 23-35. DOI: 10.12691/ajssm-8-1-5
Received May 14, 2020; Revised June 16, 2020; Accepted June 23, 2020

Abstract

Upper limb injury often requires repetitive and long-term physical rehabilitation which can result in low adherence due to the repetitive and internally motivated nature of the exercises. Immersive Virtual Reality (iVR) systems enhanced with games can address these challenges. These systems provide a platform for adaptable sensing and analytical tools to track progress, personalize therapy, and increase long term engagement. This paper explores such a system, through an iVR-based experience for upper-extremity rehabilitation called “OpenButterfly,” where users follow movements to protect a virtual butterfly. OpenButterfly enables a dynamically controllable environment for individual exercise by utilizing motion capture, a biomechanical model of torque and angular momentum, and a biometric pipeline for brainwave, heartrate, and skin conductance analysis. We examine this experience for five adult users with varying degrees of injury over the course of eight weeks. Our results suggest that experiences like OpenButterfly provide strong platforms for long-term physical therapy engagement, analysis, and recovery. Lastly, this paper concludes with considerations for future research into adaptive iVR physio-rehabilitation.

1. Introduction

Shoulder pain affects 18-26% of the U.S. population 1, which can make daily activities difficult. The shoulder girdle is a complex joint consisting of five large muscles and four rotator cuff muscles. It is capable of movements in all three planes resulting in at least 16,000 distinct arm positions 2. Consequently, there are multiple factors that cause shoulder pain and a limited range of motion. This can include acute injury to muscles, tendons, or bones, as well as joint replacement surgery; a stroke; osteoarthritis, or an infection 1. Regardless of the cause of shoulder injury, intensive and long-term rehabilitation is critical to restore the shoulder’s normal function 3. As with most physical therapy, shoulder rehabilitation begins with restoring the shoulder’s range of motion, then incorporating muscle-strengthening exercises 4, 5, 6, 7, 8. This is typically under the supervision of a therapist who monitors progression and increases exercise difficulty when appropriate.

Eventually, the supervised rehabilitative sessions will end, and the therapist will recommend maintenance exercises that and strength is improved, patients begin to do exercises at home. While conventional in-person therapy has been proven effective, there is still often a problem of compliance with at-home physiotherapy. Non-compliance is difficult to measure as there is no gold standard for measuring adherence and usually relies on self-report, but evidence suggests that noncompliance to at-home exercises for therapy ranges from 30-50% 9, 10, 11. Long-term, supervised therapy is commonly expensive, and home-based exercises can be repetitive, lacks positive feedback, which often leads to boredom and lack of compliance 11, 12, 13. Without proper adherence to exercises the patient may never fully recover and possibly start to retrograde the progress previously made with a therapist 11, 14.

Virtual environments and the increasingly recent use of games for health could yield the potential to address these issues. The ability to create stimulating and re-configurable virtual worlds has been shown to increase therapy compliance, accessibility, and performance analysis 12, 15, 16, 17. Research has shown that engaging in a virtual environment during treatment can distract from pain and discomfort while motivating the user to achieve their therapy goals 18, 19. Considerable success has been reported in using virtual environments for a broad range of therapeutic interventions of a psychological and a physiological nature 20, 21, but the success of these environments has been constrained due to prior cost and hardware limitations 22.

The term VR was coined long before the advent of recent immersive virtual reality (iVR) systems. This has led to differences in how the term VR is applied, and these differences can be seen within the existing literature. For the purposes of this review, we define VR as non-immersive systems that utilize a monitor and allow user interaction through conventional means such as keyboards, mice, or custom controllers 22. VR systems that provide a head-mounted display (HMD) with a stereoscopic omni-orientation monitor, along with appropriate three-dimensional spatialized sound, are categorized as iVR. These systems are similar in how they present movement-based tasks with supplementary visual and auditory feedback but differ in their interaction methods 23. Moreover, the past five years have seen explosive growth in the field of iVR systems, stemming from a projected 200 million head mounted displays systems sold on the consumer market since 2016 24. This mass adoption has been in part due to a decrease in hardware cost and a corresponding increase in ease of usability. From these observations, we argue that the integration of iVR with the physical rehabilitation process can offer a less expensive and more computationally adept option for long term therapy.

1.1. Why Immersive Virtual Environments for Rehabilitation?

Immersive virtual environments can engage users and motivate them to overcome difficulty using virtual task goals in the context of rehabilitation. This leads to positive effects such as reduced discomfort, increased compliance, and flexibility 16, 19, 25, 26. To maximize immersion, iVR Head Mounted Display systems (HMDs) may be a promising tool that can fully engross the user in a virtual world.

Other researchers, e.g., Lindner et al. demonstrated the efficaciousness of therapist-guided psychotherapy through a low-cost iVR HMD system 27. The authors found that the use of iVR devices successfully provided practical benefits for self-led and therapist-led intervention 27. In a review by Won et Al., iVR was found to be promising in assisting with the management of acute and procedural pain for adolescent patients by the process of neuromodulation through stimulating experiences 28. In another survey, Li et al. explores and demonstrates the benefits of iVR applied to rehabilitation, disability management, surgical training, psychological disease treatment, and analgesic modality 29. In Laver et al.’s review for VR therapy with stroke survivors, non-immersive VR therapy has been shown to improve arm function and activities of daily living for stroke survivors despite being less effective than conventional therapy 30. Laver et al. also concluded that researchers designing VR rehabilitation programs should conduct pilot studies to evaluate usability and validity of the system and evaluate user’s motivation, engagement and enjoyment. In this paper, we evaluate our system, assess user experience, and highlight the analysis of musculoskeletal simulation and biometric response during the rehabilitation process.

1.2. Musculoskeletal Simulation

Dynamic simulations can aid in analyzing performance as well as estimating the internal loading of the musculoskeletal system 31. These simulations are extremely valuable in the context of rehabilitation and health. It is critical to find a balance for efficient exercise and speed of recovery, as overexerting strength and ROM may injure muscles. Finding this balance can be assisted by use of a modelling software such as OpenSim.

OpenSim is an open-source software system for developing musculoskeletal models and creating dynamic simulations of various movements 31. The goal of OpenSim is to build a freely available library of movement simulations for the biomechanics community that has been validated and is ready for treating movement pathologies. The capabilities of OpenSim are vast and have been used to understand many applications such as human gait 32, 33, 34, design of assistive devices 35, 36, 37, 38, characterization of injuries 39, 40, and animal movement analysis 41, 42. Gait mechanics has been well explored with OpenSim, but as of now upper-body contributions are limited.

For our study, we desired to contribute multiple analysis techniques of various shoulder movements by utilizing the upper extremity model developed by Delp et al 31. We chose this specific model, as it includes all of the large muscle groups and the full ROM of the shoulder and elbow 31. This simulation may prove valuable as it provides shoulder joint torques that can be tracked over an extended period. Torque is important as it used to describe the movement and force produced by the muscles surrounding the joint 43, 44, 45, 46. Prior research has examined the torque of upper-body exercise for more in-depth injury assessment; for example, Perrin et al demonstrated that bilateral torque enables clinicians to more accurately set guidelines in rehabilitation of varying athletic groups 47. Another metric we focus on is angular momentum 48; this provides a metric to monitor user movement performance over several exercises, ensuring safety and preventing overuse. Several other studies have explored the benefits of quantifying angular momentum for robotic assistance 49, the severity of lower body gait impairment 50, 51, and how it contributes to whole-body muscle movement 52. By examining average torque and angular momentum per session, we illustrate the average forces and amount of movement performed during gameplay for each user. In addition to describing the mechanics of the shoulder for rehabilitative purposes, we also examine the physiological response to the game.

1.3. Biofeedback and Physiological Response

Emotion is considered a critical element of recovery and healthcare 14, 21. However, understanding the internal physiological activity generated by the emotional and physical response of a patient is often challenging or overlooked during the physical therapy process. In this study, we complement the data acquired by iVR HMDs with wearable sensors that can infer emotional states. Specifically, this study explores three commonly used biosignals: brain waves, heart rate, and galvanic skin response.

The collection of brain activity through Electroencephalography (EEG) has been previously used to infer cognitive state 53, 54, 55. Heart Rate (HR) enables the quantification of physical intensity and has been used as such with VR exercise environments 56. Galvanic Skin Response (GSR) or the change in the electrical impedance of the skin caused by the sweat gland has been correlated to physiological arousal 57, 58. All of these biosignals have been extensively explored in the past in a range of studies from non-immersive trials to understanding the physiological response to gaming, television, and music 59, 60, 61, 62. The ability to track and understand the physiological response of emotion during rehabilitation may be immensely insightful for therapists, and many researchers have recognized this potential for robotic assisted applications.

For example, Novak et al utilized psychophysiological responses during robotic stroke rehabilitation tasks to measure psychological state during rehabilitation 63. The authors found that GSR offers the most potential as a psychological state indicator during physiotherapy, with HR and other measures as useful supplementary information. When applying these measures as a biocoorperative feedback loop for upper extremity rehabilitation, the Novak et al found that these biofeedback metrics were not reliable as a primary data source in motor rehabilitation for adjusting task difficulty for users with and without upper extremity impairment 64. The authors argued that metrics such as GSR and HR are ideal for supplementary information to help contextualize the rehabilitation process 63, 64. Similarly, many researchers have been examining these metrics for flow during robotic assisted gait training. Guerrero et al demonstrated that physical human-robot interaction has a positive impact on experience, challenge, and skill of human motor performance when biofeedback is utilized to estimate physical and mental states 65. Koenig et al utilized biofeedback for real-time closed loop control of cognitive load in neurological patients during robotic assisted gait training 66. The study demonstrated that a pre-trained adaptive classifier could be used to automatically adapt exercise difficulty for both healthy and post-stroke subjects. The authors argue that measuring physiological signals for treatment in the clinic require extensive effort from therapeutic staff and that future studies must consider the tradeoff between effort of staff with attachment of sensors 66.

From our literature review, these studies have demonstrated that the degree of physiological activation can predict the degree of engagement in rehabilitation and that physiological responses can be used in a closed-loop context to dynamically adapt exercise difficulty during rehabilitation. Many of these studies suggest that biofeedback highly useful for supplementary information to consider rehabilitation performance and experience. However, there is an apparent need to factor in complexity of the system and analysis when utilizing biofeedback for rehabilitation systems. Considering these findings, our system illustrates the use of commercially available biofeedback sensors and automated analysis routines for outpatient recovery to address this challenge. Additionally, we present an immersive virtual experience that enables remote monitoring, multi-modal rehabilitation analysis, and gamification of upper extremity exercise for post-therapist intervention.

1.4. Study Goals and Contribution

The goal of this study is to evaluate our iVR therapy system over the course of 8 weeks. Our target users are those recovering from musculo-skeletal injuries who have completed conventional therapy and need to continue therapy exercises without the monitoring of a therapist. We perform our study in a lab setting to evaluate user performance and experience through musculoskeletal simulation, game analytics, questionnaires, and biofeedback response. We coin this new system of rehabilitation as OpenButterfly. OpenButterfly is a heavily modified version of Project Butterfly by Elor et al 67. We also developed a new game tool to easily record and implement custom exercise movements into the game, run repetitive personalized exercise sets with individual users, and have developed a pipeline for multi-modal analysis. Specifically, the goals of this study are to evaluate the following:

1. User performance using game play analytics.

2. Forces and total movement during gameplay sessions using biomechanical simulation.

3. User experience by measuring physiological response to gameplay and gather user feedback.

Through this work, we hope to highlight methodologies for other researchers interested in diving deeper into the rehabilitation process with immersive virtual environments, biomechanical analysis, and biofeedback.

2. Experimental Design

Our target user group consisted of outpatients recovering from shoulder injuries that were pre-cleared for participation. Additionally, these users were patients who failed to complete their at-home exercises, which, as explained prior, can lead to incomplete recovery and increase the risk for re-injury. To recruit study participants, a survey was emailed to interested college university students with general questions about their desire to participate in the study, if they had a relevant injury, if they participated in physical therapy, and what stage of recovery they were currently in. Follow up interviews were conducted with respondents to get more information about their injury and long-term recovery goals to determine if they met the user group criteria. After such screening, five students (one female, four males) with ages ranging from 21 to 28 were chosen, and each student provided informed written consent to participate in both studies. All participants were continuing normal daily living activities but claimed to have limited strength, and/or a limited range of motion. This study received IRB approval from the Office of Research Compliance at the University of California - Santa Cruz. To document user participation, we established a data-collection pipeline and methodology.

2.1. Methodology and Data Collection

The effects of OpenButterfly were examined during an eight-week period through a multimodal analysis of biomechanical, biometric, and gameplay responses. To enable such an analysis, we designed the OpenButterfly pipeline, as shown in Figure 1, and applied the analysis after testing individual users per exercise session. Our user testing protocol can be seen in Figure 1. The sessions consisted of the following methodology stages:

1) Preparation: The study administrators sanitized the iVR equipment, made sure all biometric sensors were fully charged and ran a session of OpenButterfly with all sensors active to ensure data communication quality.

2) Baseline: All biometric sensors were placed on the user in the exercise area. The administrator instructed the user to remain still and relax. After a 15 second period of adjustment, a 30-second baseline was recorded to mark each users’ resting state for every session.

3) Rest: The user was instructed to relax for 90 seconds before performing the exercise with OpenButterfly. This was done before every new exercise was prescribed.

4) Exercise: Users completed 60 seconds of gameplay using OpenButterfly with the iVR headset and biofeedback/biomechanical data recording system. Upon completion of one set, the Rest stage was repeated.

5) Survey: After all exercises were completed, users filled out a brief survey indicating preference, pain, immersion, and self-reported advancement toward long term recovery goals. Such survey questions can be seen in Table 3.

Two researchers were always present to monitor user experience and followed a strict written protocol when interacting with users. This ensures a consistent method of tracking progression in the course of the eight weeks. Below is the list of equipment that we used in the study. Every sensor was chosen with accessibility and cost as a factor.

1) HTC Vive: Through Vive and the Unity Game Engine, motion capture and game data are recorded during runtime at 90 Hz using a data exportation method developed in previous studies by Elor et al 16, 67.

2) OptiTrack: A motion tracking system of 10 reflective markers was recorded at 120 Hz using 8 OptiTrack 13W cameras 68.

3) InteraXon Muse 2 - Brain Sensing Headband 69: Muse is a commercially available headband that records EEG on the pre-frontal cortex (TP9-10, AF1-7) with dry contact electrodes. The headset has built-in internal noise suppression with 2uV RMS noise floor and a 50 or 60Hz regional notch filter. Muse was connected wirelessly to the Muse Monitor app on a mobile device and employed a Cooley-Tukey FFT 70 to extract brainwave band power in bels. Muse has successfully been used in other studies to infer mental state, analyze event potentials, and record biofeedback 71, 72, 73. Foreheads were sterilized with saline wipes before gameplay.

4) Neulog GSR logger sensor NUL-217 74: The NUL217 is a GSR logging sensor that measures the conductivity of the skin between the fingers. The logger connects to a USB-200 Module and records GSR in micro Siemens with a 10nS resolution at a max sample rate of 100Hz. The two finger electrodes were sterilized with saline wipes before gameplay.

5) Polar OH1 - optical heart rate sensor 75: the Polar OH1 is a 6 LED optical heart rate sensor that is used with an armband to record beats per minute through Bluetooth at 1Hz sampling frequency.

These devices are easy to set up for a user at home and are a more affordable solution compared to clinical grade sensors; i.e. these biofeedback sensors do not utilize single use components, unlike more conventional systems that may use EEG gels or sticktrodes.

  • Figure 1. OpenButterfly Protocol & Data Pipeline Illustration for both Pilot [A] and Revised [B] Studies. OpenButterfly Protocol indicates the general outline for each experimental session. As shown in Gameplay and Data Extraction, the HR, GSR, and EEG were independently collected for a baseline, and then collected with game data, motion capture, and video capture during gameplay. Our survey was administered at the end of each session. After each session, we compiled all the data files through synchronization achieved via Python. MATLAB R2018B [76] was used to run statistical analysis on biometric data, and OpenSim [31] utilized the tracking data to calculate shoulder joint kinematics and dynamics
2.2. Game Mechanics

Our game, titled “OpenButterfly”, consists of a virtual butterfly that moves within reach of the participant to guide the user through their required movements. OpenButterfly gameplay can be seen in Figure 2. It is an adaption of Project Butterfly by Elor et al, previously designed for upper-extremity impairments for older adults while using a soft robotic wearable 67. Specific new contributions to the OpenButterfly software includes a new system that records and prescribes custom exercise movements, runs automated repetitive personalized exercise sets with individual users, and provides increased stimuli for movement guidance. These contributions were designed through feedback sessions with collaborating physical therapists across Santa Cruz, California.

To guide movements, projectile crystals emanate from a 15m distance and move on a collision path with the butterfly. Users were informed that the goal of the game is to protect the butterfly from these crystals. The player holds an orb in their hand that they can place over the butterfly to protect it. The crystals explode when they hit the orb, letting the player know they successfully protected the butterfly and earned a point.

For prescribing custom movements, the path of the butterfly can be predetermined and set using a simple interface. The therapist can enter the ”Path Development” game mode, where they see the butterfly in their hand. When the trigger is pressed, they can move the butterfly in any path they desire at any speed. These movements can be saved and accessed through internal comma-separated value files. Through these movement files, the butterfly will follow recorded exercises that are automatically normalized to each user’s arm length, target arm, and prescribed speed of movement. These implementations were done through utilizing the Unity3D Game Engine’s Microsoft .NET File I/O C# Libraries 77.

Our study examined this new game mode by recording and prescribing seven new exercises in collaboration with therapists. The distance of the butterfly from the user is scaled based on arm length, which was measured for each participant at the beginning of the study using the relative position of the game controllers to the headset. Thus, the game automatically scales exercise paths to the user’s arm length -a feature requested by our collaborating physical therapists. Such game paths can be seen in Figure 3. These changes were done as a means for therapists to adjust the game to fit their users’ needs and to enable dynamic customization and calibration. While some of the game’s stimuli have been explored by Elor et al through Project Butterfly, OpenButterfly will be one of the first studies to examine these new exercise features when applied to iVR therapy over the course of eight weeks.

2.3. User Feedback

At the end of each session, participants were asked several Likert scale questions about their iVR experience that day. These questions were taken from a Jennett et al. survey for immersion in games 78 and was modified to focus more on user engagement. Such survey questions can be seen in Table 3. The surveys were utilized to track self-perception for users at the end of each session. These questions were used to evaluate if the users would remain engaged, entertained, and immersed over the eight-week period. Additionally, an exit interview was conducted at the end of the eight weeks to determine what modifications users would want to help improve rehabilitation. This enabled us to establish a mixed- method approach of gameplay, biomechanical, biometric, and survey responses for OpenButterfly.

2.4. Data Processing

Each of the biosensors, the OptiTrack motion capture system, and HTC Vive produced their own output files with their respective recording frequency. Approximately 1,200 data files were produced during the Pilot and Revised Study. Python 79 scripts were written to structure the file management system and then sync all the files for each user session. OpenSim 31 simulations were ran using the motion tracking files from the OptiTrack system, generating approximately another 1,100 files. Statistical analysis was then conducted on the data files using MATLAB 76. The full pipeline for collecting and processing our data can be seen in Figure 1.

3. Pilot Study

The goal of the pilot study was for participants to perform common daily movements with an incremental and gradual amount of weight increase on their arms. The movements chosen were Forward Arm Raise (FAR), Side Arm Raise (SAR), and Horizontal Abduction (HA). These movements are simple single plain movements. We were careful to start with low-intensity movements to ensure participant’s safety.

Additionally, we hypothesized that participants would initially be unable to have full ROM through each exercise but would be able to progress to full ROM with a small amount of wrist weight by the end of the pilot study.

3.1. Protocol of the Pilot Study

The pilot study was performed for the first four weeks.

Each week consisted of two sessions where users performed 30-45 minutes of exercise (time includes rest). During each session, exercises were performed in the following order: FAR, SAR, and HA in order for a total of three rounds. The first round was played without weight for a warmup, and the subsequent two rounds were performed with the appropriate weight per user. Additionally, users had a 90-second rest between exercises, and each exercise was performed for 60 seconds at ten repetitions per minute. Aspects of this protocol are highlighted in Figure 1.

Full ROM for these movements was attained before adding weight to the user’s wrist. The weight was added in small increments to elicit a strength progression, and users’ average weight per session can be seen in Table 1. Weight was only increased when participants could comfortably perform two consecutive rounds of all three exercises for a given weight. To account for the participant’s responses being influenced by the novelty of the VR game and or headset, an initial session was performed to introduce the game mechanics and enable the participants to be familiar with the OpenButterfly environment and movements.

3.2. Results of the Pilot Study

The averages of collected data can be seen in Table 1. The most prominent observation was that all users were able to complete the entire ROM of each exercise quickly, indicated by the high compliance rate, which allowed us to begin using weight early on in the study. To understand user engagement, effort, and immersion we employed a modified survey from Jennet et al. 78. Table 3 shows the questionnaire asked at the end of each user testing session. Generally, users agreed that the game was engaging; they put a lot of effort into participating and felt immersed throughout gameplay sessions.

  • Table 1. OpenButterfly Protocol Pilot Study [A] And Revised Study [B] Session Averages Between All Users (session number is indicated after A or B). Parenthesis Indicates Standard Deviation. Exclamation Mark Indicates Resting-State Change (Note All Biometric Measurements Indicate Change Induced From Gameplay Compared To Baseline Measurements)

  • Table 2. Wilcoxon Significance For Pilot Study [A] Vs. Revised Study [B] Results. The Protocols Were Found To Be Significantly Different From Each Other At 95% Confidence In All Data Categories. “Sig” Indicates The Significance Level. Superscript (A) Indicates Resting-State Change (Note All Biometric Measurements Indicate Change Induced From Gameplay Compared To Baseline Measurements). Bolded Values Indicate Significant At 95% Confidence From Wilcoxon Testing. Pilot Study Indicates Higher Game Performance As Well As GSR And EEG. Revised Study Shows Higher EEG Performance As Well As Blinks And Jaw Clenches. Note That Na And Nb Is The Total Number Of Samples Found By Number Of Sessions Number Of Users Number Of Exercises

  • Table 3. OpenButterfly Survey Table. Results Without Asterisks Are In Likert Type Scale Where One Indicates Strongly Disagree And 5 Indicates Strongly Agree. “Sig” Indicates Wilcoxon Significance Level. Superscripts Indicate: (A) Scale Of “Not At All” To “A Lot”, (B) Scale Of “Very Poor” To “Very Well”, (C) Ten-Point Likert Scale For “Not At All” To “A Lot”, (D) Indicates It Was A Reverse Question And The Response Average Is Represented In The Inverse To Keep All Values On The Same Scale

All elements of player behavior and biometric events (with the exception of user jaw clenches) were found to be significant, as seen in Table 1 A-s1 to A-s5. Users were able to acclimate to a 100% increase in weight resistance while moving their weak arm at a total average of 30m of distance per session. Additionally, users were able to successfully protect the butterfly at a compliance rate of a mean 96%, where compliance is defined as the time protecting the butterfly divided by the total time of the exercise session. In terms of both compliance and arm travel distance, these results held a low range of standard deviation, indicating that user performance was fairly constant between all users for these sessions. From the biometric data starting at a resting heart rate, the exercise sessions induced an average mean increase of 11.78 beats per minute, indicating increased physiological intensity from the exercises (shown in Table 2 [A]). Galvanic skin response was also found to be at a positive increase for all pilot protocol sessions, with a mean 1.47 micro Siemens resting-state change indicating arousal from gameplay, as shown in Table 2. For brainwave activity, the pilot protocol generally held a mean increase of all wavebands for alpha, beta, delta, theta, and gamma powers -this may confirm that users were well mentally stimulated and physiologically challenged during gameplay.

3.3. Influence on the Revised Study

One thing that we learned from the pilot study is that the exercises of the Pilot Study were effective in increasing general strength, as can be seen in Table 1 where average weight between each session increases consistently. However, our users had a more substantial initial ROM than we anticipated. For our revised study, we needed to help our users progress more in ROM than the exercises in the Pilot Study required.

We performed ROM expansion by adding two common adduction/abduction movements: External Rotation (EXR) and Abducted Rotation (ABR), as well as two multiplanar movements: Mixed Press (MXDPR) and MixedCircles (MXDCR). These movements can be seen in Figure 4 in the green region. Since these new movements focused more on stretching, no weight was used while performing these four movements.

  • Figure 4. OpenButterfly Movements and OpenSim Outputs are shown above. The pilot study included the movements with the red background, while the revised study included both the red and green background movements. The movements are: FAR = Forward Arm Raise, SAR = Side Arm Raise, HA = Horizontal Abduction, EXR = External Rotation, ABR = Abducted Rotation, MXDPR = Mixed Press, and MXDCR = Mixed Circles. The white dotted line shows the path the butterfly traveled for each movement. On the graphs, the blue line shows the relevant angular displacement of the shoulder, and the red shows the torque placed on the shoulder throughout the movement

In the Pilot Study, the first round was always played without weight as a warmup. Since the new stretching games were played without weight, we decided to do two rounds first of the non-weighted movements followed by two rounds of the weighted movements. This is further explained in the Revised Protocol Section.

4. Revised Study

Learning from the pilot study results, we modified our game to have a more appropriate protocol for our users. To address insufficient ROM exercises, new exercises were created, as shown in Figure 4 (EXR, ABR, MXDPR, MXDCR). These movements require a greater ROM at different angles than the pilot games. FAR, SAR, and HA games were kept to specifically address increases in strength by still utilizing the wrist weight progression protocol.

4.1. Revised Protocol

The revised study lasted four weeks, with twice a week session consisting of 30-45 minutes of exercise. During each session, two rounds of EXR, ABR, MXDPR, and MXDCR were performed with a one-minute rest between each exercise. Each movement was performed for 60 seconds at a slow tempo to allow for stretching at the limit of each subject’s ROM. These exercises were always performed without any wrist weight as stretching was the goal, not strength.

Afterward, two rounds of FAR, SAR, and HA were performed with a one-minute rest between each exercise. This followed the same weight increase protocol as the pilot study to ensure a safe progression in strength exercises.

4.2. General Results of the Revised Study

From the revised protocol, users engaged in greater weight resistance than the Pilot Study [A], and subsequently, there was far more variability between users. Arm travel distance was less in total, but the movements were far more complex and slower. HR and GSR were found to be less than the Pilot Study’s [A] mean resting state change but still elevated by nearly 10bpm and 1.13 micro Siemens, respectively. The lowered heart rate may be an artifact of the slower tempo in movement, and declining GSR may further indicate acclimation to the game with a lowering rate of arousal (however, it was still elevated far above resting state). Table 2 lists these results. As this table shows, the Revised Study [B] results were significantly different from the Pilot Study’s results [A] for all data sets. Specifically, the Pilot Results [A] had a greater compliance rate, weak arm movement, HR change, and GSR change. In contrast, the revised study’s results show higher levels of brain activity for all wavebands as well as Blinks and Jaw Clenches. This may indicate that the Revised Study was more mentally stimulating while both increasing weight resistance and game compliance, as shown in Table 1.

4.3. Biomechanical Performance

Using OpenSim with the motion tracking data from each session, we were able to determine the amount of torque placed on the shoulder for each exercise, as shown in Figure 5. We took the integration of torque with respect to time to determine the amount of angular momentum the shoulder generated from exercise.

While the average torque for each user dropped between the Pilot Study [A] and the Revised Study [B], the average angular momentum for each user increased between the studies, as shown in Table 2. This decrease in mean torques is a result of more exercises being performed without weight. For example, 8 of the 14 exercises in the Revised Study [B] were played without weight, whereas only 3 out of the 9 exercises in the Pilot Study [A] were without weight. The increase in average angular momentum between the Pilot Study [A] and the Revised Study [B] occurs because more games are played in the Revised Study [B]. Figure 5 shows each user’s average torque and angular momentum for every session.

We expected to see more of a steady increase for torque and angular momentum in each study since the users were lifting the same or more weight than their previous session. However, we observed that users’ average torque and angular momentum fluctuated a bit from session to session in the same protocol rather than continuously increasing. We believe this is important to show as we can see which sessions the users were not performing as expected. Users told us that some days they would come in more stiff or sore than other days. This data may be indicative of these cases and theoretically could be used to help adjust the exercise protocol. These results may be helpful for other researchers interested in expanding upon this work for a more personalized and reactive therapy regime. Understanding day to day fluctuation through this data could lead to new algorithms for a more customizable rehabilitation through adapting to users’ capabilities. Even if users decline or have a setback (i.e., sore from previous day’s activities such as yard work), these insights could be used to tailor the difficulty to the most important muscle groups for maximizing therapeutic benefit.

From this data, we can also see the most significant changes in torque and angular momentum come from the different amount of games played; in short, the more games played, the more angular momentum was gained. This is useful when deciding a rehabilitation plan as the variables to consider are weight resistance or total movement. So, by utilizing the angular momentum, we can determine the exact amount of motion of a specific game, no matter the movement path of the arm. Such an analysis of angular momentum could be used to build a more intelligent progression.

4.4. Qualitative Performance

A significant difference between our two studies was found from Wilcoxon significance testing at 95% certainty for the qualitative user surveys, as shown in Table 3. Users reported the Revised Study [B] to be significantly more challenging ([Q7]), progressive ([Q10, Q11]), and liked ([Q1]) the protocol more than the Pilot Study [A]. Conversely, users reported that they felt the revised protocol provided significantly less distraction ([Q4]) from their real-life compared to the pilot protocol. These results may indicate that the protocol choices for the Revised Protocol [B] were successful in challenging each user from an engagement and effort perspective.

5. Discussion

From our multimodal analysis of our eight-week study, we show that OpenButterfly accomplishes our goals of increasing ROM and increasing strength. This was a multi-step process that required two stages to adjust and further customize the game to the users’ capabilities as rehabilitation necessitates. The Pilot Study [A] was useful to help determine the capabilities of our users and how to set achievable goals for them. This stage showed that the users achieved a full ROM for the first three exercises (FAR, SAR, HA) and were ready to start training with weights very quickly. Starting with simple movements was a safeguard against exercises that were too advanced for their state of recovery. The insights from the Pilot Study enabled us to create more complex movements and continue to work on strength.

For the Revised Study [B], we created four new movements (EXR, ABR, MXDPR, and MXDCR) that were performed without weight to target enhanced ROM. At the same time, the original exercises from the Pilot Study [A] were carried over to focus on strength building. We saw ROM increased to meet these challenging and further-reaching movements indicated by the increase in the rate of the compliance recovery during the Revised Study [B], as shown in Table 1. Also, in Table 1, results from average weight indicated a successful increase in strength. The Revised Study [B] displayed an improvement where more exercise difficulty was leveraged to safely challenge the users. Users also enjoyed the new exercises and stated it was similar to ”unlocking a new level in a game.” This is a consideration as we move forward: making levels that are of different movements so that the game remains challenging and does not become repetitive.

Additionally, the users’ physiological recordings and self-reported responses indicated that users were able to remain engaged with the game beyond the novelty effect period for the course of the eight weeks, as seen in Table 1 and Table 2. Consequently, technology like OpenButterfly may become a promising tool for addressing the problem of adherence to a rehabilitation program. The more enjoyable and engaging the program, the more likely users will continue the program. This adherence with OpenButterfly is particularly exciting, as other researchers may be able to utilize similar iVR physical therapy experiences for long-term treatment.

5.1. Contributions from OpenButterfly

Through our study, we believe several insights can be useful to game developers and researchers. First, we learned that the ability to easily and quickly create custom paths for arm movements during gameplay allowed us to efficiently adjust our games between the Pilot Study [A] and the Revised Study [B]. This adjustment needed to occur because all users were able to complete a full ROM with added weight within a few sessions. The Pilot Study [A] exercises proved valuable as a baseline ROM and for improving strength. The Revised Study [B] had more complex movements targeting ROM and kept the now proven original exercises for targeting strength. These game modifications were guided by our collaborating physical therapists to increase the difficulty of an appropriate progression in strength and ROM. The ability to record custom motion paths and normalize movements to each user’s arm length and height proved to be a valuable tool.

Another useful tool was the biomechanical simulation, as it offered more in-depth analytics into user performance through analyzing performance during a session. In traditional PT, the therapist can monitor progression through measurements of ROM and strength, typically pounds lifted or level of a resistance band. Through our study with OpenSim, we are able to provide this data and, in the future hope to have everything streamlined so that no matter the movements performed, simple or complex, we can provide a thorough representation of the amount force placed on the working joint for a therapist to examine. With further user testing, perhaps researchers can build more sophisticated models for simulation that will provide individual muscle for training and rehabilitation for any of the movements performed in the game. This can help the therapist target specific muscles to aid in a focused recovery.

Additionally, biometric data from OpenButterfly may help researchers understand users’ physiological responses to the iVR experience. This helps with recovery as more enjoyable user experience is likely to lead to better adoption of a rehabilitation program. Our data indicates users had higher brain activity for the Revised Session [B], which should be further explored in considering rehabilitation monitoring and game adaption for future studies. These metrics also provide a possibility for determining how much effort the user is putting into the game on a physiological level. Since this isn’t a strenuous workout, we want to make sure users are working at an appropriate level. We learned that the levels we chose were enough to elicit a strength increase response, but not so much that user is at risk of injuring themselves.

We believe our study has shown this game’s feasibility for helping with the recovery process, and fellow researchers, developers, engineers, and therapists may find aspects of our research useful for their endeavors. Collecting HR, GSR, and EEG may provide a deeper understanding of a user’s engagement and physical effort with iVR exercises. This can help with game development in creating exciting experiences to help increase a user’s desire to play the game. Biomechanical simulation can provide valuable metrics to a monitoring therapist and also give a progress log over an extended period of time. OpenButterfly itself shows that other games can be created to aid with recovery, and we suggest from guidance with our collaborating therapists that in future games, there is a way for a therapist to dictate the movements of the game easily, so it is customizable to the user’s needs.

5.2. Study Limitations

There are several limitations that may impact generalizability of the results. The study examined 5 users with OpenButterfly, but in the future iterations we plan to recruit more users. While the study lasted 8 weeks it would be helpful to understand more long-term effect by conducting the study for 12-16 weeks since our goal is adherence of users. Additionally, we limited the frequency to 2 sessions per week to ensure adequate time for recovery, but having the users progress to 3 and 4 times per week could yield better benefits. While our long-term goal is at-home use, we conducted our study in a lab to examine performance and user experience of our system. The next step would be applying what we learned from our system feasibility study and conduct an at-home user study.

6. Conclusions and Future Work

The purpose of OpenButterfly was to create an effective and feasible iVR physical therapy game to help users with shoulder injuries through multimodal rehabilitation analysis. This was accomplished by working with therapists and enabling game recordings to mimic movements found in physical therapy targeted at ROM and strength training. Through OpenButterfly, we present a novel study that is a long-term, customizable highly immersive virtual reality game for shoulder rehabilitation that analyzes physiological response and uses biomechanical simulation to identify the joint kinetics and dynamics. Working with therapists, we have identified useful tools and data sensors to aid in developing games targeted at recovery that we believe other serious game researchers will find helpful. This multimodal rehabilitation analysis will help with the next iteration of our game to ensure user engagement and that users are working at an appropriate threshold, not too intense to risk injury but difficult enough to elicit a physiological adaptation.

We explored two experimental studies: a Pilot Study consisting of three single everyday movements targeted at a basic ROM and strength, and a Revised Study that incorporated four new movements aimed at ROM from insights gained from the multimodal rehabilitation analysis of the Pilot Study. Our results indicate that users were able to overcome the novelty effect of iVR through extended exposure to gameplay over eight weeks. We were also able to measure heart rate, galvanic skin response, and electroencephalography while our users played the game, allowing us to understand their physical strain and emotional response while playing. With the motion capture data, we were able to determine the kinematics and dynamics of the shoulder during gameplay through biomechanical simulation. We believe this data would be useful for physical therapists as it helps quantify the forces of the joint for an entire session and would provide a method for remote therapists to quickly understand users’ exercise session.

In the future, we plan to explore the design of new levels within the game that contain more complex and less predictable movements to challenge users physically and mentally. Our long-term goal is to develop an at-home recovery game that is capable of providing meaningful game data remotely to the therapist. Subsequently, we plan to explore a more complex biomechanics model capable of identifying individual muscle force contribution to movements. The incorporation of runtime biomechanical models to identify muscle weaknesses may further aid in custom movements for an individual user to help maximize their recovery by ensuring the targeted muscles are being used for a given movement. We hope to deploy this system for at-home use to make OpenButterfly more accessible for users in need. To this end, there are more butterflies to protect as we continue working with therapists to modify the game and models.

Acknowledgements

The authors thank the student participants that helped make this study possible. The authors would like to thank Dana Seth, PT, DPT, for her guidance on developing progression movements for the game to ensure user safety, and Rebecca Friedman, DSC, for her advice and suggestions for therapist needs. Additionally, the following UC Santa Cruz students are acknowledged for their support: Teresa Faasolo, Evanjelin Mahmoodi, and Conrad Esch.

References

[1]  C. H. Linaker and K. Walker-Bone, “Shoulder disorders and occupation,” Best practice & research Clinical rheumatology, vol. 29, no. 3, pp. 405-423, 2015.
In article      View Article  PubMed
 
[2]  J. Perry, “Normal upper extremity kinesiology,” Physical therapy, vol. 58, no. 3, pp. 265-278, 1978.
In article      View Article  PubMed
 
[3]  C. Sicuri, G. Porcellini, and G. Merolla, “Robotics in shoulder rehabilitation,” Muscles, ligaments and tendons journal, vol. 4, no. 2, p. 207, 2014.
In article      View Article
 
[4]  N. D. Boardman III, R. H. Cofield, K. A. Bengtson, R. Little, M. C. Jones, and C. M. Rowland, “Rehabilitation after total shoulder arthroplasty,” The Journal of arthroplasty, vol. 16, no. 4, pp. 483-486, 2001.
In article      View Article  PubMed
 
[5]  J. J. Brems, “Rehabilitation following total shoulder arthroplasty.” Clinical orthopedics and related research, no. 307, pp. 70-85, 1994.
In article      
 
[6]  D. D. Brown and R. J. Friedman, “Postoperative rehabilitation following total shoulder arthroplasty,” Orthopedic Clinics of North America, vol. 29, no. 3, pp. 535-547, 1998.
In article      View Article
 
[7]  M. Hughes and C. S. Neer, “Glenohumeral joint replacement and postoperative rehabilitation,” Physical therapy, vol. 55, no. 8, pp. 850-858, 1975.
In article      View Article  PubMed
 
[8]  S. Jackins, “Postoperative shoulder rehabilitation.” Physical medicine and rehabilitation clinics of North America, vol. 15, no. 3, pp. vi-643, 2004.
In article      View Article  PubMed
 
[9]  R. Argent, A. Daly, and B. Caulfield, “Patient involvement with homebased exercise programs: can connected health interventions influence adherence?” JMIR mHealth and uHealth, vol. 6, no. 3, p. e47, 2018.
In article      View Article  PubMed
 
[10]  S. Bassett, “Measuring patient adherence to physiotherapy,” J Nov Physiother, vol. 2, no. 07, pp. 60-66, 2012.
In article      View Article
 
[11]  E. M. Sluijs, G. J. Kok, and J. Van der Zee, “Correlates of exercise compliance in physical therapy,” Physical therapy, vol. 73, no. 11, pp. 771-782, 1993.
In article      View Article  PubMed
 
[12]  H. Mousavi Hondori and M. Khademi, “A review on technical and clinical impact of Microsoft Kinect on physical therapy and rehabilitation,” Journal of medical engineering, vol. 2014, 2014.
In article      View Article  PubMed
 
[13]  G. C. Burdea, “Virtual rehabilitation-benefits and challenges,” Methods of information in medicine, vol. 42, no. 05, pp. 519-523, 2003.
In article      View Article  PubMed
 
[14]  C. for Disease Control, Prevention et al., “Brfss survey data and documentation 2017,” 2019.
In article      
 
[15]  D. Corbetta, F. Imeri, and R. Gatti, “Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review,” Journal of physiotherapy, vol. 61, no. 3, pp. 117-124, 2015.
In article      View Article  PubMed
 
[16]  A. Elor, M. Teodorescu, and S. Kurniawan, “Project star catcher: A novel immersive virtual reality experience for upper limb rehabilitation,” ACM Transactions on Accessible Computing (TACCESS), vol. 11, no. 4, p. 20, 2018.
In article      View Article
 
[17]  A. Elor, M. Powell, E. Mahmoodi, N. Hawthorne, M. Teodorescu, and S. Kurniawan, “On shooting stars: Comparing cave and hmd immersive virtual reality exergaming for adults with mixed ability,” ACM Transactions on Computing for Healthcare.
In article      
 
[18]  H. G. Hoffman, W. J. Meyer III, M. Ramirez, L. Roberts, E. J. Seibel, B. Atzori, S. R. Sharar, and D. R. Patterson, “Feasibility of articulated arm mounted oculus rift virtual reality goggles for adjunctive pain control during occupational therapy in pediatric burn patients,” Cyberpsychology, Behavior, and Social Networking, vol. 17, no. 6, pp. 397-401, 2014.
In article      View Article  PubMed
 
[19]  H. G. Hoffman, G. T. Chambers, W. J. Meyer, L. L. Arceneaux, W. J. Russell, E. J. Seibel, T. L. Richards, S. R. Sharar, and D. R. Patterson, “Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures,” Annals of Behavioral Medicine, vol. 41, no. 2, pp. 183-191, 2011.
In article      View Article  PubMed
 
[20]  P. J. Standen and D. J. Brown, “Virtual reality in the rehabilitation of people with intellectual disabilities,” Cyberpsychology & behavior, vol. 8, no. 3, pp. 272-282, 2005.
In article      View Article  PubMed
 
[21]  J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and A. Muhlberger, “The impact of perception and presence on emotional¨ reactions: a review of research in virtual reality,” Frontiers in psychology, vol. 6, 2015.
In article      View Article  PubMed
 
[22]  P. J. Costello, Health and safety issues associated with virtual reality: a review of current literature. Advisory Group on Computer Graphics, 1997.
In article      
 
[23]  D. E. Levac, S. M. Glegg, H. Sveistrup, H. Colquhoun, P. Miller, H. Finestone, V. DePaul, J. E. Harris, and D. Velikonja, “Promoting therapists use of motor learning strategies within virtual reality-based stroke rehabilitation,” PloS one, vol. 11, no. 12, p. e0168311, 2016.
In article      View Article  PubMed
 
[24]  M. Beccue and C. Wheelock, “Research report: Virtual reality for consumer markets,” Tractica Research, Tech. Rep., Q4 2016. [Online]. Available: https://www.tractica.com/research/virtual-reality-for-consumer-markets/.
In article      
 
[25]  A. Elor, S. Kurniawan, and M. Teodorescu, “Towards an immersive virtual reality game for smarter post-stroke rehabilitation,” in 2018 IEEE International Conference on Smart Computing (SMARTCOMP). IEEE, 2018, pp. 219-225.
In article      View Article
 
[26]  G. Riva, C. Botella, P. Legeron, and G. Optale, “13 immersive virtual´ telepresence: Virtual reality meets ehealth,” 2004.
In article      
 
[27]  P. Lindner, A. Miloff, S. Fagernas, J. Andersen, M. Sigeman, G. An-¨ dersson, T. Furmark, and P. Carlbring, “Therapist-led and self-led onesession virtual reality exposure therapy for public speaking anxiety with consumer hardware and software: A randomized controlled trial,” Journal of anxiety disorders, vol. 61, pp. 45-54, 2019.
In article      View Article  PubMed
 
[28]  A. Won, J. Bailey, J. Bailenson, C. Tataru, I. Yoon, and B. Golianu, “Immersive virtual reality for pediatric pain,” Children, vol. 4, no. 7, p. 52, 2017.
In article      View Article  PubMed
 
[29]  L. Li, F. Yu, D. Shi, J. Shi, Z. Tian, J. Yang, X. Wang, and Q. Jiang, “Application of virtual reality technology in clinical medicine,” American journal of translational research, vol. 9, no. 9, p. 3867, 2017.
In article      
 
[30]  K. E. Laver, B. Lange, S. George, J. E. Deutsch, G. Saposnik, and M. Crotty, “Virtual reality for stroke rehabilitation,” Cochrane database of systematic reviews, no. 11, 2017.
In article      View Article  PubMed
 
[31]  S. L. Delp, F. C. Anderson, A. S. Arnold, P. Loan, A. Habib, C. T. John, E. Guendelman, and D. G. Thelen, “Opensim: open-source software to create and analyze dynamic simulations of movement,” IEEE transactions on biomedical engineering, vol. 54, no. 11, pp. 1940-1950, 2007.
In article      View Article  PubMed
 
[32]  A. Rajagopal, C. L. Dembia, M. S. DeMers, D. D. Delp, J. L. Hicks, and S. L. Delp, “Full-body musculoskeletal model for muscle-driven simulation of human gait,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 10, pp. 2068-2079, 2016.
In article      View Article  PubMed
 
[33]  E. M. Arnold, S. R. Hamner, A. Seth, M. Millard, and S. L. Delp, “How muscle fiber lengths and velocities affect muscle force generation as humans walk and run at different speeds,” Journal of Experimental Biology, vol. 216, no. 11, pp. 2150-2160, 2013.
In article      View Article  PubMed
 
[34]  T. K. Uchida, J. L. Hicks, C. L. Dembia, and S. L. Delp, “Stretching your energetic budget: how tendon compliance affects the metabolic cost of running,” PloS one, vol. 11, no. 3, p. e0150378, 2016.
In article      View Article  PubMed
 
[35]  H. Baskar and S. M. R. Nadaradjane, “Minimization of metabolic cost of muscles based on human exoskeleton modeling: a simulation,” Int. J. Biomed. Eng. Sci, vol. 3, no. 4, p. 9, 2016.
In article      View Article
 
[36]  B. J. de Kruif, E. Schmidhauser, K. S. Stadler, and L. W. O’Sullivan, “Simulation architecture for modelling interaction between user and elbow-articulated exoskeleton,” Journal of Bionic Engineering, vol. 14, no. 4, pp. 706-715, 2017.
In article      View Article
 
[37]  G. S. Virk, U. Haider, I. Nyoman, N. Masud, I. Mamaev, P. Hopfgarten, and B. Hein, “Design of exo-legs exoskeletons,” in ASSISTIVE ROBOTICS: Proceedings of the 18th International Conference on CLAWAR 2015. World Scientific, 2016, pp. 59-66.
In article      View Article
 
[38]  D. S. Pina, A. A. Fernandes, R. N. Jorge, and J. Gabriel, “Designing the mechanical frame of an active exoskeleton for gait assistance,” Advances in Mechanical Engineering, vol. 10, no. 2, p. 1687814017743664, 2018.
In article      View Article
 
[39]  J. A. Dienes, X. Hu, K. D. Janson, C. Slater, E. A. Dooley, G. J. Christ, and S. D. Russell, “Analysis and modeling of rat gait biomechanical deficits in response to volumetric muscle loss injury,” Frontiers in Bioengineering and Biotechnology, vol. 7, p. 146, 2019.
In article      View Article  PubMed
 
[40]  M. Marieswaran, A. Sikidar, A. Goel, D. Joshi, and D. Kalyanasundaram, “An extended opensim knee model for analysis of strains of connective tissues,” Biomedical engineering online, vol. 17, no. 1, p. 42, 2018.
In article      View Article  PubMed
 
[41]  Z. F. Lerner, B. C. Gadomski, A. K. Ipson, K. K. Haussler, C. M. Puttlitz, and R. C. Browning, “Modulating tibiofemoral contact force in the sheep hind limb via treadmill walking: Predictions from an opensim musculoskeletal model,” Journal of Orthopaedic Research, vol. 33, no. 8, pp. 1128-1133, 2015.
In article      View Article  PubMed
 
[42]  J. W. Rankin, J. Rubenson, and J. R. Hutchinson, “Inferring muscle functional roles of the ostrich pelvic limb during walking and running using computer optimization,” Journal of the Royal Society Interface, vol. 13, no. 118, p. 20160035, 2016.
In article      View Article  PubMed
 
[43]  J. M. Burnfield, K. R. Josephson, C. M. Powers, and L. Z. Rubenstein, “The influence of lower extremity joint torque on gait characteristics in elderly men,” Archives of physical medicine and rehabilitation, vol. 81, no. 9, pp. 1153-1157, 2000.
In article      View Article  PubMed
 
[44]  L. Ballaz, M. Raison, C. Detrembleur, G. Gaudet, and M. Lemay, “Joint torque variability and repeatability during cyclic flexion-extension of the elbow,” BMC sports science, medicine and rehabilitation, vol. 8, no. 1, p. 8, 2016.
In article      View Article  PubMed
 
[45]  A. K. Gillawat and H. J. Nagarsheth, “Human upper limb joint torque minimization using genetic algorithm,” in Recent Advances in Mechanical Engineering. Springer, 2020, pp. 57-70.
In article      View Article
 
[46]  K. Kiguchi and Y. Hayashi, “An emg-based control for an upper-limb power-assist exoskeleton robot,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 42, no. 4, pp. 1064-1071, 2012.
In article      View Article  PubMed
 
[47]  D. H. Perrin, R. J. Robertson, and R. L. Ray, “Bilateral isokinetic peak torque, torque acceleration energy, power, and work relationships in athletes and nonathletes,” Journal of Orthopaedic & Sports Physical Therapy, vol. 9, no. 5, pp. 184-189, 1987.
In article      View Article  PubMed
 
[48]  J. Hamill and K. M. Knutzen, Biomechanical basis of human movement. Lippincott Williams & Wilkins, 2006.
In article      
 
[49]  M. T. Farrell and H. Herr, “Angular momentum primitives for human turning: Control implications for biped robots,” in Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots. IEEE, 2008, pp. 163-167.
In article      View Article
 
[50]  S. M. Bruijn, P. Meyns, I. Jonkers, D. Kaat, and J. Duysens, “Control of angular momentum during walking in children with cerebral palsy,” Research in developmental disabilities, vol. 32, no. 6, pp. 2860-2866, 2011.
In article      View Article  PubMed
 
[51]  C. Nott, R. R. Neptune, and S. Kautz, “Relationships between frontalplane angular momentum and clinical balance measures during poststroke hemiparetic walking,” Gait & posture, vol. 39, no. 1, pp. 129-134, 2014.
In article      View Article  PubMed
 
[52]  R. R. Neptune and C. P. McGowan, “Muscle contributions to wholebody sagittal plane angular momentum during walking,” Journal of biomechanics, vol. 44, no. 1, pp. 6-12, 2011.
In article      View Article  PubMed
 
[53]  J. F. Lubar, “Discourse on the development of eeg diagnostics and biofeedback for attention-deficit/hyperactivity disorders,” Biofeedback and Self-regulation, vol. 16, no. 3, pp. 201-225, 1991.
In article      View Article  PubMed
 
[54]  J. Lubar, M. Swartwood, J. Swartwood, and D. Timmermann, “Quantitative eeg and auditory event-related potentials in the evaluation of attention-deficit/hyperactivity disorder: Effects of methylphenidate and implications for neurofeedback training,” Journal of Psychoeducational Assessment, vol. 34, pp. 143-160, 1995.
In article      
 
[55]  G. Deuschl, A. Eisen et al., “Recommendations for the practice of clinical neurophysiology: guidelines of the international federation of clinical neurophysiology,” 1999.
In article      
 
[56]  M. Lotan, S. Yalon-Chamovitz, and P. L. T. Weiss, “Improving physical fitness of individuals with intellectual and developmental disability through a virtual reality intervention program,” Research in developmental disabilities, vol. 30, no. 2, pp. 229-239, 2009.
In article      View Article  PubMed
 
[57]  H. D. Critchley, “Electrodermal responses: what happens in the brain,” The Neuroscientist, vol. 8, no. 2, pp. 132-142, 2002.
In article      View Article  PubMed
 
[58]  W. Boucsein, Electrodermal activity. Springer Science & Business Media, 2012.
In article      View Article
 
[59]  R. Ramirez and Z. Vamvakousis, “Detecting emotion from eeg signals using the emotive epoc device,” in International Conference on Brain Informatics. Springer, 2012, pp. 175-184.
In article      View Article
 
[60]  V. N. Salimpoor, M. Benovoy, G. Longo, J. R. Cooperstock, and R. J. Zatorre, “The rewarding aspects of music listening are related to degree of emotional arousal,” PloS one, vol. 4, no. 10, p. e7487, 2009.
In article      View Article  PubMed
 
[61]  M. S. Cameirao, I. B. S. Bermudez, E. Duarte Oller, and P. F. Verschure,´ “The rehabilitation gaming system: a review,” Stud Health Technol Inform, vol. 145, no. 6, 2009.
In article      
 
[62]  R. W. Picard, Affective computing. MIT press, 2000.
In article      View Article
 
[63]  D. Novak, J. Ziherl, A. Olensek, M. Milavec, J. Podobnik, M. Mihelj, and M. Munih, “Psychophysiological responses to robotic rehabilitation tasks in stroke,” IEEE Transactions on neural systems and rehabilitation engineering, vol. 18, no. 4, pp. 351-361, 2010.
In article      View Article  PubMed
 
[64]  D. Novak, M. Mihelj, J. Ziherl, A. Olensek, and M. Munih, “Psychophysiological measurements in a biocooperative feedback loop for upper extremity rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 4, pp. 400-410, 2011.
In article      View Article  PubMed
 
[65]  C. R. Guerrero, J. C. F. Marinero, J. P. Turiel, and V. Munoz, “Using hu-˜ man state aware robots to enhance physical human-robot interaction in a cooperative scenario,” Computer methods and programs in biomedicine, vol. 112, no. 2, pp. 250-259, 2013.
In article      View Article  PubMed
 
[66]  A. Koenig, D. Novak, X. Omlin, M. Pulfer, E. Perreault, L. Zimmerli, M. Mihelj, and R. Riener, “Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 4, pp. 453-464, 2011.
In article      View Article  PubMed
 
[67]  A. Elor, S. Lessard, M. Teodorescu, and S. Kurniawan, “Project butterfly: Synergizing immersive virtual reality with actuated soft exosuit for upper-extremity rehabilitation,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 1448-1456.
In article      View Article
 
[68]  I. D. O. NaturalPoint, “Optitrack - industry leading precision motion capture and 3d tracking systems for video game design, animation, virtual reality, robotics, and movement sciences.” O. M. C. Systems, Ed., 23.01.2020. [Online]. Available: https://optitrack.com/.
In article      
 
[69]  InteraXon, “Featured research with muse,” M. Research, Ed., 29.06.2019. [Online]. Available: https://choosemuse.com/muse-research/ ,developer.choosemuse.com/tools/available-data.
In article      
 
[70]  J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297-301, 1965.
In article      View Article
 
[71]  N. Kovacevic, P. Ritter, W. Tays, S. Moreno, and A. R. McIntosh, “‘my virtual dream’ : Collective neurofeedback in an immersive art environment,” PloS one, vol. 10, no. 7, p. e0130129, 2015.
In article      View Article  PubMed
 
[72]  O. E. Krigolson, C. C. Williams, A. Norton, C. D. Hassall, and F. L. Colino, “Choosing muse: Validation of a low-cost, portable eeg system for erp research,” Frontiers in neuroscience, vol. 11, p. 109, 2017.
In article      View Article  PubMed
 
[73]  S. Bhayee, P. Tomaszewski, D. H. Lee, G. Moffat, L. Pino, S. Moreno, and N. A. Farb, “Attentional and affective consequences of technology supported mindfulness training: a randomised, active control, efficacy trial,” BMC psychology, vol. 4, no. 1, p. 60, 2016.
In article      View Article  PubMed
 
[74]  Neulog, “Gsr logger sensor nul-217,” Neulog, Ed., 29.06.2019. [Online]. Available: https://neulog.com/gsr/.
In article      
 
[75]  Polar, “Polar oh1: optical heart rate sensor,” Polar, Ed., 29.06.2019. [Online]. Available: https://www.polar.com/us-en/products/accessories/ oh1-optical-heart-rate-sensor.
In article      
 
[76]  H. Moore, MATLAB for Engineers.Pearson, 2017.
In article      
 
[77]  Unity Technologies, “Unity real-time development platform — 3d, 2d vr ar,” Internet: https://unity.com/ [Jun. 06, 2019], 2019.
In article      
 
[78]  C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, and A. Walton, “Measuring and defining the experience of immersion in games,” International journal of human-computer studies, vol. 66, no. 9, pp. 641-661, 2008.
In article      View Article
 
[79]  G. Rossum, “Python reference manual,” 1995.
In article      
 

Published with license by Science and Education Publishing, Copyright © 2020 Michael Ora Powell, Aviv Elor, Mircea Teodorescu and Sri Kurniawan

Creative CommonsThis work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Cite this article:

Normal Style
Michael Ora Powell, Aviv Elor, Mircea Teodorescu, Sri Kurniawan. OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy. American Journal of Sports Science and Medicine. Vol. 8, No. 1, 2020, pp 23-35. http://pubs.sciepub.com/ajssm/8/1/5
MLA Style
Powell, Michael Ora, et al. "OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy." American Journal of Sports Science and Medicine 8.1 (2020): 23-35.
APA Style
Powell, M. O. , Elor, A. , Teodorescu, M. , & Kurniawan, S. (2020). OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy. American Journal of Sports Science and Medicine, 8(1), 23-35.
Chicago Style
Powell, Michael Ora, Aviv Elor, Mircea Teodorescu, and Sri Kurniawan. "OpenButterfly: Multimodal Rehabilitation Analysis of Immersive Virtual Reality for Physical Therapy." American Journal of Sports Science and Medicine 8, no. 1 (2020): 23-35.
Share
  • Figure 1. OpenButterfly Protocol & Data Pipeline Illustration for both Pilot [A] and Revised [B] Studies. OpenButterfly Protocol indicates the general outline for each experimental session. As shown in Gameplay and Data Extraction, the HR, GSR, and EEG were independently collected for a baseline, and then collected with game data, motion capture, and video capture during gameplay. Our survey was administered at the end of each session. After each session, we compiled all the data files through synchronization achieved via Python. MATLAB R2018B [76] was used to run statistical analysis on biometric data, and OpenSim [31] utilized the tracking data to calculate shoulder joint kinematics and dynamics
  • Figure 2. A view of OpenButterfly gameplay. The protective transparent blue orb is outlined in white. The purple arrow shows the next incoming crystal cluster that heads towards the butterfly. To earn points, users need to place the orb over the butterfly to protect it from the crystal. Each crystal that is blocked earns a point
  • Figure 3. The “Path Development” custom game mode for therapist movement implementation. The right picture showcases a researcher using the iVR control to trace the path of the butterfly. The left picture indicates the movement’s path, traced in red, so the researcher can see where exactly the path is located
  • Figure 4. OpenButterfly Movements and OpenSim Outputs are shown above. The pilot study included the movements with the red background, while the revised study included both the red and green background movements. The movements are: FAR = Forward Arm Raise, SAR = Side Arm Raise, HA = Horizontal Abduction, EXR = External Rotation, ABR = Abducted Rotation, MXDPR = Mixed Press, and MXDCR = Mixed Circles. The white dotted line shows the path the butterfly traveled for each movement. On the graphs, the blue line shows the relevant angular displacement of the shoulder, and the red shows the torque placed on the shoulder throughout the movement
  • Figure 5. Average torque (top) and angular momentum (bottom) for each session is shown with each color representing an individual user
  • Table 1. OpenButterfly Protocol Pilot Study [A] And Revised Study [B] Session Averages Between All Users (session number is indicated after A or B). Parenthesis Indicates Standard Deviation. Exclamation Mark Indicates Resting-State Change (Note All Biometric Measurements Indicate Change Induced From Gameplay Compared To Baseline Measurements)
  • Table 2. Wilcoxon Significance For Pilot Study [A] Vs. Revised Study [B] Results. The Protocols Were Found To Be Significantly Different From Each Other At 95% Confidence In All Data Categories. “Sig” Indicates The Significance Level. Superscript (A) Indicates Resting-State Change (Note All Biometric Measurements Indicate Change Induced From Gameplay Compared To Baseline Measurements). Bolded Values Indicate Significant At 95% Confidence From Wilcoxon Testing. Pilot Study Indicates Higher Game Performance As Well As GSR And EEG. Revised Study Shows Higher EEG Performance As Well As Blinks And Jaw Clenches. Note That Na And Nb Is The Total Number Of Samples Found By Number Of Sessions Number Of Users Number Of Exercises
  • Table 3. OpenButterfly Survey Table. Results Without Asterisks Are In Likert Type Scale Where One Indicates Strongly Disagree And 5 Indicates Strongly Agree. “Sig” Indicates Wilcoxon Significance Level. Superscripts Indicate: (A) Scale Of “Not At All” To “A Lot”, (B) Scale Of “Very Poor” To “Very Well”, (C) Ten-Point Likert Scale For “Not At All” To “A Lot”, (D) Indicates It Was A Reverse Question And The Response Average Is Represented In The Inverse To Keep All Values On The Same Scale
[1]  C. H. Linaker and K. Walker-Bone, “Shoulder disorders and occupation,” Best practice & research Clinical rheumatology, vol. 29, no. 3, pp. 405-423, 2015.
In article      View Article  PubMed
 
[2]  J. Perry, “Normal upper extremity kinesiology,” Physical therapy, vol. 58, no. 3, pp. 265-278, 1978.
In article      View Article  PubMed
 
[3]  C. Sicuri, G. Porcellini, and G. Merolla, “Robotics in shoulder rehabilitation,” Muscles, ligaments and tendons journal, vol. 4, no. 2, p. 207, 2014.
In article      View Article
 
[4]  N. D. Boardman III, R. H. Cofield, K. A. Bengtson, R. Little, M. C. Jones, and C. M. Rowland, “Rehabilitation after total shoulder arthroplasty,” The Journal of arthroplasty, vol. 16, no. 4, pp. 483-486, 2001.
In article      View Article  PubMed
 
[5]  J. J. Brems, “Rehabilitation following total shoulder arthroplasty.” Clinical orthopedics and related research, no. 307, pp. 70-85, 1994.
In article      
 
[6]  D. D. Brown and R. J. Friedman, “Postoperative rehabilitation following total shoulder arthroplasty,” Orthopedic Clinics of North America, vol. 29, no. 3, pp. 535-547, 1998.
In article      View Article
 
[7]  M. Hughes and C. S. Neer, “Glenohumeral joint replacement and postoperative rehabilitation,” Physical therapy, vol. 55, no. 8, pp. 850-858, 1975.
In article      View Article  PubMed
 
[8]  S. Jackins, “Postoperative shoulder rehabilitation.” Physical medicine and rehabilitation clinics of North America, vol. 15, no. 3, pp. vi-643, 2004.
In article      View Article  PubMed
 
[9]  R. Argent, A. Daly, and B. Caulfield, “Patient involvement with homebased exercise programs: can connected health interventions influence adherence?” JMIR mHealth and uHealth, vol. 6, no. 3, p. e47, 2018.
In article      View Article  PubMed
 
[10]  S. Bassett, “Measuring patient adherence to physiotherapy,” J Nov Physiother, vol. 2, no. 07, pp. 60-66, 2012.
In article      View Article
 
[11]  E. M. Sluijs, G. J. Kok, and J. Van der Zee, “Correlates of exercise compliance in physical therapy,” Physical therapy, vol. 73, no. 11, pp. 771-782, 1993.
In article      View Article  PubMed
 
[12]  H. Mousavi Hondori and M. Khademi, “A review on technical and clinical impact of Microsoft Kinect on physical therapy and rehabilitation,” Journal of medical engineering, vol. 2014, 2014.
In article      View Article  PubMed
 
[13]  G. C. Burdea, “Virtual rehabilitation-benefits and challenges,” Methods of information in medicine, vol. 42, no. 05, pp. 519-523, 2003.
In article      View Article  PubMed
 
[14]  C. for Disease Control, Prevention et al., “Brfss survey data and documentation 2017,” 2019.
In article      
 
[15]  D. Corbetta, F. Imeri, and R. Gatti, “Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review,” Journal of physiotherapy, vol. 61, no. 3, pp. 117-124, 2015.
In article      View Article  PubMed
 
[16]  A. Elor, M. Teodorescu, and S. Kurniawan, “Project star catcher: A novel immersive virtual reality experience for upper limb rehabilitation,” ACM Transactions on Accessible Computing (TACCESS), vol. 11, no. 4, p. 20, 2018.
In article      View Article
 
[17]  A. Elor, M. Powell, E. Mahmoodi, N. Hawthorne, M. Teodorescu, and S. Kurniawan, “On shooting stars: Comparing cave and hmd immersive virtual reality exergaming for adults with mixed ability,” ACM Transactions on Computing for Healthcare.
In article      
 
[18]  H. G. Hoffman, W. J. Meyer III, M. Ramirez, L. Roberts, E. J. Seibel, B. Atzori, S. R. Sharar, and D. R. Patterson, “Feasibility of articulated arm mounted oculus rift virtual reality goggles for adjunctive pain control during occupational therapy in pediatric burn patients,” Cyberpsychology, Behavior, and Social Networking, vol. 17, no. 6, pp. 397-401, 2014.
In article      View Article  PubMed
 
[19]  H. G. Hoffman, G. T. Chambers, W. J. Meyer, L. L. Arceneaux, W. J. Russell, E. J. Seibel, T. L. Richards, S. R. Sharar, and D. R. Patterson, “Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures,” Annals of Behavioral Medicine, vol. 41, no. 2, pp. 183-191, 2011.
In article      View Article  PubMed
 
[20]  P. J. Standen and D. J. Brown, “Virtual reality in the rehabilitation of people with intellectual disabilities,” Cyberpsychology & behavior, vol. 8, no. 3, pp. 272-282, 2005.
In article      View Article  PubMed
 
[21]  J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and A. Muhlberger, “The impact of perception and presence on emotional¨ reactions: a review of research in virtual reality,” Frontiers in psychology, vol. 6, 2015.
In article      View Article  PubMed
 
[22]  P. J. Costello, Health and safety issues associated with virtual reality: a review of current literature. Advisory Group on Computer Graphics, 1997.
In article      
 
[23]  D. E. Levac, S. M. Glegg, H. Sveistrup, H. Colquhoun, P. Miller, H. Finestone, V. DePaul, J. E. Harris, and D. Velikonja, “Promoting therapists use of motor learning strategies within virtual reality-based stroke rehabilitation,” PloS one, vol. 11, no. 12, p. e0168311, 2016.
In article      View Article  PubMed
 
[24]  M. Beccue and C. Wheelock, “Research report: Virtual reality for consumer markets,” Tractica Research, Tech. Rep., Q4 2016. [Online]. Available: https://www.tractica.com/research/virtual-reality-for-consumer-markets/.
In article      
 
[25]  A. Elor, S. Kurniawan, and M. Teodorescu, “Towards an immersive virtual reality game for smarter post-stroke rehabilitation,” in 2018 IEEE International Conference on Smart Computing (SMARTCOMP). IEEE, 2018, pp. 219-225.
In article      View Article
 
[26]  G. Riva, C. Botella, P. Legeron, and G. Optale, “13 immersive virtual´ telepresence: Virtual reality meets ehealth,” 2004.
In article      
 
[27]  P. Lindner, A. Miloff, S. Fagernas, J. Andersen, M. Sigeman, G. An-¨ dersson, T. Furmark, and P. Carlbring, “Therapist-led and self-led onesession virtual reality exposure therapy for public speaking anxiety with consumer hardware and software: A randomized controlled trial,” Journal of anxiety disorders, vol. 61, pp. 45-54, 2019.
In article      View Article  PubMed
 
[28]  A. Won, J. Bailey, J. Bailenson, C. Tataru, I. Yoon, and B. Golianu, “Immersive virtual reality for pediatric pain,” Children, vol. 4, no. 7, p. 52, 2017.
In article      View Article  PubMed
 
[29]  L. Li, F. Yu, D. Shi, J. Shi, Z. Tian, J. Yang, X. Wang, and Q. Jiang, “Application of virtual reality technology in clinical medicine,” American journal of translational research, vol. 9, no. 9, p. 3867, 2017.
In article      
 
[30]  K. E. Laver, B. Lange, S. George, J. E. Deutsch, G. Saposnik, and M. Crotty, “Virtual reality for stroke rehabilitation,” Cochrane database of systematic reviews, no. 11, 2017.
In article      View Article  PubMed
 
[31]  S. L. Delp, F. C. Anderson, A. S. Arnold, P. Loan, A. Habib, C. T. John, E. Guendelman, and D. G. Thelen, “Opensim: open-source software to create and analyze dynamic simulations of movement,” IEEE transactions on biomedical engineering, vol. 54, no. 11, pp. 1940-1950, 2007.
In article      View Article  PubMed
 
[32]  A. Rajagopal, C. L. Dembia, M. S. DeMers, D. D. Delp, J. L. Hicks, and S. L. Delp, “Full-body musculoskeletal model for muscle-driven simulation of human gait,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 10, pp. 2068-2079, 2016.
In article      View Article  PubMed
 
[33]  E. M. Arnold, S. R. Hamner, A. Seth, M. Millard, and S. L. Delp, “How muscle fiber lengths and velocities affect muscle force generation as humans walk and run at different speeds,” Journal of Experimental Biology, vol. 216, no. 11, pp. 2150-2160, 2013.
In article      View Article  PubMed
 
[34]  T. K. Uchida, J. L. Hicks, C. L. Dembia, and S. L. Delp, “Stretching your energetic budget: how tendon compliance affects the metabolic cost of running,” PloS one, vol. 11, no. 3, p. e0150378, 2016.
In article      View Article  PubMed
 
[35]  H. Baskar and S. M. R. Nadaradjane, “Minimization of metabolic cost of muscles based on human exoskeleton modeling: a simulation,” Int. J. Biomed. Eng. Sci, vol. 3, no. 4, p. 9, 2016.
In article      View Article
 
[36]  B. J. de Kruif, E. Schmidhauser, K. S. Stadler, and L. W. O’Sullivan, “Simulation architecture for modelling interaction between user and elbow-articulated exoskeleton,” Journal of Bionic Engineering, vol. 14, no. 4, pp. 706-715, 2017.
In article      View Article
 
[37]  G. S. Virk, U. Haider, I. Nyoman, N. Masud, I. Mamaev, P. Hopfgarten, and B. Hein, “Design of exo-legs exoskeletons,” in ASSISTIVE ROBOTICS: Proceedings of the 18th International Conference on CLAWAR 2015. World Scientific, 2016, pp. 59-66.
In article      View Article
 
[38]  D. S. Pina, A. A. Fernandes, R. N. Jorge, and J. Gabriel, “Designing the mechanical frame of an active exoskeleton for gait assistance,” Advances in Mechanical Engineering, vol. 10, no. 2, p. 1687814017743664, 2018.
In article      View Article
 
[39]  J. A. Dienes, X. Hu, K. D. Janson, C. Slater, E. A. Dooley, G. J. Christ, and S. D. Russell, “Analysis and modeling of rat gait biomechanical deficits in response to volumetric muscle loss injury,” Frontiers in Bioengineering and Biotechnology, vol. 7, p. 146, 2019.
In article      View Article  PubMed
 
[40]  M. Marieswaran, A. Sikidar, A. Goel, D. Joshi, and D. Kalyanasundaram, “An extended opensim knee model for analysis of strains of connective tissues,” Biomedical engineering online, vol. 17, no. 1, p. 42, 2018.
In article      View Article  PubMed
 
[41]  Z. F. Lerner, B. C. Gadomski, A. K. Ipson, K. K. Haussler, C. M. Puttlitz, and R. C. Browning, “Modulating tibiofemoral contact force in the sheep hind limb via treadmill walking: Predictions from an opensim musculoskeletal model,” Journal of Orthopaedic Research, vol. 33, no. 8, pp. 1128-1133, 2015.
In article      View Article  PubMed
 
[42]  J. W. Rankin, J. Rubenson, and J. R. Hutchinson, “Inferring muscle functional roles of the ostrich pelvic limb during walking and running using computer optimization,” Journal of the Royal Society Interface, vol. 13, no. 118, p. 20160035, 2016.
In article      View Article  PubMed
 
[43]  J. M. Burnfield, K. R. Josephson, C. M. Powers, and L. Z. Rubenstein, “The influence of lower extremity joint torque on gait characteristics in elderly men,” Archives of physical medicine and rehabilitation, vol. 81, no. 9, pp. 1153-1157, 2000.
In article      View Article  PubMed
 
[44]  L. Ballaz, M. Raison, C. Detrembleur, G. Gaudet, and M. Lemay, “Joint torque variability and repeatability during cyclic flexion-extension of the elbow,” BMC sports science, medicine and rehabilitation, vol. 8, no. 1, p. 8, 2016.
In article      View Article  PubMed
 
[45]  A. K. Gillawat and H. J. Nagarsheth, “Human upper limb joint torque minimization using genetic algorithm,” in Recent Advances in Mechanical Engineering. Springer, 2020, pp. 57-70.
In article      View Article
 
[46]  K. Kiguchi and Y. Hayashi, “An emg-based control for an upper-limb power-assist exoskeleton robot,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 42, no. 4, pp. 1064-1071, 2012.
In article      View Article  PubMed
 
[47]  D. H. Perrin, R. J. Robertson, and R. L. Ray, “Bilateral isokinetic peak torque, torque acceleration energy, power, and work relationships in athletes and nonathletes,” Journal of Orthopaedic & Sports Physical Therapy, vol. 9, no. 5, pp. 184-189, 1987.
In article      View Article  PubMed
 
[48]  J. Hamill and K. M. Knutzen, Biomechanical basis of human movement. Lippincott Williams & Wilkins, 2006.
In article      
 
[49]  M. T. Farrell and H. Herr, “Angular momentum primitives for human turning: Control implications for biped robots,” in Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots. IEEE, 2008, pp. 163-167.
In article      View Article
 
[50]  S. M. Bruijn, P. Meyns, I. Jonkers, D. Kaat, and J. Duysens, “Control of angular momentum during walking in children with cerebral palsy,” Research in developmental disabilities, vol. 32, no. 6, pp. 2860-2866, 2011.
In article      View Article  PubMed
 
[51]  C. Nott, R. R. Neptune, and S. Kautz, “Relationships between frontalplane angular momentum and clinical balance measures during poststroke hemiparetic walking,” Gait & posture, vol. 39, no. 1, pp. 129-134, 2014.
In article      View Article  PubMed
 
[52]  R. R. Neptune and C. P. McGowan, “Muscle contributions to wholebody sagittal plane angular momentum during walking,” Journal of biomechanics, vol. 44, no. 1, pp. 6-12, 2011.
In article      View Article  PubMed
 
[53]  J. F. Lubar, “Discourse on the development of eeg diagnostics and biofeedback for attention-deficit/hyperactivity disorders,” Biofeedback and Self-regulation, vol. 16, no. 3, pp. 201-225, 1991.
In article      View Article  PubMed
 
[54]  J. Lubar, M. Swartwood, J. Swartwood, and D. Timmermann, “Quantitative eeg and auditory event-related potentials in the evaluation of attention-deficit/hyperactivity disorder: Effects of methylphenidate and implications for neurofeedback training,” Journal of Psychoeducational Assessment, vol. 34, pp. 143-160, 1995.
In article      
 
[55]  G. Deuschl, A. Eisen et al., “Recommendations for the practice of clinical neurophysiology: guidelines of the international federation of clinical neurophysiology,” 1999.
In article      
 
[56]  M. Lotan, S. Yalon-Chamovitz, and P. L. T. Weiss, “Improving physical fitness of individuals with intellectual and developmental disability through a virtual reality intervention program,” Research in developmental disabilities, vol. 30, no. 2, pp. 229-239, 2009.
In article      View Article  PubMed
 
[57]  H. D. Critchley, “Electrodermal responses: what happens in the brain,” The Neuroscientist, vol. 8, no. 2, pp. 132-142, 2002.
In article      View Article  PubMed
 
[58]  W. Boucsein, Electrodermal activity. Springer Science & Business Media, 2012.
In article      View Article
 
[59]  R. Ramirez and Z. Vamvakousis, “Detecting emotion from eeg signals using the emotive epoc device,” in International Conference on Brain Informatics. Springer, 2012, pp. 175-184.
In article      View Article
 
[60]  V. N. Salimpoor, M. Benovoy, G. Longo, J. R. Cooperstock, and R. J. Zatorre, “The rewarding aspects of music listening are related to degree of emotional arousal,” PloS one, vol. 4, no. 10, p. e7487, 2009.
In article      View Article  PubMed
 
[61]  M. S. Cameirao, I. B. S. Bermudez, E. Duarte Oller, and P. F. Verschure,´ “The rehabilitation gaming system: a review,” Stud Health Technol Inform, vol. 145, no. 6, 2009.
In article      
 
[62]  R. W. Picard, Affective computing. MIT press, 2000.
In article      View Article
 
[63]  D. Novak, J. Ziherl, A. Olensek, M. Milavec, J. Podobnik, M. Mihelj, and M. Munih, “Psychophysiological responses to robotic rehabilitation tasks in stroke,” IEEE Transactions on neural systems and rehabilitation engineering, vol. 18, no. 4, pp. 351-361, 2010.
In article      View Article  PubMed
 
[64]  D. Novak, M. Mihelj, J. Ziherl, A. Olensek, and M. Munih, “Psychophysiological measurements in a biocooperative feedback loop for upper extremity rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 4, pp. 400-410, 2011.
In article      View Article  PubMed
 
[65]  C. R. Guerrero, J. C. F. Marinero, J. P. Turiel, and V. Munoz, “Using hu-˜ man state aware robots to enhance physical human-robot interaction in a cooperative scenario,” Computer methods and programs in biomedicine, vol. 112, no. 2, pp. 250-259, 2013.
In article      View Article  PubMed
 
[66]  A. Koenig, D. Novak, X. Omlin, M. Pulfer, E. Perreault, L. Zimmerli, M. Mihelj, and R. Riener, “Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 4, pp. 453-464, 2011.
In article      View Article  PubMed
 
[67]  A. Elor, S. Lessard, M. Teodorescu, and S. Kurniawan, “Project butterfly: Synergizing immersive virtual reality with actuated soft exosuit for upper-extremity rehabilitation,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 1448-1456.
In article      View Article
 
[68]  I. D. O. NaturalPoint, “Optitrack - industry leading precision motion capture and 3d tracking systems for video game design, animation, virtual reality, robotics, and movement sciences.” O. M. C. Systems, Ed., 23.01.2020. [Online]. Available: https://optitrack.com/.
In article      
 
[69]  InteraXon, “Featured research with muse,” M. Research, Ed., 29.06.2019. [Online]. Available: https://choosemuse.com/muse-research/ ,developer.choosemuse.com/tools/available-data.
In article      
 
[70]  J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297-301, 1965.
In article      View Article
 
[71]  N. Kovacevic, P. Ritter, W. Tays, S. Moreno, and A. R. McIntosh, “‘my virtual dream’ : Collective neurofeedback in an immersive art environment,” PloS one, vol. 10, no. 7, p. e0130129, 2015.
In article      View Article  PubMed
 
[72]  O. E. Krigolson, C. C. Williams, A. Norton, C. D. Hassall, and F. L. Colino, “Choosing muse: Validation of a low-cost, portable eeg system for erp research,” Frontiers in neuroscience, vol. 11, p. 109, 2017.
In article      View Article  PubMed
 
[73]  S. Bhayee, P. Tomaszewski, D. H. Lee, G. Moffat, L. Pino, S. Moreno, and N. A. Farb, “Attentional and affective consequences of technology supported mindfulness training: a randomised, active control, efficacy trial,” BMC psychology, vol. 4, no. 1, p. 60, 2016.
In article      View Article  PubMed
 
[74]  Neulog, “Gsr logger sensor nul-217,” Neulog, Ed., 29.06.2019. [Online]. Available: https://neulog.com/gsr/.
In article      
 
[75]  Polar, “Polar oh1: optical heart rate sensor,” Polar, Ed., 29.06.2019. [Online]. Available: https://www.polar.com/us-en/products/accessories/ oh1-optical-heart-rate-sensor.
In article      
 
[76]  H. Moore, MATLAB for Engineers.Pearson, 2017.
In article      
 
[77]  Unity Technologies, “Unity real-time development platform — 3d, 2d vr ar,” Internet: https://unity.com/ [Jun. 06, 2019], 2019.
In article      
 
[78]  C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, and A. Walton, “Measuring and defining the experience of immersion in games,” International journal of human-computer studies, vol. 66, no. 9, pp. 641-661, 2008.
In article      View Article
 
[79]  G. Rossum, “Python reference manual,” 1995.
In article