BDCC-06-00055

BDCC-06-00055

Big Data Cogn. Comput. 2022, 6(2), 55; doi:10.3390/bdcc6020055

Article

Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience

Francesco Chiossi 1,*Orcid, Robin Welsch 1,2Orcid, Steeven Villa ¹Orcid, Lewis Chuang ³Orcid and Sven Mayer ¹Orcid

¹

Institute for Informatics, LMU Munich, 80337 Munich, Germany

²

Cognitive Psychology and Human Factors, Institute of Psychology, Faculty of Behavioural and Social Sciences, Chemnitz University of Technology, 09107 Chemnitz, Germany

³

Humans and Technology, Institute for Media Research, Faculty of Humanities, Chemnitz University of Technology, 09111 Chemnitz, Germany

*

Correspondence: francesco.chiossi@um.ifi.lmu.de

Academic Editors: Valentin Schwind and Thomas Kosch

Received: 1 April 2022 / Accepted: 5 May 2022 / Published: 13 May 2022

Abstract

:

Virtual reality is increasingly used for tasks such as work and education. Thus, rendering scenarios that do not interfere with such goals and deplete user experience are becoming progressively more relevant. We present a physiologically adaptive system that optimizes the virtual environment based on physiological arousal, i.e., electrodermal activity. We investigated the usability of the adaptive system in a simulated social virtual reality scenario. Participants completed an n-back task (primary) and a visual detection (secondary) task. Here, we adapted the visual complexity of the secondary task in the form of the number of non-player characters of the secondary task to accomplish the primary task. We show that an adaptive virtual reality can improve users’ comfort by adapting to physiological arousal regarding the task complexity. Our findings suggest that physiologically adaptive virtual reality systems can improve users’ experience in a wide range of scenarios.

Keywords:

adaptive interface; biocybernetic control loop; physiological computing; social VR

1. Introduction

Virtual Reality (VR) has, in the last decade, grown from being a niche technology to one that we see employed across diverse application domains, including work [1], training [2], and education [3]. VR can provide an immersive simulation of the physical environment enriched by digital artifacts, providing more control over environmental variables. On the one hand, developers can achieve higher realism by manipulating environmental parameters, such as increasing the number of polygons, texture details, lighting realism, and animation [4]. This even allows for creating realistic Non-player characters (NPCs), such as pedestrians or co-workers, with human-like agent behavior. Thus, adding a social component to the virtual environment [5]. On the other hand, increasing the immersive properties of VR simulations could introduce complexity that requires more user effort to process [5,6,7,8]. This could diminish users’ cognitive spare capacity to execute primary tasks. In other words, highly immersive VR scenarios could overload the user, potentially hindering their ability to achieve their objectives by distracting, or even overwhelming, the user.

Researchers have shown that experienced cognitive workload can influence human physiological activity [2,9,10,11]. For instance, when we perform cognitively demanding tasks, the physiological arousal is heightened, manifesting in increased physiological activity such as skin conductance level (SCL) [12], heart rate [13], or pupil dilation [14]. This has given rise to the plausible notion that VR complexity can be adapted in real-time [15] in response to the user’s physiological activity in order to maintain an optimal physiological arousal level and, hence, the optimal performance level [16] within the VR scenario.

This aligns with the Motivational Intensity Model (MIM), which describes a ’tipping point’ where allocated cognitive resources decline with overload or unachievable demands [17]. The MIM model theorizes that increased cognitive effort due to increased task demands results in increased engagement until the user reaches overload, and task disengagement occurs. Here, physiological measures of mental effort can detect this [18,19]. With this in mind, biocybernetic control loops could prevent users from reaching cognitive under- or overload by dynamically adjusting the source that induces arousal [20]. Such systems could accept real-time inputs from physiological sensors, such as electroencephalography (EEG) [21], electrodermal activity (EDA) [22], or electrocardiography (ECG) [21,23], and adapt task difficulty accordingly. EEG [24] and ECG [25] contributed to the physiological underpinnings of MIM within biocybernetics loops. Pope et al. [26] developed a pioneering version of such a system in which EEG signals were exploited to adapt the automation level in a simulated aviation task to support task engagement and avoid attentional drops. Muñoz et al. [27] implemented a training scenario to support the users’ readiness level by adapting the task difficulty based on the computation of R-to-R intervals and ECG frequency domain parameters. Although EDA is relatively unexplored, in our previous work [28] on which this work is based on, we presented an initial investigation into real-time adaptation exclusively using EDA in a Social VR scenario of which this paper presents a more in-depth analysis.

Social environments are an emerging trend in VR that is especially critical in supporting remote collaborations [5,29,30]. Social VR leverages the flexibility of VR to enable multiple players and NPCs to interact. The inclusion of multiple agents can increase environmental complexity. In addition, it can also lead to social crowdedness, whereby the violations of social norms (e.g., personal space) [31,32,33] could induce heightened physiological arousal [34,35]. In the following, we use social crowdedness in a Social VR environment to induce arousal [32,36]. Such environments typically have a primary task, e.g., learning, training, or group conversations, while other characters perform their task nearby. Here, we present an online physiologically-adaptive VR system, which adjusts the secondary task based on the users’ EDA, allowing for optimal performance of the primary task. In detail, the user is performing an n-back task as the primary task, simulating a high cognitive load such as training or work. The secondary task is to engage with a stream of non-player characters bypassing in close proximity. By adapting the stream (number of non-player characters per minute), we change the visual complexity, which is itself linked to task difficulty [6,7].

In this paper, we present an evaluation of our novel physiologically adaptive VR system that aims to increase user satisfaction by optimizing the secondary task’s complexity to maintain an optimal physiological arousal state. Our system recognized changes in EDA, which we treat as an index for arousal and cognitive workload. Based on this, the system then adapts the stream of non-player characters of a secondary task in real-time to minimize EDA changes. With this, we make the following contributions: (1) we investigated the feasibility and usability of VR adaptation based on EDA changes; (2) we show that online adaptation of the VR environment improves the user’s perceived workload compared to a non-adaptive system; and (3) we provide the recorded dataset of behavioral performance as well as the EDA, EEG, and ECG physiological signals. With this, our study offers evidence that physiologically adaptive VR environments can improve the experience by reducing the complexity of the surroundings, which allows the user to focus on the primary task.

2. Related Work

In the following, we review three areas essential for this work: (1) current social virtual reality experiences; (2) electrodermal activity and its link to cognitive workload; and (3) previous physiologically adaptive systems.

2.1. Social Virtual Reality

Social VR is an emerging research topic [29,37,38] with many facets and applications. Generally, all instances of Social VR support social interactions between multiple users [39] for an initial application taxonomy. Current investigations mainly focus on how the user characters should be displayed [5,40,41] and how to facilitate user interactions with each other [42], which are considered critical factors for a good user experience [43]. Established findings from real-world proxemics often generalize Social VR experiences [29,44]. Hecht et al. [44] could show that the personal space is shaped circularly in physical and virtual environments. McVeigh-Schultz et al. [45] could show that establishing boundaries in Social VR is important [31]. Thus, they need interactions that are genuine to Social VR, e.g., by restraining close-by VR teleportation. Therefore, characters respecting a personal space can be considered as important in VR as it is in physical spaces, and Social VR needs interaction concepts that can take into account human social capabilities to enable a satisfactory user experience.

Note, however, that not only proxemics is relevant, but also that the level of crowdedness affects how people feel with respect to their environment [32]. In VR, crowd simulation studies show that people restrain their space when walking in a crowd [46]. Dickinson et al. [47] show that crowdedness produces negative feelings and can, thus, disrupt the user experience in Social VR. Moreover, fine-grained experimental work from Llobera et al. [36] can put this into context by experimentally varying the number of NPCs approaching in VR. Here, four characters approaching produces more physiological arousal compared to one. In line with this, Latoschik et al. [5] describe that the number of users in the VR scene impacts subjective performance such as fluidity, synchrony, and, more importantly, annoyance in the user when displaying more than 25 social avatars.

While it is clear that crowded Social VR scenarios can increase arousal and affect the user experience, there are no interaction concepts that can mitigate this effect or define what the optimal number of characters is per unit of virtual space. Nevertheless, from a social-crowding perspective, we can infer that more users in a scene increase the user’s arousal, which, in turn, induces overload [32,36,47]. Finally, crowdedness inherently raises the visual complexity of the scene, defined as the amount of detail, clutter, and objects in a scene [48].

2.2. Electrodermal Activity and Cognitive Workload

When cognitive resources are allocated, they impact physiological arousal in the form of increased activation of the sympathetic branch of the autonomic nervous system [49,50]. This allows for the implicit evaluation of the demands posed by a VR environment to users [51]. Across sympathetic peripheral physiological measures, EDA represents a noninvasive, easy-to-use, and robust method for detecting the effect of cognitive workload on arousal [52]. It provides information related to availability or decrease in efficiency of cognitive resources such as workload detection [53], task engagement [54], and stress [51]. In particular, the relationship between workload, engagement, and arousal is consistent with the MIM [17], where optimal task engagement correlates with increased arousal state just before it drops, resulting in disengagement of the task at hand, cognitive overload, and hypoarousal [55,56,57]. As an index of physiological arousal, EDA is characterized by two components: phasic and tonic. Phasic activity is indexed by rapid skin conductance responses (SCRs) reflecting discrete and stimulus-specific responses for the measurement of novelty, significance, and intensity of the stimuli used [58]. Tonic activity has a slower and increased response; it is an index of the general state of parasympathetic activation.

Measures of tonic activity are particularly suitable for assessing the effect of continuous stimuli (such as long-duration stimulations or tasks) and have, therefore, been used to assess changes in arousal under conditions of high cognitive load or stress in HCI. Son and Park [59] estimated a driver’s cognitive workload with SCL that correlated with behavioral performance. Mehler et al. [60] showed how SCL correlated with changes in cognitive workload at different points in a demand curve, reflecting the spare capacity of participants’ cognitive resources. Fairclough and Venables [12] reported that high stress increased sympathetic activity, as indexed by tonic activity, in a high-demand multi-component task. Similarly, when increasing task load to an effective task in immersive virtual reality, tonic activity increased in conditions of high workload [61]. In conclusion, EDA can be widely applied to physiologically estimate and monitor a user’s workload [62].

2.3. Physiological Adaptation

A physiologically adaptive system can monitor its user’s physiological activity and iteratively adapt its interactions with the user to optimize some aspect of user performance or user experience. This approach is inspired by classic control theory [63], which encompasses physiological data acquisition and processing (i), transformation into a system response (ii), which then shapes the future or expected psychophysiological response from the user (iii). Specifically, biocybernetic control loops employ negative-control loops, where deviations from the optimal state are detected and exploited to cue changes in the system to promote a desirable user’s state [64]. Biocybernetic loops found applications in different domains, such as cockpit automation [65], computer-based learning [66], and robotics [67]. Nevertheless, VR applications are relatively overlooked, with a small number of exceptions, e.g., [68]. For instance, Stach et al. [69] inferred and supported the user’s motivation from Heart Rate Variability to provide an optimum exercise protocol in an immersive exergame. Alternately, adapting task difficulty based on peripheral and central measures of arousal can support task engagement in a Virtual Reality Stroop Task (VRST) [70] or in a virtual object selection task to increase training effects based on EEG alpha frequency [71].

In the context of the cognitive workload, increasing visual complexity can induce cognitive overload, which is reflected by increased physiological arousal [72,73,74]. Thus, adapting the environment to optimize the user’s arousal can support the user in processing information and executing actions. Visual complexity adaptation based on users’ sympathetic arousal can support the user experience in various VR applications, ranging from simulator sickness [75] to VR productivity and interruption management [76].

The goal of this paper is to evaluate if online dynamic adjustments of visual complexity based on a peripheral physiological measure of arousal can support user experience and performance as compared to stable levels of visual complexity in a Social VR scenario.

3. Online Physiologically-Adaptive System

Here, we propose an online adaptation of virtual environments based on EDA. More specifically, we developed a real-time physiologically adaptive system based on the user’s arousal as indexed by EDA. This system determines if its user can handle more visual complexity or reduce complexity to optimize cognitive load. Consistent with prior literature [77], the online signal processing pipeline derives a mean EDA value by applying a 20-s median moving-average filter that replaced each data point with the average of its 20 neighboring data points.

To enable the user-dependent adaptation of the system, the physiologically-adaptive system is initialized with a baseline recording b 0 ⋯ b i . For later baseline comparison, we calculate Δ b as follows (Note: we denoted the mean value as x ¯ ):

Δ b = b j − i ⋯ b i ¯ − b 0 ⋯ b i ¯ ,

(1)

where i is sampleRate ∗ windowBaseline and j is sampleRate ∗ timeBaseline . The baseline period is used to compute the baseline slope. In our implementation, this is the slope between the averaged values of the first and last 20 s over the three minutes using the aforementioned signal processing pipeline.

Δ s = s j − i ⋯ s i ¯ − s 0 ⋯ s i ¯ ,

(2)

where i is sampleRate ∗ windowOnline and j is − sampleRate ∗ window . Thus, this determines the current slope of physiological arousal. Due to the other external influences, such as baseline arousal of the virtual environment, this alone is not sufficient for adaptation.

In the final step to determine how to adapt the virtual environment, we compare the baseline measurement Δ b to the online measurement Δ s . The difference then determines how to adapt the environment, which we define as follows:

a d a p t a t i o n ( Δ s ) = increase of Stream if Δ s ≤ Δ b − t decrease of Stream if Δ s ≥ Δ b + t

(3)

where t is the threshold parameter that enables the physiological-adaptation system to work with a delay, counteracting high-frequency changes in EDA and, thus, preventing rapid and unstable adaptations. Moreover, for a stable adaptation, Equation (3) is only used for adaptation every 20 s.

4. User Study

We conducted a study to evaluate whether our physiologically adaptive system can support users’ comfort and usability in the virtual environment. To do so, participants performed a visual working memory n-back task (primary task) [78] and a visual detection task (secondary task). Both tasks were simulated as real-world activities in a virtual environment. While the primary task simulates a high workload task, the secondary task simulates situations where the user is distracted by, for instance, other players [38], notifications [79], or graphical artifacts [80].

In the n-back (i.e., 1-back) task, participants determined if the color of a presented sphere matched the color of the last presented sphere, see Figure 1. Participants were presented with colored spheres that they had to place in either a left or right bucket depending on their color (mis)match with the previous sphere. Spheres were either green, red, blue, or black [81]. A sound cued the appearance of each sphere. Then the participant had to pick up the sphere within 4 s; otherwise, it would count as an error. New spheres would appear either after 4 s or when the current sphere was placed into one of the two buckets. We counted missing a sphere as an error.

In the secondary tasks, we asked participants to inspect the museum tickets of a Stream of virtual NPCs. In detail, participants were required to distinguish museum visitors who had a ticket either in their left or right hand from those without a ticket. Museum visitors were represented by virtual NPCs that were walking past the participant on either the left or right side. We set the percentage of virtual NPCs who have no ticket to be 15%, which we found to be a suitable value in the informal pilot tests for this task. These had to be identified by clicking on them with the right input controller. All NPCs had randomized visual characteristics (i.e., hair, shirt color), and their shirt color turned red when clicked on. The NPCs’ distance from the user was kept at a minimum of 2.5 m to avoid proxemics confounds with the level of arousal of participants [36].

The five non-adaptive conditions had fixed Streams of 7, 22, 37, 52, or 67 NPCs per minute entering. In the adaptive condition, the system adapted the number of NPCs and added more NPCs when the physiological activity of the user did not indicate physiological arousal or removed NPCs. We randomized the six test conditions. Participants were not aware whether they would experience the adaptive condition or one of the non-adaptive conditions.

4.1. Participants

Eighteen volunteers ( M r a n g e = 23 − 31 ; M a g e = 27.9 , S D a g e = 2.9 ; male = 9 , female = 9 ) participated in our study, of which we had to exclude three from the analysis as the EDA electrodes lost contact. We recruited the participants using our institutional mailing lists and social networks and using convenient sampling. Exclusion criteria required participants to not experience intense physical activity or consume any caffeine or nicotine in the 3-h pre-study period [82]. None of the participants reported a history of neurological, psychological, or psychiatric symptoms. They were also required to submit a negative test for SARS-CoV-2 within 48 h prior to participation.

4.2. Apparatus

We designed the virtual environment for the study in Unity (Version 2019.4.24f1), see Figure 1, and presented it via an HTC VIVE VR headset with a display resolution of 2160 × 1200 pixels, refresh rate of 90Hz, and an average field of view of 110°. The environment used for the experiment is a replica of the Neue Nationalgalerie in Berlin, Germany. The 1-back task takes place in the entrance of the building, see Figure 1 and, for the detection task, agents with and without tickets in their hands entered the entrance.

We performed the EDA data recording according to the guidelines by Babaei et al. [82]. We placed Ag/AgCl electrodes (7 mm surface diameter) on the participant’s non-dominant hand (inner distal phalanges of the index and middle fingers). An electrolyte solution (0.5% NaCl) was applied to the acquisition site to ensure proper hydration and to minimize the effect of individual differences. We attached the electrodes 10 min later, with double-sided adhesive collars. We exploited the exosomatic recording principle with a direct current (DC) of a constant 0.5 V voltage.

We used Equation (3) to adjust the number of NPCs in the adaptive condition. In detail, when the EDA slope computed in the 20-s window was greater than the baseline slope added to the threshold slope, two NPCs were removed from the scene. On the contrary, if the EDA slope was lower, four NPCs were additionally spawned. A fixed range of 7–67 NPCs was maintained regardless of the participants’ measured physiological activity. These settings were determined empirically during a number of test sessions.

4.3. Measurements

We recorded three physiological measurements: (1) EDA signal via the LiveAmp amplifier (BrainProducts GmbH, Germany), using a 250Hz sampling rate; (2) ECG (Polar H10 chest strap, Finland) at 130 Hz; and (3) EEG signal (DSI-VR 300, Wearable Sensing, San Diego, CA, USA) at 250 Hz. Physiological data were streamed and recorded within the Lab Streaming Layer framework (LSL) (https://github.com/labstreaminglayer/, accessed on 1 April 2022). We only used the EDA signal as an indicator for physiological arousal. In addition, performance accuracy metrics were computed for both tasks. For the 1-back task, errors were represented by the proportion of times the sphere was placed in the wrong bucket. For the visual detection task, errors were represented by missing an NPC or selecting an NPC with a ticket.

Responses on three standardized questionnaires and two custom Likert items were collected to evaluate user experience and workload. First, raw subjective workload measures (NASA-TLX [83,84]). Second, perceived gamefulness of system use (Game Experience Questionnaire (in-Game Core Module) [85]). Here, we only recorded the subscales on Competence, Immersion, and Positive Affection allowing for content validity [86]. Third, the Fast Motion Sickness scale (FMS) to control for motion sickness [87]. Finally, participants rated two general usability statements on a 5-point Likert scale (strongly disagree - strongly agree); questions: “I would like to use the system in the future,” and “The flow of the people was appropriate”.

4.4. Procedure

Upon arrival, we briefed the participants on the study procedure, and we answered any open questions, which were followed by signing the informed consent form. Next, they performed a Snellen visual acuity test. Finally, the EDA sensor, EEG-VR headset, and ECG chest strap were attached to the participants.

The study began with a trial phase to allow participants to familiarize themselves with the VR environment. The VR trial phase started with participants practicing the 1-back task until they reached an accuracy level of at least 95% within a sequence of 80 spheres. Next, three minutes of EDA recording was performed, during which the participant was asked to ignore the NPCs (Stream: 37 agents/min) and to perform the 1-back task. This provided the baseline measurement for the physiologically-adaptive system.

The testing phase consisted of six blocks for each test condition. Each block lasted for six minutes. Visual feedback for the 1-back task was provided, and participants were instructed to maintain 90% performance. In between blocks, participants filled in questionnaires (FMS, NASA-TLX, In-game GEQ subscales, ad-hoc questionnaires) on the previous block and rested for 2 min to stabilize their physiological state. The entire experiment lasted approximately one hour. We compensated participants with 10 Euros for their participation.

5. Results

We evaluated the usability of a physiologically adaptive system in a Social VR scenario. In the following section, we report the results of our study. As no participant aborted due to simulator sickness, however, electrodes were not attached correctly for three participants; the following analysis is based on the data of 15 participants. Results of the analyses can be found online on the DaRUS Open Data Platform, at [88].

5.1. Analysis

We analyzed indicators of physiological arousal, performance, and subjective experience across the adaptive and non-adaptive conditions in R [89,90,91]. First, we present the results for the five non-physiologically adaptive conditions and present a comparative analysis of the adaptive condition to determine if the physiologically adaptive condition produces: (1) superior performance; (2) a more enjoyable game experience; and (3) lower levels of perceived workload. We fit a linear mixed model (estimated using a restricted maximum likelihood approach and nloptwrap optimizer) [92,93] to predict our outcomes as a function of Stream. We computed the p-values using the Wald-Approximation for the calculation of degrees of freedom. To account for the repeated-measures structure in our data, we added a random intercept for every participant to our model. We use Welsh-corrected t-tests for comparing means or Wilcoxon-rank tests for ranks.

5.2. Non-Adaptive Conditions

For mean EDA, the model intercept is at −0.16 (95% CI [−0.67, 0.36], t ( 71 ) = − 0.60 , p = 0.551 ). Within this model, the effect of Stream is statistically significant and positive (beta = 0.004, 95% CI [0.00, 0.007], t ( 71 ) = 2.54 , p < 0.01 . In other words, every additional NPC increases mean EDA by about 0.004 standardized µS units (see Figure 2). This is consistent with the raw NASA-TLX score, see Table 1. Increasing the number of NPCs in the simulation elevated the NASA-TLX score by about 0.1 per NPC. In contrast, Stream affected performance accuracy negatively in both tasks, see again Table 1. Increasing Stream reduced accuracy up to 10% in both the n-Back and the visual detection task. Together with this, we supplement our analysis with a SCL computation as a measure of tonic activity. To compute SCL, EDA data were filtered with a 3Hz, high-pass, fourth-order Butterworth filter to remove high-frequency noise and decomposed into tonic and phasic components by means of a non-negative deconvolution analysis [94]. For SCL, the model intercept is at 8.89 (95% CI [7.40, 10.39], t ( 71 ) = 11.86 , p < 0.001). The effect of Stream is statistically significant and positive (beta = 0.01, 95% CI [0.00, 0.02], t ( 71 ) = 2.54 , p < 0.013 . This result implies that the addition of every NPC increases SCL by about 0.01 standardized µS units. We found that the subjective experience scales (GEQ Immersion, GEQ Competence, GEQ Positive affect, desire to use, and Appropriateness of Stream) were not significantly by our manipulation for all p > 0.05 , see Table 1.

5.3. Adaptive Condition

On average, participants adapted to having a Stream of 32.89 ( SD = 18.6 ) NPCs in their environment. This represents a medium level of Stream, relative to the fixed range (i.e., 7-67) that was adopted for the non-adaptive conditions. Figure 3 illustrates how Stream adapted over time for one participant.

The recorded measures from the non-adaptive conditions cannot be directly compared to their counterpart in the physiologically adaptive condition. This is because Stream was a fixed value in the former and an adapted value in the latter. Thus, we computed individual expected values from the five non-adaptive conditions using linear mixed models with Stream as a predictor and calculated the expected value as predicted from our statistical model of the non-adaptive conditions, see Table 2 in the Prediction column and at the local maxima in Stream for the adaptive condition.

In other words, any aspects of the user experience that are different in the physiologically adaptive condition compared to the non-adaptive conditions would be revealed if the actual measurement in the condition is significantly higher or lower than the corresponding value that would be predicted from a regression of the measurements of the non-adaptive conditions.

5.3.1. Electrodermal Activity Measures

The physiologically adaptive condition did not significantly increase physiological arousal in either the EDA mean or SCL according to the linear mixed model. Individual regression predicted a mean EDA of -0.023 µS ( SD = 0.957 ), while in the adaptive condition, the actual mean EDA was 0.040 µS ( SD = 1.074 ) after a stable Stream was achieved, see Figure 4b. Similarly, the adaptive algorithm did not significantly increase tonic arousal as indexed by SCL. Actual SCL was 9.456 µS ( SD = 3.100 ), reflecting predicted values of 9.275 µS ( SD = 2.761 ), see Figure 4c. SCL and EDA results are summarized in Table 2.

5.3.2. Workload and Performance

The physiologically adaptive condition did not increase subjective workload. Individual regressions predicted a mean raw NASA-TLX score of 8.31 ( SD = 2.25 ), while workload was 8.57 ( SD = 3.20 ) in the adaptive condition after a stable Stream was achieved, see Figure 2 and Figure 4 and Table 2. This was mirrored for both the performances in the detection task and the n-back task. The adaptive algorithm regulated performance based on the individual linear fit to a local optimum and, thus, we found no significant differences between the expected and actual performance for the adaptive condition as seen in Figure 5 and Table 2.

Figure 6 illustrates the negative relationship between workload and mean EDA. The algorithm allowed participants to perform tasks at their individual optimal levels of physiological arousal. Participants with relatively higher mean EDAs experience lower subjective workload r ( 13 ) = − 0.62 , p = 0.013 . This shows the successful adaptation of the algorithm and that participants approached an individual local maxima in perceived workload.

Table 2. Mean predicted and actual means across measures with Welsh-corrected t-test or Wilcoxon-signed-rank test depending on the Shapiro test and Cohen’s d. We compare the predicted value from the LMM at an optimized Stream with the actual value at this rate of Stream.

Table 2. Mean predicted and actual means across measures with Welsh-corrected t-test or Wilcoxon-signed-rank test depending on the Shapiro test and Cohen’s d. We compare the predicted value from the LMM at an optimized Stream with the actual value at this rate of Stream.

| Actual | Prediction | Diff | t-Test/Wilcoxon | Shapiro
| M | SD | M | SD | M | SD | d | t/Z | p | W | p
n-back Acc. [%] | 94.892 | 2.662 | 94.471 | 2.007 | 0.422 | 1.801 | 0.234 | 93.000 | 0.064 | 0.825 | 0.008
Visual Det. task Acc. [%] | 95.478 | 3.837 | 94.696 | 2.766 | 0.781 | 1.706 | 0.458 | 1.774 | 0.098 | 0.944 | 0.440
Raw NASA | 8.578 | 3.205 | 8.317 | 2.250 | 0.261 | 1.952 | 0.134 | 0.518 | 0.613 | 0.938 | 0.363
EDA Mean (std) | 0.040 | 1.074 | −0.023 | 0.957 | 0.063 | 0.364 | 0.172 | 0.668 | 0.515 | 0.976 | 0.937
SCL | 9.456 | 3.100 | 9.275 | 2.761 | 0.181 | 1.051 | −0.172 | 0.666 | 0.516 | 0.976 | 0.939
GEQ—Competence | 2.767 | 0.623 | 2.441 | 0.374 | 0.326 | 0.436 | 0.746 | 2.890 | 0.012 | 0.933 | 0.298
GEQ—Positive Affection | 2.300 | 0.978 | 2.396 | 0.262 | −0.096 | 0.866 | -0.111 | 52.000 | 0.679 | 0.864 | 0.027
GEQ—Immersion | 1.500 | 0.707 | 1.067 | 0.334 | 0.433 | 0.574 | 0.755 | 2.924 | 0.011 | 0.903 | 0.105
Stream Appropriate | 3.467 | 0.743 | 1.669 | 0.058 | 1.798 | 0.729 | 2.465 | 9.548 | <0.001 | 0.902 | 0.104
Desire To Use | 3.533 | 0.640 | 1.728 | 0.101 | 1.805 | 0.591 | 3.053 | 120.000 | <0.001 | 0.827 | 0.008

5.3.3. Subjective Evaluation

The subjective feeling of being immersed, measured by the GEQ-immersion scale, was increased in the adaptive condition ( M = 1.556 , SD = 0.705 ) as compared to its expected value ( M = 1.032 , SD = 0.323 ), as predicted from the non-physiologically adaptive conditions. Stream was also considered to be more appropriate in the adaptive condition, compared to the predicted value derived from the actual measures of the non-adaptive conditions. This converges with a desire to use the system as it increased when compared to the actual and the expected desire to use the system, see Figure 7a. In summary, participants favored the physiologically adaptive condition in terms of Stream appropriateness, the desire to use, gaming immersion (GEQ), and their feelings of competence (GEQ) and positive affection (GEQ). Those results are graphically summarized in Figure 7b.

6. Discussion

We presented a design of an online physiologically adaptive VR system. In detail, our system can adapt to the complexity of a VR environment based on the user’s arousal as measured using EDA. We tested this system in a VR environment where the adaptive system optimizes the stream of NPCs. In our evaluation, participants completed a primary task (n-back) and a secondary task (visual detection) while either being presented with five static levels of Streams of NPCs or an adaptive Stream of NPCs.

First, we investigated the effectiveness of the Stream modulation. Here, we lend credibility to earlier results on crowdedness [32,36] and visual complexity [6,7]. In detail, a higher Stream of NPCs impacts EDA and the task performance, albeit in both the primary and the secondary task. Thus, the desire to use the system is negatively affected by increasing Stream. Thus, we found that an online adaptation of the Stream is feasible for modulating workload. Therefore, we investigated the effectiveness of such a physiologically adaptive VR system. Here, we compared the baseline from the non-adaptive conditions to the adaptive condition using a prediction model. We found significant differences between adaptive and non-adaptive conditions for the desire to use the system, GEQ competence, and GEQ immersion. Interestingly, participants also rated the flow significantly more appropriate than the baseline, while we kept the range of possible flows within the same bounds. Thus, we conclude that our physiologically adaptive VR system can improve VR experiences, especially in complex environments such as this simulated Social VR. This is a promising step toward online adaptive VR environments based on physiological sensor data. Our study adds to other findings in physiological computing, namely that sympathetic arousal, here indexed by EDA, can depict task engagement [22,95,96]. Additionally, our subjective results are in line with studies that have adapted VR environments based on physiological arousal. Thus, our results are in line with prior work in terms of perceived workload [24] and task engagement [27] when interacting in a physiologically adaptive VR system. These results are promising for the implementation of EDA measures in a biocybernetic loop for adaptation, confirming subjective results obtained with other measures i.e., EEG and ECG.

At first, it is counterintuitive that EDA was not lower when using the adaptive system, as the system optimizes in order to not overload the user. However, we argue that lowering the EDA was not expected nor intended. In our implementation, we used a −2/+4 adaptation i.e., if NPCs need to be removed, we removed two, while we added four in the more adaption. Thus, this pushed the user to always take a bit more at once while reducing it only gradually. This prevents the user from just tuning out but also keeps up the pressure while not overloading. Moreover, in a real-world scenario, keeping as many characters in the environment as possible is important to not lose an understanding of the Social VR scenario one engages in, but also to prevent lowering the level of immersion and presence. We further see this manifested in the significantly higher rating of immersion. Here, we argue participants were less likely to be bored or stressed, as we optimized for optimal arousal according to the MIM [17] where flow-experience and task engagement correlate with rising demands [97]. As a result, they focused on the environment and, thus, achieved a higher level of immersion.

Behavioral results indicated that Stream adaptation did not significantly impact n-Back and detection accuracies. This might point toward an insufficient increase in task complexity. However, it is not corroborated by either the EDA processed by our algorithm or from the non-negative deconvolution analysis. Second, as distractor processing is influenced by increased executive load [98], future studies should investigate task complexity adaptation under increased working memory load conditions. This conclusion is shared with the work of Dey et al. [71] and with Ewing et al. [24] when considering increased task demands.

Our findings have a broader impact on collaborative environments; virtual agents can assist or perform tasks within a collaboration VR environment in future VR applications. Such systems could exploit physiologically adaptive systems to modulate the VR complexity (e.g., amount of virtual coworkers or concurrent visually displayed information). This allows supporting concurrent goals without the need to prioritize performance over comfort while preventing cognitive overload. We argue that our results are not limited to Social VR per se. For instance, physiological adaptation can be used to transition on the reality-virtuality continuum [99]. Here, the VR complexity can overload the user with the virtual component, and, thus, it could gradually fade out the digital overlay. Thus, varying the blend of realities according to users’ physiological arousal can be the next step toward a physiologically adaptive mixed reality system.

EDA data acquisition is low-cost and unobtrusive; skin-interfaced wearable systems are easy to implement in real or virtual scenarios, i.e., biofuel cell-based self-powered wearable sensors [100] or conductive fabric gloves [101]. Consequently, we argue that the next step is to embed physiological sensing into VR controllers, allowing practitioners to use them without a tedious setup process. This is consistent with recent developments in entertainment computing, where a player’s emotional state as inferred from physiological correlates was used to predict actions in a game [102] or dynamically adapt the game narrative to induce states of arousal [103].

Limitations

EDA activity has a one-to-one relationship with the state of physiological arousal [95], which can serve as a workload indicator. However, the adaptation pipeline used in this study, although with a low computational cost, could benefit from more advanced and standardized methods of EDA component deconvolution [94,104]. So far, this has not yet been used for online adaptation.

Tonic changes occur with a slow frequency and, therefore, may not be adequate for adaptive interactions that occur at a faster pace. Here, other physiological measures that are experimentally more demanding, but have better time-resolution, such as EEG, could be more suitable for adaptive systems [24]. Nevertheless, for adapting visual complexity in our Social VR scenario, adaptation in a third of a minute was suitable with regard to fostering immersion; here, faster adaptation could potentially break immersion as layer characters would appear and disappear seemingly randomly.

We targeted the complexity exclusively in the visual domain. Still, evaluating multimodal physiologically adaptive systems could better support users’ comfort as multimodal integration facilitates task-relevant information selection [105,106]. For example, in a social VR scenario, the visual representation could disappear, and voices from afar could be toned down to support the user in focusing on the primary task.

According to Braithwaite et al. [107], about 10% of the general population do not exert a strong electrodermal response attuned to sympathetic arousal. This is especially relevant when considering commercial applications of EDA sensing, e.g., for adaptive VR games where potentially 10% would not benefit from a solely EDA-based system. Thus, supplementing EDA with other arousal measurements such as the heart rate or eye-gaze could make measurements more robust and make the technology more accessible to the general population. Together with this, we had to exclude three participants via medical-grade instrumentation for high data quality due to a lack of data quality. This is consistent with EDA guidelines [107] and could be attributed to either technical failures and to individual variations in EDA. While we controlled for other factors such as physical stress [108] or caffeine consumption [109], using EDA in applied scenarios can not be shielded from such influences.

Finally, we acknowledge that only the data of 15 participants were used in the analysis. While on the one hand, this is a low number, we could still show large effect sizes for the appropriateness of the stream, and most importantly for the desire to use the adaptation. As such, larger sample sizes will be needed only for the measures with small and medium effect sizes.

7. Conclusions

Our study reports on the utility and viability of a VR system that adapts the environment to the physiological input, namely the user’s electrodermal activity. By adjusting the visual complexity of the environment in response to the changing arousal levels of the user, we were able to maintain an ideal working environment in a complex scenario (i.e., Social VR). This is the first step toward a physiologically adaptive immersive system that supports users’ satisfaction and cognitive capacity without the need for explicit inputs (i.e., verbal, button-presses). Future studies could use physiologically adaptive systems to optimally blend mixed reality, enabling more use cases by ensuring signal processing robustness and by integrating complementary physiological measures (i.e., EEG) or embedding physiological sensors into the hardware of VR systems (e.g., VR-controllers).

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/bdcc6020055/s1.

Author Contributions

Conceptualization: F.C. and S.M.; Data Curation, F.C.; Formal analysis, F.C. and S.M.; Methodology, F.C. and R.W.; Project administration, L.C. and S.M.; Software, S.V. and F.C.; Visualization, F.C. and R.W.; Writing—original draft, F.C.; Writing—review & editing, F.C., S.M., R.W. and L.C. All authors have read and agreed to the published version of the manuscript.

Funding

F.C. was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project-ID 251654672—TRR 161. L.C. is supported by the DFG, under the Project-ID 416228727—SFB 1410. S.V. is funded by the European Research Council (ERC AMPLIFY, no. 683008) and the Federal Ministry of Education and Research of Germany (BMBF Hive, no. 16SV8183).

Institutional Review Board Statement

According to the fast-track conditions of our local institutional ethics board (Ethikkommission der Fakultät 16, LMU Munich), the study qualified as fast-track, meaning that participants are not subject to any risk (e.g., deception, stress beyond normal levels, recording of sensitive information).

Informed Consent Statement

Informed consent was obtained from all participants involved in the study.

Data Availability Statement

The data presented in this study are openly available in DaRUS Open Data Platform at [88].

Conflicts of Interest

The authors declare no conflict of interest.

References

Knierim, P.; Kosch, T.; Schmidt, A. The Nomadic Office: A Location Independent Workspace through Mixed Reality. IEEE Pervasive Comput. 2021, 20, 71–78. [Google Scholar] [CrossRef]

Czarnek, G.; Strojny, P.; Strojny, A.; Richter, M. Assessing engagement during rescue operation simulated in virtual reality: A psychophysiological study. Int. J. Hum. Comput. Interact. 2020, 36, 464–476. [Google Scholar] [CrossRef]

Radianti, J.; Majchrzak, T.A.; Fromm, J.; Wohlgenannt, I. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comput. Educ. 2020, 147, 103778. [Google Scholar] [CrossRef]

Giordana, F.; Efremov, V.; Sourimant, G.; Rasheva, S.; Tatarchuk, N.; James, C. Virtual Production in ’Book of the Dead’: Technicolor’s Genesis Platform, Powered by Unity. In Proceedings of the ACM SIGGRAPH 2018 Real-Time Live! Vancouver, BC, Canada, 12–16 August 2018. [Google Scholar] [CrossRef]

Latoschik, M.E.; Kern, F.; Stauffert, J.P.; Bartl, A.; Botsch, M.; Lugrin, J.L. Not Alone Here?! Scalability and User Experience of Embodied Ambient Crowds in Distributed Social Virtual Reality. IEEE Trans. Vis. Comput. Graph. 2019, 25, 2134–2144. [Google Scholar] [CrossRef] [PubMed]

Ionescu, R.T.; Alexe, B.; Leordeanu, M.; Popescu, M.; Papadopoulos, D.P.; Ferrari, V. How Hard Can It Be? Estimating the Difficulty of Visual Search in an Image. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2157–2166. [Google Scholar] [CrossRef]

Mnookin, J.; Kellman, P.J.; Dror, I.; Erlikhman, G.; Garrigan, P.; Ghose, T.; Metler, E.; Charlton, D. Error Rates for Latent Fingerprinting as a Function of Visual Complexity and Cognitive Difficulty; National Institute of Justice, US Department of Justice: Washington, DC, USA, 2016.

Dorneich, M.C.; Whitlow, S.D.; Ververs, P.M.; Rogers, W.H. Mitigating cognitive bottlenecks via an augmented cognition adaptive system. In Proceedings of the SMC’03 Conference Proceedings, 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme—System Security and Assurance (Cat. No.03CH37483), Washington, DC, USA, 8 October 2003; Volume 1, pp. 937–944. [Google Scholar] [CrossRef]

Lackey, S.J.; Salcedo, J.N.; Szalma, J.L.; Hancock, P.A. The stress and workload of virtual reality training: The effects of presence, immersion and flow. Ergonomics 2016, 59, 1060–1072. [Google Scholar] [CrossRef]

Luong, T.; Martin, N.; Raison, A.; Argelaguet, F.; Diverrez, J.M.; Lécuyer, A. Towards Real-Time Recognition of Users Mental Workload Using Integrated Physiological Sensors Into a VR HMD. In Proceedings of the 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Porto de Galinhas, Brazil, 9–13 November 2020; pp. 425–437. [Google Scholar] [CrossRef]

Bläsing, D.; Bornewasser, M. A Strain Based Model for Adaptive Regulation of Cognitive Assistance Systems—Theoretical Framework and Practical Limitations. In Proceedings of the Intelligent Human Systems Integration 2019, San Diego, CA, USA, 7–10 February 2019; pp. 10–16. [Google Scholar] [CrossRef]

Fairclough, S.H.; Venables, L. Prediction of subjective states from psychophysiology: A multivariate approach. Biol. Psychol. 2006, 71, 100–110. [Google Scholar] [CrossRef]

Splawn, J.M.; Miller, M.E. Prediction of perceived workload from task performance and heart rate measures. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Torino, Italy, 1 October 2013; Volume 57, pp. 778–782. [Google Scholar] [CrossRef]

Kosch, T.; Hassib, M.; Buschek, D.; Schmidt, A. Look into My Eyes: Using Pupil Dilation to Estimate Mental Workload for Task Complexity Adaptation. In Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018; pp. 1–6. [Google Scholar] [CrossRef]

Mishra, J.; Anguera, J.A.; Gazzaley, A. Video games for neuro-cognitive optimization. Neuron 2016, 90, 214–218. [Google Scholar] [CrossRef]

Kukolja, D.; Popović, S.; Dropuljić, B.; Horvat, M.; Ćosić, K. Real-Time Emotional State Estimator for Adaptive Virtual Reality Stimulation. In Proceedings of the Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, San Diego, CA, USA, 19–24 July 2009; pp. 175–184. [Google Scholar] [CrossRef]

Wright, R.A. Refining the prediction of effort: Brehm’s distinction between potential motivation and motivation intensity. Soc. Personal. Psychol. Compass 2008, 2, 682–701. [Google Scholar] [CrossRef]

Richter, M.; Gendolla, G.H. The heart contracts to reward: Monetary incentives and preejection period. Psychophysiology 2009, 46, 451–457. [Google Scholar] [CrossRef]

Richter, M.; Friedrich, A.; Gendolla, G.H. Task difficulty effects on cardiac activity. Psychophysiology 2008, 45, 869–875. [Google Scholar] [CrossRef]

Fairclough, S.H. Fundamentals of physiological computing. Interact. Comput. 2009, 21, 133–145. [Google Scholar] [CrossRef]

Dorneich, M.C.; Ververs, P.M.; Mathan, S.; Whitlow, S.; Hayes, C.C. Considering Etiquette in the Design of an Adaptive System. J. Cogn. Eng. Decis. Mak. 2012, 6, 243–265. [Google Scholar] [CrossRef]

Kosch, T.; Karolus, J.; Ha, H.; Schmidt, A. Your Skin Resists: Exploring Electrodermal Activity as Workload Indicator during Manual Assembly. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Valencia, Spain, 18–21 June 2019. [Google Scholar] [CrossRef]

Kamkuimo, K.S.A.; Girard, B.; Menelas, B.A.J. Dynamic Difficulty Adjustment Through Real-Time Physiological Feedback for a More Adapted Virtual Reality Exposure Therapy. In Proceedings of the Games and Learning Alliance, Laval, France, 9–10 December 2020; pp. 102–111. [Google Scholar] [CrossRef]

Ewing, K.C.; Fairclough, S.H.; Gilleade, K. Evaluation of an adaptive game that uses EEG measures validated during the design process as inputs to a biocybernetic loop. Front. Hum. Neurosci. 2016, 10, 223. [Google Scholar] [CrossRef] [PubMed]

Wright, R.A.; Kirby, L.D. Effort determination of cardiovascular response: An integrative analysis with applications in social psychology. Adv. Exp. Soc. Psychol. 2001, 33, 255–307. [Google Scholar] [CrossRef]

Pope, A.T.; Bogart, E.H.; Bartolome, D.S. Biocybernetic system evaluates indices of operator engagement in automated task. Biol. Psychol. 1995, 40, 187–195. [Google Scholar] [CrossRef]

Muñoz, J.E.; Quintero, L.; Stephens, C.L.; Pope, A.T. A psychophysiological model of firearms training in police officers: A virtual reality experiment for biocybernetic adaptation. Front. Psychol. 2020, 11, 683. [Google Scholar] [CrossRef]

Chiossi, F.; Welsch, R.; Villa, S.; Chuang, L.L.; Mayer, S. Designing a Physiological Loop for the Adaptation of Virtual Human Characters in a Social VR Scenario. In Proceedings of the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, Christchurch, New Zealand, 12–16 March 2022. [Google Scholar] [CrossRef]

Williamson, J.; Li, J.; Vinayagamoorthy, V.; Shamma, D.A.; Cesar, P. Proxemics and social interactions in an instrumented virtual reality workshop. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 8–13 May 2021; pp. 1–13. [Google Scholar] [CrossRef]

Moustafa, F.; Steed, A. A Longitudinal Study of Small Group Interaction in Social Virtual Reality. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, Tokyo, Japan, 28 November–1 December 2018. [Google Scholar] [CrossRef]

Hall, E.T. The Hidden Dimension; Doubleday: Garden City, NY, USA, 1966; Volume 609. [Google Scholar]

Desor, J.A. Toward a psychological theory of crowding. J. Personal. Soc. Psychol. 1972, 21, 79. [Google Scholar] [CrossRef]

Welsch, R.; von Castell, C.; Hecht, H. The anisotropy of personal space. PLoS ONE 2019, 14, e0217587. [Google Scholar] [CrossRef]

Worchel, S.; Teddie, C. The experience of crowding: A two-factor theory. J. Personal. Soc. Psychol. 1976, 34, 30. [Google Scholar] [CrossRef]

Kennedy, D.P.; Gläscher, J.; Tyszka, J.M.; Adolphs, R. Personal space regulation by the human amygdala. Nat. Neurosci. 2009, 12, 1226–1227. [Google Scholar] [CrossRef]

Llobera, J.; Spanlang, B.; Ruffini, G.; Slater, M. Proxemics with Multiple Dynamic Characters in an Immersive Virtual Environment. ACM Trans. Appl. Percept. 2010, 8. [Google Scholar] [CrossRef]

Li, J.; Vinayagamoorthy, V.; Schwartz, R.; IJsselsteijn, W.; Shamma, D.A.; Cesar, P. Social VR: A New Medium for Remote Communication and Collaboration. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020; pp. 1–8. [Google Scholar] [CrossRef]

Li, J.; Vinayagamoorthy, V.; Williamson, J.; Shamma, D.A.; Cesar, P. Social VR: A New Medium for Remote Communication and Collaboration. In Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 8–13 May 2021. [Google Scholar] [CrossRef]

Jonas, M.; Said, S.; Yu, D.; Aiello, C.; Furlo, N.; Zytko, D. Towards a Taxonomy of Social VR Application Design. In Proceedings of the Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts, Barcelona, Spain, 22–25 October 2019; pp. 437–444. [Google Scholar] [CrossRef]

Freeman, G.; Maloney, D. Body, Avatar, and Me: The Presentation and Perception of Self in Social Virtual Reality. Proc. ACM Hum.-Comput. Interact. 2021, 4, 1–27. [Google Scholar] [CrossRef]

Heidicker, P.; Langbehn, E.; Steinicke, F. Influence of avatar appearance on presence in social VR. In Proceedings of the 2017 IEEE Symposium on 3D User Interfaces (3DUI), Los Angeles, CA, USA, 18–19 March 2017; pp. 233–234. [Google Scholar] [CrossRef]

Sykownik, P.; Masuch, M. The Experience of Social Touch in Multi-User Virtual Reality. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology, Virtual, 1–4 November 2020. [Google Scholar] [CrossRef]

Gunkel, S.; Stokking, H.; Prins, M.; Niamut, O.; Siahaan, E.; Cesar, P. Experiencing Virtual Reality Together: Social VR Use Case Study. In Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online Video, Seoul, Korea, 26–28 June 2018; pp. 233–238. [Google Scholar] [CrossRef]

Hecht, H.; Welsch, R.; Viehoff, J.; Longo, M.R. The shape of personal space. Acta Psychol. 2019, 193, 113–122. [Google Scholar] [CrossRef] [PubMed]

McVeigh-Schultz, J.; Márquez Segura, E.; Merrill, N.; Isbister, K. What’s It Mean to “Be Socia” in VR? Mapping the Social VR Design Ecology. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems, Hong Kong, China, 9–13 June 2018; pp. 289–294. [Google Scholar] [CrossRef]

Koilias, A.; Nelson, M.G.; Anagnostopoulos, C.N.; Mousas, C. Immersive walking in a virtual crowd: The effects of the density, speed, and direction of a virtual crowd on human movement behavior. Comput. Animat. Virtual Worlds 2020, 31, e1928. [Google Scholar] [CrossRef]

Dickinson, P.; Gerling, K.; Hicks, K.; Murray, J.; Shearer, J.; Greenwood, J. Virtual reality crowd simulation: Effects of agent density on user experience and behaviour. Virtual Real. 2019, 23, 19–32. [Google Scholar] [CrossRef]

Olivia, A.; Mack, M.L.; Shrestha, M.; Peeper, A. Identifying the Perceptual Dimensions of Visual Complexity of Scenes. Available online: https://escholarship.org/uc/item/17s4h6w8 (accessed on 31 March 2022).

De Manzano, Ö.; Theorell, T.; Harmat, L.; Ullén, F. The psychophysiology of flow during piano playing. Emotion 2010, 10, 301. [Google Scholar] [CrossRef]

Mandryk, R.L.; Inkpen, K.M. Physiological indicators for the evaluation of co-located collaborative play. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, Chicago, IL, USA, 6–10 November 2004; pp. 102–111. [Google Scholar]

Martens, M.A.; Antley, A.; Freeman, D.; Slater, M.; Harrison, P.J.; Tunbridge, E.M. It feels real: Physiological responses to a stressful virtual reality environment and its impact on working memory. J. Psychopharmacol. 2019, 33, 1264–1273. [Google Scholar] [CrossRef]

Nourbakhsh, N.; Wang, Y.; Chen, F.; Calvo, R.A. Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks. In Proceedings of the 24th Australian Computer-Human Interaction Conference, Melbourne, Australia, 26–30 November 2012; pp. 420–423. [Google Scholar]

Ghaderyan, P.; Abbasi, A. An efficient automatic workload estimation method based on electrodermal activity using pattern classifier combinations. Int. J. Psychophysiol. 2016, 110, 91–101. [Google Scholar] [CrossRef]

VaezMousavi, S.M.; Barry, R.J.; Clarke, A.R. Individual differences in task-related activation and performance. Physiol. Behav. 2009, 98, 326–330. [Google Scholar] [CrossRef]

Matthews, G.; Desmond, P.A. Personality and multiple dimensions of task-induced fatigue: A study of simulated driving. Personal. Individ. Differ. 1998, 25, 443–458. [Google Scholar] [CrossRef]

Hockey, G.R.J. A Motivational Control Theory of Cognitive Fatigue; American Psychological Association: Washington, DC, USA, 2011. [Google Scholar]

Thayer, R.E. The Biopsychology of Mood and Arousal; Oxford University Press: Oxford, UK, 1990. [Google Scholar]

Boucsein, W. Electrodermal Activity; Springer Science & Business Media: Berlin, Germany, 2012. [Google Scholar] [CrossRef]

Son, J.; Park, M. Estimating Cognitive Load Complexity Using Performance and Physiological Data in a Driving Simulator. Available online: https://www.semanticscholar.org/paper/Estimating-Cognitive-Load-Complexity-Using-and-Data-Son-Daegu/347eb082dc4969dc65a813db8ad72f6b99061609 (accessed on 31 March 2022).

Mehler, B.; Reimer, B.; Coughlin, J.F.; Dusek, J.A. Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transp. Res. Rec. 2009, 2138, 6–12. [Google Scholar] [CrossRef]

Parsons, T.D.; Courtney, C.G. Interactions between threat and executive control in a virtual reality stroop task. IEEE Trans. Affect. Comput. 2016, 9, 66–75. [Google Scholar] [CrossRef]

Charles, R.L.; Nixon, J. Measuring mental workload using physiological measures: A systematic review. Appl. Ergon. 2019, 74, 221–232. [Google Scholar] [CrossRef]

Wiener, N. Cybernetics or Control and Communication in the Animal and the Machine; Technology Press: Cambridge, MA, USA, 1948. [Google Scholar]

Heylighen, F.; Joslyn, C. Cybernetics and second-order cybernetics. Encycl. Phys. Sci. Technol. 2001, 4, 155–170. [Google Scholar]

Scerbo, M.W.; Freeman, F.G.; Mikulka, P.J. A brain-based system for adaptive automation. Theor. Issues Ergon. Sci. 2003, 4, 200–219. [Google Scholar] [CrossRef]

Kort, B.; Reilly, R. Analytical Models of Emotions, Learning and Relationships: Towards an Affect-Sensitive Cognitive Machine. Available online: https://web.media.mit.edu/~reilly/its2002.pdf (accessed on 31 March 2022).

Rani, P.; Sarkar, N. Operator engagement detection and robot behavior adaptation in human-robot interaction. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2005; pp. 2051–2056. [Google Scholar]

Fairclough, S.H.; Zander, T.O. Current Research in Neuroadaptive Technology; Elsevier Science & Technology: San Diego, CA, USA, 2021. [Google Scholar]

Stach, T.; Graham, T.C.N.; Yim, J.; Rhodes, R.E. Heart Rate Control of Exercise Video Games. In Proceedings of the Graphics Interface 2009, Kelowna, BC, Canada, 25–27 May 2009; pp. 125–132. [Google Scholar]

Wu, D.; Courtney, C.G.; Lance, B.J.; Narayanan, S.S.; Dawson, M.E.; Oie, K.S.; Parsons, T.D. Optimal arousal identification and classification for affective computing using physiological signals: Virtual reality stroop task. IEEE Trans. Affect. Comput. 2010, 1, 109–118. [Google Scholar] [CrossRef]

Dey, A.; Chatburn, A.; Billinghurst, M. Exploration of an EEG-based cognitively adaptive training system in virtual reality. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3d User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 220–226. [Google Scholar] [CrossRef]

Makransky, G.; Terkildsen, T.S.; Mayer, R.E. Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn. Instr. 2019, 60, 225–236. [Google Scholar] [CrossRef]

Harper, S.; Michailidou, E.; Stevens, R. Toward a definition of visual complexity as an implicit measure of cognitive load. Acm Trans. Appl. Percept. (TAP) 2009, 6, 1–18. [Google Scholar] [CrossRef]

Ragan, E.D.; Bowman, D.A.; Kopper, R.; Stinson, C.; Scerbo, S.; McMahan, R.P. Effects of field of view and visual complexity on virtual reality training effectiveness for a visual scanning task. IEEE Trans. Vis. Comput. Graph. 2015, 21, 794–807. [Google Scholar] [CrossRef]

Kelaiah, I.; Kavakli, M.; Cheng, K. Associations between simulator sickness and visual complexity of a virtual scene. Front. Psychol. Behav. Sci. 2014, 3, 27–35. [Google Scholar]

George, C.; Janssen, P.; Heuss, D.; Alt, F. Should I Interrupt or Not? Understanding Interruptions in Head-Mounted Display Settings. In Proceedings of the 2019 on Designing Interactive Systems Conference, San Diego, CA, USA, 23–28 June 2019; pp. 497–510. [Google Scholar]

Staib, M.; Castegnetti, G.; Bach, D.R. Optimising a model-based approach to inferring fear learning from skin conductance responses. J. Neurosci. Methods 2015, 255, 131–138. [Google Scholar] [CrossRef] [PubMed]

Kirchner, W.K. Age differences in short-term retention of rapidly changing information. J. Exp. Psychol. 1958, 55, 352. [Google Scholar] [CrossRef] [PubMed]

Rzayev, R.; Mayer, S.; Krauter, C.; Henze, N. Notification in VR: The Effect of Notification Placement, Task and Environment. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, Barcelona, Spain, 22–25 October 2019; pp. 199–211. [Google Scholar] [CrossRef]

Selzer, M.N.; Gazcon, N.F.; Larrea, M.L. Effects of virtual presence and learning outcome using low-end virtual reality systems. Displays 2019, 59, 9–15. [Google Scholar] [CrossRef]

McMillan, K.M.; Laird, A.R.; Witt, S.T.; Meyerand, M.E. Self-paced working memory: Validation of verbal variations of the n-back paradigm. Brain Res. 2007, 1139, 133–142. [Google Scholar] [CrossRef]

Babaei, E.; Tag, B.; Dingler, T.; Velloso, E. A Critique of Electrodermal Activity Practices at CHI. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 8–13 May 2021. [Google Scholar] [CrossRef]

Hart, S.G. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2006, 50, 904–908. [Google Scholar] [CrossRef]

Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology; Elsevier: Amsterdam, The Netherlands, 1988; Volume 52, pp. 139–183. [Google Scholar]

IJsselsteijn, W.A.; de Kort, Y.A.; Poels, K. The Game Experience Questionnaire; Eindhoven University of Technology: Eindhoven, The Netherlands, 2013; Volume 46. [Google Scholar]

Law, E.L.C.; Brühlmann, F.; Mekler, E.D. Systematic review and validation of the game experience questionnaire (geq)-implications for citation and reporting practice. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, Melbourne, Australia, 28–31 October 2018; pp. 257–270. [Google Scholar]

Keshavarz, B.; Hecht, H. Validating an efficient method to quantify motion sickness. Hum. Factors 2011, 53, 415–426. [Google Scholar] [CrossRef]

Chiossi, F.; Welsch, R.; Villa, S.; Chuang, L.; Mayer, S. Virtual Reality Adaptation using Electrodermal Activity to Support User Experience. DaRUS Open Data Platform. Version 1.; 2022; pp. 2693–2700. Available online: https://darus.uni-stuttgart.de/dataset.xhtml?persistentId=doi:10.18419/darus-2820 (accessed on 31 March 2022).

R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2021. [Google Scholar]

Wickham, H.; Averick, M.; Bryan, J.; Chang, W.; McGowan, L.; François, R.; Grolemund, G.; Hayes, A.; Henry, L.; Hester, J. Welcome to the tidyverse. J. Open Source Soft. 2019, 4, 1686. [Google Scholar] [CrossRef]

Dahl, D.B.; Scott, D.; Roosen, C.; Magnusson, A.; Swinton, J. xtable: Export Tables to LaTeX or HTML. R Package Version 1.8-4. 2019. Available online: https://rdrr.io/cran/xtable/ (accessed on 31 March 2022).

Kuznetsova, A.; Bruun Brockhoff, P.; Haubo Bojesen Christensen, R. lmerTest: Tests in Linear Mixed Effects Models. R Package Version 3.1-3. 2020. Available online: https://cran.r-project.org/web/packages/lmerTest/index.html (accessed on 31 March 2022).

Makowski, D.; Lüdecke, D.; Ben-Shachar, M.S.; Patil, I. Report: Automated Reporting of Results and Statistical Models. R Package Version 0.4.0. 2021. Available online: https://easystats.github.io/report/ (accessed on 31 March 2022).

Benedek, M.; Kaernbach, C. Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 2010, 47, 647–658. [Google Scholar] [CrossRef]

Cacioppo, J.T.; Tassinary, L.G. Principles of Psychophysiology: Physical, Social, and Inferential Elements; Cambridge University Press: Cambridge, UK, 1990. [Google Scholar]

Pecchinenda, A. The affective significance of skin conductance activity during a difficult problem-solving task. Cogn. Emot. 1996, 10, 481–504. [Google Scholar] [CrossRef]

Peifer, C.; Schulz, A.; Schächinger, H.; Baumann, N.; Antoni, C.H. The relation of flow-experience and physiological arousal under stress—can u shape it? J. Exp. Soc. Psychol. 2014, 53, 62–69. [Google Scholar] [CrossRef]

De Fockert, J.W. Beyond perceptual load and dilution: A review of the role of working memory in selective attention. Front. Psychol. 2013, 4, 287. [Google Scholar] [PubMed]

Milgram, P.; Takemura, H.; Utsumi, A.; Kishino, F. Augmented reality: A class of displays on the reality-virtuality continuum. In Proceedings of the Telemanipulator and Telepresence Technologies. International Society for Optics and Photonics, Boston, MA, USA, 31 October–4 November 1994; Volume 2351, pp. 282–292. [Google Scholar] [CrossRef]

Ghaffari, R.; Yang, D.S.; Kim, J.; Mansour, A.; Wright, J.A., Jr.; Model, J.B.; Wright, D.E.; Rogers, J.A.; Ray, T.R. State of Sweat: Emerging Wearable Systems for Real-Time, Noninvasive Sweat Sensing and Analytics. ACS Sens. 2021, 6, 2787–2801. [Google Scholar] [CrossRef] [PubMed]

Lee, Y.; Yoon, S.; Lee, C.; Lee, M. Wearable EDA sensor gloves using conducting fabric and embedded system. In Proceedings of the World Congress on Medical Physics and Biomedical Engineering 2006, New York, NY, USA, 30 August–3 September 2006; pp. 883–888. [Google Scholar] [CrossRef]

Laufer, L.; Németh, B. Predicting user action from skin conductance. In Proceedings of the 13th International Conference on Intelligent User Interfaces, Lisboa, Portugal, 14–17 February 2012; pp. 357–360. [Google Scholar] [CrossRef]

Gilroy, S.; Porteous, J.; Charles, F.; Cavazza, M. Exploring passive user interaction for adaptive narratives. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces, Lisboa, Portugal, 14–17 February 2012; pp. 119–128. [Google Scholar]

Greco, A.; Valenza, G.; Lanata, A.; Scilingo, E.P.; Citi, L. cvxEDA: A convex optimization approach to electrodermal activity processing. IEEE Trans. Biomed. Eng. 2015, 63, 797–804. [Google Scholar] [CrossRef] [PubMed]

Talsma, D.; Doty, T.J.; Woldorff, M.G. Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cereb. Cortex 2007, 17, 679–690. [Google Scholar] [CrossRef]

Dinh, H.Q.; Walker, N.; Hodges, L.F.; Song, C.; Kobayashi, A. Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings of the IEEE Virtual Reality, Houston, TX, USA, 13–17 March 1999; pp. 222–228. [Google Scholar] [CrossRef]

Braithwaite, J.J.; Watson, D.G.; Jones, R.; Rowe, M. A guide for analysing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments. Psychophysiology 2013, 49, 1017–1034. [Google Scholar]

Healey, J.; Seger, J.; Picard, R. Quantifying driver stress: Developing a system for collecting and processing bio-metric signals in natural situations. Biomed. Sci. Instrum. 1999, 35, 193–198. [Google Scholar]

Barry, R.J.; Rushby, J.A.; Wallace, M.J.; Clarke, A.R.; Johnstone, S.J.; Zlojutro, I. Caffeine effects on resting-state arousal. Clin. Neurophysiol. 2005, 116, 2693–2700. [Google Scholar] [CrossRef]

Figure 1. Game view capture of a single trial of the VR n-back ( n = 1 ) and the visual detection tasks. Participants were required to place a sphere into the corresponding bucket. If the sphere matched the color of the previous sphere one step before, participants placed it into the right bucket. If not, the sphere had to be placed into the left bucket. The visual detection task required participants to monitor if visitors of a museum either possessed a ticket to enter the building or not. To signal a missing ticket after detection, the participant had to select the NPC (see Supplymentary Materials).

Figure 1. Game view capture of a single trial of the VR n-back ( n = 1 ) and the visual detection tasks. Participants were required to place a sphere into the corresponding bucket. If the sphere matched the color of the previous sphere one step before, participants placed it into the right bucket. If not, the sphere had to be placed into the left bucket. The visual detection task required participants to monitor if visitors of a museum either possessed a ticket to enter the building or not. To signal a missing ticket after detection, the participant had to select the NPC (see Supplymentary Materials).

Figure 2. Individual predicted standardized mean EDA from the optimal Stream for the non-adaptive condition (crosses) with individual regression lines, as well as the actual mean EDA (points) at local maxima of adaptation.

Figure 2. Individual predicted standardized mean EDA from the optimal Stream for the non-adaptive condition (crosses) with individual regression lines, as well as the actual mean EDA (points) at local maxima of adaptation.

Figure 3. Adaptation across time for one participant. The pink line indicates Stream, the green line indicates the z-scored mean EDA signal that was used for adaptation. Grey areas indicate whether the algorithm chose to increase (light grey) or decrease (dark grey) the Stream in a time window of 20 s.

Figure 3. Adaptation across time for one participant. The pink line indicates Stream, the green line indicates the z-scored mean EDA signal that was used for adaptation. Grey areas indicate whether the algorithm chose to increase (light grey) or decrease (dark grey) the Stream in a time window of 20 s.

Figure 4. The relative difference for (a) raw NASA-TLX score difference, (b) standardized mean EDA, and (c) averaged SCL scores.

Figure 4. The relative difference for (a) raw NASA-TLX score difference, (b) standardized mean EDA, and (c) averaged SCL scores.

Figure 5. The relative difference for overall task accuracies in the n-Back and visual detection tasks.

Figure 5. The relative difference for overall task accuracies in the n-Back and visual detection tasks.

Figure 6. Standardized mean EDA at local maxima of adaptation as a function of raw NASA-TLX for the adaptive condition. There is a significant negative correlation between EDA and workload, r(13) = −0.62, p = 0.013.

Figure 6. Standardized mean EDA at local maxima of adaptation as a function of raw NASA-TLX for the adaptive condition. There is a significant negative correlation between EDA and workload, r(13) = −0.62, p = 0.013.

Figure 7. The relative difference for (a) usability questions measured on a 5-point Likert scale and (b) GEQ subscales (Competence, Positive Affection, and Immersion). * indicates that measurements are significantly different from the no-adaptation baseline. Outliers were defined as data points with a value greater than 2 SDs on the log-scale from its participant-mean. Outliers are represented as bold dots.

Figure 7. The relative difference for (a) usability questions measured on a 5-point Likert scale and (b) GEQ subscales (Competence, Positive Affection, and Immersion). * indicates that measurements are significantly different from the no-adaptation baseline. Outliers were defined as data points with a value greater than 2 SDs on the log-scale from its participant-mean. Outliers are represented as bold dots.

Table 1. Means across non-adaptive conditions with the slope of linear mixed models (LMM) and their t-values estimated by Wald approximation, as well as their respective p-value.

Table 1. Means across non-adaptive conditions with the slope of linear mixed models (LMM) and their t-values estimated by Wald approximation, as well as their respective p-value.

| Stream 7 | Stream 22 | Stream 37 | Stream 52 | Stream 67 | LMM
| M | SD | M | SD | M | SD | M | SD | M | SD | b | t | p
n-back Acc. [%]. | 97.423 | 1.607 | 95.381 | 2.660 | 94.401 | 2.430 | 92.133 | 3.182 | 91.055 | 4.162 | −0.107 | −9.396 | <0.001
Visual Det. Acc. [%] | 97.208 | 3.073 | 95.579 | 4.647 | 94.834 | 2.670 | 93.067 | 3.116 | 90.949 | 2.613 | −0.100 | −7.083 | <0.001
Raw NASA TLX | 5.267 | 2.520 | 7.644 | 2.268 | 9.500 | 3.616 | 9.133 | 3.282 | 11.833 | 3.262 | 0.097 | 8.101 | <0.001
EDA Mean (std) | −0.091 | 1.031 | −0.083 | 1.001 | −0.034 | 0.896 | 0.003 | 1.083 | 0.166 | 1.056 | 0.004 | 2.536 | 0.014
SCL | 9.078 | 2.974 | 9.102 | 2.888 | 9.242 | 2.585 | 9.348 | 3.127 | 9.819 | 3.047 | 0.012 | 2.535 | 0.014
GEQ—Competence | 2.667 | 0.939 | 2.600 | 1.039 | 2.200 | 0.862 | 2.600 | 0.687 | 1.967 | 0.834 | −0.009 | −2.149 | 0.036
GEQ—Pos. Affec. | 2.400 | 0.806 | 2.500 | 0.655 | 2.467 | 0.550 | 2.433 | 0.729 | 2.100 | 0.761 | −0.004 | −1.318 | 0.193
GEQ—Immersion | 0.833 | 0.724 | 1.200 | 0.841 | 1.133 | 0.876 | 1.067 | 0.821 | 1.167 | 0.939 | 0.004 | 0.906 | 0.369
Stream Appropriate | 1.400 | 0.632 | 1.600 | 0.737 | 2.000 | 0.756 | 1.933 | 0.799 | 1.467 | 0.743 | 0.003 | 0.754 | 0.454
Desire To Use | 1.467 | 0.640 | 1.800 | 0.775 | 1.800 | 0.941 | 2.267 | 0.799 | 1.333 | 0.617 | 0.001 | 0.309 | 0.759

| Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.