Is achieving high levels of emotional response a must-have when measuring success for a virtual reality (VR) application? For many VR developers, the answer is yes, and according to a unique study from the Clemson University School of Computing, the way a virtual human looks could play a major role in achieving that desired emotional response.

It goes without saying that the more realistic the images, the more true-to-life the experience. But for VR developers, it’s more complicated than that. First of all, it’s expensive to create realistic 3D virtual humans. The shadowing, body movements, facial expressions, and close-up physical details need to be just right, which takes expertise and countless hours to capture. And secondly, VR developers are working against something called the “uncanny valley” effect, which hypothesizes that when mock features look and move almost, but not exactly, like natural human beings, it makes people uncomfortable. (This is why human-like robots, dolls, and zombie animations have a long history of being seen as creepy).

At the same time, engaging consumers emotionally using virtual characters is central for the arts and entertainment industry. And in a different way, the same can be said for job-training in professions that can be stressful or painful, which are increasingly turning to VR to simulate dangerous situations. In that space and using virtual humans, emotional response is tested to build interpersonal skills, measure job performance and help with memory retention. Some industrial applications of virtual humans in inter-personal training include healthcare, fire and police forces, psychotherapy, transportation services, and high-stake business negotiations.

So regardless the VR application, developers face an important question: is designing realistic virtual humans worth the high investment and risk? To find answers, PhD student Matias Volonte and his adviser Sabarish V. Babu from the Clemson Virtual Environments Group designed an empirical study to evaluate—for the first time in VR—how virtual human aesthetics correlate to emotional response. In particular, they examined the effects of an interactive photorealistic vs. non-photorealistic virtual patient named Bob.

“Creating realistic virtual humans is incredibly complex, and often can’t be pulled off in the right way,” said Volonte. “At times, it’s just as well to go with a cartoonish alternative. But we need to better understand the when and why to help developers make this serious decision.”

In the study, the researchers used a Rapid Response Training System (RRTS) platform designed to train medical nurses in patient monitoring. The training was a simulation that started with Bob the virtual patient, and tracked his rapidly declining health over a short period of time, expressed through physical changes and body language. The team recruited 62 participants (36 males, 26 females) to go through the training, each assigned to one of three conditions in the continuum from photorealistic to non-photorealistic rendering: 1. Bob appeared as human-like as possible (realistic), 2. Bob appeared as a cartoon (non-realistic), or 3. Bob appeared as a charcoal sketch (non-realistic).

Three renderings of a virtual human: photorealistic to non-photorealistic

 

To measure results objectively, the team had participants wear an EDA sensor on their wrist to measure electrodermal activity caused by physiological arousal (e.g., sweat). The team also had participants fill out four separate questionnaires as their patient worsened, evaluating how emotionally impacted they were from beginning to end of the simulation. This included a survey to rate Interest, Enjoyment, Surprise, Sadness, Anger, Contempt, Fear, Guilt, Shame and Shyness on a scale of 0-9.

Overall, the results proved that VR human appearance does have an effect on emotional response and perceptions of personality, though the study begs further questions as to precisely how and why.

Here is a summary of the key findings:

  • Unexpected gender trends: The objective EDA results didn’t differ significantly between Bob’s aesthetic forms, though the team noticed a fascinating pattern: male participants exhibited much higher levels of physiological arousal than females throughout Bob’s regression. The team hypothesizes males generally have more attraction to and experience playing video games, and therefore could have been more excited by the simulation.
  • Cartoon inspires same/higher intensity of emotion: From the surveys, negative emotions rose in all three conditions as Bob’s state worsened, but, contrary to expectations, the intensity of negative emotion was higher with the cartoon Bob rather than the photorealistic one. This could be because in the realistic scenario, participants were more likely to pick up subtle hints and gestures of pain in the face, which didn’t lead to as much surprise/sadness when Bob collapsed.
  • … But real Bob better communicates social cues: Participants with realistic Bob experienced significantly lower levels of enjoyment and higher levels of shame and shyness compared to the other groups. Since shame and shyness are social constructs, this proves that Bob was perceived as a more human-like conversational partner. The realistic Bob group was also more consistent in the way it described Bob’s personality traits and changes compared to the other groups.

“While cartoons can elicit very strong and authentic emotion, visual realism may allow users to perceive more subtle channels of information,” said Volonte. “This enables them to more accurately pick up on a diversity of traits that are as minor as they are critical.”

Even if the VR humans are a bit creepy.

In the future, the team hopes to further explore the gender gaps in physiological response to the virtual human simulation, the emotional effects of environmental realism in VR (not just humans), and the effects on user sociality and job performance levels. They also plan to incorporate eye-tracking into the RRTS platform to measure visual attention and interest based on appearance and behavior of the virtual human.

Read Part 1 and Part 2 on humanizing VR, or find out more about the uncanny valley effect in IEEE Xplore.